What is Data Pipeline? | Why Is It So Popular?

Подписаться 852 тыс.

Просмотров 75 тыс.

50% 1

Get a Free System Design PDF with 158 pages by subscribing to our weekly newsletter: bit.ly/bytebytegoytTopic
Animation tools: Adobe Illustrator and After Effects.
Checkout our bestselling System Design Interview books:
Volume 1: amzn.to/3Ou7gkd
Volume 2: amzn.to/3HqGozy
The digital version of System Design Interview books: bit.ly/3mlDSk9
ABOUT US:
Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview series.

Наука

Опубликовано:

10 июн 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 59

@petenjs3500 13 дней назад

3:13 typo *AWS Glue. Love these vids, thanks!

@Nonenone-rj9yp 6 дней назад

bruh had me googling whats AWS glow

@SudhanvaDixit 13 дней назад

0:49 Shouldn't the last one be 'Consume'?

@prasenjeetrathore 11 дней назад

Amazing explanation, so far the most easy to digest video about data pipelines.

@husseineldeeb 13 дней назад

Amazing video. Thanks for your great efforts!

@mrseanpaul81 5 дней назад

I love the short video format, as I can dive deeper on topics and terms I am interested in on my own time :)

@jaykukreja7125 11 дней назад

Love it. This jargon cleared now

@atomtamadas 13 дней назад

Spark is widely used in stream processing too, not only batch, see spark structured streaming.

@gus473 13 дней назад

💯 Looking like your channel is on track for 1 million subscribers by year end! Great stuff! 😎✌️

@ttehir 13 дней назад

Why do we mostly talk about data pipelines for BI or ML when many times we also need it for functional applications?

@personalbranddata 12 дней назад

Those functional applications should likely use the same data platform for their functional applications, the only difference is how you're serving the transformed result. What's the difference then that you think should be talked about?

@manishshaw1002 11 дней назад

Functional applications are most likely consume very small amount of data while BI and AI ML models required way more likely gb to TB amount of data to work with. There's no possible way you can load 1gb of data in your web app or sql it just makes your app clogging and time consuming.

@JB-ve8sk 11 дней назад

Because more and more non-traditionally technical business roles are leveraging data for business intelligence - so the demand for understanding these concepts is greater there (than in complex application architectures where more traditional technical skill accumulates).

@deadohiosky1701 8 дней назад

Just call it messaging and you’re good to go

@zobaidulkaziex 13 дней назад

Very good discussion

@immanuelt613 8 дней назад

Top quality work as always

@raj_kundalia 4 дня назад

Thank you for doing this!

@sreenivasreddypallerla9941 10 дней назад

Very informative !! But how you do all these animations ??what product do you use !!

@jordanfarr3157 13 дней назад

Always so so good

@virtuoso_hub 4 дня назад

I like your presentations. What do you use to make them?

@rishiraj2548 13 дней назад

Thanks

@vlplbl85 6 дней назад

Great video. Small remark: the AWS service for ETL is called AWS Glue, not Glow

@bladethirst1 12 дней назад

Maybe some examples of simplified pipeline on specific application would make this video even better.

@mikedepacina8588 13 дней назад

Aws glow or aws glue?

@mwanthidaniel1254 13 дней назад

Which tool do you use to create these animated presentations?

@johnson51200 7 дней назад

Trade secret 😂

6 дней назад

Is GA4 consider a data stream? And big query a storage and transform tools?

@saratpoluri 15 часов назад

Bravo!

@user-data_junkie 13 дней назад

What do you use to create these animations/info graphics

@knighthawk095 5 дней назад

I think it could be either figma or canvas.

@user-data_junkie День назад

@@Biostatistics is there a video out there that shows how that is done in power point? I see these data like infographics a lot these days

@Biostatistics День назад

@@user-data_junkieit’s says in the description of this video, he used Adobe illustrator and after effects. 😊

@user-data_junkie День назад

@@Biostatistics thanks. I did check at the time and did not see anything. Appreciate the update

@yongguangli3304 7 дней назад

请问这些精美的图是怎么画的？太赞了

@VishnuVijayan7 12 дней назад

Did not make a mention on data lakehouse

@marcgentner1322 8 дней назад

So i need to build a way so retrieve man many emails and categorize them with a ml model and then save them in the right system. Do i build this with kafka and pyspark? Or how can this be done easaly

@johnson51200 7 дней назад

Kafka dear

@eddielim8888 13 дней назад

AWS Glow or Glue?

@markwallstrom9994 11 дней назад

No mention of Apache Iceberg and such technology?

@Mr.Andrew. 9 дней назад

Your diagram had compute arrows twice when you verbally said compute and consume for the last two phases.

@johnson51200 7 дней назад

"Trade Secret" name of the tool used to create the animations ...😂

@VikramPatilvp 4 дня назад

Looks like your examples are only AWS or Google stack. Why not cover examples from MS Azure stack as well?

@andreslasvegas30 13 дней назад

I dont know why but the gain of the microphone is too high, there is a little background noise and its a bit noticeable, keep it in check. Great video, as always in the channel.

@internetexplorer1593 13 дней назад

Leaving out all Azure tools... really a shame

@scottedmiston6566 12 дней назад

Maybe it's intentional. Many serious data scientists aren't fond of the Azure UI for big data pipelines.

@JB-ve8sk 11 дней назад

Microsoft training has that covered

@thesimplicitylifestyle 9 дней назад

😎🤖

@vickyg1877 13 дней назад

Rest api

@checkerist 10 дней назад

apache hive logo is on acid

@dimitrikalinin3301 6 дней назад

AWS Glue, not Glow

@albinantony4998 13 дней назад

looks like you need to change the mic you are currently using. there is some crackling noise when you talk.

@johnsmith21123 13 дней назад

Hadoop is dead

@praveens2272 13 дней назад

Why, what's the reason

@JohnS-er7jh 13 дней назад

they said that about Mainframe computers 30 years ago, but they are still here/in production. Large organizations are not going to adopt the latest solutions for all there data needs (for instance data that isn't accessed that often/specific use cases, or they might have support staff that is more familiar with legacy tools and they don't see the need to adopt latest methods at the moment). So I can guarantee Hadoop is NOT completely dead.

@angryktulhu 12 дней назад

Lol it’s not dead at all, and its ecosystem tools are still widely used

@shilashm5691 12 дней назад

😂 most uses hdfs as data lake, when you say hadoop.is dead be precise and say mapreduce.is dead, bcoz hadoop ecosystem is large and still functioning

@personalbranddata 12 дней назад

@@shilashm5691 Most use AWS S3 as storage for their datalake, others Azure Data Lake Storage. MapReduce is dead and HDFS is on the brink of obscurity as well. I pity those who still have to work with some inhouse hdfs from the darkest and most painful era of data engineering (hadoop era)