Learn Apache Airflow in 10 Minutes | High-Paying Skills for Data Engineers

Подписаться 136 тыс.

Просмотров 127 тыс.

50% 1

Join My Data Engineer Courses Here: datavidhya.com/courses
➡️ Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?pl...
USE CODE: COMBO50 for 50% off
What is Apache Airflow and How To Learn? This video will discuss Apache Airflow, its popularity, basics, and everything around it.
👦🏻 My Linkedin - / darshil-parmar
📷 Instagram - / datawithdarshil
🎯Twitter - / parmardarshil07
🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
3 Books You Should Read
📈Principles: Life and Work: amzn.to/3HQJDyP
👀Deep Work: amzn.to/3IParkk
💼Rework: amzn.to/3HW981O
Tech I use every day
💻MacBook Pro M1: amzn.to/3CiFVwC
📺LG 22 Inch Monitor: amzn.to/3zk0Dts
🎥Sony ZV1: amzn.to/3hRpSMJ
🎙Maono AU-A04: amzn.to/3Bnu53n
⽴Tripod Stand: amzn.to/3tA7hu7
🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
👉Data Engineering Complete Roadmap: • Data Engineer Complete...
👉Data Engineering Project Series: • Data Engineering Proje...
👉Become Full-Time Freelancer: • Best Freelancer Series...
👉Data With Darshil Podcast: • Podcast Series - Data ...
✨ Tags ✨
✨ Hashtags ✨
#airflow #dataengineering #bigdata

Опубликовано:

6 окт 2023

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 102

@DarshilParmar 8 месяцев назад

100k when? 😭 Hit that LIKE button, Subscribe if you are new and Comment something for better reach 🥹

@sathyamanikanta2784 8 месяцев назад

Soon❤

@sathyamanikanta2784 8 месяцев назад

All the best ❤for 1m subs

@sathyamanikanta2784 8 месяцев назад

Can I get opportunity if i know only Airflow???

@AshisRaj 5 месяцев назад

This 10 mins video is much better than lengthy but hollow videos of hours. Thanks bro

@saminyead1233 Месяц назад

Wow! The amount of value you bring to the viewer in just 12 minutes is just incredible! Like you were saying "the hassle of managing pipelines in python scripts", I made a data project a while back, and I really can understand what you mean. I heard that Airflow makes everything easier, but I kept deferring Airflow learning, partly because "maybe this is good for large data pipelines, but my one consists of just a few small tasks," and also because most RU-vid videos of Airflow out there was a couple of hours long. And that's great if you are already committed to learn Airflow, but I was really looking for a video like yours - something that will give me a practical demo of how Airflow actually works, but it will be around 10-15 minutes. And this video was just perfect. Once again, thanks a whole lot! I'll definitely be integrating Airflow into my project now! And my resume too, God willing!

@VIJAYSHREEVAIBHAV 2 месяца назад

00:05 A data engineer builds data pipelines to extract, transform, and load data onto a target location. 01:50 Apache Airflow is a viral and widely adopted open-source project for managing data pipelines. 03:33 A workflow in Apache Airflow is called a DAG (Directed Acyclic Graph). 05:10 Apache Airflow has different operators and executors which make task creation and execution easier. 06:46 DAG with Dummy Operator in Airflow 08:12 Apache Airflow allows you to manage DAGs and track their statuses in one place 09:40 Airflow allows you to create complex pipelines and execute Python functions 11:16 Airflow overview and recommended project

@gunjankothari7471 6 месяцев назад

Got clear understanding of basics of Apache Airflow.. Very helpful

@abdulazizmamun 4 месяца назад

Just Smart video, to the point, no unnecessary content. thanks a lot

@krishnakumarmishra5406 8 месяцев назад

Hi Darshil , please make these type of videos for Hive , cassandra , kafka ,data bricks ,snowflake,snowpark,mongodb

@DarshilParmar 8 месяцев назад

Yes, I am planning to do the same, already did it for Spark and Airflow (now), I will do it for DataBricks and others too

@shubhambhosale8467 8 месяцев назад

@@DarshilParmardo for kafka using azure services as well

@surajpadhy6442 8 месяцев назад

Hey Darshil, I find your videos really helpful both conceptually and practically. I have been able to explain these concepts in my interviews and crack them. Keep on doing your amazing work with such explanatory videos

@DarshilParmar 8 месяцев назад

Thank you Suraj

@user-uf5lv9ik4g 3 месяца назад

Really clear and interesting explanation. Thank you

@hiltonsumit 6 месяцев назад

Thanks for this video.. very informative and educative.. thanks again

@pallavirajput9779 4 месяца назад

Thanks. Quick and clear 👍

@suchetahardikar2687 8 месяцев назад

Please make thesr type of videos on Snowflake and Apache Kafka. This would be very useful. You can explain the topic in simple way so that a learner can understand it easily. 👌👌👌

@rajatmw1999 2 месяца назад

Great job explaining in just 10mnts. Thanks!

@tarukhmakhtoh9726 11 дней назад

Thank you for all the help brother... I have started learning from your videos..🙏🏻

@PoojaM22 7 месяцев назад

Excellent explanation Darshil! Keep it up

@hasiiabbasi 6 месяцев назад

Really helpful.keep it up good work.

@jobisha6704 12 дней назад

I like these kind of small videos to get to know about a new thing. If we want to master it, then we can go for the full course or other videos. Thank you!

@guhanathanprathish9704 3 месяца назад

Thanks buddy, very useful 🙌🏻

@vishalbhardwaj85 8 дней назад

Super useful video. Best beginner video available on this topic.

@Sonurathore3107 7 месяцев назад

please dont apply background music

@rudroroy1054 6 месяцев назад

will you be uploading similar kinds of videos for Jenkins and Kafka as well? I feel there is always confusion between airflow, Jenkins, and Kafka and what a new data engineer should be learning when starting.

@KamranAli-yj9de 5 месяцев назад

Amazing Explanation!!!! :)

@Ninja_info 2 месяца назад

Great video, thank you..

@nandubindu2566 3 дня назад

Very helpful . Thank you

@yatshunlee 6 месяцев назад

Thank you for your fruitful content and I have smashed the like button of the video. May I ask why you can directly save the csv to s3 without using boto3?

@atraveller7155 8 месяцев назад

Very informative video Darshil, btw which font style do you use?

@VishalVerma-wn4by 7 месяцев назад

very nice video bro. keep it up

@adityaf17 8 месяцев назад

Basically, we can also create a python script and do the same work, the only thing which airflow is helping developers is that we don't have to track these scripts individually, here we are getting one portal kind of thing which manages and it also has different functions which can track individual script and if something fails we can track it.

@veer66 Месяц назад

Great explanation!

@prachisharma6765 8 месяцев назад

Hey darshil, very informative and detailed video, thanks a lot for this one.. 😊 have a request though, please upload more such videos consisting of small E2E projects on other apache tools.

@DarshilParmar 8 месяцев назад

Working on it

@SuhasKM-tl1rg 3 месяца назад

Amazing stuff

@arpittapwal4651 4 месяца назад

Thank you for making this video Darshil. I just watched the snowflake video and then couldn’t, resist to watch this one. Keep making such awesome content.

@DarshilParmar 4 месяца назад

Thank you so much 😀

@AmitKumar-ym8uz 8 месяцев назад

Hi Darshil, are you also proving course on Azure for data engineering?

@harshdoshi9919 8 месяцев назад

It is very helpful, learned a lot. Thanks for the video🙂🙂🙂

@DarshilParmar 8 месяцев назад

Thank you Harsh

@gialamhuynh5448 3 месяца назад

Hi Mr Parmar, Thank you for this short introduction to Airflow.

@DarshilParmar 3 месяца назад

You are most welcome

@looklook6075 6 месяцев назад

Does your data engineering class on your website teach airflow? Does it teach how to debug python code too?

@worldtour666 8 месяцев назад

Hi @DarshilParmar, I have gone through couple of your video and posts on youtube and linkedin. All are very helpful for me. I need your help to resolve my one problem as below: I need to create aws quicksight dashboard using my developer team designed ui template. I didnt not any option to use custom Ui option in qucksight? Please help me.

@VishalYadav-bj4ls Месяц назад

Nice explanation

@ishitasrivastava5923 8 месяцев назад

Definitely best overview of Airflow till date

@DarshilParmar 8 месяцев назад

Thank you!

@rohitgade2382 8 месяцев назад

Can you share he the emoji bro

@rahulb9058 Месяц назад

Great content. One suggestion- Remove/tone down the background music. Had trouble paying attention to your voice.

@xeankun5420 8 месяцев назад

Great video! I hope you can make a comparison between airflow, prefect and dagster

@DarshilParmar 8 месяцев назад

Great, I will put this in my list

@venkatadurgarao8892 8 месяцев назад

Hi darshil, can you please make a tutorial on pyspark

@pallavibrahpuria5806 6 месяцев назад

You must talk about internals and workings too.

@anandukrishnan3054 2 дня назад

On what compute service the python code runs?

@mpumisetshedii 8 месяцев назад

Please make an intro course for Apache using Windows

@Errormakesmarter 8 месяцев назад

Bro can i get trained in jspiders as a full stack developer is it worth . Does its certificate have value

@skateforlife3679 8 месяцев назад

Airflow is really tricky for me, got problems calling functions in my dag from other files that are not in airflow folder structure ...

@user-bz7uc7lg3g 8 месяцев назад

Hello Darshil!! Your content is very good. Please start pyspark tutorial. It will be very helpful for me.

@DarshilParmar 8 месяцев назад

Thank you, I am working on it

@chesswithmoiz 3 месяца назад

Can you please make a crash course of Apache Airflow in which following topics are discussed. Installation (Using Docker, EC2 and PIP) Multiple DAG creation Executors Different configurations like using airflow along with kafka how we can use airflow on EC2.

@JeffersonOsorio-gx1gw 8 месяцев назад

Nice Video!!!

@DarshilParmar 8 месяцев назад

Glad you liked it

@laxmanravula9199 4 месяца назад

good video

@shubairabbas5480 8 месяцев назад

Really crisp explanation

@DarshilParmar 8 месяцев назад

Glad you liked it

@shubairabbas5480 8 месяцев назад

@@DarshilParmar Bro you are a Data Engineer Guru. Thanks for making life simple. Bro can you create a video how we get a full time role as a Data Engineer permanent Remote. You are doing freelancing but can you make how we get remote jobs and earn in USD. Because on RU-vid everyone is just faking and what they mentioned are for highly skilled people

@geekyprogrammer4831 8 месяцев назад

Please create video on Mage AI.

@param8592 8 месяцев назад

Hi darshil, My coding skills are not so much good and currently i am working as data engieer and i am not using any cloud platform. Could please tell me which cloud platforms is best where we have less coding for pipline,transformation and deployment . So i can start learning for that cloud platform like azure ,gcp, aws

@DarshilParmar 8 месяцев назад

Hi, Cloud platforms has nothing to do with amount of coding you will be doing, as most of them are similar at fundamental level, it totally depends on the tool you are working on. If you are working with tools like Alteryx then it is mainly drag and drop but Airflow, Writing Spark script all of them requires coding If your coding is not good then I suggest start working on it as it is important skillset to have

@mrafam626 5 месяцев назад

@@DarshilParmarI am a beginner at learning Python and data engineering fundamentals in general with hopes of being one in the near future. I was tempted to see how these tools work and to be honest, I am a bit intimidated by the coding and complexity of it all. I pray I am able to learn it all.

@naushadsheik2136 8 месяцев назад

Bring on end-end course on airflow with projects @Darshil

@DarshilParmar 8 месяцев назад

Working on it

@awsmaster7788 3 месяца назад

Jai Jinendra 🙏

@izhanabdullah 8 месяцев назад

Hey Drashil, As you said Twitter API is not valid anymore. Now which API we can use for Twitter Data Pipeline project? How to use another API like tweepy? Could you please explain about this?

@DarshilParmar 8 месяцев назад

RU-vid, Maps, Spotify, Crypto there are so many just search about it