Azure Data Factory Mapping Data Flows Tutorial | Build ETL visual way!

Подписаться 201 тыс.

Просмотров 224 тыс.

50% 1

With Azure Data Factory Mapping Data Flow, you can create fast and scalable on-demand transformations by using visual user interface. In just minutes you can leverage power of Spark with not a single line of code written.
In this episode I give you introduction to what Mapping Data Flow for Data Factory is and how can it solve your day to day ETL challenges. In a short demo I will consume data from blob storage, transform movie data, aggregate it and save multiple outputs back to blob storage.
Sample code and data: github.com/MarczakIO/azure4ev...
Next steps for you after watching the video
1. Azure Data Factory introduction video
- • Azure Data Factory Tut...
2. Check mapping data flow documentation
- docs.microsoft.com/en-us/azur...
3. Helpful tips and samples
- github.com/kromerm/adfdataflo...
Want to connect?
- Blog marczak.io/
- Twitter / marczakio
- Facebook / marczakio
- LinkedIn / adam-marczak
- Site azure4everyone.com

Наука

Опубликовано:

4 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 307

@Haribabu-zj4hd 2 года назад

So nice of your talent explaining the data flow in simple way. Thank you so much Mr.Adam.

@bifurcate-ai 4 года назад

Adam, I have been watching many of your videos. As someone new to Azure, i find your videos immensely valuable. Keep up your great work, really appreciate!

@AdamMarczakYT 4 года назад

Awesome, thank you!

@subhodipsaha7608 4 года назад

I just find your videos while searching for ADF tutorials in youtube. The materials are fantastic and really helping me to learn. Thank you so much!!

@AdamMarczakYT 4 года назад

Happy to help! :)

@dimitarkrastev6085 Год назад

Great video! Most videos seem to focus mostly on the evertisement material straight from Azure. At best they show you the very dumb step of copying data from a file to DB. This is the first video I saw where you actually show how you can do something useful with the data and close to real life scenario. Thank you.

@waklop4384 4 года назад

Just discovered the channel. Your material is hight quality. It's excellent work. I will go watch more. Thank you Adam !

@AdamMarczakYT 4 года назад

Thank you. This means much :)

@pradeeps3671 3 года назад

Hello Adam, pls let me know how to connect to dynamic crm .. Pls send detail to pradysg@gmail.com

@joshuaodeyemi3098 Год назад

I love you, Adam! I have been struggling with using expression builder in Data Flow. I can't seem to figure out how to write the code. This video just made it look less complex. I'll be devoting more time to it.

@hovardlee 3 года назад

-1979 and ,12 This is why complex logic is needed. Nice tutorial :)

@gunturchilli767 4 года назад

Thanks you so much Adam. I was able to crack an interview with the help of your videos. I prepared notes according to your explanation & 3 hrs before the interview i watched your videos again it helped me alot

@AdamMarczakYT 4 года назад

Fantastic!

@alanzhao8074 3 года назад

The best video about Azure Data Flows I can find. Thank you Adam!

@AdamMarczakYT 3 года назад

Wow, thanks! :)

@hassy9118 4 года назад

Excellent presentation of ADF Data Mapping...we love to see more Data Factory ETL videos. Thank you Adam!

@AdamMarczakYT 4 года назад

Thanks, feel free to check my ADF playlist.

@fadiabusafat5162 2 года назад

Nice one Adam. Cool one. Keep doing fabulous videos always fella. Many THanks.

@lavanyay2767 Месяц назад

very very detailed work flow , i tried this and able to understand Data flow process so easily . Thank you for the wonderful session.

@aarontian5979 2 года назад

Your channel is totally underrated, man

@omarsantamaria6871 4 года назад

Awesome video. I've seen a lot of site & videos and they are so complicated, but all yours are very crystal and anyone can understand.

@AdamMarczakYT 4 года назад

Thanks Omar :)

@subhraz 2 года назад

Very well explained and demonstrated. Really helpful to get started with Data flows.

@sapecyrille5487 Год назад

Great! You are the best Adam.

@Raguna 2 года назад

Very good explaining the Data Flow. Thanks Mr.Adam.

@hrefname 4 года назад

Thank you so much for this. I subscribed immediately, very informative and straightforward azure info. Will definitely recommend your channel. Keep up the great work!

@AdamMarczakYT 4 года назад

Awesome, thank you!

@icici321 4 года назад

I am new to data and ETL stuff but your video's are too good. Excellent examples and very clear explanation so anyone can understand. Thanks very much.

@AdamMarczakYT 4 года назад

Thank you, always happy to help!

@ngophuthanh 2 года назад

Thank you, Adam. As always, you rock.

@yashmeenkhanam3451 4 года назад

Outstanding !You just made Azure easy to learn. Thank you.

@AdamMarczakYT 4 года назад

Awesome, thank you!

@johnfromireland7551 2 года назад

ADF is but just one part of about 100 significant tools and actions in Azure. :-(

@CallousCoder 2 года назад

Hi Adam, is it possible to create these pipelines as code as well? Or somehow create them from my actual Azure pipeline? It would be sheerly insane (but it is a Microsoft product) to require and maintain two pipeline one that’s yiur Azure pipeline for CI and CD and one for the ADF. I really would want the Azure pipeline to be able to fill/create the ADF pipeline. But I haven’t found anything yet.

@susanmyers1 4 года назад

@ Work I'm having to build out a Data Mart with no training on my own. You are literally saving my hide with your videos. THANK YOU!

@AdamMarczakYT 4 года назад

Glad to help! :)

@sameerdongare1113 4 года назад

Another awesome video. The best part of Mapping Data Flow was the Optimization...where we could do Partitioning.

@AdamMarczakYT 4 года назад

Thank you! Glad you like it :)

@Lego-tech 4 года назад

Very crisp and clear information, I watched many videos but Adam's contents are awesome!! Thanks dear!! All the best for future good work!!

@AdamMarczakYT 4 года назад

Thank you so much 🙂

@mohmmedshahrukh8450 Год назад

best video on azure I have ever seen❤❤

@eramitgoswami 3 года назад

Your way of explaining is outstanding, after watching it feel like Azure is very easy to learn. kindly keep sharing good videos Thank You..

@AdamMarczakYT 3 года назад

Thanks a ton :)

@arulmouzhiezhilarasan8518 3 года назад

Impeccable to know reg Mapping Data Flow, Thanks Adam!

@AdamMarczakYT 3 года назад

My pleasure!

@sarahaamir7457 3 года назад

Thank you so much Adam! this was very clear and great video and a big help for my interview and knowledge.

@AdamMarczakYT 3 года назад

Very welcome! Thanks for stopping by :)

@lowroar5127 2 года назад

So helpful! Thank you very much Adam!

@shantanudeshmukh4390 3 года назад

Wow ! Fantastic explanation.

@AdamMarczakYT 3 года назад

Glad you liked it!

@skillT01 2 года назад

👍 Its amazing , Practical implementation of Data Flow.

@SIDDHANTSINGHBCE 3 года назад

These videos are great. Helping me so much! Thanks Adam

@AdamMarczakYT 3 года назад

Glad you like them!

@abhijitk7363 4 года назад

Adam, Thanks for this excellent video. You explained almost every feature available there in data flows. Looking forward a video on Azure SQL DWH. I know it will be great to learn about it from you.

@AdamMarczakYT 4 года назад

Glad it was helpful! I'm just waiting for new UI to come to public preview then the video will be done :)

@soumikdas7709 3 года назад

Your videos are very informative and practical oriented. Keep doing .

@AdamMarczakYT 3 года назад

Thank you, I will!

@GiovanniOrlandoi7 3 года назад

Great video! Thanks Adam!

@AdamMarczakYT 3 года назад

My pleasure!

@rqn2274 4 года назад

Nice video Adam. Professional as always

@AdamMarczakYT 4 года назад

Wow, thanks!

@BijouBakson 2 года назад

It must be very challenging to do all this thing in English for you I imagine, Adam! Congratulations for pushing through despite the difficulty. 🙂

@nick6s 2 года назад

Excellent tutorials

@NehaJain-ry9sr 4 года назад

Awesome videos Adam, your videos are great help to learn Azure. Keep it up :)

@AdamMarczakYT 4 года назад

Thanks, will do!

@bharatruparel9424 4 года назад

Hello Adam, I just finished this video. Very well done indeed. Thanks and regards. Bharat

@AdamMarczakYT 4 года назад

Thanks Bharat :)

@eddyjawed 5 месяцев назад

Thank you Adam Dzienkuje, this is a great tutorial.

@wojciechjaniszewski9086 4 года назад

very well done on explaining principles of mapping data flows!!!

@AdamMarczakYT 4 года назад

Thanks a lot!

@grzegorzz4025 3 года назад

Adam, great tutorial! Kudos!

@AdamMarczakYT 3 года назад

Glad you liked it!

@isurueranga9704 2 года назад

best tutorial ever... 💪🏻💪🏻💪🏻

@generaltalksoflife 3 года назад

Hi Adam, Thank for helping us in learning new technologies. You are awesome 👌🏻👌🏻👌🏻👏👏.

@AdamMarczakYT 3 года назад

My pleasure!

@KarthikeshwarSathya 3 года назад

This was explained very well. thank you.

@AdamMarczakYT 3 года назад

You're very welcome!

@chandrasekharnallam2578 3 года назад

excellent explanation with simple scenario. Thank you.

@AdamMarczakYT 3 года назад

Glad it was helpful!

@avinashbasetty 4 года назад

Features are very interesting. want to try with the different partitioning techniques. Thank you for sharing such amazing stuff

@AdamMarczakYT 4 года назад

My pleasure! Thanks! :)

@achraferraji3403 2 года назад

Amazing Video, we want other parts !

@Lakshmi-y4x 11 дней назад

Thank you, very helpful tutorials

@skybluelearner4198 Год назад

Good explanation there.

@valentinnica6034 4 года назад

that was actually not so hard. thanks man, you're awesome.

@AdamMarczakYT 4 года назад

No! you are awesome! :)

@balanm8570 4 года назад

As useful another Awesome video Adam !!!. Excellent. It was to the POINT !!!. Keep up the good work which you have been doing for plenty of users like me. Eagerly waiting for more similar videos like this from you !!!. Can you please have some videos for Azure Search ...

@AdamMarczakYT 4 года назад

Thank you so very much :) Azure Search is on the list but there is so many news coming from Ignite that I might need to change the order. Let's see all the news :).

@jayong2370 Год назад

Thank you Adam.

@biswajitsarkar5538 4 года назад

Thanks Adam !! very informative video.Liked it a lot..

@AdamMarczakYT 4 года назад

Thanks and you are most welcome! Glad you hear it.

@RahulRajput_018 3 года назад

Thanks buddy ...Great work

@AdamMarczakYT 3 года назад

My pleasure

@rahulkota9793 3 года назад

Very useful. Thank you so much.

@AdamMarczakYT 3 года назад

Glad it was helpful!

@Cool2kid 4 года назад

Your video content is awesome!!! Your video is very useful to understand Azure concept specially for me who just started Azure journey. I would like to have one video where we can see how to deploy code from Dev to QA to Prod. How to handle connection string, parameter etc during deployment. thanks again for wonderful video content.

@AdamMarczakYT 4 года назад

ADF CI/CD is definitely on the list. It's a bit complex topic to get it right so it might take time to prepare proper content around this. Thanks for watching and suggesting ;)

@JoeandAlex 3 года назад

Brilliant way of explanation

@JoeandAlex 3 года назад

Subscribed to your channel

@AdamMarczakYT 3 года назад

Thank you, appreciated 🙏

@DrDATA-ep6mg 3 года назад

Very nice tutorial 👍

@AdamMarczakYT 3 года назад

Thank you! Cheers!

@MrVivekc 3 года назад

very good explanation Adam. keep it up.

@AdamMarczakYT 3 года назад

Thanks, will do!

@MrVivekc 3 года назад

@@AdamMarczakYT Adam do we have trail version of Azure for Learning purpose?

@dataintelligence6262 4 года назад

Great job! Thanks for all

@AdamMarczakYT 4 года назад

Thank you too!

@sudarshanbhattacharjee4411 2 года назад

Thanks for such good video

@atiry839 3 года назад

Wow,I like your video, I did it today. and I had good result. thanks for your good explanation.

@AdamMarczakYT 3 года назад

Great job! Thanks!

@horatiohe 3 года назад

thanks for the great content!! you are the man :)

@AdamMarczakYT 3 года назад

I appreciate that!

@dintelu 3 года назад

Wow..lucid explanation..

@AdamMarczakYT 3 года назад

Glad you think so!

@rajanarora6655 2 года назад

Your videos are really great and helped me understand lot of concepts of Azure. Can you please make one using SSIS package and show how to use that within Azure Data Factory

@Rafian1924 3 года назад

Lovely bro!!

@AdamMarczakYT 3 года назад

Thanks 🔥

@arun06530 3 года назад

nice & detailed video.

@AdamMarczakYT 3 года назад

Thank you!

@yashgemini4024 3 года назад

Appreciate you content. Thanks.

@AdamMarczakYT 3 года назад

My pleasure! :)

@harshapatankar484 3 года назад

Amazing videos.

@AdamMarczakYT 3 года назад

Glad you think so! :)

@javm7378 2 года назад

I really like your tutorials. I have been looking for a "table partition switching" tutorial but haven't found any good ones. May be you could do one for us? I am sure it'll be very popular as there aren't any good ones out there and it is an important topic in certifications :-)

@jayakrishna9153 4 года назад

very good explanation..keep doig

@AdamMarczakYT 4 года назад

Thanks. Will do.

@thisiszico2006 4 года назад

Awesome again.

@AdamMarczakYT 4 года назад

Glad you think so!

@carlossalinas8497 4 года назад

Adam you have an ability at explaining complex things, this tutorial made my day, thanks

@AdamMarczakYT 4 года назад

Glad it helped! Thanks!

@mikem8915 4 года назад

Outstanding.

@AdamMarczakYT 4 года назад

Thanks Mike ;)

@niteeshmittal 4 года назад

Adam, Your content is always easy to grab, excellent work mate. Could you please explain how to create a pipeline which has a copy activity followed by a mapping data flow activity.

@AdamMarczakYT 4 года назад

Thanks, just drag and drop copy activity and data flow blocks on the pipeline and drag a line from copy to data flow activity.

@khurramshehzad980 Год назад

very nice

@gbw5679 4 года назад

Thanks Adam!

@AdamMarczakYT 4 года назад

My pleasure!

@RahulJain-wk2wv 3 года назад

awesome!

@AdamMarczakYT 3 года назад

Thank you! Cheers!

@549srikanth 3 года назад

I would say this is the best content I've seen so far!! Thank you so much for making it Adam! Just wondering, is there a Crtl+Z or Crtl+Y command in case we did some changes in the dataflow and restore it to previous version?

@AdamMarczakYT 3 года назад

Awesome, thanks! Unfortunately not, but you can use versioning in the data factory which will allow you to revert to previous version in case you broke something. Highly recommended. Unfortunately not reverts for specific actions.

@549srikanth 3 года назад

@@AdamMarczakYT Excellent!! Thank you so much for your reply!

@johnfromireland7551 2 года назад

@@549srikanth I publish each time I create a significant new step in the pipeline and I use data preview before moving on to the next step. Also, you can , I think, export the code version of the entire pipeline. Presumably you can, then, paste that into a new Pipeline to resurrect your previous version.

@JoshuaDHarvey 4 года назад

Great video thank you

@AdamMarczakYT 4 года назад

Thanks again! :)

@01sanjaysinha Год назад

Thanks!

@bpittala 5 дней назад

very good

@desparadoking8209 3 года назад

Thanks for the informative and detailed video adam, 😊👌. Your content is practical. Can you make a video on how load the data from Oracle table to azure data factory? It would be helpful for audiences.

@AdamMarczakYT 3 года назад

Thanks! That's the plan :)

@anubhav2020 2 года назад

Hello Adam, thanks a bunch for this excellent video. The tutorial was very thorough and anyone new can easily follow. I do have a question though. I am trying to replicate an SQL query into the Data Flow, however, I have had no luck so far. The query is as follows: Select ZipCode, State From table Where State in ('AZ', 'AL', 'AK', 'AR', 'CO', 'CA', 'CT'...... LIST OF 50 STATES); I tried using Filter, Conditional Split and Exists transforms, but could not achieve the desired result. Being new to the Cloud Platform, I am having a bit of trouble. Might I request you please cover topics like Data Subsetting/Filtering (WHERE and IN Clauses etc.) in your tutorials. Appreciate your time and help in putting together these practical implementations.

@willydonkeng339 Год назад

Perfect

@RC-nn1ld 3 года назад

Love these videos so easy to understand, do you have a video on new XML connector

@AdamMarczakYT 3 года назад

Great, thanks! Not yet, maybe in near future :)

@velupalani2496 4 года назад

Velu well done

@AdamMarczakYT 4 года назад

Thank you!

@abu1479 3 года назад

Adam, excellent presentation of ADF concept. I find all your videos really helpful in understanding the ADF concept. One question in regards to the sink dataset in dataflow, how can I create dynamic folder in my blob storage based on the year, month and day when this dataflow was triggered?

@AdamMarczakYT 3 года назад

Depends on what do you want to achieve. You can either set partitioning by date column which will split the data by date. Or if you want to put entire dataset in one folder using date then use formatDateTime expression like formatDateTime(utcNOw(), "yyyy/MM/dd") as path.

@prashanthkumar6731 3 года назад

Thanks adam

@AdamMarczakYT 3 года назад

My pleasure!

@alfredsfutterkiste7534 2 года назад

Great

@chsrkn 4 года назад

Features are very interesting and thanks for your clear explanation , could you explain bit more reusable data flows , I mean to use same data flow for multiple tables/files like reusable pipelines.

@AdamMarczakYT 4 года назад

Thanks for suggestion, I will try to make video on data flow parametrization in the future. Thanks for watching :)

@samb9403 2 года назад

Great video. Question: Under "New Datasets", is there a capability to drop data into Snowflake? I see S3, Redshift, etc. I appreciate the video and feedback!

@kunalr_ai 3 года назад

Thanks

@AdamMarczakYT 3 года назад

You're welcome!

@erick1992santos 4 года назад

Excellent content!! I just have one doubt, where I can find documentation about the functions of scala? I do not know anything about it. Just have subscribed your channel! Thanks a lot!!

@AdamMarczakYT 4 года назад

Tank you Erick! I think the list of available functions are standard for Spark. I never checked if they all match but you can find them here spark.apache.org/docs/2.4.4/api/sql/

@nidhisharma-rb7nx 2 года назад

Adam, great video.I m new to Data Flow and I have one doubt, I want to implement File level checks in Data Flow but not able to do it. All tasks are performing data level checks like exist or conditional split. Is it possible to implement File level check like whether file exist or not in storage account?

@tenghover 3 года назад

Would you plan to make video for introduction of each transforamtion components? Thanks

@gursikh133 4 года назад

Adam, FOr using transformation do I need to learn scala. Or just refer the documentation you specified for scala functions and write the transformation?

@AdamMarczakYT 4 года назад

Documentation should be enough. MDF is targeting simple transformations so in most cases documentation alone will suffice.

@adiky 3 года назад

Hi Adam, that's a great tutorial, many thanks for it. I have a question that can we write the transformation functions in different language like Python or R instead of Scala? If yes can you please share some details on it?

@AdamMarczakYT 3 года назад

Unfortunately not right now :( If you need those then use Azure Databricks instead.