Azure Data Factory Parametrization Tutorial

Подписаться 201 тыс.

Просмотров 174 тыс.

50% 1

Parametrization in Azure Data Factory is essential to achieve good design and reusability as well as low cost of solution maintenance. Using parameters also speeds up implementation of new features in your pipelines.
In this video I will cover basics of Data Factory Parametrization using common Blob to SQL loading scenario.
Source: github.com/MarczakIO/azure4ev...
Want to connect?
- Blog marczak.io/
- Twitter / marczakio
- Facebook / marczakio
- LinkedIn / adam-marczak
- Site azure4everyone.com
more to come..
Next steps for you after watching the video
1. Check data factory docs docs.microsoft.com/en-us/azur...
2. Linked service parametrization example
docs.microsoft.com/en-us/azur...
3. System variables
docs.microsoft.com/en-us/azur...
See you next time!

Наука

Опубликовано:

4 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 371

@rajdeepmandal1470 4 года назад

The quarantine days wouldn't be this useful without your teaching :) thank you Adam. You are the best!!!

@AdamMarczakYT 4 года назад

Wow, thank you! :)

@tatianebrandao5487 8 месяцев назад

I'm just starting with Synapse/DF and I found your video extremely helpful for beginners. I managed to go through your video, complete the exercise on my own and everything made so much sense! Your way of explaining is very clear! Thank you very much for creating the video :)

@nupoorsingh3323 2 года назад

It's great support for people like me who do not have a mentor. Thanks for uploading these videos and helping me to understand the workflow and survive.

@nikhilpatil2021 4 года назад

Thank you Adam, this is the best tutorial on Azure Data Factory I have seen! I am definitely going to try these demos. Thank you!

@AdamMarczakYT 4 года назад

Great to hear! Cheers!

@johnd3601 3 года назад

Thanks this is a big help, its amazing how much you don't know you know. This has opened up a lot of possible automation and saved me loads of time once I manage to apply this to my situation.

@carl33p 4 года назад

Demos are the best way to learn, and your demo is as good as it gets. Build, run, build, run- the incremental style allows viewer to understand slowly.

@AdamMarczakYT 4 года назад

Couldn't agree more! Thanks!

@ghay3 3 года назад

My goto RU-vid channel for anything Azure - you are doing the greatest service by sharing your knowledge and responding to all the queries! many thanks, Adam. Hope you can also enlighten the community on AWS :)

@AdamMarczakYT 3 года назад

Wow, thank you! glad to help!

@nicholasgregory4774 3 года назад

Another great one Adam, consider me a new student! Short enough in length while fitting all the relevant information in + easy to digest, without overly complicating things. Your PowerPoints with your explanations are second to none.

@AdamMarczakYT 3 года назад

Awesome, thank you!

@eerosiljander4622 Год назад

Thanks Adam for these hands on videos. Best wishes,ES, Data engineer & Scientist.

@raviv5109 4 года назад

Are you real? You really explain and show demo in such a flawless manner. Thanks you so much, your Azure videos are happy at every step.

@AdamMarczakYT 4 года назад

Thank you so much :)

@santanughosal9785 2 года назад

Awesome, simple and hands on video to understand the concept. Thank you

@sankaji20 3 года назад

You are doing great job ADAM, 20-30 minutes videos of your's is equal to 60minuts of others. And quality of your vdo is un match-able. Now I am GCP guy but also very much interested in learning azure from your videos only.. keep uploading azure videos. Thanks so much.. doing wonderful job for learners.. you will get a lot of blessings.. Thumbs-up for you..

@AdamMarczakYT 3 года назад

Wow, thanks! Appreciated!

@barryjohnson8915 4 года назад

Really excellent tutorials. Very simply explained, easy to follow, fantastic stuff.

@AdamMarczakYT 4 года назад

Thanks Barry :) Glad to have you here.

@ashishjuneja6574 2 года назад

Super awesome !! Great going Adam.

@lehai9172 3 года назад

Fantastic tutorials, I've learned a lot from your videos. Thank you for sharing your experiences, Adam.

@AdamMarczakYT 3 года назад

My pleasure!

@mashagalitskaia8642 4 месяца назад

Thanks a ton for this video! It helped me a lot. Very precise and detailed, I had no questions left:)

@joshiabhinav 4 года назад

Man you are awesome. you present so well and explain so well. Love the way you teach !

@AdamMarczakYT 4 года назад

I appreciate that!

@TeymurRzayev 2 года назад

Actually liked it before watching the video ;) Brilliant content, perfect explanation!

@mayur6816 3 года назад

Thanks, Adam. This is the best tutorial on ADF.

@AdamMarczakYT 3 года назад

Glad you think so! Thanks!

@IntelliCloudTech 2 года назад

Clearly explained at every point. thanks for the video.

@SuperSanjays 2 года назад

Great information you have shared Adam. It helps me a lot.

@igiq 5 лет назад

Keep it up Adam, your videos are very helpful !

@AdamMarczakYT 5 лет назад

Thanks! will do!

@iChrisBirch 7 месяцев назад

Great tutorial, answered my questions as I had them, great logical flow.

@eatingnetwork6474 3 года назад

great video, once again very clear and concise, thanks Adam

@AdamMarczakYT 3 года назад

Thanks 👍

@sazidaanjumsheikh6817 2 года назад

Thank you Adam, in a very short video you made me understand the ADF with a demo instead of Theory.. Nice content... :-)

@arnabmondal5596 4 года назад

Awesome clip. Currently, better content than many available now.

@AdamMarczakYT 4 года назад

Thank you so much :) glad that you enjoyed it.

@jayong2370 2 года назад

Awesome tutorial! Thank you.

@sraoarjun 3 года назад

One word - FANTASTIC !! ... Thank you Adam , keep up the good work in educating several people .Also request you to put some videos on Synapse analytics ...

@AdamMarczakYT 3 года назад

Thanks! Will do!

@joseantonioestebanrodrigue448 3 года назад

Perfect explanation!

@dabay200 4 года назад

This is a really amazing video, so much better than Microsoft's tutorials.

@AdamMarczakYT 4 года назад

Thanks again Dinal. You are too kind :).

@srinivasreddysankasani5140 3 года назад

Thank you Adam for your videos. your explanation is really awesome. looking for more videos

@AdamMarczakYT 3 года назад

Glad you like them!

@MarkDiamond 3 года назад

Excellent content, thank you so much for the effort!

@checomax1980 3 года назад

Thank you Adam, Excellent tutorial.

@AdamMarczakYT 3 года назад

Glad you enjoyed it!

@kalangarahouse 2 года назад

nice tutorial...please continue making such videos

@oscararmandocisnerosruvalc8503 Год назад

Your videos are amazing bro !!!!!

@baladenmark 3 года назад

Wow. Thank you Adam. I really can't thank you enough.

@AdamMarczakYT 3 года назад

Glad I could help!

@Bubu020174 2 года назад

Thank u so much to you that you have share this valuable information !

@venkataramakrishnamagapu7645 3 года назад

Excellent content. Simple and informative.

@AdamMarczakYT 3 года назад

Much appreciated!

@angelamessina7187 4 года назад

Thanks a lot Adam this will be realy helpful for my today's task. Be blessed

@AdamMarczakYT 4 года назад

Great! Thanks Angela, stay safe!

@malayku8781 4 года назад

Very nicely explained Adam!

@AdamMarczakYT 4 года назад

Thank you kindly! :)

@soumenchandra7633 4 года назад

What a fascinating presentation !!!

@AdamMarczakYT 4 года назад

Thank you! :)

@BojackCartman 3 года назад

nice tutorial, very clear and concise

@AdamMarczakYT 3 года назад

Glad you think so!

@fsfehico 4 года назад

Very well structured content. Thank you

@AdamMarczakYT 4 года назад

Thank you, glad you enjoy it :)

@jagadishparale8632 4 года назад

Hey Adam,It was excellent tutorial.I would like you to do more videos on deployment of Azure Data Factory pipelines when we use parameters

@AdamMarczakYT 4 года назад

Thanks! You are in luck, new video tomorrow is also on ADF, stay tuned!

@docondelta 4 года назад

This is really amazing tutorial .. Thank you

@AdamMarczakYT 4 года назад

Cheers! Always happy to help!

@asrarhaq3234 4 года назад

Wonderful - practical and useful- Thank you

@AdamMarczakYT 4 года назад

Thanks! Glad you like it.

@pavankumar-nm9yu 3 года назад

Thank You Adam, Amazing Explanation...!!!

@AdamMarczakYT 3 года назад

My pleasure!

@Unbox747 2 года назад

I love your videos!

@vishwanathr3507 4 года назад

Hi Adam Thank you for excellent Tutorial and your time. Good luck Cheers! Vishwanath

@AdamMarczakYT 4 года назад

Glad you liked it! :)

@snmailist1470 3 года назад

Thank you Adam. I'm very appreciate your effort 👍👍👍👍 You give me a description what's going on the cloud.

@AdamMarczakYT 3 года назад

Glad to hear it , thank you!

@MrVoltelen 3 года назад

Thanks for the great explanation

@AdamMarczakYT 3 года назад

My pleasure!

@RohkeaNox Год назад

Very helpful, thank you :) !

@shaileshsondawale2811 Год назад

Great work...!!!!

@GhernieM 4 года назад

Thanks Adam! Very helpful

@AdamMarczakYT 4 года назад

My pleasure!

@sagarnegi 3 года назад

Great explanation. I understood everything.

@AdamMarczakYT 3 года назад

Excellent

@ricardorocha7882 Год назад

Simplesmente incrível esse vídeo. Vai me ajudar bastante.

@lorenzodelsoldato6899 6 месяцев назад

Thank you, really good content

@TheSQLPro 3 года назад

Great tutorial!

@AdamMarczakYT 3 года назад

Thank you!

@devopsexpert9906 3 года назад

Excellent Video, Best tutoring.......

@AdamMarczakYT 3 года назад

Many many thanks

@jayantKadam100 4 года назад

This is really wonderful tutorial and very useful thanks

@AdamMarczakYT 4 года назад

Thank you! :)

@danceandmanymorewithprisha5599 4 года назад

great explanation !

@AdamMarczakYT 4 года назад

Thank you!

@priyankapatel9461 3 года назад

Really very helpful!

@AdamMarczakYT 3 года назад

Thanks!

@_indrid_cold_ 4 года назад

Best content! Thank you so much!

@AdamMarczakYT 4 года назад

Awesome, cheers!

@ambrishl9273 4 года назад

Thanks a lot, This was very helpful!!!

@AdamMarczakYT 4 года назад

Thank you :)

@oscarsalas6271 3 года назад

Excelente video, me ayudo mucho

@luisabaiano8689 4 года назад

Thank you! From Italy

@AdamMarczakYT 4 года назад

And thank You for watching Luisa :)

@ivanovdenys 4 года назад

Thanks a lot !!! Really really good !!

@AdamMarczakYT 4 года назад

Thanks! And thank you for watching Denys :)

@leonkriner3744 4 года назад

very helpful. Thank you!

@AdamMarczakYT 4 года назад

Glad you liked it!

@tarvinder91 4 года назад

very informative and helpful video

@AdamMarczakYT 4 года назад

Glad it was helpful!

@amitsingh8412 3 года назад

Thanks Adam.

@AdamMarczakYT 3 года назад

My pleasure!

@Peter-cd9rp 4 года назад

cool dude. Especially pipeline parameters! :)I can easily start pipelines with logic apps! :D Wohoo

@AdamMarczakYT 4 года назад

Glad you liked it!

@geethamca88 Год назад

Excellent

@bookko2377 4 года назад

Thank you From Thailand

@AdamMarczakYT 4 года назад

My pleasure! :)

@sameerdongare1113 4 года назад

Awesome video

@AdamMarczakYT 4 года назад

Thaaaanks :)

@tanmoybhattacharjee4966 3 года назад

thank you, amazing tutorials , really helpful

@AdamMarczakYT 3 года назад

Thanks! :)

@alfredsfutterkiste7534 2 года назад

Love it

@sonamkori8169 2 года назад

Thank You Sir

@kannan114 2 года назад

Thanks Adam for the great video. I have a question: how do we put this entire pipeline in loop? Let's say I create a list of source files and destination tables and run this pipeline for each set of values in that list. That way I don't want to enter the value manually during every run/debug. Is this something that can be done at trigger? Any suggestions on that?

@blse2000 4 года назад

awesome bro

@AdamMarczakYT 4 года назад

Thank you! Happy and joyful x-mas!

@rishikabapat6737 4 года назад

Thank you Adam for such great videos! I'm not sure if you have covered this in any video, but my question is around transformation of the csv before its loaded into the SQL DB. For example I have a csv file with years as columns and I need to translate them into data fields. Is there any way to do that using Data Factory, or is the only way to re-format in excel before I load it?

@AdamMarczakYT 4 года назад

Yes, absolutely! Check out my video on Data Factory Mapping Data Flows it shows exactly what you need ;) thanks for watching!

@atulsalunke6675 3 года назад

Awesome content and delivery Adam !! Thanks!! Please if possible add real Project based content using ADF

@AdamMarczakYT 3 года назад

Great suggestion! I do plan to have few videos on actual implementations :)

@mahendramanchekar 2 года назад

Thank you

@julianramirezvasquez3164 2 года назад

Hi Adam, great video as always! I have a question, is there a way to configure Data Factory in order to send files automatically from a Blob storage and then transform the files automatically as well? if that's possible, how can I do that? Thank you so much!

@Viqram1 Год назад

Hi Adam, Thank you for making such informative videos. I was trying to follow the instructions that you've mentioned in the video. However I wasn't able to load both the files using parameterisation. When I try to load cars.csv, it loads properly, and then when I try to load planes.csv, it throws an error saying column's not found. In order to reolve it, I just clicked on "Import Schemas" under sink tab and I was able to load planes.csv. Why do I have to import schemas while changing source files?

@mumair1979 3 года назад

Hi Adam, This is fantastic. How can we take parameters from a SQL table, by using a stored procedure, instead of putting them in each time into the Datasources ?? can you please advise.. Thanks

@mrboloban25 3 года назад

Great video Adam, thanks. If I am able to get the schema and table list from a db with a Lookup, how could I export this to a csv file? Thanks.

@sumashree6351 2 года назад

Thank you very much good explanation.i have one dout which parameter used to improve the performance of the company for the past 6 years

@srinivasareddy3336 2 года назад

Hi Adam, thanks for the great video tutorial. May I know how to pass variable values from one pipeline to another pipeline in Azure Data Factory.

@wouldyoudomeakindnes 4 года назад

thnk you so much!!

@AdamMarczakYT 4 года назад

You're welcome!

@krishnakoirala2088 2 года назад

Adam Thanks for all those great videos. A question: I have a parameterized stored procedure as an activity and I am passing parameter values with in ADF. I want to capture value of one parameter dynamically that is passed on the stored procedure, how do I do that? I need to capture this value within databricks notebook via widget. I have defined a Base parameters in databricks notebook activity something like this (@activity('DeleteDaysBack').output), here DeleteDaysBack is the stored procedure and parameter of which I am trying to capture in databrick activity. Only the Input has that parameter, but ADF does not support Input, throws an error. Any insight would be helpful, thanks in advance.

@rajexpert8544 2 года назад

Hi Adam, Nice tutorial. How can I parameterize the pipeline if I want to loop through all the containers in blob storage and with the corresponding sql table name? I don't want to enter parameter values manually at run time

@chen5576 4 года назад

Hi Adam! This was a great tutorial! Do we always need to pre-define a table in SQL before moving data from blob? Could we create a new table if the target sink name does not exist inside the pipeline?

@AdamMarczakYT 4 года назад

Thanks! For the table, you can select checkbox on the copy activity to create table if it doesn't exist. Just make sure that the schema mapping is there, otherwise your table schema will be pretty bad. Although this option is not recommended as you should change your DB schema in more controlled manner.

@maxgobel 3 года назад

HI Adam, thanks for the great tutorial, was very helpful. Would I be able to follow this procedure with a dataflow instead of a copy example? I am struggling with implementing this between a rawinputBlob and a cleanedOutputBlob

@AdamMarczakYT 3 года назад

It's a little different but mapping data flows support parametrization too: docs.microsoft.com/en-us/azure/data-factory/parameters-data-flow?WT.mc_id=AZ-MVP-5003556 thanks for stopping by!

@samama1975 4 года назад

Hi Adam, thank you for throwing light into the world of Azure. I enjoy following your tutorials. Question though, Could you please point me to where I can download the PowerPoint doc you used for this tutorial? It's simply the road map that is needed to guide you as you navigate your project. Thank you

@AdamMarczakYT 4 года назад

Thank you for watching and commenting. Unfortunately while the source code and samples are open source on github, entire content are free, I decided that at this point in time I'm not sharing PowerPoint presentations as I want to maintain copyright over my materials. Thank you for understanding.

@Navi24 4 года назад

Thanks Adam for this video, this really helped. Could you also do a video/use case where-in parameters and/or variables that are set in Pipeline are used in 'Mapping Data Flow'. Also, looking forth for video on 'Wrangling Data Flows'. Thanks.

@AdamMarczakYT 4 года назад

Great suggestion! Thanks for watching :) I'll figure out if I can fit this in the schedule :).

@jonahnio5642 4 года назад

Thanks alot Adam for the video. WOuld you be able to show me how I can pass variables across different pipelines? Thanks very much

@AdamMarczakYT 4 года назад

Thanks. Check out MS blog entry on that cloudblogs.microsoft.com/industry-blog/en-gb/technetuk/2020/03/19/enterprise-wide-orchestration-using-multiple-data-factories/

@sarveshpandey1125 4 года назад

You saved a lot of my time . Thank you! can you please clarify one thing, what's the use case for using pipeline variables?

@AdamMarczakYT 4 года назад

Thanks! Parameters are for entry input, variables are for intermediate results and temporary or calculated values that change during pipeline run. They have many use cases.

@aks541 3 года назад

Nice video - just a thought: if you'd have used a FOREACH loop and inside that your copy activity was there , you probably don't need to give the name as Cars etc. Because loop took both the files from lake iteratively and pass to the copy activity one by one.

@AdamMarczakYT 3 года назад

Thanks. Correct, I show this approach in my bulk copy video ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-KsO2FHQdILs.html

@vickygana9260 4 года назад

Thanks Adam for the vedio, it would have been great if you would have also showed how to pass the filename, tablename for multiple items(cars, planes) dynamically instead of passing manually while debugging.

@AdamMarczakYT 4 года назад

I did in my dynamic bulk load and triggers video, feel free to check then out ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-KsO2FHQdILs.html and ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-uF3LOCVFHkw.html

@yyy168-wt1zq 5 месяцев назад

Hi Adam, thanks for the video, that's a clear explanation, can I ask what if I want to put Cars and Planes into a variable array, then Pipeline can have a for loop execution of doing the copy of Cars and then Planes. Is it possible?