No video :(

Part 4- Data Ingestion (2) | End to End Azure Data Engineering Project

Подписаться 20 тыс.

Просмотров 33 тыс.

50% 1

#azuredataengineer #endtoendproject #azuredataengineeringproject #azureintamil #azuredatafactory #azuredatabricks #azuresynapseanalytics #azuredatalake #datalake #powerbi #keyvault
This is a long awaited video of mine- Lets build a complete End to End Azure Data Engineering Project. In this project we are going to create an end to end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.
The tools that are covered in this project are,
1. Azure Data Factory
2. Azure Data Lake Storage Gen2
3. Azure Databricks
4. Azure Synapse Analytics
5. Azure Key vault
6. Azure Active Directory (AAD) and
7. Microsoft Power BI
The use case for this project is building an end to end solution by ingesting the tables from on-premise SQL Server database using Azure Data Factory and then store the data in Azure Data Lake. Then Azure databricks is used to transform the RAW data to the most cleanest form of data and then we are using Azure Synapse Analytics to load the clean data and finally using Microsoft Power BI to integrate with Azure synapse analytics to build an interactive dashboard. Also, we are using Azure Active Directory (AAD) and Azure Key Vault for the monitoring and governance purpose.
Part 5 will be uploaded soon. Stay tuned.
- - - Book a Private One on One Meeting with me (1 Hour) - - -
www.buymeacoff...
- - - Express your encouragement by brewing up a cup of support for me - - -
www.buymeacoff...
- - - Other useful playlist: - - -
Azure Data Factory Playlist: • Azure Data Factory Tut...
Azure General Topics Playlist: • Azure Beginner Tutorials
Microsoft Fabric Playlist: • Microsoft Fabric Tutor...
Azure Databricks Playlist: • Azure Databricks Tutor...
Azure End to End Project Playlist: • End to End Azure Data ...
Databricks CICD Playlist: • CI/CD (Continuous Inte...
End to End Azure Data Engineering Project: • An End to End Azure Da...
- - - Let’s Connect: - - -
Email: mrktalkstech@gmail.com
Instagram: mrk_talkstech
- - - Tools & Equipment (Gears I use): - - -
Disclaimer: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide, I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!
DJI Mic: amzn.to/3sNpDv8
Dell XPS 13 Plus 13.4" 3.5K : amzn.to/45KqH1c
Rode VideoMicro Vlogger Kit: amzn.to/3sVFW8Y
DJI Osmos Action 3: amzn.to/44KYV3x
DJI Mini 3 PRO: amzn.to/3PwRwAr
- - - About me: - - -
Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN ABOUT THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"
I will be creating playlist which covers the below topics (with DEMO)
1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)
After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,
1. Data Engineer
2. Data Analyst
3. Data Scientist
Can't wait to help people with my videos.
- - - Support me: - - -
Please Subscribe: / @mr.ktalkstech

Опубликовано:

24 май 2023

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 98

@parullnpriyaadarshani8512 Год назад

I must say. This is the best data engineering project. Thank you so much for all your efforts and great explanation.

@mr.ktalkstech Год назад

Thank you so much :)

@prasanthbugga8381 9 месяцев назад

Honestly, You are a legend...its just awesome, the way you are teaching is superb...i was into GCP and came across this project, with this single project, you are making me to choose between azure and GCP, thank you so much Mr.K

@mr.ktalkstech 9 месяцев назад

Thank you soo much :)

@kasireddymahesh8241 21 день назад

Good work. It really helps to start journey on azure data enginner

@chubsmash7602 2 месяца назад

Great video, simple and short, and yet covering pretty most of the important things you need to migrate to Azure.

@prabhatgupta6415 Год назад

Good Going plz bring more projects

@helovesdata8483 Год назад

The best! I'm coming from AWS and my new job needed help with an Azure initiative. Your videos are helping me to get up to speed quickly.

@mr.ktalkstech Год назад

Glad to hear that, All the best :)

@girijasahu7409 3 месяца назад

great example end to end project. This is very useful for any Azure ADF projects.

@mr.ktalkstech Месяц назад

Thank you so much :)

@user-rm7yn7us8q 11 месяцев назад

Thank you so much for all your efforts and great explanation. Plz bring more projects.

@lifeitnow9600 5 месяцев назад

So much of confidence while you deliver the content!! truly a note to make..... thanks for putting up the efforts to build the azure playlists... keep going!!! Need more such videos

@mr.ktalkstech 4 месяца назад

Thank you so much :)

@4nxh___ 6 месяцев назад

I appreciate your great effort and time given to spread Data Engineering-Project knowledge. Well Delivered!

@mr.ktalkstech 6 месяцев назад

Thank you so much :)

@user-qt8bg9ov6e 8 месяцев назад

This is awesome. I’m glad I chose the Azure cloud.

@mr.ktalkstech 7 месяцев назад

Thank you so much :)

@gaganaulakh41 4 месяца назад

The best content I have ever seen so far on youtube!!!!!

@mr.ktalkstech 4 месяца назад

Thank you so much :)

@90harvinder 6 месяцев назад

The best video on ADF i have seen. Bravo.

@mr.ktalkstech 4 месяца назад

Thank you so much :)

@anithaveeramani8909 2 месяца назад

Great Video Mr.K. I am completely new to Azure, But I am able to follow since it is well explained in short. Thank you!!!

@mr.ktalkstech Месяц назад

Thank you so much :)

@Techyprof Месяц назад

Thanks for the knowledge sharing! Another option is to concatenate in the SQL script on SSMS and you won't have to do this in ADF.

@selairendri9997 2 месяца назад

Excellente Course

@mr.ktalkstech Месяц назад

Thank you so much :)

@anjibabumakkena Год назад

Great explanantion sir, Thank you very much for providing these videos

@salmashaik6294 4 месяца назад

One of the best explanations ever. Really liked this series

@mr.ktalkstech 4 месяца назад

Thank you so much :)

@salmashaik6294 4 месяца назад

@@mr.ktalkstech Can I know if the same project applicable for incremental data?

@GloryBharat 5 месяцев назад

Very well delivered. Super impressive and helpful. Love the passion with which you deliver. Very easy and informative to understand. Keep it up

@mr.ktalkstech 4 месяца назад

Thank you so much :)

@Mark_Trail 3 месяца назад

Great stuff! I'll keep watching!

@mr.ktalkstech Месяц назад

Thank you so much :)

@shashibhushansingh1628 Год назад

great man now iam feeling higly confident in azure thanks man

@mr.ktalkstech Год назад

Glad it helped you :)

@pritidongarwal739 2 месяца назад

Very good video . I like the short video and it gives an idea of the key steps to be followed.

@mr.ktalkstech Месяц назад

Thank you so much :)

@user-of1br4hd8m 7 месяцев назад

Very Good Explanation, please come up with more Real time projects which will be helpful. May the coming year i'll reach my short term goal with your teaching and support.

@mr.ktalkstech 7 месяцев назад

Thank you so much :) Sure :)

@sraoarjun 8 месяцев назад

This is such a good presentation and to the point crisp explanation !!

@mr.ktalkstech 7 месяцев назад

Thank you so much :)

@manjunathbn9513 Год назад

Great video. Shared in whatsapp group. Pls add all videos asap. helpful to many people. thanks.

@mr.ktalkstech Год назад

Thank you so much :) Sure, will upload the rest asap

@easypeasy5523 7 месяцев назад

You are simply amazing bro thank you for these content this such a premium level

@mr.ktalkstech 7 месяцев назад

Thank you so much :)

@user-wx7os2zo8z Год назад

hi sir please upload next videos asap, and do more these kind of videos, thank you for great teaching.

@pridename2858 11 месяцев назад

Wow!! excellent sir,this is called teaching...where teacher is able to inject the the real materials in the students mind. hats off for you sir. just curious if you have covered, how we will get SSMS on our laptop, so we can use the sample DB to injest in bronze. Thanks

@mr.ktalkstech 11 месяцев назад

Thank you so much :) Please check the description of the video to import the sample DB :)

@kirtiagg5277 Год назад

wonderful work..Thank you so much for good explaination !!

@mr.ktalkstech Год назад

Thank you so much :)

@ranjansrivastava9256 Год назад

Dear Kishor, One suggestion here is :- Here you have written the SQL Script to get all the lists of tables using the sys tables and sys schema, it's really works well but suppose QA/Developers create similar kind of tables for their testing we can not control by this approach. So My opinions are [a]. We can create a table which has table name, table schema, active flag and will create SP that will fetch all the active tables by using the Lookup activity and will do the further activities. [b]. We can create json file with table names and schemas and upload into the ADLS and we can read from there using the lookup activity. Kindly correct me

@mr.ktalkstech Год назад

The synapse views will be automatically reflected as long as there is no schema change happening in the source.

@ericbernal8714 Год назад

Great job! bring more projects.

@mr.ktalkstech Год назад

Thank you :)- sure, will do.

@noufalrijal9811 7 месяцев назад

To be honest, the presentation is quite straightforward to follow, even for someone like me who comes from an AWS background. I would like to know if there is a method to create this without using drag and drop and instead use code.

@johnsonjoseph6910 День назад

Great video!.Just want to know one thing. You have copied 10 tables using copy activity and it saved as parquet format in Datalake. if this same pipleine running next time. Is it copy entire table data or incremental data? I just want to know ..How can we add incremental data?

@ChanceMinus 9 месяцев назад

Outstanding. Thank you.

@RohanKumar-mh3pt 7 месяцев назад

very amazing can you make one video for Full and Incremental Data load for Ingestion part?

@SuperAnurag121 7 месяцев назад

Awesome

@sidsan000 7 месяцев назад

Thanks a lot its really a good hands on experience. Right now I have two issues, can you pls help on this 1. in Azure blob storage a o byte file is getting created every time when a folder is generated. 2. My Cluster is not getting up. can you pls help on this

@centebatech Год назад

This looks very similar to ETL looks like Informatica? Why would you use this instead of a standard ETL tool that is portable if you decide to not use Azure and want to go to, say, AWS or GCloud? Thanks

@ranjansrivastava9256 Год назад

Dear Kishor. Keeping performance in mind , up to what maximum number of tables can be ingested from source to destination with more data coming in the future. How can we handle this situation. I am asking because I am facing the same issue, might others benefited as well from this.

@mr.ktalkstech Год назад

It depends upon the volume of each tables, mostly determined based on the trial and error method :)

@zuhailiaminuddin520 2 месяца назад

I got this error message when trigger the pipeline 'Operation on target ForEach1 failed: Activity failed because an inner activity failed' do i need to provide actual value of the parameters to preview data of @item().SchemaName and @item().TableName in the source

@zuhailiaminuddin520 2 месяца назад

ok i know where i do wrong @{concat('SELECT * FROM^', item().SchemaName,'.', item().TableName)} in this syntax i should put 'space' indicate by '^' .. i discover this when i see the input of the fail trigger activities

@user-fe4gm3qv1o 11 месяцев назад

Hello.. its a great learning video... I have 4 yrs gap & i was stuck to choose Salesforce vs data engineer field. as many say data engineering is hard & complex tasks involved. does all the azure data engineer projects follow the same/similar kind of steps as you have explained in this whole project

@pridename2858 11 месяцев назад

Hi User, I would say, keep all the things done in this project in mind a rough Idea and watch another e-to e project. They are similer, sometime tools can be different...so I would say its a great start. and solid start.

@centebatech Год назад

What is the language you are using for all the macros and scripts you are writing as you build out the pipeline? Do you have a tutorial for that? Nice job, by the way!

@mr.ktalkstech Год назад

It's called as ADF expressions. I haven't posted any videos specifically for that, but you can see the below Microsoft documentation to get familiar with that, cheers :) learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions#functions

@nilgiripaiya Год назад

very w well explained, Could you please complete the complete project flow video

@mr.ktalkstech Год назад

Thank you :) will do it asap

@hritiksharma7154 Год назад

Hii , your videos are great. Can you please create a video on unity catalog in databricks

@mr.ktalkstech Год назад

Thank you :) Sure, will do in the near future.

@naturetime5266 Год назад

can we upload data manually? instead of connecting with SQL server!

@mr.ktalkstech Год назад

Yup, you can do that- if you have CSV files then you can just upload it to the RAW zone and then process it using Databricks

@shiinobeats1329 Год назад

I'm getting pipeline fails due to this errror ErrorCode=InvalidTemplate, ErrorMessage=Unable to parse expression '(concat('SELECT * FROM ', item().SchemaName, '.', item ().TableName)}'

@mr.ktalkstech Год назад

should be something to do with the syntax based on your comment, I am seeing a space between item and () , it should be item().TableName. Please have a look if there is a space between item and ()

@shiinobeats1329 Год назад

@@mr.ktalkstech thank you for responding. I’ve tried everything, is there somewhere I can get the original syntax

@mr.ktalkstech Год назад

This is the original syntax that I used in the demo, @{concat('SELECT * FROM ', item().SchemaName, '.', item().TableName)}

@anjibabumakkena Год назад

Kindly provide the Realtime scenario series about pyspark

@mr.ktalkstech Год назад

Sure :)

@ranjeetkumarkushwah6750 Год назад

Hi Sir, I am geeting the error like " incorrect syntax near 'fromEmp,},],' Emp is my schema name

@mr.ktalkstech Год назад

Should be a typo in your script, did you fix this?

@ranjeetkumarkushwah6750 Год назад

@@mr.ktalkstech Hi Sir, I had tried many times but each and every time i got the same error "Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Incorrect syntax near 'FROMEMP'.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Incorrect syntax near 'FROMEMP'.,Source=.Net SqlClient Data Provider,SqlErrorNumber=102,Class=15,ErrorCode=-2146232060,State=1,Errors=[{Class=15,Number=102,State=1,Message=Incorrect syntax near 'FROMEMP'.,},],' "