Part 2- Environment Setup - Integrating Azure DevOps GIT Repository to Azure Databricks for CI/CD

Подписаться 19 тыс.

Просмотров 13 тыс.

50% 1

#cicd #azuredevops #devops #databricks #azuredatabricks
In this Part 2 video of CI/CD in Azure Databricks, I have discussed about the different environment setup required for building the CI/CD pipelines. Things like Integrating the Azure DevOps with Azure Databricks workspace, Branch protection and merging techniques are seen in this video with Demo.
The next parts will be uploaded soon, stay tuned!
Part #3. Creating CI Pipeline
Part #4. Creating CD Pipeline and Testing
Here’s what you’ll see in this video:
0:00:00 - Intro
0:02:10 - Creating Azure DevOps Git Repository
0:05:51 - Git Integration in Azure Databricks
0:11:23 - Branch Protection
0:20:13 - Merging the changes to Main
0:28:00 - Outro
- - - Book a Private One on One Meeting with me (1 Hour) - - -
www.buymeacoffee.com/mrktalks...
- - - "Express your encouragement by brewing up a cup of support for me" - - -
www.buymeacoffee.com/mrktalks...
- - - Other useful playlist: - - -
Azure Data Factory Playlist: • Azure Data Factory Tut...
Azure General Topics Playlist: • Azure Beginner Tutorials
Microsoft Fabric Playlist: • Microsoft Fabric Tutor...
Azure Databricks Playlist: • Azure Databricks Tutor...
Azure End to End Project Playlist: • End to End Azure Data ...
Databricks CICD Playlist: • CI/CD (Continuous Inte...
End to End Azure Data Engineering Project: • An End to End Azure Da...
- - - Let’s Connect: - - -
Email: mrktalkstech@gmail.com
Instagram: mrk_talkstech
- - - Tools & Equipment (Gears I use): - - -
Disclaimer: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide, I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!
DJI Mic: amzn.to/3sNpDv8
Dell XPS 13 Plus 13.4" 3.5K : amzn.to/45KqH1c
Rode VideoMicro Vlogger Kit: amzn.to/3sVFW8Y
DJI Osmos Action 3: amzn.to/44KYV3x
DJI Mini 3 PRO: amzn.to/3PwRwAr
- - - About me: - - -
Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN ABOUT THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"
I will be creating playlist which covers the below topics (with DEMO)
1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)
After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,
1. Data Engineer
2. Data Analyst
3. Data Scientist
Can't wait to help people with my videos.
- - - Support me: - - -
Please Subscribe: / @mr.ktalkstech
#AzureDatabricks #ApacheSpark #Sparkcompute #clusters #notebooks #magiccommands #machinelearning #ETL

Наука

Опубликовано:

2 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 24

@mr.ktalkstech 6 месяцев назад

I hope you all enjoy this video, I have spent multiple days in making this tutorial since my main intention is to cover almost all the topics in the most detailed way so that even an absolute beginner can understand the complete functionality of the CI/CD pipeline. I would highly appreciate if you could give a Like to this video and subscribe to my YT channel which will motivate me to make more such videos in the future. To get the YAML files used in this tutorial, please do the below as a token of support :) 1. Please follow my Instagram channel : mrk_talkstech (and) 2. Send me an message in Instagram with your email asking for the YAML code ( I will send you the files) ***Please Follow**** What else should I cover in my upcoming videos? Please let me know in the comments :)

@JayJay-tk2gp 12 дней назад

Mr. K, you are an amazing tutor. Thank you for your efforts for creating this series.

@TrinadhKumarMadupu 10 месяцев назад

Very Clean Explanation, Thanks. Eagerly waiting for the next part...

@sharaniyaswaminathan8760 10 месяцев назад

Clearly explained. Thank you so much. Waiting for the next part.

@mr.ktalkstech 10 месяцев назад

Thank you so much :)

@sajadbinnazir2606 10 месяцев назад

Well explained , Waiting for part 3rd ..... Thanks

@deepanshujain4119 Месяц назад

Hello Mr K - Amazing videos. You have really nailed the art of teaching someone who is a beginer to data engineer. Would you be posting the video on how we can include integration tests in CI part as well?

@yahiachammemi8267 4 месяца назад

I Highly recommend this playlist Great contents !

@mr.ktalkstech 3 месяца назад

Thank you so much :)

@CharanSaiAnnam 2 месяца назад

thanks

@jatinmadan8489 Месяц назад

Hi Mr K @mr.ktalkstech, I really enjoy your videos, but I am unable to generate the git credentials for azure repos git using azure devops. Problem: While creating a new repo, I am not able to see the advanced setting to setup the git credential to Azure DevOps Services (Azure Active Directory). Can you please help me with the same.

@hritiksharma7154 10 месяцев назад

Hi Sir, I have one doubt if someone what to create a new cluster in a workspace how he can create it by following devops process and what if unity catalog is also enabled in the workspace.

@mr.ktalkstech 10 месяцев назад

Creation of the clusters and resources will be mostly done via templates like terraform or biceps, etc

@destroyer3706 10 месяцев назад

very good explaination. I have one question. Where is the CD part? If i am not wrong, you have created repository in Azure devops and connect it to Databricks dev space. THen you pulled request and merge into main branch. When the Production environment will get updates? or files from dev to prod environment when this will happen.?

@mr.ktalkstech 10 месяцев назад

Those will be uploaded in the future parts of the video :)

@anupshastry5969 10 месяцев назад

Do we need knowledge of devops to be data engineer

@RahulKumar-dy4gr 3 месяца назад

Can you help us in estimating the full project cost in Azure... I mean how to estimate the expected cost with respect to resources ? In data engineering projects 🚩🚩🚩🚩

@mdsultanahemad Месяц назад

Hi if you could help me, I am trying to create repo . But I am not getting any option to add GIT credentials because of which I m not able to proceed. If you could help me ? I am getting only 3 options git repository url ,git provider and repository name

@jatinmadan8489 Месяц назад

Facing the similar issue, any update on it ?

@mdsultanahemad 27 дней назад

@@jatinmadan8489 no bro

@sm.prithiv7174 10 месяцев назад

Hi sir I am from banking domain planning to switch to data engineer. Your videos are very useful for my journey i am preparing for dp 203 exam. I got to know from mybfriend that in realworld we dont get access to azure portal. We dont built pipelines in azure console. Everything we do in IAC. So do i need to study teraform or template in azure. Please clarify me. Thanks in advance

@deepakchadda 10 месяцев назад

What's your approach for certification preparation?

@sm.prithiv7174 10 месяцев назад

@@deepakchadda i started doing your end to end project watched all your tutorials now i am reading microsoft documentation and book azure data engg cert by newton Alex and some youtube tutorials. I am aware of sql, python, spark and hadoop components like sqoop hive.

@mr.ktalkstech 10 месяцев назад

Yup, mostly in real-time projects, all the resources will be created via templates like terraform, biceps, etc. But that is not the skillset needed for a Data Engineer- it's more like a devops engineer / network engineer work.