End To End Data Engineering Project Using Snowflake | Real Cricket Analytics Use Case

Подписаться 44 тыс.

Просмотров 35 тыс.

50% 1

🏏 Dive into the world of one day international (ODI) cricket data with my latest data engineering project, where we've built a comprehensive end to end data engineering solution using Snowflake. This video covers everything from design considerations to the intricacies of data flow and database layer/table architecture.
🔍 Design Considerations: Learn about the key decisions made in designing the data modelling and the overall project architecture.
🌐 Data Flow: From data loading and curation to transformation into Snowflake, understand the seamless flow of information. We'll break down the steps involved in ensuring data quality, integrity, and how Snowflake enhances the efficiency of this process.
Once you complete this end to end real time snowflake based data engineering project (ETL/ELT), you would be able answer of following questions
1. How to load data from location machine to Snowflake stage location.(Small JSON files as well as many JSON files at on go).
2. How to load delta data set from local machine to internal stages?
3. How to use $ notation to query stage file?
4. How to run copy command to load JSON files into tables?
5. How to design different layers & fact/dimension tables?
6. How to automate data flow using task and task tree?
7. How to build quick dashboard using Snowsight?
🚀🚀 How To Connect
✏ Instagram: / learn_dataengineering
✏ Medium Blog: / data-engineering-simpl...
✏ Exclusive Snowflake Facebook Group: / 627874916138090
🚀🚀 Sample JSON Files & SQL Scripts
✏ / 8f8e4f0fd1d0
🚀🚀 Chapters
➥ 00:00:00 Introduction
➥ 00:01:56 Welcome Note
➥ 00:03:07 Udemy Course Offers
➥ 00:03:33 Review Match Level JSON File
➥ 00:06:19 Review E2E Data Flow Diagram
➥ 00:11:07 Part-01 - Pre Requisite
➥ 00:11:58 Part-02 - Database/Schema Creation
➥ 00:15:13 Part-02.1 - Load JSON Files Using SnowSight
➥ 00:17:35 Part-02.2 - Bulk Load JSON Files Using SnowSQL CLI
➥ 00:19:01 Part-03 - Stage to Raw Data Loading
➥ 00:28:46 Part-04 - Cleansing + Curation Introduction
➥ 00:36:09 Part-04.1 - Extract Clean Match Data
➥ 00:48:28 Part-04.2 - Extract Clean Player Data
➥ 00:54:01 Part-04.3 - Extract Clean Bowling Data
➥ 01:12:57 Step-05 - Quick Data Validation
➥ 01:15:22 Step-06 - Populate Fact & Dimension
➥ 01:41:36 Step-07 - Quick Visualisation
➥ 01:43:49 Step-08 - Automate Continuous Data Flow
➥ 01:50:12 Step-09 - Summary
➥ 01:51:00 Thank you note
🚀🚀 Other Popular Playlist
✏ Complete Snowflake Master Class - • Snowflake Tutorial
✏ Snowflake Productivity Hacks • Snowflake Productivity...
✏ Snowflake Cost Optimisation & Cost Management Techniques | Tips To Reduce Snowflake Cost • Snowflake Cost Optimiz...
✏ Snowflake Table - Complete Guide With Example • Snowflake Table - Com...
✏ Snowflake SnowPro Core Certification - Complete Guide • Snowflake SnowPro Core...
✏ SnowPro Core Exam Mock Test - • Snowflake Cerfication ...
✏ Learn Snowpark Using Python API Examples - • Snowpark Tutorial | Le...
✏ Everything About Snowpark - • What Is Snowpark | Lea...
✏ Working With JSON Data In Snowflake - • Working With JSON Data...
✏ How To Load Data Into Snowflake - Masterclass - • How To Load Data Into ...
#dataengineering
#snowflake
#snowflaketutorial
#snowflakedatawrehouse
#snowflakecomputing
#clouddatawarehouse
#snowparktutorial
Disclaimer: All snowflake-related learning materials and tutorial videos published in this channel are the personal opinions of the data engineering simplified team and they're neither authorised by nor associated with Snowflake, Inc.

Опубликовано:

28 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 51

@pravachan6577 5 месяцев назад

This is incredible and passionate work Sir God bless you abundantly

@abhijitmohanty734 8 месяцев назад

Most awaited video..Thank you so much Sir !!

@DataEngineering 8 месяцев назад

Most welcome.. and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

@pavitersingh4698 2 месяца назад

this is super helpful

@AnkitGupta-tp3ln 5 месяцев назад

The amount of effort and dedication that you have put in this video is tremendous. Great work!!

@DataEngineering 5 месяцев назад

Thank you so much 😀 if you would like to go more deep in snowflake.. you can also watch my udemy contents.. My current 3 courses are available in discounted price www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=NEW-YEAR-2024 www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=NEW-YEAR-2024 www.udemy.com/course/snowflake-dynamic-table-masterclass-e2e-data-pipeline/?couponCode=B1E84B2CB4AA82CB95E3

@vishal_uk 4 месяца назад

Amazing stuff!!! May I know what JSON visualizer you're using in this video, please?

@lu3ma304 8 месяцев назад

Thank you so much for the video

@DataEngineering 8 месяцев назад

You are so welcome! Please try to access following link if want to get access to content. ✏ medium.com/@data-engineering-simplified/8f8e4f0fd1d0 alternatively, you can join my Facebook group and 200 ODI data set is already published there. facebook.com/groups/627874916138090/?mibextid=c7yyfP and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

@volukulachaitanya5406 8 месяцев назад

hi wiil u do the video on generative AI and LLMS with snowflake detailed manner or suggest any resources for that , it will be more helpful for lot of members

@saisumanthnagasuri7741 8 месяцев назад

Excellent 👌👌

@DataEngineering 8 месяцев назад

Thanks a lot and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

@cargouvu 7 месяцев назад

Why did you need to connect to Dbeaver to see the PK/FK relationship? Could we not do that in Snowflake?

@rpafacts5574 8 месяцев назад

Thanks for the video

@DataEngineering 8 месяцев назад

You are so welcome! Please try to access following link if want to get access to sql scripts ✏ medium.com/@data-engineering-simplified/8f8e4f0fd1d0 alternatively, you can join my Facebook group and 200 ODI data set is already published there. facebook.com/groups/627874916138090/?mibextid=c7yyfP and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

@rpafacts5574 8 месяцев назад

Great video

@DataEngineering 8 месяцев назад

Glad you enjoyed it and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

@jaspermurphy8865 Месяц назад

Hi I didn't the get all the file from the gitlab or from medium source link. Could you help me

@Ashu23200 8 месяцев назад

more projects like this please using more features.

@DataEngineering 8 месяцев назад

Thanks for your note... I assume, you have already seen the other end2end project using snowpark ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-1jC98XQwBZw.html

@shubhamalsunde3230 7 месяцев назад

sir please give data file also so that we can also make the project according with you

@RGDataJourney 2 месяца назад

This video helps lot but i don't see any 8 json files in gitlab pls can u provide...

@naveensai2175 8 месяцев назад

Hi , At 41 minute how do u get result attribute in case statement --> when info:outcome.result = 'tie' then 'Tie' because we have -> "outcome": { "winner": "Sri Lanka", "by": { "wickets": 8 } we have only 2 attributes 'winner' and 'by' in outcome object

@DataEngineering 8 месяцев назад

if you try yourself, you would understand how it works.. but your question is not super clear to me... all code is available in my medium page.. so you can download and try it out..

@ouzaslnn 3 месяца назад

Hey, thanks for this nice hands-on project, just wanted to ask which tool do you use to visualize the json ?

@DataEngineering 3 месяца назад

drop me a note to my instagram

@alpitgoyal7532 28 дней назад

The resources that you have used are free or it required subscription to use them??

@akshaynaik5926 8 месяцев назад

I am stuck with multilevel and different attribute json to snowflake conversion using snowpark

@DataEngineering 8 месяцев назад

Unless, you share more detail, it is hard to extend help.. You can also watch the complete JSON playlist and try to see if that can help. ru-vid.com/group/PLba2xJ7yxHB6ybgtaIsTKmmF2Nl2wAe2S

@faizalqureshi7840 Месяц назад

Couldn't find the first 6 json files to be loaded within landing schema Can anyone please help me with the files so i can proceed further with the projy

@ashokannam1346 Месяц назад

can we use this project in our resume ?

@prateeksmithpatra4454 Месяц назад

FROM WHERE CAN I GET THE DATA THAT YOU USED IN CRICKET DATA, PLS PROVIDE THE DATA ATLEAST NO SIR

@cargouvu 7 месяцев назад

What tool are you using to visualize the data as nodes?

@sajaldadhich2743 7 месяцев назад

json crack.. it's extension is available in VS Code

@amanpirjade9 8 месяцев назад

Do we have to code in snowflake

@DataEngineering 8 месяцев назад

To interact with snowflake, you can either do it using standard ANSI SQL or you can also write program using python API or Java API or Scala API. ---- I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

@ravikreddy7470 8 месяцев назад

Do you provide any snowflake online trainings?

@DataEngineering 8 месяцев назад

Planning to do it soon, but for now.. No.... pls join my facebook group so if I do it.. you will come to know about it.

@mukundareddy2681 6 месяцев назад

Data set we do we get?

@DataEngineering 6 месяцев назад

check video description

@nagendradantam4117 8 месяцев назад

can we get sample json files please

@DataEngineering 8 месяцев назад

Please try to access following link ✏ medium.com/@data-engineering-simplified/8f8e4f0fd1d0 alternatively, you can join my facebook group and 200 ODI data set is already published there. facebook.com/groups/627874916138090/?mibextid=c7yyfP and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

@nagendradantam4117 8 месяцев назад

@@DataEngineering Thank you so much😊

@naveengvkvs1915 8 месяцев назад

Hi Thanks for the detail explanation, I have a query After loading of data from stage to raw table don't we need to move or clean the files from stage location? Copy command only stores 64 days of metadata then after 64 days if old files are still there it will be reprocessed again? Can we think of a mechanism to archive the processed files from stage location?

@DataEngineering 8 месяцев назад

I have never tried this 64 days concept and there is a parameter to control it. But thanks for the note, I will try it to check if 64 old data is re-loaded or not.