Data Engineering Pipeline / ETL Process Task & Source Audit Basics using Python - Design Walkthrough

Подписаться 2,4 тыс.

Просмотров 5 тыс.

50% 1

Data Engineering / ETL / ELT
ETL - Extract, Transform, and Load
ELT - Extract, Load, and Transform
In this video, I have explained how to write the code for Source Data Audit and Task Audit using Python.
In the previous video, I explained the Data Engineering / ETL Concepts • Data Engineering ETL V... .
The following essential concepts have been covered in the previous video:
Incremental Extract vs Full Extract
How to design the Incremental Extract
Source and Task Audit Table
Staging Area and its importance
Data Warehouse / Data Lake
Data Mart
Data Transformation and Aggregation
Defining the granularity of the data storage
Hierarchies
I have also covered how we can use Python for ETL or Data Integration tools like Pentaho Data Integration (PDI) / SQL Server Services for ETL at a conceptual level.
I hope this video would serve as a good starting point for anyone wanting to understand the Data Engineering / ETC Concepts.
Website: k2analytics.co.in
Email: ar.jakhotia@k2analytics.co.in
Mobile: +91 8939694874

Опубликовано:

30 сен 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 7

@hasanmougharbel8030 2 года назад

Hey dear, god bless your efforts in this channel. I have a general enquiry as a new sql learner. How could i create a pipeline to extract and load data from existing accounting program into our SQL server instances. How can i know if the export mechanism in the software permits me to undertake this extraction process, and how can i know if an application have an api? Thanks for taking care of my enquires. Looking forward to gain more knowledge from you

@RajeshJakhotiaAIML 2 года назад

You will have to use the API of accounting software to extract data.

@ksspqf6016 Год назад

Part of my degree was data analytics but didn't cover stuff like this whatsoever. I didn't know what etl, data modelling excels power query nor powerbi was when I left university. I feel ripped off as when I can search for a video online and it will teach me a semester's worth of content in a single video

@Velben Год назад

I always extract, transform and create the staging tables from within the python script. Is there a performance benefit in having the tables created prior?