"Data Quality is still a persistent issue in today’s modern data pipelines. The demand for data has increased as has the number of data points being analyzed. Organizations have more source systems than ever, and data quality is no longer an afterthought, there is an expectation to build data quality in line with the ETL project. This presentation will explore out of the box test functionality in dbt, data quality testing using Great Expectations and various other open-source packages that can provide a baseline for data quality testing. It will also dive into some advanced testing topics such as constraint tests, and project level tests.Presentation Topics:
Overview of dbt Tests
Configuring, Running and Storing test results
Testing sources, models
Great Expectations Overview
Integration with dbt
Configuring multiple tests
Additional open-source packages to simplify development
Advanced data quality tests
By data type
By table type
Constraints
Project level tests
Monitoring, Reporting & Notifications "
18 сен 2024
