Just like eating vegetables, no one likes writing tests. However, writing unit tests is good for your programming diet. It helps ensure that data flows from one end of the pipeline to the other without any hitches.
In this talk, Jonathan Neo, Senior Data Engineer at Cuusoo, will explain why and how you can write unit tests, and where does unit testing fit in the bigger picture.
Jonathan will demo how you can write your own unit tests in Databricks using Databricks Connect and PyTest (a popular Python testing library), and also automate the execution of unit tests using CI/CD pipelines.
--
This was the talk from the November 2021 event of the Melbourne Databricks User Group.
4 ноя 2021
