The Non-wizard’s Guide to Supercharging Data Pipelines 

14:40 - 15:10 | DATA ENGINEERING THEATRE
THE NON-WIZARD’S GUIDE TO SUPERCHARGING DATA PIPELINES
THURSDAY 21 SEPTEMBER 2023
SPEAKER: DAVIN CHEN, AIRBYTE
As data increasingly becomes the lifeblood of businesses and organizations, optimizing data pipelines is becoming more and more crucial for engineering organizations. How does one know where to start?We walk through our journey of pushing the performance boundaries of Airbyte’s pipelines to achieve a 4x speed up.We debunk the myth that performance optimization is solely the realm of engineering wizards, concocting magical algorithms and techniques behind closed doors. Instead, we showcase how understanding the system as a whole and employing iterative strategies can incrementally unlock significant performance gains with minimal engineering complexity.We illustrate some hidden pitfalls we stumbled into around pipes and backpressure, and identify some practical lessons and techniques we hope all devs can benefit from.