Data Engineering at Netflix using Apache Spark and Flink with Joan Goyeau 

This episode features Joan Goyeau, a data engineering expert at Netflix. We discuss the unique challenges of supporting a massive user base across various devices for flawless video streaming. Joan also sheds light on Netflix's innovative use of Apache Spark with Scala 3.
Links
Spark for Scala: github.com/vincenzobaz/spark-...
Joan's Kubernetes client: github.com/joan38/kubernetes-...
VMAF - Video Multi-Method Assessment Fusion open source project from Netflix: github.com/Netflix/vmaf
Data engineer job post for Netflix: github.com/vincenzobaz/spark-...
Netflix US Salaries: www.levels.fyi/companies/netf...
Sponsors
Check out ScalaJobs ( scalajobs.com/ ), the most effective platform to hire Scala Engineering talent.
Content
00:00 Introduction
00:20 Data Engineering at Netflix
03:06 CAP Theorem for Streaming
10:23 Media preprocessing
15:02 Unexpected Results from Data Analytics
17:30 Netflix Interactive Content
19:22 Spark & Flink
22:35 Tech Choices within Netflix
25:22 Custom Spark & Flink at Netflix
28:23 Chaos Engineering
30:08 Spark with Scala 3
38:45 Scala at Netflix
40:51 Mill Build Tool
46:49 Scala CLI
51:30 Hiring Process at Netflix
01:00:27 Compensation at Netflix
01:04:55 Developing an Open-Source Kubernetes Client
01:12:41 Testing and Integration with GitHub CI
#scala #dataengineering #apachespark #netflix