Advancing Spark - Databricks SQL Variables & Dynamic WHERE 

Whenever someone comes from a heavy SQL background, they find some of their favourite toys missing from the Spark SQL world - but there have been huge investments from Databricks to catch up here over the past few months. We thought it was time to talk SQL Variables, Parameterised Tables and even the WHERE (*) functionality!
So if you've been sitting in the pyspark wastelands dreaming of the good old days of SQL Variables, this is the video you've been waiting for! In this video, Simon runs through how Spark SQL parameters work, some of the neat tricks you can do using them, and has finally thought of a use case for "any column filtering"!
For more info on Variables in Spark SQL, check out the docs here: learn.microsof...