I have two follow-up questions: 1- How to avoid shuffle. Does shuffle really impact the performance in all scenarios. 2- Given a DAG or say the explain plan, what are the areas or steps that one needs to take care for performing performance optimization. @easewithdata
1. Yes shuffle impacts performance, but we cannot avoid shuffle in all cases (consider aggregations). We need to optimize those so that the step behave perfectly. 2. There are many steps for optimization of a JOB in Spark. The first step starts with Explain Plan/DAG. It only provide the way spark is going to execute the code. But each step might have some different bottlenecks that you need to look out for (e.g. skewness/spillage)
Thanks for the awesome explanation. I ran the exact code in Databricks, but it skipped the repartitioning step somehow. It is not showing the relevant steps for repartitioning in the DAG or in the explain plan.
Shuffle is only involved when the next step is a wide operation. And for KBs data, it depends on the next stage. If you have count it will first make a local count before shuffling the data (which is reduced to kbs)
If AQE is enabled you will not find 200 Shuffle partitions. It will coalesce all un necessary shuffle partitions. Checkout - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-164OKvwW8T8.html
Bro very good explanation but I have doubt...I followed all your steps but I saw instead of 229 task 217 are being created for me and 4 stages...and in the Spark job I don't see (7+5) task for repartioning the dataframes and hence 2 stages are also missing...any idead why? Thank you.
Hello, Please disable AQE and Broadcast join, to replicate the same behaviour. I did it just to explain how thing work in background. With Spark 3 AQE is enabled by default.
