24 Fix Skewness and Spillage with Salting in Spark 

Thank you for your feedback 💓 Please make sure to share it with your network over LinkedIn 👍

The 8 cores will process all the 32 partitions in 4 iterations each. (8X4 = 32)

The number partitions need to factor of cores. subhamkharwal.medium.com/pyspark-the-factor-of-cores-e884b2d5af6c

Spillage shows the amount of data it is not able to fit in memory for processing, both in deserialized and serialized form. This data will get processed as soon Spark is able to fit the same after processing some data.

@@easewithdata thanks...like in sort merge join suppose we are joining two tables let suppose after shuffling the shuffled partition of one table have very big size so that it can not fully fit in memory.. I have seen in such a scenario spark is able to join the data with the help of sort merge join how it is possible as we know both table partitions have to be fit in memory for join

The dataset is too huge to be uploaded in Github. I am trying an external source to upload the same.

Shuffle setting doesn't guarantee even data distribution among executors. In order to make sure the data is distributed properly, we are using slating technique.

If we repartition shuffle partitions after joining, then its again an extra step of exchange. And if we try to repartition before shuffle then the data will again get skewed in shuffle partitions. So, its a choice of optimisation completely based on scenario.

Watch out for AQE video for the most simpler option to fix Skewness.

@@easewithdata thanks bro, I'm going to watch.

Hello, Unfortunately GitHub is not allowing me to push GB files anymore. I am trying to locate another file sharing service to upload the bigger files.

​@@easewithdata Hi Please create file at your end using below code # Read Employee data _schema = "first_name string, last_name string, job_title string, dob string, email string, phone string, salary double, department_id int" emp = spark.read.format("csv").schema(_schema).option("header", True).load("data/employee_records.csv") from pyspark.sql.functions import lit,count emp.groupBy("department_id").agg(count("first_name").alias("count")).show() dept_3 = spark.range(40).select(lit(3).alias("department_id_temp")) dept_7 = spark.range(40).select(lit(7).alias("department_id_temp")) emp_inc_3 = emp.join(dept_3, emp["department_id"] == dept_3["department_id_temp"], "left_outer") emp_inc_3 = emp_inc_3.drop("department_id_temp") emp_inc_3.groupBy("department_id").agg(count("first_name").alias("count")).show() emp_inc_7 = emp_inc_3.join(dept_7, emp_inc_3["department_id"] == dept_7["department_id_temp"], "left_outer") emp_inc_7 = emp_inc_7.drop("department_id_temp") emp_inc_7.groupBy("department_id").agg(count("first_name").alias("count")).show() emp_inc_7.repartition(1).write.format("csv").mode("overwrite").option("header", True).save("data/output/emp_record_skewed.csv")

Hi, how can we getvthe data to practice as same​@@easewithdata