Spark performance optimization is one of the most important activity while writing spark jobs. This video talks in detail about optimizations that can be done at configuration level , file formats, serializes etc.
It will be very helpful if you provide a spark-submit command including all the optimization factors explained in this video. Thank you for such informative videos.
Hello didi , I have never came across any such video/blog where optimization factors are presented in such well organised manner. Thank you so much for that. It will be really a great help if you kindly make a video on optimising a spark job while running... Like one typical example : resolving more data getting into one executor or may be a comparative study in the spark job details page when a wide transformation is used instead of narrow etc... Anyway thank you so much for such simple & clear video on these topics .
Very useful mam. Thanks a lot for making this. Could you suggest which is the best documentation to refer to? In spark official documentation I could not find many concepts thorough enough.
