6.8 Catalyst Optimizer | Spark Interview questions 

1. Parsing - create an abstract syntax tree. 2. Analysis - Catalyst analyzer performs semantic analysis on tree. This includes resolving references, type checking, and creating a logical plan. The analyzer also infers data types. 3. Logical optimisation - Rewrite the plan into a more efficient form. This includes predicate pushdown, constant folding. 4. Physical planning - Spark stages and tasks created. 5. Physical optimisation - optimized further by considering factors like data partitioning, join order, and choosing the most efficient physical operators 6.Code Generation - generates Java bytecode for the optimized physical plan

You are precisely choosing the topics and very very accurately explaining them. Please keep it up.

Your video is very nice and specific. Apart from this sir, i want to add some thing... We will get many physical plan only when cost based optimization is enabled.

i have came across many video for catalyst optimizer. i found this is the best and well explained:)

Please try keeping volume at a higher pitch, ur videos are very educative elaborative and helpful. Please try improving the sound as well. Sometimes it is very difficult to understand and I close the video.

Clearly explained thanks

great

Add scenario based questions from Spark (Core , SQL , Streaming) . .. also add Questions for Scala

Good explaining of optimizer

very good explanation

Very nicely explained 👍

Great contents :)

also pls make a video on spark RDD vs df vs sparksql performance and which one outperforms other and in which case.

Good info. Can you publish a video showing dataframe vs dataset difference with an example.

Great and helpful video but voice is low

Thanks for the detailed explanation. However I am slightly confused now after watching previous video. There you mentioned logical plan , dag , execution plan is the pattern. Could you please connect that in this detailed context. Is DAG is part of this catalyst optimizer?

Thanks for your spark explanation. Can you please make a video on serialization, deserialization? thanku

Hi sir, great video! Could you please let us know what is whole stage code generation ? Is it the RDD code which is generated after picking up the most optimized plan ?

Regardless of using any join in my code does optimizer converts it in (terms of physical plan) into most efficient join like you said map side/broadcast/hash join??

would suggest before publishing it .plz check if it is audible or not

Hi Sir, Thanks for sharing valuable spark interview questions with us. could you please tell us the difference between Tungsten and Catalyst optimizer? can we create more than one spark context for an application, I have confusion with the allowMultipleContext property while creating a spark context? Kindly share any information with us on this. Thanks

Apki awaz bahut dheere hai.. Saabhi vedios mai.. Please make louder vedios. Content is awesum...

Is there any video on Spark optimization techniques ? I did not found so please help me with this. Thanks in advance.

REAL TIME SCENARIOSSS PLS

Hi. Can you explain about Case classes

Could you please suggest a good spark tutorial?

Volume very low sir

logical plan is lineage and physical plan is DAG, pls confirm?

cost based optimizer and rule based optimizer eliminated catalyst from spark 2??

sound aa thaaan pesaan da

please increase the audio.

from your next video kindly speak louder

please improve audio..

Would you speak a little louder, please.

Pathetic sound quality