Dynamic Partition Pruning | Spark Performance Tuning 

Thank you Rahul :)

Thanks Ravindra... Thanks for feedback :)

Thanks Sarfaraz... :)

Thank you Sandip... Sure, will create more of these videos

Thanks for answering the question.. your answer is right... Please join our telegram group.. we discuss interview questions there

@@DataSavvy what is the telegram group name

Thank you for your kind words Bhumitra :)

Thanks :)

Thanks Ananatha :)

Thank you Rahul... Sure, I will create more videos on spark3

Sure bro... Thanks for suggestions... I have added this in my list... Please provide feedback.. how can I improve this channel

Thank you Jitu :)

Thanks Leena

Thanks Madhav... Partition by helps in achieving partition pruning. if your table is not partitioned, you cannot get partition pruning

Thanks Sheetal :)

Drop me a message at 9243024759... We can talk about it

Thanks Anek :)

Imagine you have a table which generate 100M events per day and you table have data for last 3 years. 100M*365*3 events. Now you want to get and analyze last 3 months of data. Partition pruning will help you to only read that much reducing IO drastically, your cluster might not be that big to process the whole 3 year of data and it will definately get OOM if you don't use partitioning. Now imagine you need to join this data with another table. Imagine the shuffling. Irrespective of the tasks created, you spark performance is also the factor of memory and IO. If data is not pruned you will either have t create one big cluster now this will be very very slow and if your executor goes OOM then it will splii on disc one record at a time. Imagine doing it for 100M*365*3 records one record at a time. It will take forever

I think this query doesnt require dynamic partition pruning

Hi Siby... Can you elaborate that what do u find wrong with that

@@DataSavvy can u pls check the heading ..for me it looks like the diagram or heading is interchanged

Please ignore the direction in arrows.. it seems to be misleading the concept I. Trying to explain. This image should be read bottom to top

Can contact me ...I have started interview ...can drop ur what app number u will crack

Thanks Mani for great suggestion... I had not thought in that direction... I will try that

Actually, I have seen international udemy instructors explaining just baiscs and that content is not even worth, still people pay and average earnings is 16000 students × 2 dollar(6 dollar is udemy comission). Imagine what if you contribte these kind of interview videos , projects etc it gonna affect to a mass . Dont beleive me , just read about kiril ermenko (super data science) and how he earned millions jut by making courses I am not here to advertise any mooc /learning platforms, its just I feel disappointed and sad when gems like you from india , casually make wonder . Give a shot and kindly take a good camera and mic and edit good videos and make interactive ppts/animations. to get guide on how to make good ppt , watch josh starmer(stat quest) channel . and last but not the least , do stuff is such a way that a noob can master most advance stuff. Hope you will listen to my advice ,and be the best on that or any other mooc

I agree with you Mani... Thanks a lot for this suggestion... I will definitely check RU-vid channels that you have suggested and see how can I create udemy course... Thanks for encouragement :)