Understanding Stages in Spark UI for a Spark Job | Spark Interview Questions

Подписаться 5 тыс.

Просмотров 15 тыс.

50% 1

Hi Friends,
In this video, I have explained Spark internal machanism on how driver creates the logical plan and then how it will turn to the run-time execution plan, how the stages will be created etc details.
Please subscribe to my channel for more interesting learnings.

Опубликовано:

21 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 30

@vanajar7345 2 года назад

Explained clearly. It took me 10 -15 video before reaching to this video and no other video was as clear as your's

@sravanalakshmipisupati6533 2 года назад

Thanks a lot.

@naveennoel9496 5 месяцев назад

Explained in a very clear manner. Thanks.

@bhavanivani448 Год назад

Nice explained, really helpful for interview

@sravanalakshmipisupati6533 Год назад

Thank you 😊

@nagendran-wn6ws 5 месяцев назад

Nice explanation ..appriciated madam

@sushmamc8904 2 года назад

You have explained it so well. Thank you so much

@sravanalakshmipisupati6533 2 года назад

Thanks a lot :)

@sonutyagi149 2 года назад

You Channel is Gold Mine to me - Thank you So So Much .

@vaishnavidhoke8174 Год назад

Amazingly explained! Thankyou so much for making this, it’s extremely helpful, kudos!!

@technologyexplorer9193 2 года назад

Nice explanation. Could you please explain little bit more on how exactly on what bases tasks are created. For each stage

@sravanalakshmipisupati6533 2 года назад

Thank you very much. Sure.

@sravankumar1767 2 года назад

Nice explanation sravani

@sravanalakshmipisupati6533 2 года назад

Thanks a lot Sravan :)

@sriadityab4794 8 месяцев назад

Can you please explain more with good example on how to identify time taking jobs and optimizing them?

@reddammareddy2999 Год назад

Super mam

@vishnureddym5389 2 года назад

Hi @sravana, it was good explanation. I have a doubt in this. In the second job we can see two stages and the first stage is skipped which means it is broadcasted or cached right?. But job 1 executed the same and in the second job again that stage executed or first job itseems cached and using the result as input in the second job second stage?

@sravanalakshmipisupati6533 2 года назад

Hi Vishnu, the cached data will be sent to second stage as input.

@vishnureddym5389 2 года назад

Thank you for the reply. My question is we have two jobs . Job 1 is having one stage and job 2 is having two stages. So job 1 stage 1 is same as job 2 stage 1 or it is different?. If same then can we use job1 stage1 results as input to job2 stage 2.

@sravanalakshmipisupati6533 2 года назад

@@vishnureddym5389 , @ 3:30, you can see that there are 2 jobs present - Job 31 & 32. Job 31 cached result is sent to exchange at the end and you can see that the input is read from exchange in job32. This way, Job1 output is sent to job2 as input. Also, job2 will have the stage1 (which is greyed out) and stage2.

@vishnureddym5389 2 года назад

@@sravanalakshmipisupati6533 so job 31 stage out put is sending to job 32. But job32 stage1 is not executing again right which same as job 31 output

@sravanalakshmipisupati6533 2 года назад

@@vishnureddym5389 Yes, Job 32 stage 1 will not be executed again, that's why it is showing as greyed out. The cached output from exchange will be considered directly as input to stage2.