Broadcast vs Accumulator Variable - Broadcast Join & Counters - Apache Spark Tutorial For Beginners 

The best explanation so far I found on RU-vid...easily explained

In addition to the Resilient Distributed Dataset (RDD) interface, the second kind of low-level API in Spark is two types of “distributed shared variables”: broadcast variables and accumulators. These are variables you can use in your user-defined functions (e.g., in a map function on an RDD or a DataFrame) that have special properties when running on a cluster. Specifically, accumulators let you add together data from all the tasks into a shared result (e.g., to implement a counter so you can see how many of your job’s input records failed to parse), while broadcast variables let you save a large value on all the worker nodes and reuse it across many Spark actions without re-sending it to the cluster.

You are the best trainer on RU-vid bro. Keep up the good work.

Hi, Many people have questions how accumulator is getting update. Accumulator variable on each worker node is a local copy and there is a global copy which is in driver node and it can be accessed only by the driver process... Hence each worker node will return the count of blank lines to the driver process and the driver process will cumulate and update the global copy.

I've been looking so many videos...It is only now i got the concept...thanks so much for the explanation.

Bro best Explanation WOW>>!!!.Also, do you have a video of explaining entire SPARK-SUBMIT command as to how the worker nodes are created and data is distributed across multiple partitions and task and jobs?It would be really helpful

Excellent explanation... Great video to learn the concept in so a simple way. Please make another video so that we could learn all such concepts easily. Thanks.

Thank you sir, with simple example

It's clear and nice explanation. this is one of best vedio so far in this concept thanks

Excellent explaination.

super bayya, nice explanation

It's informative, Can you also let us know in what situations accumulators is useful?

Excellent Teaching

clear explanation thanks buddy

Awsome explanation

You said that accumulator value is being updated in each worker node, does worker node 2 will wait until worker node 1 empty lines count updated done? since you are updating the value........... AS SPARK JOB IS A PARALLEL HOW COULD IT GET UPDATED SEQUENTIALLY?

Time stamp 3.55 spark submit .... You didn't mentioned about cluster manager role in spark submit background process As u mentioned drive program initiate and connect to worker ....yet driver connect with cluster manager and cluster manager wil connect to workers

Nice explain sir.. 👌👌👍👍

excellent video bro.Great explanation and very thorough

Excellent explanation!

Thanks for the clear explanation

I don't think in broadcast variable example what you showed that w1 contains only USA and w2 only IND is correct. Data is distributed in random fashion and code map can be used as lookup within that worker. Please correct if understanding is wrong.

Yes Thank you.. my questions is how the situation will be handled in case we have encountered blank lines at same time on three worker nodes?

Bro u r champ nice explaning

Awsome explanation..!!

amazing!

Thanks for sharing such a nice video can please share me spark scala training videos

thank's for this explanation

When the data is getting analyzed parallelly, then how come the Accumulators get incremented. For example partition 1 has 1 space line and partition 2 has one space line, when these two processed simultaneously, both partitions can update the accumulator as 1 right. Could you please clarify

can i have the accumulator variable pass the value to broadcast variable? Like some worker nodes update the accumulator variable which is copied to a broadcast variable and inturn read by some other worker nodes

Didn't mention how to retrieve record from broadcast variable

This is not the correct explanation for Accumulator variables from the start. Kindly edit the video to add factual information

This guy said, driver will create worker node. I think he should review his video before posting. Every single person is just want to make money by starting his own channel but does not want to spend time in giving quality videos.

So many ads