Deep Dive: Apache Spark Memory Management 

Its available at executors/storage tab, port 4040 by default.

kk8866 you run out of memory when you try to accumulate results that have a size greater than the memory you have on driver as a result of some action like take(). it also happens when you are making large objects.

@@verma.chitral This is the case when the driver runs out of memory. But why executors run out of memory?

Did you get the answer to why executors run out of memory when memory can be spill to disk?

Easy, it happens due to gc overheads and oom s. When you're creating objects at greater rate they can be gc'd.