16 Late Data Processing | Watermarks | Tumbling and Sliding Window Operations in Spark Streaming 

2 more.

​@@easewithdatathank you. You are teaching very well.

Yes Spark manages the state in memory of the executors. And for fault tolerance it uses the checkpoint directory to manage restart and handle failure of application.

So it will maintain two states both in memory and checkpoint directory which is in disk ?

@@venkatasai4293 No it utilizes checkpoint directory only to keep state for fault tolerance (in case your app restarts or run into exception). Only state in memory are used for processing data.

@@easewithdata got it . Suppose we process the data incrementally on daily basis to perform aggregations in complete mode and if we use one time trigger . How it will maintain state. Because we are starting and stopping the cluster .

You will have to define an external checkpoint location. Spark will maintain the state in checkpoint. But it will be loaded in memory every time while processing the data.

Yes you can connect with me over topmate topmate.io/subham_khandelwal