Thanks for the video . Great explanation. Just a doubt in complete mode how it will keep track the state of previous micro batches .is it in memory of executors ?
Yes Spark manages the state in memory of the executors. And for fault tolerance it uses the checkpoint directory to manage restart and handle failure of application.
@@venkatasai4293 No it utilizes checkpoint directory only to keep state for fault tolerance (in case your app restarts or run into exception). Only state in memory are used for processing data.
@@easewithdata got it . Suppose we process the data incrementally on daily basis to perform aggregations in complete mode and if we use one time trigger . How it will maintain state. Because we are starting and stopping the cluster .
You will have to define an external checkpoint location. Spark will maintain the state in checkpoint. But it will be loaded in memory every time while processing the data.