Distributed Job Scheduler - System Design | Basic to Expert | On Whiteboard 

If each job does the same maintain the final state if final state is reached no more update, even the second job try to update (some state machine logic)

Like he mentioned, we will hash the job_id. If we have 5 master DBs, the hash will generate a number upto 5 and that job will be saved on that DB.

Task is something that you want the computer to execute either by a specific time (ie: every 30 minutes) or execute once at some point in the future. Take RU-vid as an example. When a user uploads a video. RU-vid will create a task for converting the video and distributing the video to CDNs. However, it's not possible for RU-vid to process all of these uploaded videos at once. So, it will create a task in the database and whenever a video processor (executor) is free. It will pick up the task to convert the video. If something happens, the failed task will be picked up to execute again in the future.

@@varshard0 Thanks for your response. I got what a task is from a theoretical standpoint, but I'm still trying to find what does it look like practically. For example, in your example above, what exactly will be stored so that executor will know it is video processor which need to do some video conversion. I'm assuming this system will need to do multiple types of tasks and not just video conversion, so the executor should somehow know how to execute the task. Basically some example of what is stored in the job detail which helps to execute the job will be good.

@@SP-yf1ib The task can be anything you like. If can be a code uploaded by the customer, which needs to scheduled and run. For example, I can upload a python script which checks the dollar vs inr and emails me when it reaches a certain threshold. But I need a scheduler to run it once a day for me.

@@rishabhjain2404 Right, but that is one important part of the scheduler which I think is missing here. How does the scheduler run it when it can be anything? For eg. in case of a python script, it need to know how to store and execute the python script. So one or two examples of how it is getting stored and executed would've helped.