Great Video actually. I was looking for Stock Exchange design and believe me i have gone through the other videos which have higher views and I can guarantee this has the best content.
I think those queues design is problematic, if the queue is the normal queue. Instead it should be the priority queue which are sorted by price and timestamp.
That is actually the task of matching engine as per my understanding. It can use redis to maintain sorted sets and do the matching. Queues are primarily introduced to achieve async behaviour.
@@architguleria989 agreed.i think the queues should only be used for maintaining the chronological order of placed orders for fairness and downstream system should do the logic of matching.
This is great stuff! Would really love to see how you would translate this system diagram into an Object-oriented software architecture. Can you please make a vide on that?
Excellent, well thoughout course. Would you have time to do a follow up video on the executor service to expand on how you'd design that part. I know you mentioned the airline reservation ticket but would be nice if you actually went in detail on the executor service here. Thanks again though !
Really Nice Content! One point - There should be one more service that will consume the Matched order Topic messages to debit/credit the actual stocks from the Demat of the Customers.
Great insightful video!! one design considration. we can use one kafka cluster here . with different topic for each stock list. this kafka can also help in notification and analtyyics. purpose
Nice video , just one comment @37.58 when you are discussing about limit orders , imagine there are 1 million limit orders for a stock which is going through a nose dive the matcher system might get overwhelmed , second i am assuming queue is not ordered and say a stocks price is 100 and somebody has placed a limit buy order at 105 at 11.00 am and someone else at 102 11.02 am , but since the matcher is taking them out sequentially it might end up matching 11.00 order since it would be fetched first , to a sell order , if the matcher has not pulled the whole queue , This can be a big bottleneck or more so incorrect behaviour
Good effort but lot of things missing like estimation of storage facilities, if the market time is 10 hrs then how come 86400 (24 hrs) for calculation? Order request missing type like Market/Limit Order.
I think the matcher can be designed with using queues, we can simply keep 2 tables for sell and buy orders, each time customer makes a sell or buy request just put the request in a queue, which is then picked up by the matcher. If it is a buy request matcher queries the sell table for the unmatched sell orders, if appropriate order is found, it marks both the records as match in their respective tables then passes the request to executor queue. if it is a sell request matcher queries the buy table for the unmatched buy order and does the same. Adding a the unmatched orders in the queues again will prohibit us from giving fifo service.
That's my exact design for an interview and from the feedback, it's not optimal enough with the db overhead. It's better to have the matcher also store trees of buy/sell and match them in memory instead. The buy and sell tables can exist for record keeping and restoring the tree if the matcher die.
I have my doubts about choosing order request db as NOSQL. 1. I don't believe order request data is going to be in hundreds of TB because this will only store one day worth data - this should less than 1 TB data. Matched orders is where we will maintain the history of orders but then there again we can split that into data for current day and history database. 2. Order request database does require high consistency. Often clients would like to modify their orders which are yet to be matched or cancel these. 3. Lastly, I have my doubt on order_id being the primary query key because trading mostly happen via brokers like icici direct or zerodha so will zerodha poll status for each order_id.. I guess that, they might poll based on their broker id or something which gives info in bulk.
Hey Kshitij will try to answer your doubts based on my experience and understanding in the order you have asked 1. Keeping both ongoing order request and past order data in order request table make sense as we aggregate all order related info int o its own collection which makes our data denormalized that avoid unnecessary joins and can have a flag to make active order this will help us in our future API's hence no-sql storage 2. So based on our initial assumption we are letting user update their ongoing order hence data is consistent by nature 3. Yes in matched order there should a broker_id by here the order status user is requesting is abstracted wrt our order request status and yes internally we need to expose a webhook endpoint to get the realtime status and according updating our order request table service coulmn
Thanks for the video, very insightful. I am just wondering about the matching service which takes the events from buy and sell queues and matches them. Since the queue maintains the messages in order, and a buy event can match with a sell order which is present in later part of the queue, how do we match the orders in that case here?
Why matching engine runs once in 5 seconds? Why this system can handle 60k rps for every stock, but match them only once in 5 secs? I don't understand. HFT will drop this exchange (imagine if its real) at 6 second and never come back )
How do you manage consistency between when you get a message from the queue (SQS), have a positive match, send it to the Executor service and then delete the message from SQS? Do you have an intermediate step in the Matcher service in which you synchronously check and update the state of those orders (maybe in the db) before deleting the message from SQS? My concern is: If you have orders Buy A with Sell B matching, you process them, send another message to the queue for the Executor Service and right after you fail to delete those messages from SQS (in the Matcher service) Then retry happens and you get more messages from both Buy and Sell queue. A and B but now also Sell C, this time the algorithm matches A and C, sends it to the Executor service successfully and deletes A and C from SQS successfully. We are not in an inconsistent state, The executor service cannot de-dup on Buy A and we've effectively lost a Sell C order. Thanks
Great questions. I think we should use a persistent storage with 2PC capability. When we successfully notified the Queue then the storage will be settled and fire the event.
As a software engineer who are developing systems using Nasdaq exchange stack, i can this video is crap! Nobody nowadays will design an exchange like this.
Care to go into detail about how it is crap? It's also meant to be a system design interview question so being the most "up-to-date" iteration is not necessarily the goal (especially since not everyone will actually understand how a stock exchange even works). The goal is to have a conversation and see how the interviewee responds.
