🎯 Key points for quick navigation: 00:02 *📈 Overview of Stock Exchange Systems* - Overview of major stock exchanges (NSC, BSC, NASDAQ, NYSE, etc.). - Challenges in designing a stock exchange system. - Key aspects: scalability, fairness, limit orders, and matching engine. 04:09 *📋 Functional Requirements of Stock Exchange Systems* - Core functionality: buying and selling stocks. - Support for limit orders. - Orders valid for the duration of market hours. - Payment integration via third-party providers. - Partial matching of orders. 08:07 *🌐 Non-Functional Requirements and Scale* - High availability and consistency requirements. - Ensuring trade fairness for all participants. - Handling a peak load of 60,000 transactions per second (TPS). 26:00 *📊 Introduction to Queues in Stock Exchange Systems* - Introduction to the necessity of introducing queues in the system, - Reasons for using separate queues for buy and sell orders, - Importance of segregating queues for different stocks. 29:15 *⏰ Workflow of Order Validation and Storage* - Workflow from client request to order validation and storage, - Notification process to clients with order IDs, - Initial handling of buy and sell requests into respective queues. 30:10 *🔄 Scaling and Operation of Matcher Service* - Scaling considerations for matcher nodes pulling from queues, - Implementation of Amazon SQS-like visibility timeout for message consumption, - Frequency and handling of matcher nodes pulling from buy and sell queues. 33:06 *💱 Matching Logic and Handling Partial Orders* - Logic for matching orders based on best available prices, - Handling partial orders and requeuing unmatched portions, - Ensuring orders are fulfilled or updated in the order status database. 38:31 *📜 Execution of Matched Orders and Notification* - Separation of match creation and execution using a queue system, - Tasks performed by the executor service post-match, - Importance of publishing match execution notifications to scalable services like Kafka or SNS. Made with HARPA AI
This sharing was so good, thanks! One thing I want to point out that the real fact is that the number infinitely approximates 1 because "retry" is always set to 3 :)
How are the media files cached in real time? Considering they are significantly larger than manifest file. Do the clients directly pull it from cdn servers or does it route through the backend then cdn. I have observed the clients directly pull the media from cdn.
40:15, You may not need to rollback after payment booking. What you can do is have a SQS queue in place and a scheduler will update the Booking Info let's say after 10 mins.
How do you manage consistency between when you get a message from the queue (SQS), have a positive match, send it to the Executor service and then delete the message from SQS? Do you have an intermediate step in the Matcher service in which you synchronously check and update the state of those orders (maybe in the db) before deleting the message from SQS? My concern is: If you have orders Buy A with Sell B matching, you process them, send another message to the queue for the Executor Service and right after you fail to delete those messages from SQS (in the Matcher service) Then retry happens and you get more messages from both Buy and Sell queue. A and B but now also Sell C, this time the algorithm matches A and C, sends it to the Executor service successfully and deletes A and C from SQS successfully. We are not in an inconsistent state, The executor service cannot de-dup on Buy A and we've effectively lost a Sell C order. Thanks
Great questions. I think we should use a persistent storage with 2PC capability. When we successfully notified the Queue then the storage will be settled and fire the event.
Learnt a lot from this video. Well diagrammed and explained. Great job! Quick question though. Here you're storing the live stream to s3 and intend to serve it from there. So new streaming content will be written to the disk every second. Can the client read from the new files on the fly while the file has jsut been written without any latency? or do we need to stream the data directly to some broker that delivers the content as a data stream instead.
Thanks for these videos! One quibble - it's server-sent events. Also, there's no upgrade for SSE - it's just a normal http request with specific headers to tell the server to keep the connection alive. It might also be worth explaining why you might just do SSE if you don't need the bidirectional communication of ws - simpler, more compatible with networks and load balancers, etc..
How short are individual chunks during streaming? Given its a live stream i guess they should be 1 or 2 seconds? Does that mean manifest file gets updated every second and the client has to pull manifest file every second?
Nice video , just one comment @37.58 when you are discussing about limit orders , imagine there are 1 million limit orders for a stock which is going through a nose dive the matcher system might get overwhelmed , second i am assuming queue is not ordered and say a stocks price is 100 and somebody has placed a limit buy order at 105 at 11.00 am and someone else at 102 11.02 am , but since the matcher is taking them out sequentially it might end up matching 11.00 order since it would be fetched first , to a sell order , if the matcher has not pulled the whole queue , This can be a big bottleneck or more so incorrect behaviour
how does the movie booking companies handle the locking part, does this decision depends on the criticality of the usecase, for example we are okay with the user to retry here, we can take an optimistic locking and in other ways we choose pessimistic locking.
Thank you very much sir for informative video. This is really very helpful. It would be great if you could create video for coding for this airline system
00:07 Airline reservation system requires handling limited resources, parallel requests, flight search capabilities, and failure handling. 02:51 The main point is to understand the high-level diagram of an Airline reservation system and the functional requirements. 07:42 Certain parts of the airline reservation system prioritize availability, while others prioritize consistency. 09:58 Capacity estimation for Airline Reservation System 14:49 Designing a read-heavy system for airline reservation 17:16 Applying filters on a booking reservation system 21:31 The airline reservation system includes three tables: flight API, booking information, and payment details. 23:48 The storage requirement for flight information is very minimal. 28:05 The need for distributed systems arises when storing petabytes of data that cannot fit into one machine. 30:03 Clients can scale independently without being a point of failure for each other. 34:31 Building an interface for notification and analytics services 36:44 Booking service has multiple steps: checking availability, completing payment, updating booking information. 41:00 Implementing distributed transactions with the two-phase commit approach 42:58 Handling concurrency in parallel requests 47:20 Serialization using pessimistic locking 49:26 Implementing TTL on locks is necessary to prevent indefinite locking of resources. 53:43 Using pessimistic locking in airline reservation systems. 55:44 Handling payment failures and refund scenarios in the airline reservation system 59:58 Consistency and serialization are important in an airline reservation system Crafted by Merlin AI.
Great video. I have some feedbacks. 1. Talk about Orchaterator vs Choreographer pattern. Booking Service is acting as a Orchasterator . We can also use decentralized system like Kafka. 2. 2PC is generally not practical to achieve distributed transaction. A better approach would be SAGA pattern and having compensating events. 3. Also, talk about creating some derived data. For example, is a flight is booked completely . Keep this info in cache and keep cache in sync with DB so that cancellation would mark this booked flight to available.
Great insightful video!! one design considration. we can use one kafka cluster here . with different topic for each stock list. this kafka can also help in notification and analtyyics. purpose
Thanks for the video, very insightful. I am just wondering about the matching service which takes the events from buy and sell queues and matches them. Since the queue maintains the messages in order, and a buy event can match with a sell order which is present in later part of the queue, how do we match the orders in that case here?
Wouldn't it make sense to name the media files with descriptive names (Encoding type, Seconds, steamID,? That would allow POPs and client to predicted the next file without referencing to cache. Example? streamID: streamID12345, 1080 P, 50 Seconds. etc. So the cache can just hold the parent directory of files (s3). redis: streamID12345 -> s3: /youtube/India/Vinayak/. Example: s3: /youtube/India/Vinayak/streamID12345_1080P_01Seconds.MPEG s3: /youtube/India/Vinayak/streamID12345_480P_01Seconds.MPEG s3: /youtube/India/Vinayak/streamID12345_1080P_02Seconds.MPEG s3: /youtube/India/Vinayak/streamID12345_480P_02Seconds.MPEG ...... s3: /youtube/India/Vinayak/streamID12345_1080P_50Seconds.MPEG s3: /youtube/India/Vinayak/streamID12345_480P_50Seconds.MPEG every Second add a new file to S3. Now the client can predict and download the next "second file" as they come in.
