System Design Interview - Distributed Message Queue 

This was the first video I'd seen from this channel. This is some next-level system design content. Way more in-depth than other videos I've seen. Unfortunately, it doesn't look like there have been any recent uploads, which is really too bad.

This is absolutely the best content on RU-vid on the system design topic! No "scratching the surface" bullshit, but rather very in-depth and concrete explanation on how to navigate successfully through the system design interviews. Thank you Mikhail for your great efforts! Большое спасибо, Михаил! One thing bothers me though, there haven't been any recent updates on the channel. I'm pretty sure, all the people here will appreciate and support if Mikhail decides to continue his endeavours and uploads new videos! Anyway, thanks a lot!

Appreciate your efforts on aggregating and delivering such quality content in such a lucid manner. I don't think there is better content than this anywhere on youtube especially for system design topics. Wow ! (Y)

I've watched the whole series already, thanks for the great quality content! looking forward for more

I can’t thank you enough, I was really struggling to grasp these topics and your explanations really helped me put it all together 🙏 excellent work!

Thank you for summarizing precisely about what can be covered in a 40 minute time limit. Knowledge is one part which is built over learning and experience. Your video really helps to focus on structuring and expressing the knowledge in a coherent manner. Thank you.

Normally I play videos in 1.5x or 2x. Your videos have so much information that I am afraid of losing some fundamentals if I play in 2x. Outstanding quality. Please keep making such videos.

This is the best content on system design I've ever seen. Please consider resuming the journey! We'd love to ride along!

I almost never comment on RU-vid but this undoubtedly deserves an exception. Thank you for the superb quality content you’ve put together. I wouldn’t be surprised if this series becomes the de facto video resource for systems design and architecture interviews. Hope you keep uploading!

can't thank you enough for this video. All of these components are building blocks and the interviewer can dig deep as to how the requests are handled from client to server. Awesome video

I have been following some system design channels on youtube and this one is by far the best. Well structured, it discusses tradeoffs, solutions are clear.... In one word, AWESOME! Thank you very much.

This is pure GEM!.. Amazing crystal clear explanation on Distributed computing concepts. Best tutorial till date on whole of youtube for system design prep material... Great work

wow! This is so to the point and even the duration of the video is as good as an actual interview discussion! Touched so many "must know" topics!

I don't think any other youtube video on System Design goes this deep into explaination. Thanks for your efforts in bringing these videos to all of us.

Each word in this video is a golden word. Make sure you don't skip or neglect it. Thank you so much

I love the way you dive in every component one by one.

These videos are best resource among all I have gone through. I am surprised, why this channel is not posting any more videos. Good Work and thank you !

this is the best video for systems design I've ever watched. I listened to it at 50% speed to write down every word whereas I usually watch YT vids at 1.5x -- each sentence was invaluable.

Amazing video series that goes beyond high level fluff, thank you so much!

This is the best distributed message queue system design I have seen so far. Many good concepts introduced and summarized , I feel it very helpful by using it as a guideline and read the other documents for more details such as the AWS SQS document. Looking forward to more content in the future.

Very thorough! Really appreciate your hard work. I can tell your channel will become huge for engineering resources.

On the system design topic, video very in-depth and concrete explanation on how to navigate successfully through the system design interviews. Thank you for this tutorial! ❤❤

Mikhail is the best. His system design videos are systematic and top notch with respect to quality and crispness. Presenting how to finish a complete design in 35 to 40 mins Wow, this is a great feat and only way for us. Prior to watching these videos my ideas and presentation is messy and dis-organized, but these videos gave me the basic sense what is expected in System Design and how to approach it. I take down the notes while watching the videos and try to apply these principles to other interview questions. This is greatly helping with my preparation. Can't thank you enough Mikhail, you are one of my best teachers.

This is quality content. Really appreciate your efforts.

I have seen a bunch of youtubers for system design interview...this one is one of the better ones...good way of using graphics while talking instead of mundane approach of heavy talking and using a whiteboard....Great job buddy...I am gonna explore all your video now

Please keep uploading! This is great! Thanks

Hands down best System Design Channel on youtube. I cannot wait for more videos!

Best course available on the Internet.

This is by far the best curated content on system design Wish I had come here before . Keep up the good work

I wish this guy was still making these videos. By far the best of the system design interview content out there (I've watched them all, lol).

thank you for working on the subtitles, makes it easier to consume your good content

Finally finished going through your videos. Thanks so much!

The quality of the videos that you make are really make. Please continue making such videos.

Very good content. Enjoyed going through the video. Thank you. Hope you continue this series

Great work and Explanation . Thanks a lot. This is the best explanation and walk through to prepare for a System design interview.

Just awesome...your approach to solving a system design is amazing. Great content. Thank a lot for your efforts.

This is amazing, thank you so much! Very detailed explanation and exactly what I was looking for.

This is excellent! Looking for more such videos.

Hi Mikhail, excellent video on the system design course series. Very nicely presented & explained. One of the best channel.

One of the best system Design video I have watched, Awaiting more videos.

Thank you for all the hard work and such a great explanation of complicated topics. This is way superior to other paid content. I would love to see video on API design that covers how and what. For example, question that say design an API to upload photos from iOS, how do you go about it? What are the good characteristics of an API? What are key components you need to think about while designing an API and so on.

Thanks allot @Mikhail. Your videos are so fun and easy to watch. I feel it's one of the best specifically for system design and you sound like some Russian Pro coder to top it;) I request you to make a video about all the possible components (load balancers, CDN, etc) in a system design interview that will ever be used, because you keep using few different components for different problems. If we get to know all the components, then we too can arrive at a better solution. Thanks for the great content and keep creating new videos! 👍

Such a great video with so much information! Many thanks!

Really great content and well paced delivery. Waitiing for more content.

Just awesome ! Your approach to problem solving is very generic. Really liked it and keep posting such fantastic system design interview questions. This is the best material i have seen till date on the topic. Thanks a lot.

Just the right amount of detail. Surprised it has fewer views. Shows view count is not very reliable. Thank you for the great effort

This is crisp and clear. Many thanks.

Thank you for your great work! Please keep it up with more great content! :)

best system design video I had watched.

5:10 load balancer availability/single point of failure, can assign multiple A DNS records for our domain (multiple IPS ) when we want multiple load balancers 8:00 usd cases for frontend service 9:15 request deduplocation to achieve exactly once or at most once semantics (the ack response could be lost when sending back to user, so user will retry and we don’t want to process duplicate requests that we already housed) 12:00 reusable components for other system designs

Thanks a lot for your videos! Very helpful! I wonder if it's possible for you to have a mock interview video (e.g. you're on the interviewee side), covering things like what tools/apps would you use for real-world SDIs for better productivity, etc.

I am expecting you write a SD book that gives us the same amount of useful information as here. outstanding job!

Your videos are Superb ! Most useful videos on System design. Please start making more videos like this ! More on each component details which helps in System design :) !

keep up the good work! please upload more system design videos!

Really comprehensive coverage of the topic! Although, would've loved to see more discussion on Asynchronous out-cluster replication. It's tricky to design, eg. what happens when you receive a get request while the data hasn't fully replicated across hosts in a cluster? can we hit all hosts in a cluster? what happens if we receive a get request when the message deletion replication is in progress etc.

Really great content. Please keep uploading more videos!

It is full of quality content, It took me around 2-3 hours to completely watch along with preparing notes for this video. And 9 pages of notes it is, initially I thought it would be in 3-4 pages only. It's full of rich content, that I had to note everything down. Thank you. I am looking for a better way to prepare for System Design questions. With this speed, I am not sure how much time It will take for me.

I bought your course. Arguably the best investment I made for system design courses so far. Thanks for putting together contents and explaining them so lucidly. I look forward to other two modules.

17:08 to clarify option 2, queues are basically each a cluster in this case(each cluster contains a set of queues). Instances are the replications. Interestingly replication for inmemory hosts/instances are handled similarly to nosql nodes

This is awesome!! Thank you!

This video is great quality. I think sequential writes should have been suggested in the video. This seems to me the best way to achieve high throughput.

Summary(notes): 1. Problem statement: Producer sends data and exactly one of the consumers gets the data 2. Resolving ambiguity in the problem statement by asking questions such as scale, priority, and so on... 3. Just focus on the core set of requirements - sendMessage(messageBody), receiveMessage() 4. SLA numbers for the non-functional requirements 5. Components: LB, Control Plane(Metadata-Service), Data Plane-1(Frontend), Data Plane-2(Back-end) 6. FE: Required Cross cutting concerns such as billing, throttling, the most important - routing to Backend since the Backend is stateful and so on. 7. Metadata Service: Caching Layer for routing information and metadata ( high consistency required in case of very few writes, R/W Ratio) 8. Backend Service: API Handling Layer, Storage and so on. Since Backend has to be HA and fault tolerant as it requires a consensus service like ZooKeeper or In-Cluster and Out-Cluster management strategy. ---- Extend the above design of queue creation with queue deletion, message deletion, message replication, delivery semantics( exactly once delivery not supported because it requires 2PC) and Pull vs Push messages, security and monitoring. --- Scalability Bottlenecks, use-case exntensability and use-case supported/limitations?

Great video! Thanks for uploading it. I have a question about the replication piece. Once an item is popped off the queue in the leaderless, how do we ensure consistency? Is there some sort of quorum that has to happen to avoid another pop request coming to another cluster?

Thank you so much for your videos. I have watched all videos in your channel. Waiting for your distributed database video. Hopefully you are still running this channel

Brilliant content! Please start uploading again!

Big thanks for the videos, great material !

Awesome Content, Thanks a ton :)

13:17 Strictly speaking, using a database as a backend message store is a valid option (because database = storage engine + high level type system). The problem is that FIFO message ordering semantics, that is usually expected from queues, is not easily achievable on the top of off-the-shelf databases. Because the latter are usually designed for a different query/update patterns than it's common for queues. But some databases can handle this custom case perfectly, and some queues (Kafka) are even positioning themselves as a databases.

Very helpful! Another super effective way to prepare system design interviews: Do mock interviews with FAANG engineers at Meetapro.

Thanks for such a great content.

Wow this is great stuff!

Thank you! This covers a lot of background. One thing i feel should have been covered how strict message ordering is achieved across partitions?

Thanks a lot. That is awesome!

Thank you for great quality content. Excellent explanation, I can't miss single word. Why the series is stopped? Any other resources would you recommend?

Covered many points similar to that I used in my daily work. Practical and useful!

Great video! I need to watch it a few more times. I got a question thought: Is Kafka a distributed message queue? How is it different than your desgin?

Please do more if you can :) excellent content

Golden quality.

Awesome content!

Outstanding!

great explanation, thank you!

This is golden!!!

This is more just Distributed MQ. The video covers the fundamental approaches in modern data intensive distributed systems.

great content, thank you

Hi! for option B: Small cluster of independent machines We know FrontEnd would look up the mapping info within MetaService to know which cluster to send the message to. But I am not really clear about how FrontEnd could connect to a random machine within a cluster. Does the mapping info with metaService also contain the list of machines within that cluster? Thank you!

Great content and explanation, thank you alot

Really helpfull videos, thank you!!!!

Reall enjoy the video, I wish I could upvote multiple times. One thing missing is how to persist message into file system.

It would be helpful if you also give a brief intro to RabbitMQ, Kafka and Kinesis/SQS. You talked about SQS briefly, that was really appreciative!

Overall great video. Speaker is very knowledgeable.

Great video!

Great content. Thanks a lot. Just one feedback: Would have been great had there been subtitles as I find a bit difficult to get the words.

Thanks for the great video. Just a question below: If we have multiple consumers, say, application deployed in a cluster and assuming consumers poll for new messages, how do we ensure that only one instance of them gets the message? Do we need to have some distributed lock on the message so that only one consumer would get it?

really good explanation. Thanks !!!

Thanks for the video, had one suggestion on rate limiting. Token bucket is widely used and not leaky bucket.

What will be the most suitable database for metadata service? Also, while summarizing the design at the end, why did you not include zookeeper in the block diagram? We are gonna use zookeeper as the out-cluster manager right?

Really great content. I have one doubt. In the out cluster management option, let's say we split each queue into multiple partition across multiple clusters, and each partition is being handled by separate clusters with replication of data in all the nodes inside the cluster. What if an entire cluster goes down? How will we ensure durability of the message belongs to the partitions managed by that cluster.. Please help me to resolve the doubt.

For monitoring, it’s be helpful to monitor size of the queue. Also number of messages getting queued or dequeued for each host at the queue level.

Thanks for sharing the design. May I ask a question? As you mentioned at 1:22 of the video, in case of queue, the message is received by one and only one consumer. Does it mean, we should create a queue for every consumer?

Great content! Thank you for making this. One thing that seems not clear (at least to me) ~min15-min17 you show the flow of the message from FE to backend node and to receiver. It is unclear how do you separate persisted information between MS (database) and in-cluster manager (ZooKeeper). Both seems to store mapping between Queue name and Leader Host. Do you really need to store this information in two places? How are they synchronized? Why would you need to keep this info in the MS in the first place? Isn't ZooKeeper enough for queue-to-node mapping?