Notification Service System Design Interview Question to handle Billions of users & Notifications 

Thanks for the kind words. Do share it in your circle. Crawler is coming soon. Rest I'll try to do when there is some free time from work :)

@@codeKarle Api limiter is also a very common thing being asked and would love to see your extensive design perspective on the same :)

Could please share video link for these topics

I agree. No unnecessary noise and bulshit only hit on bull's eye. Great content

Yes. He is the best. But not sure why he has stopped and also not able to reach him. Absolutely awesome explanation with crystal clear clarity.

@@Nexatek He is working at Facebook . He might be busy. But , he really best

Thanks! Good to hear that you liked it! Do share this with your group :)

Thanks! Glad that you liked it 😊

Thanks :)

+1. SaaS customer level rate limiting will be better served if done right at the start. User level rate limiting can be done later.

Can you pls Clarify more?

There are two scenarios here: 1. If you have the same message( or message template) with same attachment(or attachment template) being sent to millions of users. If that's a use case that we need to handle, then I'll modify this a bit, and keep a message generator component that creates that message, and kafka would just have a message Id. once the email handler processes the message, before sending out, it can query the message generator with the message Id, and fill in the content and send out to the user either directly or via a vendor. 2. If there are different messages being sent out to different users, then i'll stick to the current design, and we'll have to have all the data being put into Kafka. I'll still keep the attachments separate though, in an attachment service. Also, as far as putting 1M messages in Kafka is concerned, that can be decided based on how frequently this use case comes in. If it's a frequent thing, it's better to have each component process bulk requests in one message. If it's not, the regular flow can be used. PS: By message template I mean replacing just the user email, name etc and everything else remains the same. And Thanks for the feedback on Audio. We are trying out a few different microphones and if the channel grows big, we'll be investing in a proper recording studio, but I guess that's far away :)

may be we can have callback from notification handler and callback topic in kafka which will notify failure and then our system will retry sending notification

I think this can be very well implemented with the existing solution as well. Just enabling auto commit offset to false, if the notification delivery failed, can help us with re-attempts. To add to it, we could have some monitoring enabled on top of it, which would show how many notifications deliveries have failed, and as such alert the frontend service about the failure. This would work something similar to Hysterix.

A reasonable solution to this could be to use different topics for different priorities, and having more consumers on topics that are having high priority messages. Low priority messages can be worked upon by a fewer number of consumers at their own pace. We could do something more smarter by keeping a running counter of messages of all priorities, and then at runtime spawn more workers for dealing with high priority messages if the traffic surges, but that might be too much over-engineering for a notification system.

@@codeKarle please make more videos

see consistent hashing

1. Typically, this thing can be achieved by staging the messages before dispatch to handler in some storage (or if Kafka has capacity, then in some specific topic). Basically it can be handled as "kind" of back-pressure. Now there would be some meta information flowing along with message to be delivered that would help identify this restriction and using the same, such messages can be segregated separately. Note that final delivery of messages happen via 3p service, unless complete integration (e.g. for SMS with telecom operators, for emails with SMTP platform and so on), so from this notification engine at max, the dispatch to such 3ps can be controlled. This is typical business rule enforcement. 2. When the SMS or Email is sent, again 3p service provider can send back ACK. Now that may depend on 3p vendor, if it wants to send ACK to some webhook exposed by this platform or by other mechanism. But ACK is quite important and SLA is enforced with 3p vendor so that for high priority messages (OTP e.g.) if ACK is not received within certain interval, the message is retried. This is mostly driven by 3p vendor agreement. In case of email, generally postfix logs from SMTP or equivalent servers are parsed. You can assume webhook as mechanism for ACK with message id embedded and can do the aggregation (from above example, using data from Cassandra) to drive certain intelligence or you can have separate topic in queue/kafka for messages pending ACK for retrial purpose if SLA is violated. Hope it helps!

We'll be storing this data in a NoSQL data store. In this video we have chosen a Mongo as a datastore to store all user transaction info. Reason was to do bulk queries like find all users who bought any mobile phone and send them a notification.

Fantastic use case!! I think chunking it and then putting it into Kafka would be a decent idea here. We can say that each message would have let's say 100-200 users, and we can put a few 1000 odd messages into Kafka. This might cause an issue in case some of the messages error'ed out while putting to kafka or during processing. Retry might be a bit tricky to implement in this case, because we wouldn't want to send the same notification to the same user again. But I think it's still a good way to implement this use case.

Or you can have dedicated partitions for different priorities

Let's say this system needs to scale upto many millions of Notifications per minute kind of a scale, and if we have a requirement to store all the notifications that we are sending out. Then we need a data store that can scale upto that many number of writes, one for each notification. This becomes a dumping ground for all the notifications in our system. Cassandra works well for that kind of a use case because it can scale horizontally. You can find a more detailed explanation here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-cODCpXtPHbQ.html

@@codeKarle thank you.

Sure, We'll try to make that, but it'll take a while.. a bunch of other videos are planned before that.

It would take time surely. Two use cases can happen: (1) Transactional system with massive traffic: I'll throw more hardware to scale it. Since we have millions of transactions, we can afford some extra hardware. (2) Promotional messages: I would rather stagger it and send the messages during those hours of day when there is lesser traffic. Normally you'll have a combination of these in any real world system, so for that you can use a hybrid approach.

Most of them were using draw.io

Thanks @@codeKarle

You can use a clustered Redis to avoid Single point of failure, not just here, in a lot of other scenarios as well.

We are in process of adding that. There are subtitles on most other videos now! The Automatic ones are very incorrect!! But some of the videos now have proper subtitles.

我懂你

We are using Kafka, so i dont think you meant network failure here, if the end service is not up the message will just stay in the queue until its back up and read again. Now the final delivery is handled by the end services, all the end services have to be fault tolerant and present with a retry mechanism (Services like SMTP for email and messaging are assumed to usually not fail), but just in case if they do i think the source will be present as one of the destinations and through the same notification service we can send them another notification about the failure.