What changes would you do in your design if a requirement comes in where notifications have different priority, like P0 notifs need to be sent before any P1 and P2 notifictaion?
Probably just ensure that they're going to the same kafka queue via some sort of partitioning key. Kafka queues are consumed one at a time, so that should ensure the ordering for me. If you mean that you expect the P2 to be delivered after a P0 but it is published first, well I think you'd have to clarify your requirements a bit more. Clearly you'd need to buffer the events for a bit and use some sort of priority queue to occasionally poll one and send it to the world.
Why do we need to use Zookeeper to store metadata like partitions in a topic , IP Addresses of those partitions etc. Why can't we use a DB for that purpose? What are the advantages of using Zookeeper over a normal key value DB
Zookeeper is strongly consistent which is very important to have when dealing with actual configuration of your architecture so you don't make stale reads Recommend watching my ZooKeeper video
Great design ! Of restricting user to just one web socket connection with server by queuing messages ! Couple of questions ! 1-What’s the implication of user subscribing to topic (topic to user list mapping I mean ) topics are internal to our notification service ! For example user May be interested in getting notified for some product X coming in stock , so our general topic can be product in stock but user is only interested for product X in stock notification ! ? Thanks
what about celebrity problem? ... would you do pull vs push in this case? you also mention user queues - how do you batch the messages for a user? ... imagine the user doesnt want to receive 20 messages per day - instead 3 or 4 message might be a better user experience...
Yeah I think using dedicated celebrity queues which the user connects to makes a lot of sense there. If you did need to batch messages, you could use stream processing to do some sort of time windowed joining (see the stream processing concepts video)
Thank you for video. I still have one question. How to to avoid sending duplication messages? You say that unique id is attached to each notification message. How to use it then? It which step?
A consumer will receive messages on a message queue, it will then cache the unique IDs locally. If it's seen that I'd before, don't handle the message again.
@@jordanhasnolife5163 So, where is cached? Because it might be some consumers (backend) and each one caches locally, so one consume does not know about cach of another, right?
@@povdata if the message gets delivered to a node but the node is unable to send an ack back to the broker due to network issues the broker may deliver it a second time
Questions here: I'm assuming that the publishMsg API will finally trigger the service to push SSE msg to the Frontend Users, so the consumer servers as the ones who pushes the messages, but how does it know that the client is connecting to itself, what if the notification consumer picks up a message and find that the client is not connected to it ? Is it being controlled by the UserId Sharding queue ? so that the queue always know which consumer to send to ?? (and what if a consumer is down, does the queue know it and take care of the reconnected consumer ?)
Yeah you basically said it. We're partitioning based on userID. If the consumer goes down, consistent hashing will ensure that those messages and the client frontend will connect to a different consumer, but the same one.
Yeah very possible I forgot it here, you'd probably want some other stateful consumer that is aware of which queue a message from a given topic should be routed to.
The problem is that we have a chat app and when the app is in kill state in android or ios the push notification does not work so we go for FCM and we dont want to use firebase as google data issiues so do you think it is possible to build your own push notification in kill state like fcm. i will be looking forward for your answer
Not sure - I imagine there's a way but from what I know about mobike dev most people outsource their push notifications to FCM, one service (I think it's called), or SNS
Curious why we use SSE/websocket here instead of long polling? I would assume notification is more like uni-directional communication instead of bi-directional here.
Also not sure whether I missed anything, is HBase use single leader replication? It mentioned in this video but iirc this is the first time I heard of this.
Yeah but isn't notification more like uni directional pattern which push notification from server to client? So why we don't use long polling here?@@jordanhasnolife5163
Great design ! Of restricting user to just one web socket connection with server by queuing messages ! Couple of questions ! 1-What’s the implication of user subscribing to topic (topic to user list mapping I mean ) topics are internal to our notification service ! For example user May be interested in getting notified for some product X coming in stock , so our general topic can be product in stock but user is only interested for product X in stock notification ! ? 2- we are keeping Queue per user design which might hog resources as user base increases? Thanks
1) There will likely have to be some logic by a consumer to figure out which messages are relevant for which user 2) There's not 1 queue per user, there's just 1 user per queue. You can have many users getting messages through the same queue
Followup question.. Why do you think a userId sharded 2nd level of MQs' are necessary? In most practical applications such as say chat for example, the notifications still go through multiple mediums based on what the user subscribes to (say push notification, email but not SMS). This information could be made part of the payload as tags that the broker can aggregate on for easier consumption. For any medium, it would then make sense to have a storage mechanism to hold messages that were not delivered. I don't think the current design saves any resources on the server(threads?) on the server either since you probably need to maintain a connection with each device the user has in order to send the message. Thoughts?
Yep, the point was actually to optimize for the client device. If a client has to maintain an active connection for every single topic that they are subscribed to, they need to maintain potentially hundreds of active connections, which to me at least seems excessive. I'd agree that while perhaps a second set of message queues aren't necessary, it seems pretty clear to me at least that the client should only be pulling from one device as opposed to actively maintaining many TCP connections. For a phone at least, that could really use a lot of data/drain battery life.
@@jordanhasnolife5163 interesting. How many of these connections must one hold ? Unless it's real time (chat or something of that nature) isnt it mostly a fire and forget (email/sms) . Either ways worth having a conversation with the interviewer I guess ..
@@VivekMahadevanv If it's fire and forget I agree - however I've seen notification service talks that assume you're actually directly pushing to the service, in which case all the messages for a user have to go to the same place
@@jordanhasnolife5163 isn't the second queue also necessary to ensure the order of delivered messages for a given user? the first queue is topic-based, this is necessary as it enables per-topic heavy post-processing right after the first queue, which is quite likely. Then the second queue makes sure messages for each user are in the same queue and shard, hence ensuring the order of message consumption. Finally the notification sender server is 1-to-1 mapped to a user partition so to have only one websocket per mobile client. Am I right?
I am trying to implement an API for one of the notification services. Let me explain the architecture. There is a gateway service that has methods for creating configurations. A person can interact with the service and create a configuration for sending a message via email, for example. Then the person calls the Notify method, which accepts the following DTO: 1. Consumer information 2. Transport configuration 3. Message: [type (simple, broadcast), content] The Notify method looks at the transport and understands that the message should be sent using the email service, so it sends the DTO to the email service. Ok, so I described how gateway and email transport service work. However, there is a transport called "client" that has not been implemented yet, and it needs to be designed. The main idea is that the client can be a web application, some desktop application, etc. It is clear that delivery will happen through a message broker, but how will it all look? I don't quite understand
@@jordanhasnolife5163 maybe I’m bad in explanation and my english also bad but yes! Lets imagine we have rabbitmq broker with different topics. We want to send notification to a user so what topic to use? Maybe you have an example api?
@@jordanhasnolife5163imagine user is offline, we keep all notifications in the database, but what if there are a lot of them? if a person becomes online and calls fetch_notification, they will immediately receive a lot of notifications and this is bad. how to solve this problem?
I think this is definitely useful from the perspective of not dropping notifications, though if it's okay to drop the occasional notification then it isn't a huge deal if we don't acknowledge received notifications and stick them in a database somewhere.
On using bloom filters for idempotence: Wouldn't we start getting a lot of false positives at some point (due to hash collisions)? That would lead to erroneously rejecting non-duplicated notifications, which seems counterintuitive to idempotency. We're essentially being so "aggressive" that we're actually rejecting "good" (read: non-duplicated) notifications.
@@jordanhasnolife5163 I had thought about that but was worried about the case when you clear the BF but then process an actual duplicated notification. Probably an edge case but worth mentioning nonetheless. Appreciate the reply and thanks for putting in the time to make all these videos!
@@SlashTagged I think in the worst case scenario you'd just check if you've seen it before - remember the bloom filter can only tell you with certainty if you HAVEN'T seen something, so we have to check again if there's a chance we have seen it anyways
Hi Mate , can u please elaborate more on local state management using flink here ? Like consumers are sharded on hash of user id and topic has list of users to notify , how state management is happening here !
Basically the consumers know for each userId, which topics they are subscribed to, so they can easily send messages to the appropriate queues. We just get better data locality out of doing this than having to consult a database for this information. It effectively is just a cache.
Following up on the user id shards. Do you mean using user ids as partitions for 2nd log based queue? If yes, then will it not be unoptimised to have 1 billion partitions (as per capacity planning) even tho there is no limit on the numb of partitions?
I think that you're maybe slightly misinterpreting - being sharded by userId doesn't mean that there will be only one userId per partition, but rather that we just partition on the hash range of the userId field - there could potentially be many userIds per partition, it's just important that all the messages for a given userId are sent to the same node so that we can keep a persistent websocket connection on it
Well at the end of the day your only option for something like that is to add more resources, right? Assuming you're doing this in AWS I'd imagine you'd have some autoscaler to give you more when traffic increases.
Haha no need truthfully - I'd rather put my videos out there for people to see, as I think in the long run it's better to get the exposure. Plus, I'm not really qualified to be charging money for this stuff as I'm mostly just ripping it off of other people for the time being. I appreciate the view!
Great design ! Of restricting user to just one web socket connection with server by queuing messages ! Couple of questions ! 1-What’s the implication of user subscribing to topic (topic to user list mapping I mean ) topics are internal to our notification service ! For example user May be interested in getting notified for some product X coming in stock , so our general topic can be product in stock but user is only interested for product X in stock notification ! ? 2- we are keeping Queue per user design which might hog resources as user base increases? Thanks
