In your example at 9:00, the business logic of transfer and transfer fee is tied down to each other. so this biz logic cannot be broken down to 2 microservices in first place regardless of event driven service or pub/sub service is used.
Nice and complete explanation, some examples where they are used: 1) all notifications work this way. 2)Event driven systems are particularly lightweight, so they are extensively used for Notifier hardware(sensors) with minimal processing power/ power consumption , to broadcast events whenever their is a malfunction/low battery. 3) they are mainly used for systems which are not inherently in requirement of security, though some kind of crypt might get used to ensure the source of msg to be valid. 4) not particularly a point but, should have mentioned youtube's own publisher-subcriber system while putting a video on youtube ;-)
Very nicely explained. Publish subscribe model is also used for order fulfillment by many companies when users place orders online to buy their products.
thank you for enlightening me about this system design. i have a comment on the arrows. those arrows supposed to be bi-directional since requests go into one direction and responses go to another direction. looking forward to your videos. thanks again.
I'm a second-year student pursuing btech in case, I started watching this video for a gcloud quest topic coverage. I literally have no clue what you're talking about in this video, but it was explained thoroughly so I guess it was cool. Thumbs up.
Idempotent means that the state of the server will be changed AT MOST ONCE by a series of invocations. Thus, the publisher-subscriber architecture isn't idempotent, as demonstrated in this awesome tutorial. Thanks, Gaurav! :)
GCP Pub/Sub states that if a topic does not have subscribers, but messages are published to it, the messages won't be delivered since they were published before any subs were present. This is different than what you had stated. Why?
I’ve watched a couple of your videos so far. Very good job! I enjoyed them. At 8:56 you mention that you’ll be dealing with ways to avoid the out-of-order problems (consistency, I guess) in a later video. Have you made that yet? If so, where is it? I’ve been interested in general message passing systems for a long time because of the loose coupling aspect of it. That’s what attracted me to the Actor model. But in trying to write some small programs with the Actor model (for example I use the C++ Actor Framework) I run into problems with out of order messages similar to what you’ve described here. The nice thing about the model is that each actor is a single thread, making it easier to reason about and avoid deadlocks. But according to the theory, messages are not necessarily in order within the system. So it seems as if it’s really just pushing the complexity of the problem of deadlocks into the realm of message order. What I’m looking for is general patterns, techniques, etc. for dealing with out-of-order messages. Or I should say, dealing with cases where messages need to occur in a certain order within a system. If you know of any resources let me know since I haven’t found any so far.
Have a look at ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-dGXHHABFGnk.html. It has a tech video in it's description which talks about the solution to out of order messaging.
@@gkcs You mentioned in your video, that this system is not good for Financial systems. Well only if the request needs an immediate response. For any load based system, this is a perfect architecture. Let's say you did perform a transaction and want it to be send to your analytics services to process and store it in it's local database, this architecture can work well. Something like this Server1 => DB1 Transaction => Message placed to Queue with Primary Key generated in DB with coming transaction (at this point my queue knows DB is updated with Server1 information) => Server 2 will pick message from Queue and query DB1 with Primary Key passed with message in Queue => Pull data from DB1 and perform analytics operation on it and store results into DB2. * Here Primary Key can be any unique identifier, maybe BatchId or maybe GUID (Unique Identifier)
Very good presentation......the explanation of failure is a bit wrong from POV of design....on can simpy argue just by redesigning so that S2 only sends a message after it has processed, SO takes the commission out, then the design is not going to have the problem stated. But kudos to you for explaining this is so much clean detail.
Great video. I was expecting this publisher subscriber thing. Moreover, you can also cover publisher-subscriber design patterns also.. which is using the object oriented model.
Hey Gaurav, thanks for the amazing content. :) At 1:05 you said "even if it is asynchronous, S2 will wait for S3 and S4 responses to come". I didn't understand why S2 will wait for the responses, since it is asynchronous it should not wait right.? it can go and process next request, in future at some time when it'll get the response it can handle that in a different thread. that is what i understood by asynchronous. please correct me if i'm wrong, thanks :)
He meant, on failure case, even if system is asynchronous, say, for failure at S4, S1 will only know about it, when timeout of S2 is reached, (and then timeout of S1 is reached.). You have the correct understanding of asynchronous.
Awesome work. Very well explained with examples:) 1. Are there more videos coming out for system design ? 2. Also I have a question related to system design. I do understand that system design questions are very broad and we need to narrow it down during an interview. But there are so many technologies that need to be learnt and I am quite sure all of them cannot be learnt. So when I am understanding and learning system design problems what are the must know technologies and also do I need to know all of them in depth. Can you cover the must know technologies for system design and maybe make a separated video for the different technolgies
Thanks! 1. There will be more videos on system design. There is a lot more to cover in this topic 😁 2. Some hot topics are Zookeeper, Cassandra, Docker and Git. It's best to know how they work internally to get a sense of general consistency and availability principles.
about the poor consistency issue, can't you wrap the transactions in a callback such as rails before/after commit. then it doesnt really become an issue
RU-vid might be using it as well You upload a video with service1(S1) The notification system for subscribers is pending in the message queue waiting for S2 or S3 probably a reason why I get notified late for new videos
Thanks for explaining this so simply Gaurav. In the financial institution example, How about if we build the debit transaction in S1 and post the message for transfer to MQ? If the transfer fails we can recredit the amount. Dont we see the same thing in current banking system where we get the debit message immediately and the transfer completes after few days. Thanks again.
Hey man, amazing series and big fan of your work. Wanted to touch upon the message delivery guarantees pub-sub architectures provide, like "at-least once, at-most once or exactly once". That would be appealing to developers as it will lessen their code logic to maintain the state. Thoughts?
At least once requires idempotency on the receiver. Ay most once is like the typical request-response behaviour of web apis. Exactly once uses transactions. Yes, the guarantees help a lot.
6:45 Hi Gaurav - when it comes to the setting with financial transactions/atomicity, is it basically the following? 1. Transactions are performed in their chronological order of arrival? 2. Transactions must be fully complete; that is, in a given series of transactions (t_0,t_1, ... t_i, t_{i+1}, ... t_n ), if Transaction ( t_i ) fails, then no transaction afterwards will be executed?
Hi Gaurav, You started by saying that S1 has to wait for S0 and S2, similarly S2-> (S3, S4). If you are using a queue that means that the response does not matter to the requesting machine. So the service can just asynchronously call the each other and forget no need to wait. To be clear we should definitely use message queues for these scenarios (benefits like retry etc). I just felt that your comparison was flawed.
Hey Amrit, that is a good question! This would be true if we had an atmost once message delivery requirement. I was highlighting the atleast once message delivery requirement. We don't care about the order, but we do care about messages being sent successfully :)
Gaurav Sen in case multiple request. Spawn multiple threads asynchronously, does not make much difference. Queues are definitely better , I am just saying the first point in your comparison does not make any sense.
I understand that multiple threads can speed up processing, however you have to wait for a successful response from both of them. We cannot ignore failure in our system. It has to be recovered from or reported to the user. If you have to wait for both of them to complete, you are dependent on their response. Is the point clear now? 🙂
What will happen if, say, a tsunami destroys the Message Queue? Is there a backup necessary/possible? Even in case of a backup DB: if there is some data that hasn't been synced before the disaster, is it lost forever? Is it possible to handle this scenario? PS: Great series! Thank you for your contribution to mankind!
same as any other distributed service. your message queue will be redundant and make copies of the data hopefully across physical locations to prevent this from happening. replication is a fundamental concept in the kafka message broker service. of course, that data for replication is being transferred across the network, so if data hasn't been replicated it will indeed be lost if the original node fails.
Hi Gaurav, i am a huge fan of your presentations and appreciate your time and effort on these. QQ- In a publish subscribe model like KAFKA, the publisher S1 or S2 publish the messages to the KAFKA topics and it is upto to the subscriber group to pull the messages isn't it. Listeners need to be configured to the KAFKA topics to pull the messages by the subscribers and the message broker doesn't push them to the subscribers. Please let me know if my understanding is wrong.
great explanation..But why the invoice is getting generated before even debiting the account? It should be the other way round right? But I got your point...inconsistency because one tractionsaction is distributed across services..!!
Hey Gaurav, Great content thanks for sharing. I have a question, Kafka exactly-once semantics overcame the problem of idempotency and consitency issue right?
In your opinion what should the best core API for gps telemetry? Producer API Consumer API Streams API Connector API I think should be streams API, isn't? So event driven model should be for microservices, or not?
Excellent explain action Gaurav 👍 One question , as this is pub/sub model, is there any option in informatica cloud application integration , when a simple event can be directly loaded into one database table, our Business team wants to load Salesforce platform events into a db table, I did not find any such literature in informatica, if you provide any inputs Mangesh
Great explanation as always. Just one more advantage - Since publishers and subscribers have different capacities for publishing and consuming messages. The number of messages that can be published/consumed is not limited as we can scale our Message Queues separately. Correct?
I have always a doubt regarding this type of architecture, how to I poll the message broker??? If I have node application that if listening for en event of RabbitMq in order to send a notification to my users, How do i poll ? A "for" ? A "while" ? Does RabbitMq come with a hook ?
Great content. One query : Suppose the commission is on % basis let say 10% . And publisher pushing message along with request I'd First time s0 (10% of 900)charged for 90 rupees S2 failure . Second time s0 see the message with same request I'd will not charged because earlier it charged as 90 and that is not more then it should be (10 % of 800) is 80 means there is difference of 10 and processing the s2 ( success). Can we use/maintain a different queue instance which help us : if any failure occurs on S2 then revert the respective transaction on s0 ( by maintaining some unique I'd to do so) . Please correct me if I wrong anywhere. Thanks in advance
Hi Gaurav. Just wanted to ask a question about about Competitve programming. Do you think we are at a point where CP has become absolutely necessary to get a job at a top tier company like Google, FB, etc? Also do you think someone with a poor math background should pursue CP just for job offers?
Hey Shreyan! It isn't necessary to do CP for a job there. Practicing algorithms and math within the syllabus is still enough to clear the interviews. Of course, practicing problem solving helps, and CP in turn helps you do that. If a guy has a poor math background, he should try and improve that. I don't think it helps participating with the sole purpose of getting a job. Keep hope! Cheers!
Gaurav Sen maybe I was a little ambiguous. I meant is CP necessary for a job in top tier company like Google/Directi. Will you say that CP is still not necessary?
"You cannot use this for mission critical systems, where either there's a success or a failure. There is no atomicity in this" (9:06 - 9:13) Could you please elaborate more on why this is the case? I know atomic transactions are that they either all succeed, or all fail together. Wondering if you could provide more context on this. Thanks!
Thanks! Inconsistent data is a problem in these systems. Like a account being debited but the other not being credited. This is possible if one service works and the other fails. This is common in messaging systems.
Hi Gaurav, your videos are really good, i have a question, as you said publish subscribe model gets the job done for twitter and gaming services, but what is the alternative for financial services, build a monolith app?
When idempotency is not an option or when we need consistency across services, the publish subscribe model is ill fitting. Mission critical stuff in finance has to be dealt with in a different way. However, banks and other finance services can use this is places like analytics and book keeping.
you said that this architecture is not suitable for financial application, then which architecture is suitable for financial application. is it Request/Response or something else?
Hii gaurav, firstly I really love your videos. Big fan. and i appreciate all that you do. we get to learn a lot from these videos. I have one doubt. In the bank example, we could have configured the message queue to send the message to S2 (fund transfer ) service first and if that is successful only then send the message to S0(Invoice) service right?? This could have solved the problem right? Or is there any catch in it which I might be missing? Thanks alot again!!!!
The message queue doesn't usually have ordering/conditional logic on subscribers. It could send events to S2 and take separate 'success' events from S2, which S0 can subscribe to. If the sucess events have enough context, S0 can work on it. This would be similar to a request response architecture, with the queue being a broker.
The message queue should have some way of persistence, if the events are important. The queue service also has to be distributed to allow fault tolerance. Have a look at Apache Kafka.
While showing request response architecture, it is advisable to show either bidirectional arrow or 2 different arrows (opposite direction) between client server
@@gkcs just to share information, we are developing DDS framework for Autonomous vehicles as it can handle both request response as well as publisher subscriber system (without broker).
I have been researching these kind of topics, and everywhere people explaining applied knowledge of pub/sub ie what/how/where to implement pub/sub (already implemented tools/stacks). I want to implement my own pub/sub from scratch then afterward I would like to go for how I gonna use it.
Bro i want to ask something suppose i have site called Amazon.com, In this we need to dump lot of code. I think they use micro services like home page served by one machine, log in by another and add to bag and checkout by other server right. then those servers are remotely connected right . if we click on add to bag the request goes to add to bag server(i mean the code related to this is present in that server) right ? then if one server gets failed the dependency servers are not works right in micro services. If we use monolithic that may works but no use of working those pages example with out add to bag page there is no use of checkout page right. Is this is the way the system would be. thats what i understood, is it fine ?
How about this: S1 updates its local DB, then it fails to publish the message to the message broker. Now you have inconsistency in your microservices. Failure to publish can be caused by various reasons and it is very likely to happen, although not frequently.
No, the queue has a responsibility to maintain the task status. At each step, it completes a job like persisting to local DB. At the end of each job, the status is updated. All the jobs to be done have to be idempotent for this to work.
There is a good pattern with Message broker to solve this problem. Some call it "Listen to yourself". The idea is S1 just publishes the message to the Message Broker and returns success Ack if it succeeds. S1 also has subscribed to this message and it updates its local DB only when digesting that message from the Message Broker. The disadvantage of this pattern is S1 has to deal with eventual consistency. When it returns success Ack it is only saying that it successfully received the client's request and being processed. In other words, the request has not been rejected.
Amazon SQS can be used as a the message broker. We could also use RabbitMQ or Kafka. A message broker is something which takes the responsibility of delivering messages to it's subscribers. You could have a look at the video here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-oUJbuFMyBDk.html
Could you add some numbers/ordering to your videos? I watched it after event driven architecture because this video came afterwards in your System Design playlist.
The discount code is 'gaurav'. You can also check out my system design video course at: get.interviewready.io/courses/system-design-interview-prep Use 'earlybird' for a 20% discount :)
Hey @gaurav, I've watched your previous videos and also subscribed ur channel ..I need your help to just have a look what I made so far based on knowledge from your videos.
