Data Consistency and Tradeoffs in Distributed Systems 

For new readers, I think it should be explicitly pointed out that Consistency in ACID is very different from Consistency in CAP. ACID consistency means that the values in DB should be valid i.e. if we are making two transactions both should complete before committing to disk. CAP consistency means read requests to different nodes of a distributed system return the exact same data. A system can exist without CAP consistency but a system should never exist without ACID consistency. ACID consistency is must-have for any RDBMS to be reliable/useful while CAP consistency can be a trade-off as per requirement. To be honest C in ACID doesn't add any proper value. If a transaction is Atomic, Isolated, and Durable, it automatically becomes consistent.

Your system design videos helped me get a job at Amazon! Thank you so much for all the great work you are doing! :)

The way you teach concepts from scratch/ bottom-up way is the most favourite thing in this channel !

One thing I learnt from this video today is you can’t get super computers in super market.

You said facebook is down and it was down after a week. what a coincident!

I like ur energy and the selfless appetite to share with community...

Thanks a lot for sharing this content! this is very helpful. Although I was aware of a lot of concepts there, your samples help me to figure them out a lot better. :)

Great explanation. However 2 phase commit is not usually recommended in the current industry due to a possible failure on the coordinator or the microservies will mix the things up by keeping locking the resources, timing out etc.. Therefore there are other options like 3 phase commit or SAGA which are widely used in fintech as well as in big companies.

The Tshirt Gaurav! 😍 NITS ❣️

Great explanation. Thanks for creating this video :)

very clear as always... what does the prepare statement phase offer that the commit doesn't in the 2 phase commit? since they can both fail and both may need to be rolledback.

That 'Yeah! I made a joke!' look at 8:32 is priceless!

Thank you for this great explanation

Nice, its also worth knowing more than 2PC, we can also use TC/C (Try Cancel/commit ) which does not explicitly lock the resources; rather it works by issuing compensating tx in case of failures..

That 'C++' expression was priceless 😁

Very well explained ❤️

Thanks bro I have been doing front-end for the past 5 years. I’m just putting myself back on the market now and this video is a great refresher on the fundamentals consistency in distributed systems.

Amazing explanation! Thanks a lot! I feel more and more confident in system design daily... Thanks to you, sir!

Amazing video. So easy to understand complicated topics. I have very limited experience with this and the pace and the content was perfect.

Hi Gaurav, thanks for making such great contents regarding consistency in distributed systems. But in case of 2 Phase commit, if one of the follower node machine goes down, then the leader who got commit response from other follwers, also can't commit it and may need to rollup the transactions. So until and unless the single follower comes back, the system doesn't accept any write operations, so it still puts in a single point of failure right ?

Here are my concerns: how do you track the prepare statements sent? How will server B differentiate two duplicate prepare statements? Also, as part of the prepare statement execution can we not update a status column of a row to indicate it is in process of being updated? Also how does the leader keep track of all of the data that needs to be sent out? What if there is just too much data to be sent out for one leader to handle?

Hi Gaurav, just wondering whether high water mark index will help here and the client can be assured that the commit is acknowledged as anyways everything noways a async in nature and once all the nodes in the cluster are up to the speed we can start showing the change to the client.

In the 2 phase protocol, consider a scenario. If server C receives a begin transaction order from A, and further fails to receive a commit order, what if we lock the read requests for some amount of time from C, until a "begin" transaction orders isn't committed yet, so that the leader can keep trying to commit the change in C. Latency is increased but I guess consistency can be maintained. Is this a correct? Please let me know

I loved the pause on the C++ joke >_< as if to say yes I’m going there

Excellent Gaurav

Could you recommend any good system design concepts apart from your interview-focused course?

Can you please share the link to the video where I can find the idempotency concept?

One of my favourite topics in this domain. Up next: Consensus ??

Have a question. If all the commit/prepares fail after multiple retries. Should we roll back on master as well? And in in 2 PC, instead of prepare and commit. Why can't we just: First send the update in phase 1 i.e. commit and then in phase 2 just compare A /master to slaves. If if matches it's done. If it fails we mark the state as disputed or eventual roll back

Thanks for the video

How did you solve the latency in a consistent system. Still servers have to send request from UK to EU for db updates (to remain consistent).

Don't know if you have a video on the books/blogs you follow. 👏👌#goodcontent for #systemdesign

Gaurav, do the UK, US, and other countries' governments agree to store their data across other regions? what do you think?

Would be great if the people who disliked the video commented on why they disliked it. Did you disagree with something he said? If so, write it in the comments to educate us on why you disagree.

'C' in CAP is what 'I' in ACID. 'C' in ACID has no analogy in CAP and means that there are constraints and the DBMS makes sure they are not violated. More specifically, 'C' in CAP is linearizability, while 'I' in ACID is serializability (and not even strict serializability, which would be more similar to linearizability). While both are what is called "consistency model" or "correctness condition" (a similar concept in programming languages and in processor instruction set architectures is called "memory model"), they are not the same.

"Maybe C was updated here; maybe C++", shots fired

This video is pure gold! Great stuff, Gaurav!

What is the use of kafka if all the communication been hapening directly from Leader to follower ?

Great work. Thanks for the knowledge. However, in the explanation there's 1 server with write which makes it again a single point failure and if we increase the write servers then we're back to square one. Can you address this?

Isn't the leader a SPOF here? Just curious.. may be if the leader fails, one of the followers become the leader? Wonder how that might happen and if there are databases that make it happen automatically.

you're a genious man

Just finished watching Social Network, and right in the next video click, I see an example of the same.

The very quick smile you gave the camera when you said "c++" 🤣🤣🤣🤣

Great explanation. I am having one questions as followed. Ideally, the followers or the replicas should reside in different regions or data centers to ensure availability. So, this introduces latency in every write operation in transforming the data synchronously from leader to the followers if we follow 2 phase locking. How to handle this ?

Isn't 2PC used for maintaining atomicity in case of a distributed transaction ? Got me a bit confused about we leveraging it to achieve consistency.

Wow, nice and bright. New camera?:)

08:30 he realises how c++ language got its name. its intuitive :D

Hi Guarav, In a two-phase commit, do the transactions of all the servers be rolled back if one of the servers fails to acknowledge?

Hi Gaurav ,Thanks for such informative content. Looking forward for a series on design pattern

While rollback sent by the master, does it send to all slaves ?

again learned a lot

That C++ smile and expression at 8:30 .. haha

When i started listening to your videos ... i had no idea other e than you know your stuff, right now I can keep up quite alright...i now appreciate the level of your skill...and the chronological argument you give and use to explain. I love this man

He is wearing NITS Hacks T-shirt. I still remember when he came to NIT Silchar as a judge for NIT Hacks and we had a photo with him.

Is C in ACID properties o SQL same like CAP theorem? What u said is for CAP. But if u check the oracle Docs Consistency in ACID means Data should not violate any DB constraints FK, UK, Col types etc…correct me if am wrong.

At 1:24 there's a mis-communication and writing on board for Get profile A

Liked the new edit. Kudos to your techie video editor 🤓

Now I get to know When we went to banks and they are like servers are not responding, but actually in the backend they are keeping themselves consistent. :D

Why not use kafka or rabbitmq and store in them all of the updates, and each database will subscribe and make the changes according to the queue. The data will be eventually consistence, is this a good idea?

Hi Gaurav, can you pls make a video on lifecycle of code? i e, what happens after the coding is done?

8:32 that smile hits me every time 😂😂😂

RU-vid recommend your video at 6am today ..ever since im hooked ... prepping for technical program manager role .. hope you would make some program centric videos too

Loved the video. Just to nitpick a little, Kafka belongs more to the pub/sub family rather than the message queue family.

What does GKCS stand for

Hi Gaurav Sen, when the video on Eventual consistency will come. Waiting for it

Thanks Gaurav, I have question, which tool support 2 phase commits? Kafka can you please suggest open source tool

What about partitioning? You didn't explain or I might have missed..

Hi Gaurav, Great video! But I have a doubt. Why does communication over a long distance lead to latency? Don't servers use electromagnetic waves(speed of 3 x 10^8 m/s) to communicate with each other. So, the distance of a few 1000KM's will only cost few milliseconds. Is that enough to cause latency? Or did I miss something?

🙏🙏🙏

Next Hotstar 🙏... High in demand

So we can roll back messages from Kafka 🤔

In case of single leader, there are high chance of single point failure.

25:00 SAGA pattern ??

That gaze 8:32.

It seems like you mixed Consistency and Distributed Transactions when you talk 2PC

Retrying every 5 seconds doesn't make sense if the system is down because it's not gonna be up and running soon and hitting it at every 5 seconds leads to wastage of resources.

Sir i am very much inspired from your english conversation, will u make a video on how to do coding problems means how to build logic for that please sir it's my humble request, please make video on that

2PC is used in which Industry?

While I am watching this videos, Facebook, Instagram and WhatsApp are down

I don’t think he released the next eventual consistency video

how Inertnet work ? GB MA how jio airtel control net speed ? Any new info ? apart from underground wires 😐

We are trying to solve single point of failure and consistency with distributed databases but ending up with the same . What am i missing here

Tsunami in Harvard☺️

What happens if transaction is to add 100 rs and leader retried 100 times, so 100 rs added 100 times?

Data sharing shyness is real.

13:38 "I wont explain everything here, I'm lazy too "

one more question When we delete anything it store in our computer only in hard disk just remove the link so that we cannot find that file And now we can store another data on this data it will overlap. but how can we transfer the deleted storage files to cloud that means our disc will can work for more years

Do you know you don't need supercomputer to scale? You can scale with cloud computing. The best thing is that you don't need physical shopping. You just click and pay. 😜

Super computer at super market Harvard vs Oxford 😂

C >> C++ ... LOL

1st