Distributed Consensus and Data Replication strategies on the server

Подписаться 585 тыс.

Просмотров 155 тыс.

50% 1

We talk about the Master Slave replication strategy for reliability and data backups. This database concept is often asked in system design interviews with discussions on consistency and availability tradeoffs.
Very closely tied to the master slave architecture is the concept of distributed consensus. When designing a system, we must make sure that the individual components can agree on a particular value. (Leader election, distributed transactions, etc...) Some popular techniques are 2 phase commit, Multi Version Concurrency Control, SAGAs and Quorum.
Chapters
0:00 Problem Statement
0:53 Replication
1:24 Synchronous replication vs. Asynchronous replication
3:36 Peer to Peer data transfer
4:44 Split brain problem
Also useful when used with Master Slave replication is sharding. You can check out the video mentioned below.
Looking to ace your next interview? Try this System Design video course! 🔥
get.interviewready.io
Course chapters:
1) Design an email service like Gmail
2) Design a rate limiter
3) Design an audio search engine
4) Design a calling app like WhatsApp
5) Design and code a payment tracking app like Splitwise
6) Machine coding a cache
7) Low level design of an event bus
The chapters have architectural diagrams and capacity estimates, along with subtitled videos. Use the coupon code of 'earlybird' to get a 20% discount.
References:
Sharding: • What is DATABASE SHARD...
System Design Playlist: • System Design Playlist
Designing Data Intensive Applications - amzn.to/2yQIrxH
You can follow me on:
Facebook: / gkcs0
Quora: www.quora.com/profile/Gaurav-...
GitHub: github.com/gkcs
LinkedIn: / gaurav-sen-56b6a941
#SystemDesign #Databases #Replication

Опубликовано:

6 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 161

@gkcs 5 лет назад

Guys, sorry for the last 4 minutes of the video where the camera focus went haywire. I keep running into these technical issues every video. Maybe I should do a course on videography...any suggestions? Also, if you are preparing for a system design interview, try get.interviewready.io. All the best!

@sreyanchakravarty7694 5 лет назад

Where is your video with Tourist? Are you still in the process of editing it? And you might want to make a video about anti-patterns.

@mrtourist8066 5 лет назад

@@sreyanchakravarty7694 I thought that would be the next video but my assumption failed

@sreyanchakravarty7694 5 лет назад

@@mrtourist8066 do you know why he took the first video down? The video with tourist that is

@amitagnihotri30 5 лет назад

Use small aperture, pre focus and then set focus to manual.

@vishalpmittal 4 года назад

yaa great work bro. Just one advise, disable auto focus. Manually focus it on the board first and then start recording.

@mukundsridhar4250 5 лет назад

Excellent video as always gaurav. A few points from my side. 1. Taking snapshots at regular intervals of the db is one way to avoid the single point of failure. 2. Log reconstruction is one more way. Before writing to the db first flush to a log file. So if the db crashes the log file can be used to restore the db. 3. I think when you talk about the databases communicating with each other , you mean there is an API that sits in front of them and these API's communicate with each other. Or is it that there is an API in front of the db which writes to a queue and another process that polls the queue and talks to the second db. 4. One way to deal with the split brain problem is through the paxos algorithm. Great job and thanks !!! :)

@gkcs 5 лет назад

Excellent points Mukund, thank you! 😁

@phd9262 5 лет назад

Dude thanks very much. Your work is highly appreciated!

@vyshnavramesh9305 3 года назад

Key terms for Data Replication during interview: Atomicity (A of ACID), Master- Slave, Master-Master, 2PC, 3PC, MVCC, SAGA, Split Brain, Distributed Consensus, Regular DB Snapshots, Log reconstruction, Paxos Algorithm MIT video on 2PC: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-aDp99WDIM_4.html

@alokuttamshukla 5 лет назад

Great work brother!!! Awesome as always

@TheHp6515b 4 года назад

Great content (upvoted and subbed). Don't see any other channel with short and precise summaries of fairly complicated concepts. One more benefit of Master-Slave - Move replicated data closer (geographically) to the end user. For better user experience - lower latency. Also sharding and replication scheme are tangential and often work in tandem. Generally you will have sharded databases with each shard replicated. Replicated shard could be a slave or a master (consensus based). I vote for distributed consensus. In fact this would cover the meat of distributed systems and could be broken down into multiple smaller introductory videos Video 1> Inherent problems with distributed systems (failure detection, n/w partitions, byzantine(non) faults, clock skews). Video 2> Why do we need consensus ? Protocols to tackle #1 and what they tradeoff to offer consensus (CAP axis). Protocols - 2pc, 3pc, paxos (intro). Video 3> Real applications and tradeoff they make for better performances. (Why distributed acid applications are hard and to be avoided). I see lot of interviewees(i'm interviewer in one of the FANG) have surface knowledge of these concepts and often incorrectly adjust to a constraint (been guilty myself). Content like yours can help someone find gaps in their understanding.

@computernetworking6061 3 года назад

Awesome explanation Gaurav Sir 🤩 I learn a lot from your videos keep the good work up.

@jayantsogani8389 5 лет назад

Hey ! It was really helpful video for beginners ! Keep rocking

@forgetfulfunctor2986 5 лет назад

Solid content my dude!

@manishprajapati1171 6 месяцев назад

The point where the video gets blurry, I felt like I got hit by some sort of vision problem 😂😂 Video was very interesting and fun to learn.. Thanks Gaurav Sen

@harisridhar1668 3 года назад

Hi Gaurav - thanks for reviewing the differences between ( Master-Slave ) and ( Peer-to-Peer ) architectures : it's helpful to know that Masters have RW operatinos, but slaves have only R operations. Same for synchronous copying versus asynchronous copying : I should remember this in addition to shallow copying versus deep copying for object datatypes :-).

@shubhamqweasd 5 лет назад

Hey Gaurav, great videos :), I believe 2PC and MVCC are completely different things, they cannot be compared to each other, MVCC(weak isolation) is a way of handling transaction Isolation on a single node, whereas 2PC/3PC are ways of handling distributed consensus (as you have explained already). Transaction Isolation levels should have no relation with distributed consensus. Please correct me if I am mistaken here, will help a lot.

@gkcs 5 лет назад

Yes that is true. I think I mixed them up because of the close nature of consensus and locks.

@yashgupta417 Год назад

I was looking for this comment.

@blankblank103 5 лет назад

Great video !

@DivyanshuBansal 5 лет назад

Nice work Gaurav! This is the only channel for which I have enabled notifications. You are helping the whole community. Thanks a lot. PS: start an instagram page and post all these memes there also.😀

@gkcs 5 лет назад

Haha that's a really good idea 😁

@MarketesCM 5 лет назад

Thanks for the video!! I think it would also be interesting to explain how a consensus algorithm like paxos works and what problems it prevents.

@gkcs 5 лет назад

Yes, that's something to discuss in a future video (on distributed consensus) 😁

@sankalparora9374 Год назад

Thanks for the vid!

@gkcs Год назад

Thank you!

@PiyushSingh-vx7bx 4 года назад

Amazing content brother 👍

@jatinmotiani6378 5 лет назад

Great explanation Gaurav, You just made my interest in system design also😊

@gkcs 5 лет назад

Thank you!

@harsha8635 3 года назад

same here, I started loving this topic.

@nandanrathod 5 лет назад

Hi Gaurav, good video. Can you make a video over "how large scale Indian Unified Payment Interface works"? Focus on technical/Networking and Database aspects. Thanks !!!

@haaarshiiiit 5 лет назад

That's actually a great idea and a video I am looking forward to !

@speaktothepoint2108 5 лет назад

You mean, about the Payment Gateways of various financial institutions like credit cards, netbanking, ecommerce & digital wallets ?

@LudwigvanBeethoven2 5 лет назад

Great humor keep it up ^•^

@drakezen 5 лет назад

Your videos are great and you've done a great job in your explanations. You should do a video on book recommendations that aid people on learning about systems design thinking in this way.

@girish6064 3 года назад

Thanks buddy!

@umeshmalhotra4617 5 лет назад

Hi Gaurav, polling for next video - Reporting engine. In many systems, heavy reportings/analytics is involved. There are various solutions like using sql views/ late update using cron jobs. Using separate database or even servers. Please enlighten us on this.

@gkcs 5 лет назад

Sounds interesting, and I will add this to my list :D

@donha475 4 года назад

Good video. Subbed ;)

@prabhuroshan5762 5 лет назад

Khoob bhalo dada :)

@sdfg204 5 лет назад

Hey Gaurav, Great timing for the upload btw, Thanks for a very exhaustive discussion regarding master slave architecture. Much new stuff I came to know. Distributed consensus or "Quorom", as mentioned in your earlier videos, will be highly dependent on many factors as mentioned, but I believe that these concepts dealing with "system vote" in case of potential failure will require a branched study of its own, as it slightly outside the topics in discussion. Hey, your video was very complete regarding master-slave, can't focus more on that, but it looks like the "quorum" part also contains some of the architectural meat, and can be discussed in dedicated manner at once, for all the possible use cases.

@gkcs 5 лет назад

Thanks! That's true, and hence my call out to the "distributed consensus" topic :)

@sdfg204 5 лет назад

@@gkcs Yes, it is noticeable you are into it right now. Waiting for the resulting vid.

@ghanishsingla8270 3 года назад

Where is Distributed Consensus actually explained ? Did you make a video for that already ? Please share if you did. Thanks for the videos. Keep up the good work.

@calvinsugianto5963 2 года назад

Hi, thanks for the great work! I just have a question regarding the syncronization lag between master and slave DB. What is the best practice to solve this issue for a read heavy system ?

@Davti555 4 года назад

awesome.

@mohitsinghal5996 5 лет назад

Amazing vedio,can u give me a brief advise so that I could become a programmer like you.

@nirupam89 5 лет назад

MVCC is used by MySQL InnoDB storage engine as well.

@gkcs 5 лет назад

Wow, I had forgotten about that 😋

@rakibjahankhan4656 4 года назад

If there is single database, my application is scaling up in terms of user then the load balancer will not help, all the load will be in single database, for that how to serve fast in terms of user experience(read/write), how to scale or design the database??? i m seeking help for sql server

@sinawic1101 3 года назад

heey, great job! thanks for ur vids can i ask how do u know this much at this age?

@anupamkarn1009 5 лет назад

New hair cut! 👌

@prathmeshborle8570 4 года назад

Hi gaurav, will you make a video about system design of UPI and Bitcoin please

@niteshgupta212 4 года назад

In many Amazon interviews, they usually ask Command execution framework design and play store design. Can i have material for that?

@poornacmt7969 3 года назад

For active-active-active data centers, 3 nodes RAC cluster in each, if application has to do DB replication, what could be the best strategy?

@Dan-tg3gs 4 года назад

Is this considered isolation? Or consistency? I always get those 2 mixed up

@AakashGoplani 4 года назад

If our system has 'n' replicas, which replica amongst those 'n' stand chance to be master? What is the criteria for selecting one single replica amongst pool of replicas?

@mehulparmar9976 2 года назад

how master slave is linked to consistent hashing. Can they both work together or work as an alternatives?

@mitgundigara466 3 года назад

Hi Gaurav, In between you have explained the split brain problem, that if network partition has failed then A & B assumes that they are the master and both updates the data due to split brain problem. But however even if network partition is not failed then also there are couple of problems. What if S1 Send the Write request to DB1 and for same entry in database S3 Send the Write Request to DB2 then also both can updated and database will be in inconsistent state. So We need to make sure before performing write request to any DB Server we would have a lock on the entry in every server. How will we ensure that? Either B needs to talk to A or somehow some other system has to handle this. Because we can't blindly update DB and then tell it's other peer that I have done the update please sync. So without network partition also this needs to be handled. I am interested how this locking things will be handled can you make video on that?

@pawandeepb5967 5 лет назад

@Gaurav Sen, Any reason, you didn't talk about Paxos and Raft, for consensus? Also, @13:40 when you said co-ordinator, did you mean something like zookeeper which can do promote to master, or is that anything else ?

@gkcs 5 лет назад

I intend to talk on paxos in the Distributed Consensus video. Yes, something like Zookeeper would be a coordinator here 🙂

@mitgundigara466 3 года назад

Hi, Also there is one more question in master slave architecture we can Ideally keep only one master right if write requests are less? We can avoid 2 master to avoid locking related problems. When master dies then one of slave becomes master now again if our old master is up then this will become slave. So that will solve the locking problem I had mentioned. Ideally master -master design works good when the write requests are high otherwise should the architecture mentioned above is good? Not sure very since I am new to system design there might be problems also feel free to put comment.

@nankitable 3 года назад

What do you mean by database can crash? Generator to Hoga na!!😊 Amazing content, keep up the good work

@gkcs 3 года назад

Hahaha! 😁

@mihir469 5 лет назад

In AWS will multi AZ database work?

@yupp7352 5 лет назад

HI! i love your competitive programming videos. do u think you can make a serie of fendwick tree (some applications of it in problems, not the theory). i think it is hard to conceptualize a problem and using fendwick tree to solve it, so it would be really helpful. Thanks!

@ChandanMittal721 4 года назад

You can go through this article on Fenwick trees www.hackerearth.com/practice/notes/binary-indexed-tree-or-fenwick-tree/

@nikhilsingh2233 5 лет назад

Hey Gaurav, thanks for making a video on this topic. I had requested you for this. Could you please refer me to some links for the implementation part, as in, how do i implement this syncing of two databases in two different systems?

@gkcs 5 лет назад

Master Slave is a very common pattern, so a lot of these systems provide the functionality inbuilt. Like MySQL or Postgres. You can check out the implementation of these systems 😁

@trijit96 3 года назад

Please make a video on paxos and raft

@onkardeshpande743 5 лет назад

Can you make a video on load balancer handling the requests based on Geo-Location along with HTTP authentication tokens?

@gkcs 5 лет назад

Hopefully soon 😋

@user-ho9ff2np6u 5 лет назад

Can you make a video about proxy

@stalera 2 года назад

Good work. Just a small concern, what if there are 2 partitioning failures from a to b and c. We're back to square one. I'm guessing this wasn't the part of assumption you made. Can you throw some lights?

@denkozlov4220 2 года назад

should the slave and master be in the same subnet?

@YGorillAY 4 года назад

Hey man, awesome Videos on important topics! Actually, I haven't come across any other good source on Systems Design. Can you recommend some literature or a crash course on Systems Design? Or maybe you can make a whole course, going from topic to topic in a structured manner, more like how the topics are structured in a book, so that it can serve as a starter basic overview on the whole field. Cheers ;)

@gkcs 4 года назад

You can try: get.interviewready.io/courses/system-design-interview-prep I have the details of the course here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-beUzxZK6aKw.html

@austinjianxiaoli2625 2 года назад

The Batman meme got me dead

@jyotir124 2 года назад

thanks Gaurav for the wonderful video. Have a question on DB sharding. Do we store whole data in each shards and serve only subset basis shard key. How would shard know that I need to keep whole data but serve only subset of it. Thanks

@gkcs 2 года назад

We don't store the whole data. That would be replication without the benefits.

@pranjalpriyadarshi1289 5 лет назад

Distributed consensus can also be achieved by the Paxos algorithm, which also solves the problems of 3PC.

@gkcs 5 лет назад

Even then, Paxos is....slow. In some cases you have to go for eventual consistency to keep the business running.

@pranjalpriyadarshi1289 5 лет назад

@@gkcs Netflix and likes rely on eventual consistency right? #Clarifying Nice Content

@codeblooded 4 года назад

Hi Gaurav, awesome video. Do you happen to know which system design is probably used in the scenario when you run out of locks during distributed consensus pretty fast. Something like rate limiting for acquiring locks ?

@mysterOrel7924 5 лет назад

dude you just look like Donald Glover in The Martian ahahah great video thx !!

@gkcs 5 лет назад

😎

@rajcodingworld7768 5 лет назад

@Gaurav - can 'distributed consensus' help solve distributed concurrency issues? For example, service is incrementing no. of like of a comments. If current count is 10, if two liked at same time. Then can 'distributed consensus' make sure the count is 12 instead of 11. If not, can you cover these concurrency topic in distributed environment.

@gkcs 5 лет назад

Yes it can.

@sampath5242 5 лет назад

Auto increment, foriegn key having issue in master-master replication. How to get rid off it.

@gkcs 5 лет назад

Don't get into such a scenario if possible.

@edwardtan1773 4 года назад

5:25 you missed an opportunity to include the "Look at me. I'm the master now" meme

@gkcs 4 года назад

Hahaha!

@ARCHITSANGHVI13 5 лет назад

Why don't you make a technical meme page. Where we get knowledge with laughter. haha :P Amazing video and amazing memes in between :P

@gkcs 5 лет назад

I have an FB page you can checkout, mentioned in the description :)

@AmitSharma-rk1mg 3 года назад

I know of Raft and Paxos architecture to solve distributed consensus problem. May be Gaurav you can make a video on this sometime later. Or if you already have, then please share the link once.

@gkcs 3 года назад

I will 😁

@someshbose 5 лет назад

Hello Gaurav. Just want to ask what is your view on data science in industry field? Are you planning to switch to data science field in the near future?

@gkcs 5 лет назад

Maybe, I haven't considered it seriously yet.

@The_Personal_Picks_SnM Год назад

But won't there be redundancy of the link between A and B as well ? As you said at each step there must not be a single point of failure ,then why spof at the link between A and B?

@gkcs Год назад

That's a good point. We can add redundancy to the link.

@vrindasinghal3654 3 года назад

Why don't we just add multiple routers between A and B to avoid split-brain ?

@saurabhsingh7020 5 лет назад

Hi Gaurav, Nice video. I have some elementary doubts. 1) how communication between two DB could get broken which results in Split brain problem. My guess is they communicate on particular port. Closing of port is only way to cause this issue. 2) how master master architecture is achieved in DB. Bcz server would be configured to sent data to particular DB say DB1, howcome data could be dynamically sent to DB2. As much I know there is no load balancer behind DB set up. Thanks lot.

@gkcs 5 лет назад

Thanks! 1) There could be tons of issues, hardware or software. Some hardware failure example would be power supply loss, disaster, etc... 2) Master Master would mean you can send the update request to any server and expect a consistent state on the DB. The DB can also have a load balancer. It's a (DB) request response after all.

@saurabhsingh7020 5 лет назад

@@gkcs Thanks for replying. 1) Hardware failure ex power loss etc it means whole computer is down. It is different from Split brain which means computer is up just communication link is broken. 2) is this like config server at which source code point will list all DB instance. N using load balancer request could could go to any DB. If it's read request then any DB instance is fine provided Synchronous replication strategy is used. But if write request is also going to any instance then we have to enable this configuration of master slave take form of master master. Do you know what need to be done to achieve this switch setting.

@gkcs 5 лет назад

@@saurabhsingh7020 Those are good points, and I lost context while responding. Let me take some time and get back to you :)

@saurabhsingh7020 5 лет назад

@@gkcs Benefits of sharing knowledge. Discussion :-)

@rohan1456 4 года назад

06:00 The balance deduction thing, the 2 messages deduction 100 and deduction 50 are different messages right ? so what's wrong with deducting 150 ? I think I am missing a key point.

@munishbhatia4195 4 года назад

The problem is that serverA receives the message to deduct 100 and serverB receives the message to deduct 50 from the account, but serverA and serverB have lost their communication. So for serverA the resulting state of the account would be (120-100) = 20 and for serverB the resulting state would be (120-50) = 70 which leads to an inconsistency in the data.

@rohan1456 4 года назад

Munish Bhatia thanks man for clearing it out

@PrathamGupta2408 3 года назад

Bro, please name the video better, leaves me confused most of the time. I've noticed this in other videos as well. Other than this, your videos are nice, good explanation bro, Keep it up

@minhthongnguyen6254 5 лет назад

So finally, what is the reason about using Master-Slaves architecture but not Master-Masters architecture, after your solution about the third node part (C node). I dont understand :(

@gkcs 5 лет назад

Master slave is simpler and doesn't need 3 nodes, so it's cheaper too. The consensus is also faster.

@justinbieber9656 4 года назад

is gossip architecture implements distributed consensus or is it basically just another name for it?

@gkcs 4 года назад

It's a type of eventual consistency mechanism. Have a look at the "service discovery and healthchecks" video for a better understanding :)

@suhanigupta6701 5 лет назад

can you please make a video on SAGA?

@gkcs 5 лет назад

I will later 😁

@varunupadhyay3836 5 лет назад

Hi Gaurav, great video on the master-slave architecture. I have been asked a question about this in one of my interviews and wanted your views on the same. As you mentioned in the master-master architecture when one master fails the requests are directed to the next master. The question to me was that how much time does it take to make this switch to the new master by the load balancer (How does the switching actually happens one layer down) and what happens to the requests which are coming during the interval of the switch? How are these requests getting managed as the earlier master is down and the new master is yet not up?

@gkcs 5 лет назад

This has to do with service discovery. Let's see when we can talk about that 😁

@chiranjeevisaride 5 лет назад

Wat software u use for editing your videos???

@gkcs 5 лет назад

Adobe premiere pro

@IdreesDargahwala47 5 лет назад

Thanks for the video Gaurav. How do you implement this in real ? Like can you make a video on implementing this through code ?

@gkcs 5 лет назад

I'll check it out. You could have a look at MySQL which provides master slave features till then 😁

@prabhakarjha1168 5 лет назад

First like first comment!

@gkcs 5 лет назад

Awesome!

@krishnapurohit3494 5 лет назад

I've Amazon interview in 2 days for SDE-1, any tips?

@gkcs 5 лет назад

All the best!

@krishnapurohit3494 5 лет назад

@@gkcs Thanks!

@bootlesscoder5755 5 лет назад

don't you sleep in the night?

@SK-ur3hw 5 лет назад

What if I have millions or billions of data, say data of courier service and their mapping with the pincode of origin and pincode of destination in an ecommerce website. The mapping data can be huge. Let's say there are total 30k pincodes and say at least one courier service provides service for all 30k*30k locations/pincodes. I can't have one database for all this data plus can't have slave DB. So i thought to shard it on the basis of origin pincode but then also the data will be huge and many databases will be there again. So to have slave DB for each of them will be expensive. How to handle this situation?? Thanks for the video :)

@gkcs 5 лет назад

Thanks for the feedback 😁 Are you sure this is a real situation? I can't understand the usecase much 🙂

@SK-ur3hw 5 лет назад

@@gkcs it was asked in an interview 😅 so the question was, if you go to any ecommerce website and select a product and the seller, the application needs to check with the courier service which will collect the product from the seller and drop it to the customer. Now there might be few courier services which don't operate from the seller's location or from the pincode entered by the customer. And the few courier services which do that can serve to other pincodes too. So the use case given to me was that if there are 30k pincodes and say at least one courier service provides it's service to all of them means from each pincode to other 29999 pincodes so the mapping would be 30k x 30k in a table if you store all of them in one table. So now how to efficiently store the data. the columns are courier service id, origin pincode (seller's location) and destination pincode.

@jmoo4457 5 лет назад

I think by pincode, he means like zip code or postal code?

@nirupam89 5 лет назад

@@jmoo4457 yes. In India, it is called pincode

@napalm_exe 5 лет назад

Hit Thumbs Up button before video even started

@gkcs 5 лет назад

😁

@DaniIhzaFarrosi 5 лет назад

I'm really concerned about your title since RU-vid algorithm is sometimes misleading

@gkcs 5 лет назад

Oh, yeah that might be an issue...😛

@hpplayer3 5 лет назад

I like your hairstyle :p

@gkcs 5 лет назад

Hahaha, thanks!

@fsl4faisal 5 лет назад

is it just me or the audio isn't there..?

@gkcs 5 лет назад

It's...there 😛

@fsl4faisal 5 лет назад

@@gkcs yeah after a couple of refresh it came ..Thanks for the reply

@nameunknown007 2 года назад

I liked your video, if only I didn’t have to hear master/slave words so many times. I like the language Yugabyte uses, leader/follower. 🙂

@bostonlights2749 3 года назад

DDIA

@aneksingh4496 2 года назад

As Always good video ...but please dont include cartoon characters in between ,it breaks the flow of concentration .

@MdIrshad-me6wz 5 лет назад

Second

@gkcs 5 лет назад

Close!

@vikeng21 5 лет назад

hey can you slow down your speed a bit. looks like its 1.3x instead of 1.0x. i know you get excited but if you can tone down your speed people can follow a bit better.

@gkcs 5 лет назад

Try running it at 0.75? 0.75*1.3 is close to 1 :P