Тёмный

What is DATABASE SHARDING? 

Gaurav Sen
Подписаться 568 тыс.
Просмотров 893 тыс.
50% 1

Sharding a database is a common scalability strategy for designing server-side systems. The server-side system architecture uses concepts like sharding to make systems more scalable, reliable, and performant.
Sharding is the horizontal partitioning of data according to a shard key. This shard key determines which database the entry to be persisted is sent to. Some common strategies for this are reverse proxies.
Database interviews ask for concepts like sharding to make databases more performant and available. This makes horizontal partitioning a logical choice.
Looking to ace your next interview? Try this System Design video course! 🔥
interviewready.io
00:00 Introduction
00:52 Sharding - The problem
01:59 Horizontal Partitioning
03:20 Considerations
04:30 Potential Drawbacks
08:46 A challenge!
Video lectures, architecture diagrams, capacity planning, API contracts, and evaluation tests. It's a complete package.
Horizontal vs. Vertical partitioning: stackoverflow.com/questions/1...
Consistent Hashing:
en.wikipedia.org/wiki/Consist...
• What is CONSISTENT HAS...
Sources:
/ how-sharding-works
/ why-i-love-databases
www.quora.com/What-is-the-dif...
en.wikipedia.org/wiki/Partiti...)
Designing Data-Intensive Applications - amzn.to/2yQIrxH
#database #sharding #systemdesign

Опубликовано:

 

17 май 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 697   
@UlfAslak
@UlfAslak 2 года назад
Notes to self: * Sharding is basically a hiearchical way to index databases. * One problem is that you have to split the database somehow. What do you split on? * You only shard shards when the shard grow too big. * When shard fails you use the master/slave architecture. Writes always go to master, reads are distributed across the slaves. When the master fails one of the slaves become master.
@rishabhrastogi6076
@rishabhrastogi6076 2 года назад
Nicely put :)
@user-eq4oy6bk5p
@user-eq4oy6bk5p 2 года назад
I think another important question to ask is what happens if your shard exceeds its capacity. The way it works is that it further splits the shard into smaller shards (DynamoDB does this behind the scenes)
@sarathbaiju6040
@sarathbaiju6040 2 года назад
What will do when both master and slave failed?
@GvSharmaBKP
@GvSharmaBKP 2 года назад
@@sarathbaiju6040 replicate them atleast 3 (general practice)
@prateekgautam7398
@prateekgautam7398 Год назад
@@user-eq4oy6bk5p I just have one question. When we further break a shard into multiple shards, does those shards again go into different database servers? Or they reside in the same database server?
@stanlyjohnson5999
@stanlyjohnson5999 5 лет назад
You know you're a programmer when you count pizza slices from zero :P
@gkcs
@gkcs 5 лет назад
Hahaha!
@sardorbekomonkulov6379
@sardorbekomonkulov6379 4 года назад
Or really hungry
@mohammeddanishreza4902
@mohammeddanishreza4902 4 года назад
U r wrong... Person might be a Matlab programmer where count doesn't strts with zero.
@anktrj
@anktrj 3 года назад
​@@gkcs Did you said "No we're not going to learn RDBMS" at t=40? BTW a great video Thanks!
@MattDeLuco
@MattDeLuco 3 года назад
You know you’re a DBA when you count them from one ;)
@pavithranravichandiran6720
@pavithranravichandiran6720 5 лет назад
Can't just believe, you're uploading this for free... Please continue and we sure do want to sponsor the channel if that is in the cards!
@dhruvilpatel856
@dhruvilpatel856 5 лет назад
I see 67 likes so far. I doubt all 67 people are going to donate. XD
@ruhinapatel6530
@ruhinapatel6530 4 года назад
Keep doing what you're doing..You explain too nicely...and your channel is the most handy updated and to the point..learning was never so easy..
@ishannigam7352
@ishannigam7352 2 года назад
@@dhruvilpatel856 12k likes :)
@juggernaut420
@juggernaut420 5 лет назад
You could be a greater asset to this country if you pour yourself into research. I studied this course from the best of the best faculty in my college, who literally knows things, but you! Hands down man :)
@gkcs
@gkcs 5 лет назад
Thanks 😍
@avinashdwivedi2015
@avinashdwivedi2015 5 лет назад
You should have learned from best in world from internet..
@martintourneboeuf5282
@martintourneboeuf5282 4 года назад
He could be a teacher in your University, but I wouldn't see him from Chile. Very nice videos in general. The intro of this one !!! Good job !
@Shomerful
@Shomerful 2 года назад
lol research. you can be an asset in the industry as well.
@tusharrebaca
@tusharrebaca 2 года назад
Now I seriously doubt about your College Quality :P :P :P - You should have more focused during your JEE time. :P
@official_roboGOD
@official_roboGOD 5 лет назад
Your System Design Playlist is my favorite. I've got so much to learn. Thank you!
@gkcs
@gkcs 5 лет назад
Glad to hear that 😁
@2devanshi
@2devanshi 5 лет назад
good precise and short. exactly what is required in tutorial video with a bit of fun element in start.. awesome job
@sharathkumar1978
@sharathkumar1978 4 года назад
The initial reaction of you saying sharding and your expression is awesome which makes someone really remember the video and the content by just imagining while asked in interview. Thanks gaurav :)
@satyagunnam6324
@satyagunnam6324 5 лет назад
I am a non-tech person, looking to learn system design concepts. You do a phenomenal job explaining these concepts & also super fun. Great job Gaurav & thank you very much!
@gkcs
@gkcs 5 лет назад
Thank you!
@1125ram
@1125ram 4 года назад
I can't able to resist myself without commenting after I see your enthusiasm to teach to unknown people around the world... Hats off to all the vloggers who are sharing knowledge freely through RU-vid.
@pradeeppradyumna601
@pradeeppradyumna601 2 года назад
I like the way you present all the topics. It's so captivating, and it shows how passionate you're at solving System Design problems. Thank you for sharing your knowledge.
@kshitizsharma5469
@kshitizsharma5469 3 года назад
You are simply amazing man. I used to dread system design, but just because of you, slowly getting comfortable with it. I have actually allotted a particular share of my daily todo, just to watch your videos everyday.
@SupGhostly
@SupGhostly 5 лет назад
I love this play list and I'm learning so much but it the humor 😂 and your jokes that keep me learning great job man! Do you have a book recommendation on system design?
@theonlyarjun
@theonlyarjun 3 года назад
For the first time in life I'm actually enjoying theory
@ecandil
@ecandil 3 года назад
I don't know how I didn't come across your videos before! What an easy way to learn, well done mate!!
@lien3723
@lien3723 4 года назад
That little skit at the beginning is so funny lol. Thank you for making these system design videos, they're really helpful!
@sprihadeshpande2204
@sprihadeshpande2204 4 года назад
You are one of the best. I can't wait to watch all your tutorials for system design! :) Keep going!
@XenonSchneizel
@XenonSchneizel 4 года назад
Love your vids. The way you deliver your information is really clear and concise. Amazing quality content!
@Toy8610
@Toy8610 Год назад
You are a living legend Gaurav! I learned more from your videos than any of my previous teachers
@shivamtiwari8914
@shivamtiwari8914 3 года назад
The topics covered were organized,relevant to me, and easy to follow . I loved it so much
@chandrabhattasriram9475
@chandrabhattasriram9475 5 лет назад
The way you slipped the "Like & Subscirbe" part into the video, just one word - S L I C K
@rupal9267
@rupal9267 2 года назад
You make life simple! The intro had me hooked in till the very end! Bravo!!! 👏
@0ManishSharma0
@0ManishSharma0 5 лет назад
This inspired me to watch all of your videos. Subscribed
@GruntXIII
@GruntXIII 4 года назад
"Let's say you have pizza, and you can't have the entire thing by yourself..." you've obviously never seen me eat pizza...
@AnuragSharma-pk9pr
@AnuragSharma-pk9pr 3 года назад
I am an Oracle database developer. I loved your content man. Learned something new today. Thanks. Greetings from your new subscriber.👌👌😊
@ypucandeleteit
@ypucandeleteit 3 года назад
I recently found your channel and I just want to say I love your work. Thanks for creating fun informative content like this
@gkcs
@gkcs 3 года назад
Thank you 😁
@davidpham6330
@davidpham6330 4 года назад
Never been so interested into database sharding just from that intro. You rock! Thank you for the videos
@gkcs
@gkcs 4 года назад
😁
@interviewsortout2165
@interviewsortout2165 2 года назад
First time I was watching some video tutorials and I did not sleep.
@darshathakkar
@darshathakkar 4 года назад
Thank you for sharing this Gaurav. It means a lot.
@ALOKKUMAR-vt8bs
@ALOKKUMAR-vt8bs 4 года назад
Sir,Your Work is too much Valuable and appreciated, Thank you so much for Doing this amazing work.
@vaibhav1302
@vaibhav1302 4 года назад
Simple explanation and to the point. Keep up the good work bro. Cheers
@DivyanshuKumar-xj7gw
@DivyanshuKumar-xj7gw 5 лет назад
Your way of explanation is really cool. Super video sir.
@catman8623
@catman8623 2 года назад
Love the explanation. Digestible while still retaining the core information of this utility and its implications.
@hidayatullahkhan6194
@hidayatullahkhan6194 5 лет назад
Woah opening was super cool
@sharthakghosh970
@sharthakghosh970 5 лет назад
"indexing is cute" hahahaha
@saravanprathi6956
@saravanprathi6956 4 года назад
I had to pause the video and laugh for that :D
@Jonathan-qz9td
@Jonathan-qz9td 3 года назад
Cracked me up 😂😂😂
@trishantpahwa6554
@trishantpahwa6554 2 года назад
Cuter when all sharding can be personified as is indexing over the network, wouldn’t it? 😂
@nikhilkolhe7513
@nikhilkolhe7513 2 года назад
Me spending days to implement indexes 😭
@ranjithpals
@ranjithpals 5 лет назад
That was a great job, can you do a video on what is a microservice and differences between soa and microservice?
@abbhi1609
@abbhi1609 2 года назад
Intro is awesome man!! Love it
@adnanqz4974
@adnanqz4974 11 месяцев назад
You’ve explained this concept so well! At work, I’ve always heard my manager use this word when explaining his responsibilities with MongoDB but I’d never understand it. Now I do, thanks to you dude! PS: LOVED the ending 😂
@deactivated6737
@deactivated6737 3 года назад
Amazing explanation on database sharding, thank you for sharing.
@samart3010
@samart3010 10 месяцев назад
Initially I was not able to catch this concept when I watched this video 2-3 years back. But as of now I am with a base knowledge I able to understand. The problem with me was not really a base knowledge, but a lack of perspective towards these system, I was sticked to my perspective and was trying to build over it, so sometimes it is better to gain knowledge with clear slate.
@rahulshetty3738
@rahulshetty3738 5 лет назад
This is really fun to watch!
@praveenX
@praveenX 5 лет назад
Great work guys! I just stumble upon on this video but it turns out to be great! loved your explanation and confidence. Keep it up. I have just subscribed this channel and would love to see more videos.
@gkcs
@gkcs 5 лет назад
Thanks!
@Hotmustardgas20
@Hotmustardgas20 4 года назад
I just discovered your channel and I'm a huge fan. I have been studying to get certified in AWS and your videos have been helping me understand larger concepts.
@gkcs
@gkcs 4 года назад
Thanks! Interesting name you have there, Hotmustardgas :P
@Hotmustardgas20
@Hotmustardgas20 4 года назад
@@gkcs Lol thanks I'm glad that you like it. Do you have any experience with the major cloud platforms that are out there? If so you should make a video on them.
@akankshagarg2204
@akankshagarg2204 3 года назад
God I so miss eating Pizza in pre-lockdown days!! I have just started watching your videos for an upcoming interview and I am finding them quite useful. Thank you so much :)
@rounakkolya7350
@rounakkolya7350 2 года назад
OMG!!!Today's interview went exactly like this!!!!! except the sharding which led me to this video!!!!!
@rahulsaikia9014
@rahulsaikia9014 5 лет назад
Dude you are doing really great; learned a lot from you!!!!! Liked, subscribed, shared!!!!!
@deeraj3069
@deeraj3069 4 года назад
The ending was superb ...Enjoying the series very much
@timothy6966
@timothy6966 Год назад
This channel is a gold mind for entrepreneurs in the pizza-business.
@Anurag7376
@Anurag7376 5 лет назад
Your videos are really good. Editing and delivery are on point! Keep it up.
@gkcs
@gkcs 5 лет назад
Thanks!
@DeepakSingh-fi9tx
@DeepakSingh-fi9tx 4 года назад
Hi Gaurav, Your videos are really very helpful. Apart from videos the references you have provided are mind blowing. Thank you so much.
@seanpowers7132
@seanpowers7132 3 года назад
This cat is smart. Thanks for dumbing it down for me. This is fantastic stuff Gaurav!
@thenavigtor6867
@thenavigtor6867 3 года назад
Last line is ethusiastically official way of building interest in users...very well explained sir.
@sriharshasamana
@sriharshasamana 3 года назад
You have made it look easy! Thanks bro. I want to hear more from you :)
@poorvaparande5933
@poorvaparande5933 Год назад
Thanks a lot! you are doing an amazing job and teaching complex concepts in a interesting way! :)
@sandeepjoshi8531
@sandeepjoshi8531 5 лет назад
You are too good brother. Your passion towards computer science can be clearly seen.
@gkcs
@gkcs 5 лет назад
Thanks Sandeep!
@rohan1456
@rohan1456 4 года назад
Your creativity and content just get's better.
@AyushRaj-so3zh
@AyushRaj-so3zh 2 года назад
I had to stop the video immediately and come down here in the comments section 😊..what an intro...what a playlist..kudos Gaurav for these amazing videos
@paridhijain7062
@paridhijain7062 Год назад
Well Explained concept. 👍Understood it easy with those visuals in the starting of this video. It made the concepts more practical to understand. 🙌🙌
@ChocolateMilkCultLeader
@ChocolateMilkCultLeader Год назад
Clean and simple explanation. Great work
@NikashKumar
@NikashKumar 5 лет назад
Gaurav,, great work. Appreciate all your hardwork. It helps a lot.
@kiran-nambiar
@kiran-nambiar Год назад
I got sharing vs partitioning question in an interview. This image & whiteboard notes flashed in my head instantly, I did crack the interview but got better retention offer. Thanks man ! PS - I should've used pizza as an example starting with 0 index, maybe next time...
@AdityaDodda
@AdityaDodda 4 года назад
The first minute was Killer!
@alanclarke6096
@alanclarke6096 4 года назад
What is Database Sharding?: Effectively partitioning the pizza
@Surajkocool
@Surajkocool 4 года назад
Awesome explanation, expecting more form you. I am big fan your computer science, knowledge...
@thrilokkumarpallaki9821
@thrilokkumarpallaki9821 4 года назад
Awesome bro...while watching ur videos it feels like I came to movie
@abhijeetsoni1978
@abhijeetsoni1978 2 года назад
Clear & crisp, to help understand the core concept of Sharding😊. And no, I wasn't able to click Like & Subscribe at a time, so had to do it sequentially. 😁
@rajeshseptember09
@rajeshseptember09 7 месяцев назад
I literally lost a system design round by not saying "sharding" when asked for an optimization technique. Not that I didn't know of it, but I couldn't actively think during the interview. Yes, by just saying "sharding" and explaining sharding as a concept, one can make a big impression during the interview.
@rahulshetty3738
@rahulshetty3738 5 лет назад
Continue the great work!
@urunovtimes5816
@urunovtimes5816 3 года назад
Oh my goodness, keep going again so nice. Really awesome.
@MattDeLuco
@MattDeLuco 3 года назад
Really great explanation, thank you!
@ranzort
@ranzort 4 года назад
Going through some of the videos. I can definitely see they are more entertaining as they go.
@ranzort
@ranzort 4 года назад
Hitting like and subscribe button was pretty easy for me therefore sharding is ..?
@saurabhsharma7123
@saurabhsharma7123 5 лет назад
That interview clip with doppelganger was awesome!
@gkcs
@gkcs 5 лет назад
Hahahaha! He's almost as good as me 😋
@saurabhsharma7123
@saurabhsharma7123 5 лет назад
He can easily send shivers down the spine of an interviewee, that's for sure 😅
@sudhiriyer5118
@sudhiriyer5118 5 лет назад
Excellent work mate!
@asawari520
@asawari520 2 года назад
that interview part was toooo good. especially when you say , we are looking for somethig more serious , we have huge hugeeeeee data. :-)
@piyushkansal2647
@piyushkansal2647 Год назад
Hey Gaurav, would you say using hierarchical sharing is bettern than keeping the number of shards flexible using consistent hashing? It makes sense to me since the amount of data transfer that happens in consistent hashing when a server goes back down (Size of DB/N) is significant, albiet rare.
@aakashjain3498
@aakashjain3498 5 лет назад
I find your videos educating and entertaining as well.... Well done
@gkcs
@gkcs 5 лет назад
Thank you Aakash!
@pratikparikh8027
@pratikparikh8027 3 года назад
The ending statement, That was the best
@tryitnow5944
@tryitnow5944 5 лет назад
You are crazy bro . Excellent video
@sunilnarwaria7452
@sunilnarwaria7452 5 лет назад
Nice video. Great explanation!!
@nitinpandey7478
@nitinpandey7478 5 лет назад
Nice video , would love to see more videos on system design 😃
@KUNALSHARMA-rt9wv
@KUNALSHARMA-rt9wv 5 лет назад
Thankyou! I learnt something new today because of you.
@gkcs
@gkcs 5 лет назад
Congratulations! 😁
@manojmajumdar3815
@manojmajumdar3815 4 года назад
I am thinking about the 255 others who have disliked this video, now who on earth would dislike such a well explained video!!!
@krishnagubili1724
@krishnagubili1724 4 года назад
Very simply explained.
@ayushaggarwal7690
@ayushaggarwal7690 3 года назад
Beautifully explained, keep up the good work, hope to meet you one day in person.
@stud3671
@stud3671 5 лет назад
Ty bro for this amazing video ❤️❤️
@thecraftschool7761
@thecraftschool7761 Год назад
Hey Gaurav Your videos are really helpful, easy to understand and so much fun to watch 😂 Keep doing the good work 👍
@amitkumarkit2
@amitkumarkit2 3 года назад
thanx gaurav. great way of explanation.
@gauravsoni4423
@gauravsoni4423 5 лет назад
Great video... Lot of useful information. Although, I have one question, what if maintaining multiple servers costs too high, in that case, does database partitioning would be the right choice?
@christophersanders7483
@christophersanders7483 4 года назад
Thank you for this great video again. Better than many university professors.
@gkcs
@gkcs 4 года назад
Thank you!
@ryan-bo2xi
@ryan-bo2xi 4 года назад
Hi gaurav, how does replication effect sharding as you mentioned the point of consistency. If i have a M-S model where the write updates are propogated asynchronously to read slaves it is eventually consistent. Incase the updates are synchronous it's a performance hit because the master confirm's the commit only after the log replication and what if the slave I/O just got suspended for maintenance.
@ChandanKumar-wb9vs
@ChandanKumar-wb9vs 5 лет назад
Starting part of the video was cute and hilarious 😂. Keep up that thing.
@gkcs
@gkcs 5 лет назад
Hahaha thanks!
@MohammadbinIsmail
@MohammadbinIsmail 2 года назад
Went to Yogita"s videos ,came back here again , both are excellent masha allah
@vibhors
@vibhors 3 года назад
Bhaai toone toh rock kar diya, thank you , love you
@moulimallina7501
@moulimallina7501 4 года назад
You are an amazing person with amazing talent. Once you happened to come our clg NIT raipur, that was wonderful lectures. Please make videos frequently
@gkcs
@gkcs 4 года назад
Thanks 😁
@monikakrishn
@monikakrishn 2 года назад
oh god! the reaction when said "Sharding" by interviewee is humourous man :D
@AlfredDHull
@AlfredDHull 3 года назад
Great video! This was informative but also funny!
@Sickbeatsdd
@Sickbeatsdd 5 лет назад
I just love all your videos can you post faster those competitive programming videoss
@ralphmoran
@ralphmoran 4 года назад
Thanks so much for sharing your knowledge. I'm learning from you more than in other places. Please, keep the hard work.
@darod6098
@darod6098 5 лет назад
Excellent video! Some videos about the Map Reduce concept would be amazing!
@gkcs
@gkcs 5 лет назад
Thank you!
@mayukhdifferent
@mayukhdifferent 5 лет назад
great job done. Keep making more videos.👊
Далее
System Design: TINDER as a microservice architecture
36:41
Database Sharding and Partitioning
23:53
Просмотров 57 тыс.
Я СКУФ!
06:12
Просмотров 1,2 млн
Chips evolution !! 😔😔
00:23
Просмотров 9 млн
How do NoSQL databases work? Simply Explained!
7:38
What is CONSISTENT HASHING and Where is it used?
10:50
Просмотров 748 тыс.
What is a MESSAGE QUEUE and Where is it used?
9:59
Просмотров 935 тыс.
Data Consistency and Tradeoffs in Distributed Systems
25:42
Я СКУФ!
06:12
Просмотров 1,2 млн