Designing INSTAGRAM: System Design of News Feed

Подписаться 584 тыс.

Просмотров 620 тыс.

50% 1

This video is about designing Instagram for a system design interview. We are expected to design the server side so that all 4 mentioned features can be incorporated. System Design interviews evaluate a candidate's computer science fundamentals, architecture principles, etc.
We start by building an entity relationship diagram. Here, the columns are defined as per the requirements. We then move to designing a microservice architecture. Each microservice represents a single component of the system.
The newsfeed is built step by step with concepts like load balancing and caching as prerequisites. This completes the system design for Instagram.
Looking to ace your next interview? Try this System Design video course! 🔥
interviewready.io
With video lectures, architecture diagrams, capacity planning, API contracts, and evaluation tests. It's a complete package!
System Design playlist:
• System Design BASICS: ...
Some of the concepts discussed here:
Tinder video: • System Design: TINDER ...
Database reference: www.db-book.com/db7/
Load Balancer: • What is LOAD BALANCING...
Consistent Hashing: • What is CONSISTENT HAS...
Publish-Subscribe: • What is the Publisher ...
Chapters
00:00 Introduction
00:10 Feature Selection
03:15 DB Schema
09:45 User Followers and Following
12:02 System Design
21:32 Celebrity post fanout
#SystemDesign #DesignInstagram #DesignInterview
Become a channel member!
/ @gkcs
You can follow me on:
LinkedIn: / gaurav-sen-56b6a941

Опубликовано:

30 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 589

@elliemay1748 4 года назад

I interviewed for PayPal a few weeks ago, and this was the exact systems design question they asked. They said to me, “Do a system design for Instagram”. I smiled, because I had just watched this video a few days prior, and so I knew exactly how to answer. Thank you for this video, you helped me get a job, for real :)

@gkcs 4 года назад

Congratulations!

@sonushete4431 3 года назад

wow awesome Elli May got job in Paypal 😁

@brandonzheng1092 3 года назад

What's there to be proud of when you've seen the answer to an interview question beforehand?

@ashishsharma-nz5pq 3 года назад

@@brandonzheng1092 so one should not feel proud anyways, cuz he/she had studied that in books b4... lame perception

@pratik3106 3 года назад

@@brandonzheng1092 I sense happiness rather than pride, none of that this person said implies proudness, and even if so, why not? I'd be proud.

@Tuanvu-tb1mh 4 года назад

Hi Sen, for the database design I think you should go from Logical ERD first then derive Physical Tables from there, it is more natural approach

@samanrajaei8129 3 года назад

Damn this kid is good. Better than most of the "veteran" system architects I've worked with.

@dijoxx 3 года назад

You must have worked with some really crappy architects if that is indeed the case.

@apurvsawant5703 3 года назад

All the important concepts are explained very simply and this is what makes this video amazing.

@AtomicAkshay 3 года назад

Great video man, really appreciate the fact that you've been posting such indetail conceptual content for free.

@HarkiratSaluja 4 года назад

Probably the first video of the system design I have seen. Being a front end developer, I had a fair idea about the things but the way you explained just wow

@theakatsuki2113 2 года назад

Wow his content is really at next level I love this guy and respect his efforts and the amount of hard work he puts in each and every video

@ChitranshuVashishth 5 лет назад

I'm not even a CSE student. But watching your videos, Gaurav, actually intrigues a lot and motivates me to actually learn more about programming and design some my own scalable system one day. Thanks Gaurav.

@blasttrash 5 лет назад

Try coding train channel. You can watch most of his videos and have fun. Its like watching a movie and he does real coding. For example watch this video, even if you dont know programming, you will still understand it and its so much fun. Coding the snake game ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-AaGK-fj-BAM.html

@HemantSharma-fw2gx 5 лет назад

Thanks for explaining the practical use of all we study in our syllabus..Your videos are superb!

@gkcs 5 лет назад

Glad to hear that!

@johnfrades 3 года назад

Awesome man, im glad to see your channel, subscribed immediately! Very helpful!

@aditikhedkar8514 5 лет назад

I always wait for your videos!very good content Thankyou!! :)

@xReisk 4 года назад

As a backend developer who strugles to do projects due to thinking to much and taking a lot of time to make dumb things... I think I have found one awesome channel for me. Thanks for your videos man!

@gkcs 4 года назад

😁

@ci7alex1 5 лет назад

Explains a lot why there are so many well paid people behind each successful online service, so complex, wow

@augmentos 3 года назад

Would love to see a system design of notifications (activity feed) in twitter/IG etc. Aggregate etc.

@yog2915 4 года назад

Fantastic work it is because of people like you skills of general masses are also rising

@nitinjoshi4636 5 лет назад

Awesome content.. what I am looking for always get from your videos. Keep it up.

@devarajt6965 5 лет назад

hi Bro.. Actually the way you explained the stuff is very simple and clear.. Thanks for your time for making such videos..

@gkcs 5 лет назад

Thank you!

@ankitgoyal8556 4 года назад

Loved this playlist, thank you brother

@alexkuzmichev9051 4 года назад

I'm so glad I found your channel. Keep up the good work! Nice videos:)

@gkcs 4 года назад

Thanks 😁

@bmw1553 4 года назад

Gaurav, first of all, thank you for a fantastic, simple and clear explanation. Second of all, I can imagine the work went in to put this video, it must be humongous task in preparing the right content, taping, editing, etc. Great work!

@gkcs 4 года назад

Thank you!

@antarikshsrivastava2562 4 года назад

Absolutely loved your explanation Gaurav. Thank you :)

@amlanroy5274 2 года назад

Well, I wasn't even looking for this. Just got a random recommendation and now I'm watching this with full focus at 3AM. Never thought this day would come😂 Awesome video.

@AlphaWatt 2 года назад

Thank you very much for this. Excellent explanation through such a complicated topic. Really helped me think through a follower service I have been struggling to commit to.

@dbtechprojects2392 3 года назад

Thanks for posting this, the part on how to handle the news feed helped me out a lot, originally I could only think of the first method which the administrative tasks are way too high, precomputing the news feed is an option I didn't even think about. thanks :)!

@gkcs 3 года назад

Glad it helped 😁

@bianbian621 4 года назад

great video! the animation part is awesome. I like all your system design videos.

@gkcs 4 года назад

Thank you 😁

@harkirat1 5 лет назад

Great Explanation. Clear and concise.

@gkcs 5 лет назад

😁

@nayankhanna2367 3 года назад

Been addicted to this channel recently and binge watching it even though I have my exams ongoing xD. Man you are the best. (saying this from my experience of having watched more than 100 "Real" Coding RU-vidrs) My Systems Design knowledge is growing leaps and bounds by watching these and I plan to implement these good designs just after my tests are over. I have worked on several Applications as a backend developer, and always stressed heavily on scalability, flexibility, ACID properties. But, your channel has taught me a lot more good techniques and design concepts.

@gkcs 3 года назад

Thank you 😁

@MichaelAMomo 2 года назад

Seriously man, It is a great help

@anotherdigitalnomad9429 4 года назад

This was such a good video. Thanks!

@harisridhar1668 3 года назад

13:30 Hi Gaurav - thanks for pointing out the need for a load balancer with the snapshot technique stored onto Gateway for network routing when we horizontally scale the server-side. But why is communicating with the load balancer inefficient? Is this to avoid constant network calls ( which are slow ) and to utilize the SS, which can be stored into memory-side on the Gateway application?

@code_report 5 лет назад

Great video!

@gkcs 5 лет назад

When code_report says something, you better believe it 😎

@sankalparora9374 Год назад

Amazing stuff - not only informative - but interesting! Thanks!

@ZeroInfinityCoder 5 лет назад

@Gaurav i think it should be "postId" in place of "activityId" in "comment table" when you were explaining feature no. 2 as let say we want to find all the comments for a particular post , then we will look into comment table for column postId. correct me if i am wrong

@A.n.a.n.d.k.r. 4 года назад

I didn't knew the dbms subject was so much exciting....

@Yan-rv8mi 3 года назад

7:16 I think we do need a "type" on the Activity table. For example, suppose there's a postID being "123" and a commentID also being "123". Since both postID and commentID can be interpreted as activityID on Activity table, if there a row on activity table with activityID being "123" we don't know it's for the post or for the comment, unless we have a column "type" to distinguish between them.

@jellyrabbits375 2 года назад

True in this case, it depends on your system though. In case where the id is a UUID, then there won't be a case where postId and commetId would be equal.

@praveen3123 5 лет назад

Hey Gaurav great videos bro.. Every software engineer should know system designs to build scalable, robust applications.. keep rocking!

@gkcs 5 лет назад

Thank you!

@praveen3123 5 лет назад

@@gkcs by the way now you're a software engineer with at least 3- 4 years of experience by now. Do you still practice algorithms? I'm in this dilemma whether to practice or take it light

@gauthamhonnavara 5 лет назад

@@praveen3123 Never stop learning !

@aakashjapi4394 4 года назад

Great video! That said, in your descriptions of the database schema, you should mention hotspotting as a justification for certain decisions as well. Namely, a very good reason to not add a "likes" column to posts is that it creates a lot of contention on rows in a single table, especially because single posts can get hundreds of thousands of likes. You arrived at the same conclusion - building tables that allows for writes to avoid contention and thus reads to be aggregations (which can then utilize caching) - but I think focusing on the larger problem of hotspotting motivates your design decisions better.

@rujotheone 4 года назад

Noob question, please what is hotspotting

@osmanbaskaya7400 Год назад

@@rujotheone some records are getting queried more than others. the specific instance that contains the record will be much busier than the rest of the system. you're not balancing the load ideally uniformly.

@yuxiongzhu4249 5 лет назад

Hi Gaurav, thanks for this great post. You look so young, how could you be so knowledgeable?

@vikaspizza 3 года назад

He is actually 45 years old. He designed a system that removes aging signs from his youtube uploads...

@prakhargurawa 5 лет назад

Awesome explanation ..your videos are really helping ..keep it up bro :) :)

@shubhammehta319 3 года назад

Hi Gaurav, I love the way you explain things. This video actually sums up all the major components, including the DB structures and High-level architecture. I have one question regarding the design, which is more on low-level design, it would be great if you can create a video on that. Q. If I need to design the data storage in-memory, which data structure we should use to store the posts. likes, follower data, such that we can fulfill the given features efficiently. Thanks,

@emmanuelevbuomwan2665 4 года назад

Concise, at the same time; broad and easy to understand.

@gkcs 4 года назад

Thanks 😁

@user-mt6kv1dw7m 3 года назад

Nice video! I have one question, though: If we precompute the feed, what happens when a user scrolls past the 20 posts we have cached? Then we would have to compute the rest of the feed at the request of the user, which would be inefficient, right? Sorry if I didn't understand that correctly.

@pankilpanchal1996 5 лет назад

Just a test message weather you reads it or not. BTW very good system design playlist.

@siddharthbhola4231 4 года назад

@GauravSen: Excellent video! Please help me with a question: Can you please explain the reasoning/thought-process behind choosing a Relational database for Users and Feed Schema?? What factors do you consider when taking such a decision? Thanks and Regards

@PramodShetty 5 лет назад

Since the following table(user Id, follower id, timestamp) can grow very big, sharding or indexing would be required. On which column should that be done. If I index one of the this id columns, one of the service, getFollowing or getFollower will be affected badly.

@shradheytripathi7564 5 лет назад

Loved the explanation :) Nice work.

@gkcs 5 лет назад

Thank you!

@a2zlearninghub67 5 лет назад

Hey gourav sir😊🙏 Nice overview and well explained. You r really great person who share our personal experience. 👍

@gkcs 5 лет назад

Thanks Sandip!

@NikPnchl7 3 года назад

Will we store the post meta-data in a relational db ? Would that quickly fall apart given the scale of Insta ? Or would a nosql like Cassandra be the way to go with the tables you've described ? What do you think ?

@chathurabuddi 3 года назад

What a wonderful channel!!! just subscribed

@JaswinderSingh-uw2hf 2 года назад

Regarding Hybrid approach : practically User1 follows the ordinary user and celebrity as well. now when post done by ordinary user it will push to user1 but when post by celebrity, system/client has to pull. now how client know when it has to pull ? @gaurav sen sir, can you please explain. or correct me if I misunderstood something...

@pratiksinha5737 4 года назад

@gaurav sen Great video......it literally gave me an insight on how to use the theoritical knowledge we have gained as a CS engineer in real Practical Designing

@hellenruthes2056 2 года назад

I love your videos :) Thanks for sharing!

@tharinda 2 года назад

Much appreciated. Love it ❤🔥

@vishalarora6068 4 года назад

what you said is load balancer is work of service discovery or metadata service like zookeeper/consul/etcd and what you described as gateway is the work of load balancer or reverse proxy. Looks like you haven't used these systems practically(I am not blaming) and trying to inform others based on(your interpretation of) what you read online.

@zshanz 4 года назад

good point.

@adamberry7536 4 года назад

Can I ask a question about your comment. I totally agree what he's describing as a load balancer is actually more like Zookeeper. But I'm confused about the gateway comment. My understanding, which I may be wrong about, is that a gateway will handle authentication, authorization, and then route the incoming HTTP request to one or more services to accomplish the task at hand depending on the configuration. So yes, it sort of acts as a reverse proxy with the addition of authentication logic and possibility of making synchronous service calls. But I don't see how the gateway is a load balancer. It doesn't distribute API calls based on load. It distributed then based on function. If you wanted load balancer between a service and the gateway or a service and another service you would still need a load balancer. Is this correct?

@ivailotenevv 4 года назад

Awesome, keep up the good work :)

@rajanlagah4529 2 года назад

at 20:20 Why storing feed in cache as LRU ? I mean if feed is used it should be deleted or replaced by new one right ?

@pman-codes 4 года назад

Hi Gaurav, Greetings. I love your work, I am a subscriber and a frequent liker. However, I find an implicit assumption in your system which considers the Instagram mobile app as thin clients. The process of storing the posts in the cache in the server would result in an unscalable system. I believe the posts are cached in the user's app memory(cache and physical storage), considering that these apps have a considerable chunk of internal storage used. An added proof for this would be if you try to open up Instagram in offline mode you can still see past posts and a toast message which says "couldn't refresh feed". I would like to have the cache on the user's system and then an identifier that is stored in a place where you are storing the cache of the post on the server. (considering the news feed functionality. This can be applied for other uses too. ) Thanks.

@gkcs 4 года назад

This is a very good point. Thanks for posting 😁

@vikask7246 4 года назад

@@gkcs can you explain system design for telegram

@dustindiaz 3 года назад

@@gkcs the posts ARE stored in application cache... but it doesn't invalidate the fact that mobile apps aren't still thin clients. a user can delete an app, or visit from a third party integration (not built by instagram) -- in which case these timeline feeds are still stored in horizontal caches. you wouldn't believe the amount of money instagram/twitter/etc spend on memcache to make this happen.

@adminwadidaw576 3 года назад

@@dustindiaz so which way do i have to follow? Do i need to cache posts in client side? I confuse in like sectiob , whenever user click like, should the client side make a request?

@dustindiaz 3 года назад

admin wadidaw caching on the client is helpful when revisiting an application. This way a user can be presented with information immediately. Caching on the server, on the other hand, is necessary for large scale services to deliver things like timelines since raw sql queries based on this system design would cause the system to fall over with just a decent amount of traffic

@anuragagnihotri5238 2 года назад

@Gaurav at 20:55 You said regular polling/http requests (I assume would be using some connection pool, not establishing tcp connection for each poll request) from clients/cellPhones asking for updates from server will have bandwidth/battery issues. But even when we go with LongPolling/Websockets, its actually keeping either connection always(Wbsockets) open/connected or maintain connection for longer period(Long Polling) between client and server for realtime data transfer but this will also have battery and bandwidth issues right? So mainly how do we measure Pros/Cons for this specific Instagram case between Long Polling vs WebSockets?

@sahajarora2162 4 года назад

Hi Gaurav! Very thankful to you for sharing your knowledge with the rest of the world! I have 3 questions about the GATEWAY: 1) Is it a Micro-Service? If not, what exactly is it (i.e what does it contain)? 2) It seems like a single point of failure, looking at the diagram. 3) If we have multiple instances of a Gateway, then would the Load balancers be needed in between Client and the Gateway Service ?

@adityatiwari1022 4 года назад

Hi Man , Can you tell me how all applications handle delay in connecting and fetching real time data from external systems ? Ps: I know its a little off topic. Thanks

@darkwoodmovies 8 месяцев назад

Great video, ty. I'm building an app that does something tangential to this, really helpful for real-world work!

@abhaysoni8631 3 года назад

@Gaurav Sen , hello, thanks for the video, my question is why did you go for a table based db here , any specific reason, and i was thinking for maintaining post comment wont the subset pattern of mongodb be more better option here.

@abhishekgupta4570 5 лет назад

lot of learning with video and bro one request can u make video on your uber interview about question asked roundwise and that HR round which was pretty tough as you mentioned in video (Got job in uber).

@gkcs 5 лет назад

Thanks Abhishek! I won't be mentioning the questions asked, because we aren't allowed to. "Got hired" will turn to "Got fired". 😝 You can go through the content on the channel, it's more extensive than an interview set 😁

@AlgoHacks 5 лет назад

Awesomely explained 😊 I guess it's just not that simple as it seems after your explanation. Great work is being done behind the scene. Thanks for all the awesome videos.

@gkcs 5 лет назад

Thanks Ashutosh!

@krutikpatel906 2 года назад

Hi Gaurav, thank you for amazing content. Can you please share your thoughts on why you chose SQL database for all these data instead of NoSQL? Since the volume is high and eventual consistency seems to be ok, can we use NoSQL database for this kind of data? Thanks

@freedom1225jjy 4 года назад

also would be nice to have more videos on ER diagram ! thank you！

@vikashdhanabal8984 5 лет назад

hi Gaurav bro, it was amazing .waiting for more such videos

@gkcs 5 лет назад

Thanks!

@anastasianaumko923 Год назад

Very elaborate, thank you! 🤩

@raghav4296 5 лет назад

Awesome video series on this channel 💚. One request Gaurav- Could you please share insights on how the video&audio based systems are designed,built and the kind of algorithms/libraries that go into transcoding on scale, as they are computationally intensive tasks. Thanks.

@gkcs 5 лет назад

Thank you! I'll get to designing Netflix/RU-vid in a while. That will be fun! 😁

@AbhishekGupta-kv9ne 2 года назад

@Gaurav : For sending notifications to user why should we use Long polling or Web sockets instead of SSEs (Server side events).

@erichaymer5130 4 года назад

Fabulous video. Looking forward to more video. Can you elaborate on empirically optimize a ranked feed?

@raveenamewani7963 4 года назад

Hi Gaurav, I really like your videos they are very clear and to the point. I would like you to share in one of your videos how the Amazon Market Place Design will look and work like

@anshsharma7510 Год назад

Hey For the followers table we will surely have a lot of entries like for instagram it will be in hundreds of millions. Now to just see someone's followers should we query the whole table of that much records?? It doesn't sound effective to me

@kallyansingha3772 4 года назад

What to do with Like reaction(love,anger etc)?how to implement that ?

@gautamtyagi8846 3 года назад

thanks, really helpful explanation.

@user-cd2sg2sr1r 29 дней назад

@Gaurav , when a user post something so to immediately notify their corresponding followers ,in these scenario can't we use any Message brokers ?

@anirudhgupta2087 4 года назад

Hi gaurav, why you chose RDMS in this design. I couldn't think of reasons I should provide while choosing sql in newsfeed service. Could you pls help me with this.

@ayushiajit3088 2 года назад

Hi Gaurav, First of all excellent work on the videos :) I have a doubt on the DB selection, so basically what i am understanding is when we need to store information about user we may use Mysql cause of strong relationships etc but since the content [activity] of the user is kind of unstructured wouldn't it be better to use NoSql? by unstructured i mean, we may or may not have caption, may or may not have images, instead can have videos, or comments in that case can be recursively long..Please correct me if im not going in the right direction! Once again awesome work :)

@apoorvbedmutha457 Год назад

imo it'll be better to have a combination of noSQL and RDBMS for example tables which need to be regularly updated such as no. of likes must be kept in a noSQL DB whereas things like content of a post which are not changed so frequently are better to be stored in RDBMS

@alicebobson2868 4 года назад

this was very useful, thanks

@vibhorgupta5862 5 лет назад

Great video ! Thanks a lot for these videos, have been very helpful for me. If you get time can you also publish a video on UBER's design.

@tvnathreviews 3 года назад

NoSQL db would be better for this right?

@charan775 Год назад

how would this pre computed feed work when user keeps on scrolling the feed? do we have multiple queues of pre computed feed?

@purusottam-tb1sw 4 года назад

Great explanation man.

@practiseperfect6798 3 года назад

Hi Gaurav , Just few questions, When a user logged-in on Instagram, sometimes we see new post icon appearing right at the top and as soon as we scroll down, we get the new post. So the question is "does Instagram use pull mechanism to get notification from the account (the one which user is following)?" or Instagram sends push notification to end user? and which kind of queue mechanism is used largely for millions of push notifications in real world? Would like to hear from you , if you could clarify these questions please.

@RaviKumar-jg1hc 5 лет назад

It's amazing... please keep it going...

@gkcs 5 лет назад

On it! 😁

@RaviKumar-jg1hc 5 лет назад

Yes, hahaha...

@bhagyasripottumuthu2215 Год назад

We are going for push approach to pre compute the news feed. In case of celebrities we are going with pull model. But I don’t really understand how these two can happen together. Let’s say a user logged in and now he gets the feed from the pre generated db . Some friends posts something, his news feed will get updated immediately (push) . I am assuming this happens continuously . So celebrity posts , he won’t get an update right ? Or is it something like news feed gets calculated every 30 seconds irrespective of any posted from friend/celebrity .?

@user-bx5zi1mv5t 4 года назад

Hi Gaurav, can you explain the purpose of ActivityId? At first, I thought it was what linked a particular Like to either a Comment or Post (meaning it contains the ID of a post or comment) but seeing that the Comment table has an ActivityId column, I assume that's wrong. Do you generate an ActivityId every time you create a new Post or Comment for the sake of having an ActivityId for the Activity table? If so, why not just use the original ID of the Post or Comment in place of the ActivityId?

@ajayguru2116 4 года назад

Amazing content!

@rishabhsoni3050 Месяц назад

One doubt I had, if we keep track for all like in activity table. For example if a celebrity post a photo and getting millions of like. We are adding those into activity table. Will that not make our db more write heavy beacuse in a second there might be thousands of like. Am I thinking in correct way? Is there might be some other approaches for like count. If anyone could provide insights on that.

@AdventureTrekRide 5 лет назад

Awesome video. you are doing just great :)

@gkcs 5 лет назад

Thanks!

@xinxinzheng9324 2 года назад

I have a stupid question.... is it a good practice to design multiple tables in NoSQL DB? or we should wrap eveything into one document? Thanks!

@kavyajain8889 3 года назад

Is it better to use firebase RTD to push real-time updates instead of using web sockets?

@amitagnihotri30 5 лет назад

From Designing Tinder to Instagram, in a very short time :D

@gkcs 5 лет назад

Hahaha, just 6 months 😉

@AseshShrestha 2 года назад

Love you system design videos. Love from Nepal 👍

@MsBonesoul 2 года назад

How to solve the pull base on load model when the user followed many celes? I have got huge latency for that kind of user

@MOhan-ur4ei 5 лет назад

Your English is awesome bro!!

@zhong8842 4 года назад

The design before 11:19 seems mostly logical and doesn't seem to require too much knowledge on cs infra.

@GilP-BM 4 года назад

Great video man 💯

@jksharma7 2 года назад

Thank you sir.... a Great Teacher.

@suchismitagoswami5609 Год назад

Great explanation as always. However, I have a little doubt. Instead of sending notifications from Post Service to the user feed service directly, what iff we introduce an event bridge in between and different services will communicate with each other based on the event produced by other services?