Design Scalable News Feed System Similar to Instagram, Facebook & Twitter | System Design

Подписаться 12 тыс.

Просмотров 20 тыс.

50% 1

Design a scalable news feed system similar to feeds on Instagram, Facebook, and Twitter! We start with a simple working version and then build up to an optimized decoupled architecture while talking about the different tradeoffs that we are making.
System Design Playlist: • System Design Beginner...
00:00 Final Architecture Teaser
01:27 High-Level Requirements
02:20 Data Models
04:55 Creating A Post
06:50 Kafka CDC + Stream Processors
09:25 CDC Streams + Kafka
12:25 Getting User’s Feed
15:30 Problems with Computing Feed at Every Request
17:00 Pre-computing Feed with Redis Cache
20:45 Populating Feed Cache in Realtime
24:25 Populating Feed Cache Offline
25:55 Final Architecture Summary
29:52 Outro & Future Videos
#systemDesign #softwareArchitecture #interview
Visit me at: irtizahafiz.com
Reach me at: irtizahafiz9@gmail.com

Опубликовано:

4 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 56

@rajeshkishore7119 5 месяцев назад

So crisp and clear.

@veerendrashukla Год назад

Super explanation!

@sshian328 Год назад

good video, love it

@sun-ship 2 месяца назад

What a great channel!

@irtizahafiz 2 месяца назад

Thanks so much!

@bbi-edu 7 месяцев назад

Great . thank you

@vidhooakash7440 2 года назад

Thanks

@melk48111 5 месяцев назад

why do we need scheduled job to update cache for every user?

@dileeshaabilash5562 5 месяцев назад

🔥🔥

@amertia Год назад

One thing missing in the design is about what happens for influencers and celebrities where the Push mode would not make sense.

@vinaymiriyala4522 Год назад

Do you mind sharing how to account for this op

@HindiCartoonKahaaniyaTv Год назад

@@vinaymiriyala4522 use fanout on read model for celebrities. Just before rendering feed fetch all saved feeds along with celebs post which you follow

@mickeyp1291 3 месяца назад

3:05 Post_User does not need an ID, the primary key is as a weak entity

@kishanprajapati6170 9 месяцев назад

I have one doubt in designing data model. What would happen if We do create separate table POST_USER and include User_id in Post table.

@LuluHou Год назад

great design and clearly articulate! thanks a lot! i just wonder, why does stream processor needs to talk to feedservice? i thought feed service now just read results from redis cache. could you help clarify?

@irtizahafiz 8 месяцев назад

I should have been clearer. You are right, feed service directly reads from redis cache.

@jingjingcoming 14 дней назад

there is potential bottleneck on the post api to user-post table before the cdc Kafka. Maybe can partitioning or sharding this part

@jingjingcoming 14 дней назад

Or swap this part with nosql server

@koviroli 2 месяца назад

You made me to start thinking on a lot of things in my project. Thank you very much! A question to Irtiza or anyone: Step 1) So I fill the Feed cache with new post ids that belongs to a user, that should be displayed for user. Step 2) Probably I should remove the cached posts at a time... But when? When the user saw the post? Or should there be an expiration on each cached post?

@ujwalkumar2374 18 дней назад

1. Feels like it 2. Both seem valid but second makes more sense. Lets say there is a list of posts for some users but the users have not used the app for a while. It does make sense to remove the post after sometime

@dhiliph98 Год назад

Thanks for the amazing content! Rather than using a CDC, can we simply write a "post_created" event directly to Kafka from the post service? So the post service does 2 jobs. One, write to the database and two, write an event to Kafka.

@irtizahafiz Год назад

Yup! That works too. Totally depends on what kind of architecture you have.

@sergiim5601 Год назад

Hi, great content! Why do we need a Post_User table? We could have a UserId column in the Post table that would record an owner's ID ?

@irtizahafiz Год назад

Yeah you could do that too. But having a post_user table will let you store more fields about the relationship if needed.

@JoaoKunha Год назад

For pagination ! Lets assume you have 100 posts cached for each user. Would you consider another service to add more posts to this user cache on reaching last available posts ?

@irtizahafiz 8 месяцев назад

You can store all the IDs in your cache, and paginate there. Given you are only storing IDs, and not post details, you can add a ton there.

@juheelee568 Год назад

For the Scheduled Job, you said that you will iterate through all the users in your database and update the Feed Cache. For the Scheduled Job, if it updates the Feed for every single user in our system (let's say 5M), would you be adding 5M rows to the Feed Cache? My thought was that the Feed Cache would only store a percentage (lets say 20%) of daily users.

@irtizahafiz Год назад

Hi! You can do it both ways depending on what kind of infra you have for database and cache.

@zymasethecatalyst 2 года назад

👍🚀

@user-eq4oy6bk5p 2 года назад

Why do you need separate ID column for Friends and PostUser table when you can just use composite key (postID, userID - PostUser) which uniquely determines a row?

@irtizahafiz Год назад

I always prefer having an auto incrementing ID column for all my tables. It helps with JOINs in the future, if you are not considering all your use cases right now. And it's worth the performance tradeoff given the simplicity of that column.

@narisetiuday2906 4 месяца назад

Awesome videos. What is the name of the tool that you used for the diagrams?

@irtizahafiz 4 месяца назад

Miro :)

@JH-zd6en 3 месяца назад

I have a question. Let's say if A and B are friends. When A creates a post, it writes to the redis cache on server1 to build the feed for friend B. However, friend B gets routed to server2, which means it won't have access to this cache. In other words, if A has 100 friends, and when A creates a post, how do we update the feed cache for these 100 friends? They are in different servers and their cache will not be in server1.

@nochecku6834 2 месяца назад

what if the redis cache does not have the user id for whom the feed is getting loaded, then the feed service needs to talk to post service? Or will you return no feed for them, which is a poor experience?

@irtizahafiz 2 месяца назад

Yes. If you run into cache miss, you should always consult the DB with the "same logic".

@firezdog 3 месяца назад

What happens for the posted data when it fails moderation but is still being implemented processes by other workers / has been written into storage

@irtizahafiz 2 месяца назад

Depending on your tolerance level, you can start processing after moderation, or go ahead and delete records / evict caches after something is flagged as inappropriate by the moderation system.

@nehapurohit684 9 месяцев назад

Is Moderation service updating Post User table if any post found out to be malicious

@irtizahafiz 8 месяцев назад

Yes that would be the idea. But this design is at a very high level, so I might have not mentioned that explicitly.

@JardaniJovonovich192 Год назад

1.) Why does feedStreamProcessor need to talk to Post service? 2.) How does Feed Service fetch the information of a user whose entry isn't present in cache at all? It should be talking to Friend service, Ranking service and then fetch the relevant details and then push it to cache and return the response, right?

@irtizahafiz Год назад

1. The stream processor will need to pull details of the post. It usually deals with IDs only. 2. Yes, that's correct.

@flashliqu 2 года назад

would you upload the lecture slide

@irtizahafiz 2 года назад

Hi! Unfortunately, I don't have the slides for this one. For most of the other ones I started uploading PDFs or slides. Hope that helps!

@d33bo67 5 месяцев назад

if the posts get stored in the CDC before it hits the Modification Stream Processor, then hits the Feed Stream Processor, how is it going to prevent offending messages from being posted?

@irtizahafiz 3 месяца назад

That's a great point!

@nagarajutammineni6736 6 месяцев назад

Missing one context, Why feed stream processor interacts with feed service. You were saying "The feed of users". May I know what it is?

@irtizahafiz 3 месяца назад

The feed is a precomputed set of posts that the user sees on their home/feed page.

@meditationdanny701 2 года назад

I don't think storing age just as interger will make sense rather storing dob and parsing that to obtain age at run time is the approach

@irtizahafiz 2 года назад

Yup! I agree. The purpose of the video was to design the whole system, not dive deeper into individual data model. So I decided to keep things simple : )

@meditationdanny701 2 года назад

@@irtizahafiz gotcha

@mayankmaheshwari2544 8 месяцев назад

api gateways knows which service to hit not load balancer

@stevehyuga9216 9 месяцев назад

This is great but I don't think it is efficient to create feeds for users that you don't know if they will use the service at all. On Twitter or other social networks there must be millions of inactive users, that maybe are following Elon Musk, so everytime Elon twits you are doing a lot of unnecessary work for those millions of inactive users. Besides that, I'd like to have more details about the Raking service. On the first example I don't see efficient to get all the post in order to send them to the Ranking service.

@sarang5906 7 месяцев назад

Agreed. It is a trade-off to be made in terms of the freshness of the feed. So one solution could be to refresh the feed only if the user visits and refreshes their Newsfeed Page.

@davidmataviejo3313 5 месяцев назад

It is the price you pay for having the users feed already computed. Users will not use it if they need to wait 1 minute to be ready. And this approach only works for regular users. For users with huge amount of followers, don't follow the same approach

@dontdoit6986 2 месяца назад

Twitter actually does create feeds for every user with every new post. It’s counter-intuitive, but they do own their servers for performing the compute.