System Design Interview: Design Dropbox or Google Drive w/ a Ex-Meta Staff Engineer 

"Have you done any System Design course ? How are you so good with this subject ?" - These were the word of my interviewer. I had a High Level + Low Level system design with a start-up recently. Surprisingly the question was to design a file sharing system such as Google Drive as described in this video with some additional features. I explained the HLD with the diagram as I had learned from the the concepts of this video. After the HLD was over, the interviewer told me that I have created a very robust & elegant system. He further said, he was so satisfied with the HLD, that he no longer wants to go into the LLD. Folks, these videos are the absolutely anything that you will ever require to ace a system design interview. Do remember to learn the fundamentals used in the system. A huge thanks to #Hello Interview for putting out the best content out there.

I'm so grateful for all of your videos. I've been practicing using the Hello Interview AI interviews, booked one mock with one of your interviewers, watched all the videos. The quality is so far beyond any other content out there, and I've successfully passed 5 system design interviews. Keep up the good content, your RU-vid channel deserves to blow up and your website too #wouldinvest

These are the best System Design videos I have found. Great framework for approaching problems, clear explanations, helpful diagrams. And I really appreciate the notes about how insight’s different seniority levels might approach specific parts

With my systems design interview coming up, I was looking forward to your video. It's great as always.

This channel has the best system design content on youtube. Keep up the good work.

Great stuff again! Not only good for interview but also for daily work 1) Clear and concise structure 2) Weigh trade off rigorously and explain the final decision clearly. Every single component is well though out with real world considerations

It's pure gold! specially the parts where you are stopping the debates abouts what db to choose or if the calculations are needed. The deep dives are the best part.. no one goes that deep and thats actually what matters in an interview

Hey Evan , thank you so much for providing the great content. I really live the way you organize and put content across the board. ❤

Wow, by far the best system design videos anywhere. I love how simple you make it, and the invaluable tips!

My mind is absolutely blown by how beautifully everything is explained. I love how you understand what would be possible questions/confusions from people watching and you address them by explaining pros and cons. Thank you so much for the content! Your walkthroughs are by far the most useful and interesting.

Hi Evan, this is my first time giving system design interviews. Really glad I found this channel to learn from. Most of my prior feedback from mocks and system design have been framework related for when I explain my design. This really helps with that and I think even at work, this is a really good approach to follow for. most things. Awesome content, thanks a lot!!!

Love the content and explanation. I watched hundreds of videos on system design from last 4-5 years and also have paid subscription from few. I don't have any doubt that, your channel can become premier system design platform in no time if you keep the content quality high ( just like last 3 videos). Next video, I'd recommend to talk about messaging platform like WhatsApp or FB messenger. There are so many videos on this topic but didn't find any which explain the details and really help in the interview.

one of the best system design preparation video I have seen online.

This is the best system design video, I have watched!! Specially the deep dives, You nailed it !! Looking forward to watch your videos.

Really good approach. I love how you build up to the full solution. It makes a lot of sense to me and helps me reason these complex systems as well

Will be the best system design interview channel for sure. Neat content. Keep up with the quality Man!

Thanks for the effort and videos. Easily one of the best in business for mocks and educational material.

Great content. Thank you for posting. One of the best system design video I have come across for this design.

I really appreciate your help in this topic. I'm learning a lot! Keep the videos coming!

This is really awesome, thank you. Please keep them coming!

i dont comment much, but for this kind of explanation i gotta give it u. Hands down the best explanation on youtube . pls continue making these kind of videos . This channel will blow up

I just finished reading the article on this lol. I’ll check out the video as well.

These are like gift from God. The best SD resources I've found in the last 3 years.

This is really good, irrespective of whether we are taking interview or not, any person looking at this level of explanation and detail would try to picture software differnetly. Thanks for making such videos, would love to see some more

Excellent sys design interviews - I like the informative tone and clear approach - thanks

👏Bravo, on point as always. Thanks Evan, keep up the good work man!

Thanks for posting these! Great interview as always! I am learning a lot from these interviews. I found it interesting that you jumped ahead in order for the non-functional requirements since you knew the large file upload requirement would impact the design enough that doing the other ones first was not beneficial since they would become irrelevant. Obviously, this comes with actual experience of working on the job. May I suggest doing a follow up that uses the final design from this interview and consider how it may change if you piled on a more advanced feature like syncing only a partial set of folders or sharing folders with other people.

I really like the 5-step structure, it's the best I've seen and it effectively helps me think through the designs in a methodical way.

Hi Evan, Thankyou so much for providing this type of content. plz make a system design video on payment system

this is pure gold thank you so much

great videos, thanks for uploading these. Easily the best content about system design interviews I've found. I would probably suggest to use merkle trees for the sync functionality, seems like a natual way to diff and sync large file systems

I love the content. It has been frustrating to watch some other system design videos where they just brush off over important details and act like everything is straightforward and easy, and just make 10s of services and never really explain the nitty-gritty details of how those things would work and IF they would actually work/be efficient etc. Thank you!

One more comment on this: comparing this to the written content on hellointerview, this one seems more round and well-thought (mainly regarding using S3 notif. on chunk upload completion, which wont work). Would be cool to see it reflected there on the platform! Good job

Great video for system design preparation. Methodology, the way he approached the question was good. 5 steps. Pretty good

the content is good, keep up the good work. expecting more SD videos in more details

Very easy understand keep continue new videos, thanks you so much

One of the best and detailed explanation. Thank you so much for this content. Please upload more such videos.

Thank you for your effort! Excellent content!! Love it!

Voted on your website for payment system! Banks love these

one can argue that "supporting 50gb upload file size" is a functional requirement (you placed it under non-functional requirement) - just a call out. great video!

Thanks great content. Probably the best.

Big fan of this channel and Evan. Please solve whatsapp/messenger type chat system next if you get some time.

Great content!

First - please continue making and sharing these videos, this is incredible. Very few high quality sources available out there and this is probably the best one in my eyes. Second - how realistic is it that the download and upload happen directly b/w client and S3? Are there security concerns with this approach that should be considered? For reference, there's a Dropbox engineer's talk where uploads go through an intermediate service - this does mean additional copies of the data meaning more memory / compute but seems more realistic. In general, for any design that has media upload (eg. newsfeed), would you recommend direct upload to S3?

Love your channel !

Amazing. Some Notes: DeepDive: Chunking CDNs Adaptive Polling with only updated chunks Compression.

Loved this resouce, thank you so much! But I just wanted to point out that in my interview I was asked about sharing files with other users and I feel like this design concentrated more on just syncing files across multiple devices. In the former, I think we can talk a little more about CDN/other approaches which were hand waved here.

Thank you very much for the videos. One small nit pick. DynamoDB supports a maximum of 400KB per item and if we are storing all the chunk metadata in the item, for a 50GB file with 5 MB chunk size, assuming we need 100Bytes per chunk metadata, our item size would be around 1MB.

I truly appreciate all the effort you've put into making these amazing videos. Please keep them coming. One insignificant (not important) nitpick. 50 GB @ 100Mbps = ~ 1hr 7min. I think you just forgot to convert the decimal to minutes. You have it correct in the write up, as in 1.11 hours (0.11 * 60 = 6.6 minutes).

excellent stuff !!

I have a product architecture interview coming up. I was really looking for some good product architecture/design examples and then came across this. This is very helpful because you talk about the client, user experience, malicious users and relate it to the design decisions made. Thank you! One question, for a product architecture interview - should we go into more details about the APIs like explicitly write out requests, response, failure/success codes or the amount of discussion you did on APis is enough for senior level? Can you also tell me what topics/ points would you add over the discussion in this video if this was asked in a product architecture design round. Thanks again!

Thank you.

Love the content! Thanks for putting the efforts! nit, fingerprint probably does not fit as a good id for chunk, as hash value can change over time when the content changes.

very good content

Thanks for the great content! Your video really help me understand the "flow" about a good system design interview (which I do feel very important for staff+ engineers to direct the interview!) a few general questions: 1. Is it beneficial to mention the selection of specific DB (like dynamo), and mention to use Kafka/Spark for microbatching snapshot update etc. after evaluating the QPS / replication etc.? or keep it at abstract level would give equivalent signals for interviewers? 2. what signals / thinking might make you feel it even beyond staff (just out of curiosity, as feeling it has already been very perfect from the staff requirement)?

Your videos are v. good but your mock interview is costly for certain markets (BFS can give you more return). Wish to get interviewed soon !

Latency for upload/download? Latency for listing files and preview is important, but for transferring bandwidth is more important.

For the delta sync bit, probably should go a bit deeper into rechunking for an existing file, to perform the delta sync.

Hey Evan, this content is fantastic, thank you! I have a question regarding your solution to chunking around the 39 minute mark When we ask S3 to fetch us a pre-signed URL, do we do that for all our chunks as well? Does this happen on initial request to upload the file (metadata)? The way the File Metadata entity schema is described, it looks like we have a top-level S3Link, but also chunk-level S3 links embedded in the file metadata, so the upload flow is a little unclear to me

1. How can we handle write conflicts when we have a folder which is supposed to be consistent across multiple devices? 2. Also when two devices are disconnected from the internet and if users updates some files how does the sync happens when they come back online and when both tries to write the changes at the same time at a same file path? I am not sure if these solutions work but I think 1. We can use a Redis lock for writes with TTL same as the timeout or a little more of the pre-signed url. If connection fails in between we can just resume the upload when connected back. But this might be a problem when a user is trying to upload big files with large timeout durations since other users might have to wait till the user uploading currently is done. 2. When the user comes back online we should probably first fetch all the changes that are executed on the device and raise conflicts with the user asking what action to perform(similar to git) and acquire lock to write if required.

Thanks for the videos. 47:45 I would like to know your opinion on push approach? By push approach I meant when the File service knows there is a change in a chunk, Sync service will let the client know. And, then the client will send a request to sync/download the chunk.

Please make a video around Designing a distributed login

Hey Evan, thank you so much for the great content. One question, would you recommend any resources on the Design Amazon interview question? I don't normally see it being discussed on Alex Xu's books, hello interview, grokking de SDI, etc... Would be awesome seeing out tackle it!

Some interviewers really like seeing estimation numbers earlier on before going into the higher level design despite what you generally suggest we should do. Can you may be also create a video where you do some relevant capacity estimations early on and later guide the higher level design? I would love to see if there is a good way to get some numbers in during "requirements" phase

Hi Evan, this is great stuff! I do have a question though regarding the 3rd API endpoint. Are you assuming that the changes were made on the remote rather than locally because it seems that the user would call the "changes" endpoint to get the changed files on the remote since a particular timestamp? It is my understanding that generally files would change locally and then you would need to sync those changes to the remote.

Hey Stefan, awesome video, congrats! I've got a quick question though. Around the 49:46 mark, you mention adding an "updatedAt" to a chunk at a specific id/fingerprint. If a chunk changes, its fingerprint/hash/checksum would change too, right? So that id wouldn't really match the changed chunk anymore, would it? Doesn't that mean the old chunk gets "invalidated" and a new chunk id appears? Sorry if I'm missing something obvious here.

Thank you for this video! It is very helpful. I do have a question though - lets say the same file is modified by 2 devices at the same time. How do we handle sync in that case?

had few questions and suggestion-> 1. When really large file is divided into chunks and upload in different s3 file path , will different pre signed urls be requested for each chunk of that large file ? will this logic be maintained in file service. If we are using trust and verify along with s3 multipart api , i guess in that case s3 would created different paths for each chunk which we also need to store in metadata , would s3 return actual file path of each chunk to client so that it can updated by file service in metadata db . 2. Also generally google drive doesn't allow to update same file , it will create a new version if a new upload is made to same path , so syncing of chunks for 50gb file that has been modified is probably not a valid usecase ?(thoughts) more valid usecase is addition of files to same folder/path .(thoughts) , which i think here would work by sync service using user_id , folder_id , timestamp , returning all metadata to client which is essentially added to folder/directory for user . 3. while responses by sync service or file service for downloads are being sent to client , i guess some pre processign would be needed to get pre signed urls from actual s3 path which is stored in ddb , which can be used by clients to donwloads . But thanks for the video its super helpful .

Hi Evan, Thanks for the detailed explanation! Very helpeful! At 40:50, you mentioned that S3 exposes multi-part upload API. Does that mean on client end we don't have to handle chunking and fingerprinting given that we use S3 multi-part? Thanks!

Maybe since data integrity was a non-functional requirement it would be worth mentioning some strategy for handling two clients modifying the same file concurrently - otherwise there's a high chance you end up with a corrupt file. Since you don't handle versioning, that pretty much necessitates locking.

Thanks for this amazing video! It was really informative. I had one question on Updating the Sync Service to look for any chunks that might have been updated. Once we make some edits locally (say the file size increases from 50 Mb to 70 Mb), won't all the chunk bytes shift/change which would mean we need to update all the chunks. Is my understanding correct?

Hi question about the chunk uploading, wouldn't uploading a big file result in huge amount of requests to update the metadata? For example just updating the status if chunk is 10mb for 50GB upload would be 5000 requests to update the chunk status. Would that have performance issue on the system when in scale?

If you split the files into chunks are you able to upload all chunks using the same pre-signed URL? Or do you need to generate a new URL for each chunk? How would you piece together the file from S3 when you download it through chunks?

You're vidoes are quite informative and interesting, thanks for creating those. I have a question though, the approach we take while API design, like let's go with the flow and come back and update, how is it looked down from Seniority prospective ? Won't it look -ve for Senior/Staff candidates as it might look like "detailed thinking" is lacking or candidate isn't clear about their own thought process ?

Thanks for the great content! One question though, if we use chunk fingerprint as an id, when the chunk change the fingerprint would be changed. How are we going to sync them?

45:45 I wonder how syncing would change if instead of folder status, it's for database writes with many writers

Great content. Can u tell me which draing tool u are using?

Question - what should the file id be? It can't be based on the file name, as names can change. It also couldn't be a hash of the whole file, as the file itself can obviously change. Amazing content, thank you!!

awesome, no more mouse over typing

Hi Evan, can you explain how we keeps tracking the order of chunks in file. I mean the case when the client has downloaded chunks from S3 DB and is about to assemble it

For the client to know how to stitch together chunks, doesn't it need to know the starting offset & length for each one?

How does an event bus system clean up unnecessary work like file A changed to file B but changed back to file A again, you don’t have to walk through the events to reconstruct the folder?

Do a sells chatbot with multiple options please

One question about the chunk status verification is that with NoSQL, we cannot directly access the information about the chunk. Is it okay for us to assume that we will always be parsing through the entire list of chunk metadata? Additionally, what would happen if one chunck gets big enough that new chunk has to be created? And maybe the new chunk is super super small. are there any issues coming out of a continuously updating file where chunks start to fragment? Thanks for such an amazing content! I'd just like a bit more detail of how the database based status updates would look like!

Is storing a mimetype necessary if we have the file extension anyway from the filename?

Do you suggest a different delivery framework for system design interviews which aren't necessarily "product"?

Great video. I did have a question -- I've implemented a file upload flow before so I knew conceptually about presigned URLs. But what if you didn't and you were interviewing for a Senior+ role? I'm not quite confident in myself to have been able to come up with that on the spot. Have candidates who have never done this flow been able to come up with that?

Thanks for a great content, especially the Deep Dive part, which generally people do not discuss about. I have one question around storing the chunks as list in the DB. For 50 GB file and 5 MB chunks there will be 10K chunks created. So the chunks list will have 10K entries. Now updating one chunk list column for every chuck status change could be quite challenging. Would it be better if we have a separate table for chunks instead. Also while you do the matching of chunks with the fingerprint, You need to check 10K entries from Local DB(with separate table and indexed) vs 10K entries in the chunk list (in single table column), where former is more efficient. Kindly let me know what are your thoughts on above points ?

one question- how do we merge chunks in order after downloading to local device? is it a good idea to keep some kind of sequence number for each chunk for a file?

It would be really helpful if you can share which whiteboard tool you are using here.

Add more videos pls

2.5x speed. Interview in 19 hours. Let's go.

Hey Evan, when can we expect the next video?

Guys, do anyone know why do we need a separate service to handle fileSync?

How do we search files?

can you please do video on news feed or messenger/whatsapp?

What application is used here as board?

Did we miss point on scalability completely? Since we are building Dropbox or Google Drive, it has millions of user and we would need to talk about replication and partitioning which are essential since we wont' be able to fit the data in a single database.

21st