This is literally what I needed today. Cramming this playlist hopefully offer pulls up, I never comment on posts but u are the goat bro the goat broski if I get this offer I will send u some only fans money
Hey Jordan! Thanks as always for these awesome videos! I was wondering if you could do a quick video about how to actually structure and talk about these on an interview. Is there a structure to it that you do (or have seen), such as laying out requirements first, then doing some considerations, then diving into the design? Is it really not one size fits all? Either way could be interesting to see what tips you might have around this. Keep it up, you're incredibly helpful!
Here ya go m8 ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-IY2EPjShgc4.htmlsi=Xw4uwvd4iDBbpp_w You can always just ask your Interviewer too. Hey is it ok if I start with x?
The assertion behind the need for a derived Pending Transactions cache - that reads will hurt write throughput due to row locking - is not necessarily true if you're using a DB with MVCC (like Spanner or Cockroach). So I question the need for the separate cache.
The other reason is that you then need to run a query on disk to figure out all of the pending transactions. I'd rather just have them all precached, but agreed that if you're using snapshot isolation locking is unnecessary for such a read
@@jordanhasnolife5163 could add a local covered index to speed up the query and ensure consistency, which would slow down writes a bit but per the original requirements that's not a problem. In any case pending transactions would be a great topic to deep dive in a real interview and discuss tradeoffs, so thanks for calling it out explicitly in your video
Thanks for the video Jordan. How is polling going to work? Is there a cron job or a scheduler running every x minutes in the flink? Moreover maintaining the doubly linkedlist in the order of create time stamp would be log(n) right, as each event reaching flink could be out of order, right?
Yeah something like that, or you could just say something like on a new event, if we haven't polled in x amount of time, poll again. I would think that events reaching Flink should be coming in order on a timestamp per partition, so we could always just do a flink node per partition to maintain that invariant.
Thanks for the great video! Question about web hooks. So the Payment system is listening for web hook callbacks and the polling mechanism is only triggered when a pending payment hasn’t received a callback in a specified amount of time? Is that the idea?
17:18 - should pending payments that are `not recognized` by stripe at poll time really just be deleted from the payments table? this case might require special processing since at this point, the payment has a local DB status of `pending` but stripe has not recognized this payment. What would be a possible solution for this inconsistency?
I don't really think there is any solution, network requests to stripe can always fail. Do we want to delete the event? Maybe not but then we may find ourselves doing a lot of polling after a while.
thanks Jordan I have been watching your sd video each week have two qq regarding your design 1. could we use snowflake algorithm to generate id as idompotence key ? 2. was Flink processing part of payment service code ? if so, for fault tolerance case where payment service was down how is it going to affect flink processing ? thanks
1) Not familiar with this method, feel free to send me a link to what it is 2) Not sure what you mean by this question. Flink is just getting data from our payment db, and occasionally polling stripe to see the status of it, it is independent of any synchronous operation to do with the payment service.
@@jordanhasnolife5163 thanks for replying 1) en.wikipedia.org/wiki/Snowflake_ID 2) let me rephrase my question a little bit where was the application code to generate idempotence key logic and save to payment as one microservice and flink process another microservice or they are all clustered as one service as in payment service. if both processes are treated as one service application code and payment service was down, flink process will also halt right
I don't really know how much there is to elaborate on here beyond what I've discussed in my concepts videos. I'd take a look at something like debezium.
Can you make a video on designing Spotify? Func Requirements: - Ability to play any song. super low latency while playing any song. - User can create playlist, share playlist. - Follow other playlist, artist, get notified for any song updates by artist or in a playlist.
At least off the cuff I'll say 1) CDNs and precaching when going down a playlist 2) Use a database 3) This feels like twitter You think there are any other unique pieces to it?
@@jordanhasnolife5163 how about live streaming & podcasts? I believe that this is similar to Zoom. But would like to know if there any optimizations which can be done?
I think Cassandra's "strong consistency" is probably quorum consistency. I'd look into spanner, cockroach, yugabyte, as it seems they lean towards using distributed consensus within a replication group.
The other video I’d like to see: A distributed system for generating unique IDs, akin to Twitter Snowflake. Also with the functional requirement of how to allow people to bring along their own IDs.
Thanks for the video! Why not just use zookeeper to give us a monotonically increasing u64 for the idempotency key? This way we are guaranteed to not have any conflict, also u64 should be enough till the end of times.
1) using a monotonically increasing sequence number implies that all writes must go through the same choke point (meaning you can't shard zookeeper, which is potentially fine if we really don't care about performance) 2) We basically do this anyways, as our payments db is basically using a consensus algorithm, making it effectively the same as zookeeper
Oh man I'll have to look into this one, you may be aging yourself by asking for an RSS feed and I may be aging myself by saying I've never used one lol
