The GeekNarrator is here to make you curious, excited and inspired about Technology and Software Engineering. You will see in depth technical discussions with actionable insights from the industry experts. These insights will help you get better at Software Engineering, as it comes from the real world engineering experience.
I saw a small video of tiered storage in StarTree channel but this video is something that is much needed for in-depth understanding of what’s going on. Kudos to Neha for explaining it so clearly.
why couldn't LiteFS just observe writes to .wal and propagate only those like you are doing at Turso? Also ... if I understood correctly the primary is location based so you'll have fast primary in Europe, as example, but writings from US will inevitably need to end up there first, right? I think this is also common for other DBs, but I just would like to be confirmed reads are always fast (once in each replica) but writes are also "primary location latency" based. Is this correct? Last, but not least, please don't talk trash on JS, it's pretty awesome after all :P
Everything wonderful especial the quest. But please do not include this annoying subtitles. When I will then I enable the once from RU-vid. You're can I not disable. And when I am trying to concentrate on the story, they really annoing
Thanks for watching and your feedback. This has been a feedback by many folks so I have removed the captions from the latest episodes. I hope this won’t be a problem going forward.
@@TheGeekNarratori disagree to him, the subtitle helps a lot for non-english speaker in understanding the podcast. If you will, at least, please insert it in the caption feature (not the auto-translate one), so we can still activate or deactivate the subtitle
What a Masterclass with the Master! Its a great summary of cassandra documentation and parts of it are covered in DDIA. Also, observe how the master crafts concise and clear explanations to the questions. great questions too Kaivalya - Loved it
What is the corresponding in-memory update for the log that is written? Log is also written to disk or is it in memory and then flushed to disk? What happens when log flushing to disk fails?
A Log is flushed to disk, yes. Databases like Postgres also support fsync mode which waits for the log be flushed to the disk, which adds a little performance impact (but nothing comparable to updating data pages and indexes directly). Not using fsync mode OR in simple terms, not waiting for the OS to flush the log to the disk may result in lost committed transactions. Typically a database dealing with concurrent transactions can write log entries to disk with a single fsync which is very efficient. So choose your tradeoff, but the thumb rule I use is, enable fsync by default to ensure maximum reliability and tweak if performance ever become a problem. Does that answer your question?
First time ever enjoyed listening to tech like a story felt living in that time... Fabulous! both of you.. The silly mistakes you covered were really spot on with that idea of taking baby steps I could feel the importance of it more now.. this is my take away
Yes its on the disk. Appending to log is cheaper because it is sequential and hence no random access. Flushing actual data to disk requires random access which is slower and requires tons of IO depending on what you are inserting/updating/deleting.
But log is then also prone to getting lost in case of server crashing since they are stored in in-memory before being flushed to disk . What’s the advantage of creating log then structure then ?
@akashgoyal2567 Typically if durability is important, you would fsync the log (Databases have config for that) which means the log is persisted to the disk. Since it is a sequential write it is way faster and more importantly when DBs have high concurrency one fsync call can be used to write 100s if not 1000s of transaction log to disk. That’s when it becomes really light weight as compared to updating data pages. In short log isn’t lost if you use the right configuration.
This was a really fantastic conversation. It was so interesting to hear so many of the details behind tiger beetle. And Joran is a very very good storyteller and the way he explained everything. Just made it a pleasure to listen to. This project is really exciting. Can't wait to see where and how it gets deployed across the world!
Very insightful! For me, the main selling point of Restate is durable execution. However, how Stephan and team hide complexity of execution, communication and persistence into a single platform is a very promising approach to building distributed systems of the new era.
Nice talk and Great initiative. Question about the write path where Broker writes on the page cache. So that means that if leader node fails, data in OS cache will be lost. Does that means that storage devices being used should have the power supplies that can at least let kernel flush all it relevant caches to devices.
This talk was really awesome esp the explanations he gave were very good. The acid transactions remind me a little bit of conditional writes from Dynamo DB but those here are much better because multi table! Great questions too. The diagrams also help
