Elixir's pattern matching and guards are a perfect example of such assertions built into the language philosophy. And it doesn't have that problem of Go with assertion because servers don't crash at an exception, only single, isolated processes do.
If there is a limit to the loop then that will fix some issues and open a new path that must also be thought of. Its a good and interesting idea, but its also not without its new potential for issues.
Yeah, sorry about that, that's the first time we tried to have two people on the stream at the same time, will try to do something about that. It took like 10 episodes to get _my_ audio in order :D
Yes, we do have a comptime-assertion about the endiannes. All TigerBeetle targets are little endian, and trying to build for big endian would fail with an explicit compile-time error
@@tigerbeetledb nice, i do the same in JVM though i retain an option to completely eliminate them at compile time but performance cost is so low and failing fast with better error messages is so nice that i leave them in too. This project makes me want to seriously dive into Zig.
@@tigerbeetledb i know that state machine of the db is designed to be very decoupled so one would in theory be able to reuse all the foundational things of this db to make something completely different (reuse high availability, zero serialization etc)... but realistically how easy it is to do something like that? Have you tried to make anything else by reusing the benefits of the modules surrounding the state machine?
@@krellin Pretty easy to do. We actually already have two different state machines implemented: the double-entry accounting, the real one, and the simpler "testing" machine geared at uncovering consensus bugs quickly: github.com/tigerbeetle/tigerbeetle/blob/main/src/testing/state_machine.zig. This is totally designed to be extensible, though we are not currently providing any kind of stability guarantees around the interface.
This is really cool and the explanation is really appreciated. It's impressive how "easy" a compaction in an LSM-tree seems conceptually, but how many gotchas you can find when attempting to implement it
This one of the few channels I have *notifications on* for. 🔔 TigerBeetle is _sick_ !! Your whole team is _awesome_ !! 😍 So stoked to _finally_ be using this in a real project! 🎉 Keep up the great work. 🥷
For two reasons: * TCP checksum is actually pretty weak, there's a non-negligible probability that some corruption slips in. Its not a cryphographically-strong checksum. Plus, there's also a probability of outright bugs in TCP implementations on any machine on the path between sender and receiver. * we don't want to be bound to TCP semantics specifically. The code uses a very weak networking model, where we don't really care _how_ we got the message. So, we'll be able to switch to UDP or any other protocol easily!
Cant wait to watch these next few! I Did watch a few of the old streams/videos going over the LSM forest, but I am looking forward to getting this deep dive. I really appreciate all the work you have done on this. A guided code walkthrough TigerBeetles codebase is amazing.
Yes.... i remember back in the 90's oracle tried this system of "direct IO", Blew lots of trumpets..... and announced it HAS to be better & faster Because ....'insert reasoning here" Well you know what..... it was complete bullshit, becasue they made lots of assumptions and very little real testing. Because even you THINK you are writing directly to the "disk" YOU ARE NOT.... you are Writing to a BLACK BOX., you have absolutely NO idea of HOW or WHAT is implemented in that Black box. There may be a thousand buffer levels in that box, with all sorts of swings and roundabouts. so... no.... you are NOT directly writing to disk, such a basic lack of insight and depth of thought is a worry with this sort of "data" Evangelicalism...
Thanks Steve, I think we're actually in agreement here. That's why TigerBeetle was designed with an explicit storage fault model, where we expect literally nothing of the "disk" (whether physical or virtualized). For example, we fully expect that I/O may be sent to the wrong sector or corrupted, and we test this to extreme lengths with the storage fault injection that we do. Again, we fully expect to be running in virtualized environments or across the network, or on firmware that doesn't fsync etc. and pretty much all of TigerBeetle was designed with this in mind. However, at the same time, to be clear, this talk is not so much about the "disk" as hardware-as about the kernel page cache as software, and what the kernel page cache does in response to I/O errors (whether from real disk or virtual disk). We're really trying to shine a spotlight on the terrific work coming out of UW-Madison in this regard: www.usenix.org/system/files/atc20-rebello.pdf To summarize their findings then, while Direct I/O is (completely) not sufficient, it is still necessary. It's just one of many little things you need to get right, if you have an explicit storage fault model, and if you want to preserve as much durability as you can. At least
Zig has been on my radar for about a year as one of those languages "I want to learn someday". About a month ago I finally devoted some time to it, and I regret not having done it sooner. I initially found the syntax off-putting, but within literally a few hours, I came to really appreciate it, realizing how smoothly it allowed you to do perform many tasks and work with the language idioms. The stdlib is a little sparse in the documentation department, but I can hardly hold that against a language that hasn't even hit v1.0 yet.
I wish there were some examples of what he was talking about. Sometimes it just sounds like vague advice. "Do software good like an artist does"... Okay, but like what? Like when talking about naming things he just said: "Use the right nouns and make the names good and clear". Yes, I agree, probably, but what are some examples? How do I know what you mean by "use the right noun"? I think I do that, but I have no idea what you think that is, nor what Tiger Style says on the subject.
So far, this is the most mind-twisting episode of all the previous ones. I'm still struggling to build a mental model around the idea of each ballot is processed concurrently by the replicas, yet a replica can "assess" the result of casting for previous ballots. Are there popular media (articles, video) describing the very same fundamentals behind consensus?
Turns out I've been using tigerstyle for the past 5 years and didn't even know. It's always good to see something put in writing that you've long subconsciously practiced, because then you can focus on it and audit it. This is helpful.
Matklad, it would definitely benefit you to unlearn the "uhhh" word. I think we tend to use it in conversation so the other party doesn't butt in and take the conversation in another direction, so it makes sense that we would have learned that behaviour in normal interactions with people... But in presentations I'd say it's a bad habit
I am guessing that English is not his mother tongue and despite that he has a really good command over the language, your comment is an unnecessary nitpick instead of constructive criticism. Even native English speakers overuse uhhh. It's a common filler word And if one is giving an impromptu presentation they will end up using filler words to gather their thoughts. It's not like he is speaking off of a script.
@@YuruCampSupermacy Yeah I totally get it, maybe I'm more sensitive than others, BUT if this was a college lecture you bet the students would be talking about it after class, comes with the territory
00:03 Testing distributed systems is complex due to asynchronous communication and multiple computers. 02:39 Testing a Single-Node, Single Threaded, Distributed System 06:44 Testing for unknown unknowns is crucial for robust systems 08:41 Cost of solving bugs increases over time 12:42 Effectively testing a system requires comprehensive exploration of its state space. 14:42 Testing a distributed system can be challenging due to the difficulty in replicating bugs found by users. 18:37 Focusing testing energy on key points can significantly speed up algorithm performance. 20:38 Using Mario's x-coordinate as a guidepost for progress in the game. 24:03 Testing distributed systems often requires considering higher dimensions than initially anticipated. 25:55 Exploring incomparable and unique situations to find bugs in distributed systems. 29:39 Challenges in Branch Execution Order 31:46 Testing a single-node distributed system in 1985 revealed issues with random number generation. 35:22 Applying a mask with low probability makes random distributions more realistic. 37:15 Improving random number distributions for more natural movement and beating levels in old video games. 40:56 Developed a deterministic hypervisor for running software consistently. 43:05 Accessing the program via email subject line.
The talk defines two terms: Strategy and Tactics. Strategy is the datum you choose to optimize for as the fuzzer randomly walks the states of the system. E.g. optimize to maximize Mario's X value, or optimize for reaching all tile positions etc. Tactics is the choice of input distribution.
Correct me if I am wrong, but the static allocation is only possible because you do have few specific structures to allocate. All of these would not be possible if the client would send a completely different structure. I mean, your system would never be able to store anything but transactions, unless someone modifying the code, adds a different structure, and compile it. Am I right saying so, and is that the reason this database is so specific and cannot be used for any other use cases rather than financial transactions? I mean, even if you have some sort of metadata attached to the transaction, this has to have a fixed length, otherwise cannot be allocated statically.
Thanks for the great question. The static memory allocation happens in the foundational layer underneath the state machine, so it's orthogonal in fact to TB's choice of primitives (and specific structures). But yes, all structures at heart need to be some multiple of fixed size (e.g. you could do 64 bytes, 256 bytes, 512 bytes, 4 KiB page size etc. to accommodate variable length structures). However, this is no different to e.g. how languages like JavaScript in V8 do it (IIRC, variable strings are a minimum of 24 bytes).
These are great - I have only had time to watch the first two. I am using more Rust in my free time, but want to get into zig. Overall I find the code simple, safe, intentional, and efficient! Which is my goal. Really appreciate this deep dive series!
NOW THIS IS HOW YOU MAKE A KILLER DEV TALK INTRO. Rockstar developers. Also I find it funny that it says "Iron Beetle" when thats image is a parody of Judas Priest.
TigerBeetle is a bit of a moving target, so for this stream, I always look at that IronBeetle branch. I do update the branch periodically, just not for every stream, just to make sure that consequitve episodes look at the same code. Should probably sync with main before today's stream though...
