Rust at speed - building a fast concurrent database 

21:36 This is actually one of the best explanations of the rust ownership system I have heard yet.

Jon I really appreciate what you do. You inspired me to sit down and learn Rust. You've made a difference in my programming journey. Thanks

How you speak is perfectly paced and you hardly stutter/'um'. I'm always trying to improve my speaking abilities and you offer a great example to follow. Keep it up man

I absolutely love how you can tell, how much you are excited about the Rust programming language. Always with a big beaming smile when talking about the nice features Rust has.

I used that exact same atomic counter technique to decide when to the other copy of data is safe to write to at work. Felt quite clever when I got it to work. Really cool to see the same thing in the wild.

Jon, just found your channel today. It's rare to find someone who both covers really in-depth topics and does so in an entertaining way. Fabulous job you are doing here.

This talk is worth it's weight in gold. Thank you kind, Jon!

Well done, Jon! Fantastic talk and really interesting to hear more about what you're doing when you're not Rusting on RU-vid!

Really enjoyed this talk. The epiphany for me when hitting the borrow checker wall was when I realized that it was helping me not penalizing me for the code I've written. This small mental shift made everything all the easier and suddenly Rust became a pure joy to work with. I have to admit I didn't read through the whole Rust book from start to finish before hitting the wall. I usually never do that when learning a new language but you are absolutely right. Reading the book from start to finish will save you a lot of pain and frustration with the language.

Jon is inspiring in so many ways. The work (which he explains in the presentation) and the way he described every topic! Just awesome. I probably never watched any hour long talk in one go until this one!

I'm so glad i found this presentation again. I'm really getting into Rust and even advocating for it in my work, and recalled this presentation does a great job introducing Rust and proving how it can solve problems.

I pretty sure he love this language. Full of energy with a great talk

Thank you for uploading this, the second part of the talk is awesome and easy to follow.

Brilliant talk, thanks a lot! I really like how you've covered the ownership model, along with how you've implemented the lock free evmap datastructure and why that is safe. Well done.

Thank you for an amazing talk. As soon as Rust 1.0 came out I knew something interesting was cooking!

You’re good at explaining this - and I’m very new to exploring Rust. Thanks for posting this keynote.

Enjoyed the talk, Jon is indeed a phenomenal speaker!

Your talk was fantastic! Your passion and delivery were both amazing, and it really got me excited about Rust. Thank you for sharing your knowledge and enthusiasm with us.

This talk is amazing! You explain things in a very approachable way!

Love the idea of double increment and check if number is even. Simple and effective.

Jon, thank you and your friends for digging deep and discovering the possibilities. Since our last discussion, I am now learning how to code in python and I am learning other languages also. But my idea behind RUST is low latency and Jupyter notebook style coding. In fact Jupyter notebook was also one of my crazy ideas. Now that I am learning how to code, my writings' will be phenomenal given my skills. Expect more movies, lyrics to songs, Comedy segments, etc. Anything is possible now as always.

Every second in this video was worth watching. Awesome DS glimpse.

wow, what a great talk. Passion and enthusiasm makes such a difference. It captivated me. Thank you so much.

Thank you very much Jon, this is a valuable presentation rich with guidance and recommendations. I learned a lot from it!

I have a difference experience with rapid prototyping. To me rapid prototyping is MUCH easier with a good powerful type system if you do it in the right way. I'm still learning Rust, so I'll use Haskell as an example, but I already see it will be the same in Rust. In Haskell rapid prototyping is a breeze. Remember, that you don't need to write implementations or correct code! In Haskell you can write "undefined" and compiler will shut up. I normally, have several shorthands like "u = undefined" and then write a type and `u` as an implementation, compile, write more code replacing `u` with a more concrete implementation (that still can have more `u`s inside it) and then slowly an implementation writes itself as you replace undefined parts of your code with more and more concrete implementations. I heard this being called as type-driven or hole-driven development, because you follow the type, put holes in your code and fill those holes. Also, remember that with type inference you can replace anything with `_` in Haskell or with a sham type in Rust (like u8 when you know something is not u8) to figure its type or what type you need to pass, because it will complain and show you exactly which type something needs. Rapid prototyping is all about speeding up the feedback cycle and code-reuse. The latter is about not writing code at all, because you already have a library somewhere. Both are enabled by a strong and flexible type system. Starting with feedback, in Python to detect that your code is not working you'll have to wait until runtime .In Haskell or Rust, you'll immediately get feedback the moment you press a key. In fact, I usually wrap a function into another function that will print some useful output and I have very fast key shortcuts to both compile and to show some runtime feedback as well (like to print a datastructure I'm working on or run just tests that matter, keeping it fast). Like there will usually be a predefined name for me in a file, like `c`, which I can use at any time when working on a file. If I want to print something I just put one line in a file like `c = my_function testArg` and this is what will start getting printed out as I'm working on that function because that `c` will be like a main for that file, while I'm working on it that will constantly print output, and a script will automatically import that file and run it as I'm working on it. Same for tracing statements, pretty printing, and other tools that you would normally use. I want all of them to be a short names that I can use anywhere as I'm rapidly typing and working on something, preferably as a Swiss knife personal library that you can import in a few keystrokes into any file. Using other libraries is easier in a languages with a good type system due to a) type inference b) faster feedback cycles c) confidence in that if it compiles it will work. So, you can again just put a bunch of undefines, type holes / sham types, and slowly figure out what do you really need to some interface of that library or how to get there. In Python you can pass invalid data and you'll have to wait until runtime to figure this out. In Haskell or Rust, you will get immediate feedback that something is wrong and you have type inference to figure out EXACTLY what data needs to be there or how it should be used, and by using type holes can try to arrive at that point in any way you want by splitting the problem into smaller parts and following the types. What Rust misses here is more libraries, but when it will have a lot of libraries, figuring out those libraries will be much easier for reasons I've mentioned. Prototyping in Python can be easier DESPITE the lack of type system because there are more libraries so a lot of code you don't have to write. But when I have to write it, it's more painful. Python also has some good tools to speed up feedback cycles, like IPython, so some people naturally use workflows with better feedback cycles. But REPL is really not necessary if you have a good workflow which imitates REPL automatically. In fact, REPL can''t MAXIMIZE speed. Like you don't want to be typing code in REPL and then also in your file. You want your file to automatically give you REPL-like feedback. There are several ways to do it, beyond having scrips and conventions to automatically run things. You can have one file that acts as a RELP, imports a library you are working on and gets continuously run as you work on that library. I just find that more direct approach is faster. I guess to summarize this, I would say that if you don't fight the compiler but optimize your workflow then rapid then prototyping is easier with a more powerful type system. Rust can have no type system at all if you don't want it to, because you can ignore any parts of the code with unimplemented or panic.

Wonderful lecture ... and the switchable rw-lanes is a must takeaway... thank you!

Brilliant explanation of ownership. Simple and to the point. Better than any book I've read. Thank you for this, now it's pretty solid in my head.

Thanks for the very impressive and interesting talk! If I wouldn't love rust already, you would have convinced me to learn it ;)

Excellent description of some Rust features that will trip up the newb. Excellent video.

This was an incredible talk. Thank you. I shared this with my team - I'd love for us to consider using Rust for some lower-level video stuff. :-)

Jon is a wonderful teacher, and inspirational.

great talk, rust looks amazing

Love your work, enjoyed the session !

Amazing presentation. Now I love Rust more.

I adored this talk, thank you!

Super-linear is real, up to a point. Good memory access patterns can cause different threads to share cached data, such that 2 threads is more than 2x faster than 1 and 4 threads is more than 2x faster than 2. But in my limited experience, I never noticed past 4-6 threads, but at least beyond that was essentially perfectly linear.

This was fantastic, thank you

Great talk, many thanks for your sharing

49:16 "... your program will not compile unless everything is documented. It's fantastic!" You know you are a Computer Scientist when... 🤣

This is a really great talk!

Really interesting talk, well done

good stuff and great presentation!

Lovely talk. Thanks!

This was a fantastic talk

awesome stuff. I learned so much

thank you ! Looking forward for your new stream. Rust is the best programming language i have ever used :)

Great talk, thank you!

Thanks for sharing. Appreciate it

Great stuff

Thanks a lot, great talk

You are a genius! Thank you!

It is just incredible talk

amazing talk!

The 1 map for readers and another one for writer mechanism is like RCU - whenever the writer is ready then the pointer to the data structure is published such that the readers get the updated version.

Very interesting talk. I'm curious why you couldn't just store an atomic counter of how many readers are active per map instead of doing the whole epoch and ignore it if it's even thing.

Nice, sort of an advanced/smart “refresh ahead”, like caching systems do, that is built into the DB.

Thas was really really good.

Really interesting! you did that very well! Thank you .

I’m currently getting into rust with no basically no prior knowledge. It’s kind of a pain but it’s been fun. I’m on the fence still because I can’t find the best tutorials but I’m trying hard. I think it’s the future

Database talk start at 13:40

Hi Jon, this is a great talk and I am reading your paper as well. I found this video while searching on the topic of Read/Write separation in databases because Datomic is not open sourced thus I can't study its implementation. Have you ever evaluated Datomic? If so, what are your thoughts of Datomic and how different it is from Noria?

My inspiration

16:40 That is spectacular, I like it, will try that. (Update the read caches on write)

Hi, thanks a lot

at 52:00 he was describing that he had PTSD from fighting the Compiler ^_^

Amazing

33:30 the datastructure suggested implements a "weaker map", meaning that writes are not seen.

thank you for this great intro to noria. is there a way to try noria also ?

Interesting!

2022 and I find this talk interesting

@40:47 is that kdb with the benchmarking clause? 😅

As for the database thing, how does it work when you have multiple queries that try to change data through addition at once. Wouldn't it read the same value multiple times first an then change that multiple times instead of going in parallel?

i came here mostly for the database part of the talk :)

Bravo.

I was expecting "Multiversion concurrency control" of PostgreSQL, and CockroachDB to be compared, but then the video was over. I'll have to add that to my needs more research or testing list.

I'm interested in whether this approach is safe in shared memory multi-cpu systems. Aren't you relying on the order in which data pages are published to the different CPUs to be able to promise that an even count reflects the true state of the reader in another thread? Does Rust have some promises about the semantics of the memory architecture? (e.g. are you implicitly relying on the counter and the table pointers being in the same memory / same cache lines?)

Isn't storing nodes of a graph in a list and then using indices basically just doing pointers while hiding it from the borrow checker so it doesn't bother you?

Really awesome talk, one thing I don't get is the reasoning for an epoch counter instead of flipping between 0 and 1? edit: is it an ABA thing?

I have a question...When you exchange the reader/writer pointer, you need to ensure all the reader pointer number are even.What if you check the latter pointer, the former pointer's number change.It seems like we still need a mutex?

Quite surprised MySQL and Memcached are so fast by comparison. I'd expect more like O(10^2) difference.

Can you make tutorial about using rust + databases + web assembly to produce MVC software?

If he were creating a fairly normal relational DB in Rust, then it would be a cool way to see the advantages of Rust, but those advantages seem to be somewhat obscured by the unconventional design of his unusual database. In other words, maybe the radical design is what makes Noria cool, not Rust?

Can't postgres function idx do that?

Do you think Rust has caught up to speed on async networking stuff? I am trying to follow the Rust trends, but I am not sure what the difference between now and 4 years ago is. Tokio seems to have been doing a fair bit of work there.

37:03 Instead of counting each read begin/finish, why not just keep a boolean value, which indicates whether the thread is currently reading or not. When starting a read it will set it to true, after finishing a read, it sets it back to false.

how can I upvote this talk twice?

We did reasearch on this topic 25 years ago. The approach fail on non monotone schemas. Sometimes you have to recompute all the db. Easy to demonstrate. I can write a db in wich if you add a record in a table (cache) that cause all other cache to empty. The future is in functional databases. Where tables are functions.

Link for voting on your youtube chanel, intro video is broken :)

The idea of caching query results cannot be that new or novel isn’t it? I recall that oracle db has the feature with the name “materialized view” for a long time now

7:56 hmmm, I rely on null pointers in C++ to know about the state of my objects.

At 37:00, to check if there are no readers still reading or suspended before starting writing, he suggested to increment the counter twice. One on starting the read, the other when completed, then check if the counter is even before writing. With this approach, if there are two readers or any even number, will cause the writer to falsely assume there is no reader because the counter will be even. I would assume the right approach is increment on start and decrease when finished. Then, if the counter is 0, the writer can confidently guarantee no reader is pending finalization before start writing. Is there anything I am missing?

Isn't there a chance that just after the writer sees that all read counts are even and it is safe to do a write, and just before the writer/reader pointers get interchanged bfore the actual write, a reader starts a read, or what prevents the reader from doing that?

Thanks. Additionally, 0.8 is a reasonable speed for non-native speakers.

What happens when a read client on the partial materialized view A has an odd number of points in his access accounting, and the associated client thread died or is in an endless loop and will never end its query, i.e. won't increment its counter?

Old wine in a new bottle. This is very similar to following the even sourcing model with a materialized view (MV). or Simply use PostgreSQL 11 + CitusDB (distributed computing and sharding) + PipelineDB (continuous computation and MV)

Does anyone know if this has been abandoned? The GitHub repo hasn’t been updated in quite some time and I really want this to flourish.

Nice presentation, and I'm wondering if you've considered a potentially more simple solution to your problem: use multiple fine-grained locks, and use locks optimized for unfairness. It seems somewhat of a straw-man to compare a lock-free hashmap, with a non-concurrent hashmap protected by a single Rwlock, in the sense that one wouldn't expect the read throughput to scale linearly with the number of threads(beyond those few initial threads), when those would be contending on the same lock. Also I think you're taking a bit of a shortcut when saying that "with short critical sections, the lock acquisition/release becomes the bottle neck". With short critical sections, the point is doing work in parallel, and outside of the critical section. So if you compare the time spent acquiring and releasing a lock(entering/exiting the critical section), it should be very small compared to the time spent performing work outside of the critical section. The time spent in the critical section might be smaller than the time spent acquiring/releasing the lock, but that's besides the point, the real work is done outside of the critical section, and a thread should only enter the critical section to "publish" the result of parallel work for others to read(and take a snapshot to use in further parallel work?). ---- So I think there are two problems you're trying to solve, and going from "a single lock doesn't work", to "let's use lock-free algorithms", is a big jump from one side of the complexity spectrum to the other, one that is missing the low-hanging fruits in the middle: 1. When the individual threads in your workflow are making progress via lots of short-critical sections, you can optimize parallelism by allowing threads to re-acquire a lock they've just released, the so-called "unfairness" of the lock. 2. To improve your ability to scale the throughput of operations on the shared-data when those involve more than just a few threads, one could use many fine-grained locks, for example one around each bucket of data. Another benefit of actual locks, is that they can be paired with condvars for signalling, which can be useful if your business logic is not just about achieving lots of parallelism in reads or writes, but also cares about ordering or other types of logical synchronization between threads doing the reading or writing. I recommend the following articles: This one, which talks about optimizing locks for small size, cheap uncontended operations, and unfairness: webkit.org/blog/6161/locking-in-webkit/ This one, about locks, condvar, and concurrency in general(it has a nice paragraph on why one should avoid atomics as much as possible, and how locks can actually show better performance): www.chromium.org/developers/lock-and-condition-variable And off-course, in Rust, there is already a library offering you the kind of locks you would seem to need: docs.rs/parking_lot/

🤯

Tea leafs on the stack.

40 years from now, no one will understand how someone could even write unsafe code. You can quote me, if I am wrong.