What can game programming teach us about databases? (with Tyler Cloutier) 

I would just like to make a small correction. I was alerted to the fact League of Legends clients don't actually have full knowledge. Starcraft would have been a better example to call out.

Wow Kris! The banking analogy was so spot on. Really appreciate how focused you are while listening and absorbing new knowledge.

I did game development over a decade ago and this episode is one of the best things I heard in over a decade

Thanks for the excellent conversation! I had a lot of fun with this!

Thank you for the conversation. It was eye opening for a corporate DBA.

I feel like i leveled up just listening to this talk, thanks!

Fantastic interviewer and interviewee. This channel is an absolute diamond in the rocks. Thanks for the content!

1:00:39 (sings) "We do what we must because we can." You guys at SpacetimeDB are doing some real science!

These questions showed how deep the fundamental disconnect between classical programs and game development is. Instead of a transactional view (which also occurs a lot!), the gameplay code is much more a real time systems domain, and you manage very complex world and applications state dozens to hundreds of times a second.

My brain is still languishing in the world of relational DB's and LAMP-stack browser games. This discussion has really yanked me into the 2020's. I experienced a lot of WOW! moments while watching this.

Thank you for making the effort to produce this video. I’ve worked both in “industrial” and “game” software engineering and I’ve always wished I could mix the two. The practice of actually writing tests for your software + the data and state centric design approach 😅. You have a new subscriber !

This has honestly got to be one of the best new channels I’ve found. Lots of really interesting videos on here!

i cannot even fathom the amount of data that is being plumbed into a database considering its not just individuals' game data but also their controller events

Amazing interview! Very informative. Great questions asked and very thorough and clear to understand answers.

This was a great listen! Both spacetime and Bitcraft seem very cool. 👍

Oooohhh I liked this one. I really liked it! Thanks for the insights. It's fun to see how other engineers architect solutions to different problems.

Wonderfully explained by Tyler. Plenty to digest & think about...

@58:00 I was literally thinking the same right before he mentioned it - it all does sound like the actor model!

I wish more games would use the data-centric publish-subscribe pattern from DDS. Any agent should be able to subscribe to any data without much hassle.

super, I really enjoy this kind of content especially in game. Game development is hard and any effort that try to simplify it is awesome.

found your channel recently, very good content! congrats!

Second Life (launched in 2003) also has all these features, though it's not set up as a game where you can "kill" agents.

Implementing SpaceTimeDB in a relational database seems completely insane to me. In order to grok the 'genius' behind that implementation strategy I'd need a VERY rational explanation of why a graph database would not be significantly preferable.

wow, SpacetimeDB has some incredible potential.. and I would say as more than "just" syncing game data in real time. I'm definitely going to play with this one!

great interview!

Amazing interview, amazing guest.

btw, when he says "free-to-play game", you can read "pay-to-win game"

24:15 something important is that sometimes you just don't *want* the client to have all the info, even if you could. If the client has the data, then the player has the data, so you have to worry about players potentially extracting that data from memory and cheating. For example in a battle royale set in a map of several square km, if your client knows all the enemy positions the player could have some cheating software that will show that as an overlay or in some other way, giving them an unfair advantage. So if the client can show an accurate representation of the game world without that data it's best to just not send it. Of course it's not that easy because you also want the client to be able to predict future state to some degree so you cannot do things like "don't send the enemy position and velocity if it's behind a wall and not visible". Update: Ok he talks about it at 47:00 Amazing interview, guys!

This is great, thanks for sharing

kdb+ (Kx Systems) also can uses the database as the center of the deployed environment. q (or k) code is run inside the database and can be updated without taking the server down. For OLTP or streaming queries, the database state is kept in RAM and updates can be written to a log for replay in the event of a system interruption.

Thank you for show notes

As an academic truly appreciate👏🏾

Is this the decade event sourcing hits mainstream? So many people are discovering or re-discovering it. People want to swap out read models, they want historical data, and appending to a log (append only file) is basically a foot note in terms of cost.

Wow, sounds useful for what I'm working on :o

The challenge here is physics based sandboxes. At some level all of the example are really turn-based games that follow rules engines. Real time physics sandboxes make this stuff much harder.

I'm planning development for a MMORPG this was really helpful!

I think the architecture will be quite difficult to do model updates on. Because each server node and client node needs to use the same code/model. I can see a full shutdown of all clients required. Then let the server nodes sync up. Validate this. Then do model/code updates in single user mode. Link up again and validate shared state. Then update clients and re-connect them. *As for storing all historical transactions* Model changes and game rule changes mess with historical data. Historical transactions cannot really be replayed fully after a change, if it are actions (that would require the code as it was back then, bugs included). But if it is just a log of what was decided and approved, then information is lost and it becomes increasingly harder to interpret the data due to changes over time. I kind of look at it in the same way I look at persistent memory (computers that never boot up, programs that never shut down). That I will never trust, every bug is permanently ingrained in the OS/program state and permanent. For the same reasons object databases that simply act as virtual persistent memory are not trustworthy. Overal, I think it is an interesting idea, but fear that in practice it will be too restrictive and clog up over time.

Great Listen

I'd be interested in possible applications outside of gaming, things like AI and other types of products

Sounds like this streams MQTT subscriptions and periodically updates the location "cell data" (I presume a few hundred meters hexagon - or overlapping squares), so one cell has the same state for every user in the cell and only the changes are streamed (UDP?), presumably in a compressed binary stream. Predictions of changes can be run on the client's state and corrected within the server tick - which could even be dynamic depending on the number of events. For persistence I guess you can snapshot parts of the server memory and write the snapshots to disk in a db like clickhouse, spark, hive etc.

I love your nail polish, it looks so good

The question is which aspect of software are on the top of the concerns. As in deed games are in general good at multi users, data flow, responsiveness. BUT often they lack in safety and cheating. Industry and bunking software this is priority over all and the constant service accessibility. As games should be also resistant to that in reality often it's not a case. It's not the end of the world if someone breaks your stats by cheating. But if loose savings of your life from your account, totally different story. Further more game developers agree to some level of vulnerability to be able to develop more responsive game in acceptable time and budget. So it's a often case of tradeoffs.

this man is reinventing the internet

good stuff

For a multiplayer game, I would send player and bullets/bomb position to both server and closest peer players to avoid server round trip for very close actions.

At 13:00 - I might honestly be naive to the problem domain as I don't have insight directly into the code or the performance, but this seems like the approach would be a normal "piece table"? Where edits to the world is what is going across the wire and then at some interval serialize the edits. I don't see what is so hard about it. The pattern (piece table) is decades old. The only problem is making it as efficient as you want or need it to be.

If type lazy load conect db with ai???

It surprises me how much innovation I have been seeing in the db domain recently. Just in this podcast I guess I heard also of TigerBeetle and DuckDb (from which I am a huge fan). Maybe you should interview Glauber from Turso.

that first 10 min. I already feel the mans internal pain that only the modern world could fathom

This seems like an excellent use case for NATS Jetstream...

Runescape ticks at 600ms intervals. But there are some exceptions to this. I'm wondering about security implications of this. How easy is it for my client to subscribe to things I'm not supposed to know? For example does the game have pvp areas? Can I subscribe to certain player movements that i'm not really supposed to know (yet)? Can I subscribe to the locations of all the bombs in my area? What's stopping me from doing this?

Why not using a Graph database and vectors ? SQL on the mapreduce computing framework is not ideal for fast transaction on the model required for games

Very interesting podcast. I my mind I see this as a continuous merge engine. Actor events coming in - Merge that with the current environment state - Stream new environment filtered on rendering distance to subscribers. So you actually subscribe to the rendering distance. All objects in the world are actors. A house that blows up is an actor like an enemy is an actor, that just runs in a pod, server side. What I am wondering about is this predictive element. I am wondering if this will be done client side. Is it possible to have the client send to most probable action based on his current state (Like where is his gun pointing or is his gun loaded), along with what he just did, so that you can merge that to send the most probable future state to all subscribers?

Doesn't Clojure/Datomic (Rich Hickey) solve both the "expensive retention of historical data" problem, and the "copying of 99% inactive data" problem?

It would have been easier to explain from the point of diffing. The Server is storing a state and the change that was made to get to that state. Just like git this allows you to roll back state without storing huge amounts of data. The Servers also use sharding for players and only send the "diff" of nearby state changes once the current state is fully loaded/cached.

I don't know if i missed the question, but why on earth would u use WASM for this 🤨 feels like a weird decision, but I hope theres a good reason for it because I'm kinda hyped for this DB.

How does it compare to SpatialOS and why it will perform better? How much will it cost to run this (fixed costs) and to scale up (per player hour)? What are the latency guarantees (excluding network RTT)?

As an enterprisey RDBMS guy (OLAP/OLTP) who learned about ECS for the first time in context of hacking on an RTS game, what this guy is saying is music to my ears. Yes, everything is and should be a database, and if you think your application is not really a database, this just reveals your ignorance about databases.

when your primary motivation is data mining, but to get the data out of the plebs you also need to develop some game they will find addictive enough.

minecraft often had block lag.. would bit craft really be much different with it's lag with one giant server of people constantly requesting off of the database? really sounds better to have several small databases all connected changing one database doesn't really effect another. So like a per chunk basis the character can push around stuff in their chunk write stuff to the chunk they are in. Players don't have to load more than the databases required. So I guess there could be one giant game database that relies updating on the small chunk databases that change making one simple database save file for the game state but I'd say when it wants to go through 'server maintance'/turning off the database and rebooting from the save state you be sure to update the game state database with the chunk databases first. Yeah I'm kinda just overthinking how I would set stuff up.

This would create opportunities for more multiplayer games developed by indie devs, won't it?

Now I'm wondering if Valve could fix their CS2 sub-tick update system with SpaceTime DB lol

Very interesting to listen this dev talk!! just wondering dose he break the NDA by talking about the core game mechanism in the beginning? hope he wouldn't .

27:47 as long as the server still validates every client input as according to the game rules, this is ok. But there is no way around the server needing to have game knowledge, can't trust clients to send valid data according to game rules!

interesting, i can see this as a good approach for games that have complex hierarchical data like rpgs, but for simulation type games like bf/cod for example i think it's best to have everything sent to client and using strategies "delta views" (i think thats what its called) - which is just using extra compute to only send differences between states instead of duplicate states each frame to reduce bandwidth...there is a good talk on this by some engineer building battlefield/cod multiplayer server but forgot the name of the video...

Listening at the moment - I'm getting the impression that they have invented QuakeWorld circa late 1990s ?

so i'm not the only coder with 'christmas tree ball's metallic purple' nail polish! 🎉 the venn diagram of coder & metallic purple nail polish must have a very small cross-section...

game devs are built different

Coming from Unreal this sounds really weird because the server does a lot of things for you. Let's say I'm a cheater: On the client I'm gonna disable all collisions for a walls in a game. If I would walk through the wall on the client the server would correct me and teleport me back. Because on the server the collision is enabled. And the server has authority. How would I do something like this on SpaceTimeDB, how are collisions/physics handled? Because the server needs to know what the collision mesh of the level looks like. The way physics are simulated need to match how the physics are simulated on Unreal Engine. Something as simple* as movement seems like it would already be a huge undertaking.

Yeah you're not getting "automatic" prediction, You're just doing prediction on everything. I guess that's a way to do things.

at minute 4 may I know the game name? THe one Tyler say you can feel like a king?

Bookmark 31:44

Okay, sounds like firebase. How does it scales?

Space-time? Shout-out to Alfred Korzybski.

Why is Kris' webcam set for such a low frame rate? Sharp images but a bit jarring.

sounds like Game of War-Fire age, or Evony

There’s not much more to it than moving data around in memory or recreating it.

Jusf wow

regarding the several million dollars stuff worth bought in the game, I'd like to point people to the great James Stephanie Sterling and the jimquisition (ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-7S-DGTBZU14.html). No not everybody who spends an exorbitant amount of money actually has the funds, gambling addiction is a big problem, and un-surprinsingly the video game industry is using the same terms and "exploits" as the gambling industry, but due to loopholes in lawmaking video games are much less regulated than gambling sites.

Take a look at Star Citizens programming problems they're trying to solve! They're doing about as advanced an architecture as I've ever seen

Distrubuted systems are complex ? Only if your model is bad or heaven forbid code is spagetti.

21:38 "everything is a database" Depends on how you mean it, if that is true or not. If it means all state can be persistent, then if it not true. But if it means, every state can be accessed as if it were a database, sure, there is something to be said about that. Treating state itself as something that is persistent is enormously rigid and error prone. Every now and then someone or some institution thinks they found the light and memory needs to be persistent. That might work out well for biological systems that are inaccurate and forgetful and have Neuroplasticity. But it is a terrible concept for anything else. Think of error accumulation, having bugs forever persistent in ones program state is just plain wrong. What if a developer wants to use a different algorithm, how to transition the current state to that new one? You can see the nightmare and potential for bugs developing right in front of your eyes now, don't you?

That was a weird talk. I feel like his need to log historical data of everything is unnecessary for most of what you look at as a dev. It's solving retention issues without actually looking at the gameplay. Usually retention is due to failure in your gameplay's fun. If you try and balance your game around data then you're just building a Skinner box. I guess the 'run a local server client side' can make sense. But you'd need to be very selective about what can be run and determined client-side. There's plenty of instances where games made too many things deterministic based on client-side events. If it is purely to predict the next few packets, then it seems over engineered for most cases? A fireball that is going in a straight line will continue in a straight line. I dunno. It felt like he was trying to sell his DB more than talk about games as a whole after experiencing a mobile whale hunter and trying to make his own Minecraft MMO. There's I three different common setups each with their pros and cons. The Starcraft2 one where the server simply handles communication between the two clients to allow them to update each other's client info without exposing IP addresses. MMO servers which handle all of the claculations and AI on their end, then only expose as much as is necessary for the client to update their cooldowns, buffs, draw enemies/allies on screen. Peer-to-peer where usually one person hosts the game and all other players join. All the game logic is run on one player and distributed to the other players. Other players send their input events to the host who manages the logic of the input. It's the closest to playing split screen. But instead of the screen being split the events are replicated on the peer's game clients and rendered on their screen. Notice how the server doesn't rely on the 'database' to handle events and send data? Usually the database is separate from the server's loop because what's happening in real time does not need to be recorded. You don't need to read/write the position of a fireball as it travels through the air. But you do need to know when someone plopped down a tent so that other players know where it's at. A server is not a streaming database... How do you manage calculations and mutations to how skills should function. Yes you need to lookup the player stats. That's done when they log in and then held in memory. Updated when their gear or statuses change. But then processing that data is part of the server function... Which is volatile data which doesn't NEED to be recorded to a database unless you want to bloat your hard drives. At best that goes into a log file. The stuff updating the database is the current player health ONLY in case you have a game where they could log out of an instance and you want them to log in with the same health pool in the same place. There are interesting questions about how to structure your database based on what information needs to always be loaded, who should have that information at what time and how it should be accessed. Otherwise you end up with issues like in D4 where they load the ENTIRE player's stash and send that to every user who sees them. How would you manage lifetime of poison stacks on an entity? Every single poison stack has its own duration with its own damage and ticks down rounded to the nearest server tick... Are you really writing all of that to a database at every instance or are you holding that in ram and managing that stack? I'm still a beginner but the 'trust me bro it works fine' seems very fishy. Then equating it to an OS? Yeah, in a broad sense everything is a database because everything is reliant on accessing memory but... accessing memory is not the same as using it to determine gameplay events. So yeah, there's a lot of data being passed between a gamer and the server. But there's not always a ton of DB updates unless you make a DB that tracks every single event. For some reason. How much is it going to cost you to run a system like that for 1M players for 10 years? A bit more digging : BitCraft: The New Frontier in Blockchain Gaming Ah. I see. That's why he liked smart contracts. Please let me know if I missed the mark. Because with what I understand of gaming infrastructure it seems very very odd. Like trying to make a blockchain server work for real-time gaming.

Didnt warcraft 3 have rrplays 23 years ago?

Dear lord, not stored procedures...

it's a sickness that everything looks like related table structures, when you do a lot of relational db. most stuff is actually tree data structures, which are simulated on relational db, and performance suffers compared to native tree implementation. banking and gaming do indeed need fast transactions but, once in an relational db, realtime performance is not required, to say do data analysis in sql. which brings me to last point, relational db is capable of doing great big joins and filters on indexed data, which is all that sql does in the end..and that's not typically the result data you need in massive online worlds. so now that you're stuck in a relational engine for everything, you'll overcome most problems..that you've created. very similar to java (bad) developers making whole thing into objects, and then having to solve all those attendant problems, which werent original problems. i'm sure you're a capable developer but..dont fall for the falacy that everything looks like a nail, because you got a hammer in hand.

Rubber banding due to ping is a bad experience, I don't see how using global database is a good idea for this MMORPG?

I absolutely despise Machine Zone

4X: Explore, Expand, Exploit ... EXTERMINATE... EXTERMINATE... EXTERMINATE...

"speed ups" aka pay to win

The guy looks like an old wizard from lord of the rings lol

6:38 the consequences of falling behind in a capitalist economy. No wonder, more people turn to online and fps games it's how they cope, inadvertently this widens the wealth gap even further.

"Everything is a database". Data is defined as the lack of uniformity in the perceived/perceivable. So if the universe was all the same thing there would be no data. But any difference is data. Everything is data, dedomena is the word. But data is not information, although it can carry it.

Runescape updates every 0.6 seconds.

Can openAI help???

Why not us surrelql

isn't he basically describing blockchains?

240hz ...

why couldn't you use a distributed ledger like a blockchain specifically for the game as the database for a game? smart contracts if coded right can have transactions per second as fast as your router can send data. the system where the server disagrees with the client side is bs and usually unfair. it requires a high tickrate server to not piss people off which can cost more.

so its a blockchain xD