98% Cloud Cost Saved By Writing Our Own Database 

😂😂😂🎉😂😂😂

Read my mind

WARNING Irony detected

Why would that be a problem? Was anything shown here that hard to understand? You should know how to do file and network io. What's left is a bit of domain knowledge and a one-time effort to encode and decode a rather simple data structure. Sometimes I really don't get that "let's better not do it ourselves" mentality.

Skill issue.

What's the 1%

@@SimonBuchanNz the frontend

You are a true genius. Are you available to optimize our IT department?

tom is a genius jdsl deez

99.9% for me because it's connected to libreoffice calc. I'm even saving on Microsoft Office costs !

Medium-driven-development

RDD - resume driven development

Did these numbers before getting out of the garage office stage

How about bdedd, read bidet: big-dick-energy-driven-development

I call it fun.

Its not GPS, its the tesselators that process it.

We used to average over N readings to get pretty good sub meter precision, but I don't think GPS is any better than 5 to 10 meters today. The trade off was battery life. More readings allows better precision, but burns battery. Less precision means the device can run longer without charge or replacement.

Isn't it a restriction put in place for government?

@@Hyperlooper The US Gov used to limit, but no longer: www.gps.gov/systems/gps/modernization/sa/

I only know about Trimble because of the fucking Tractor edits where they play on that whole "missile guidance system" meme lmfao

you forgot to put in the nerd emoji 🤓

@@krux02 is it nerdy to know the absolute basics of CS?

Log file stores entire stream of data, and they seem to store both «last state» data (as last as possible), and a log file, too. So technically it’s kinda like a simple database after all. From log file they can probably write entire path of car movement for example, as it is a series of data. For rare cases when you truly need this history. In database, they have their current position, battery/fuel levels, etc. For common cases.

Then no-sql dbs are not dbs?

@@monolith-zl4qtconsidering how self taught everyone is: yes, sort of

Let's expand this a little for the curious. GNSS works by measuring the timing of a signal between the receiver and the (constellation of) satellites. Since we know the speed of these signals, we can calculate the distance from the timing. In 3D Space, we need 3 (linearly independent) measurements to pin-point a location. In Space-Time (4D) we need 4 (Time is unknown because all clocks are imprecise, and since we are talking speed of light, every nano-second matters, by about 30cm :) ). This is how we know the position from the GNSS. However, there are factors which need to be considered. The signal traverses from space to the ground, trough atmosphere. There, the signal is corrupted by various effects. Some of these are tracked and accounted for (may look-up Tropospheric correction and Ionospheric correction). After these you may get your position accuracy down to few meters. However, there are still some errors that could be predicted left. But, you would need a closeby station to measure those effects. RTK is when you use a station with known position to measure these residuals. Then, you could use the same correction for any nearby device to improve their accuracy. Depending on conditions you may get centimeters-decimeters accuracy. However, generally speaking, in a city I would not expect more than a meter accuracy. I would probably not trust it to be that precise either. RTK relies on having similar conditions and there may be interference that isn't similar. Or it may be just my paranoia. Let me know if I have missed anything, or made a mistake.

And GPS is not better than dead reckoning, it just corrects data location drift :^)

It is just like data interpolation right?

Read this post like 3 weeks ago and that was my question. Kafka, just use kafka!

Perhaps it wasn't considered due to the fact that majority of customers deploy on-prem?

​@@retagainez you can deploy Kafka on prem easily enough. Also they said they were coming from AWS Aurora so their on prem thing is a bit weird.

Exactly, there's plenty of data streaming stuff available. If you don't need exactly-once delivery, NATS Jetstream is also worth a look.

I was just thinking this...

Meaning they only have very vague idea how their system performs and scales. Not a great sign to be honest. They should have emulated your production load by themselves to figure out the scaling issues beforehand.

Follow up note: I can see why this is particularly dangerous for binary, but it certainly applies to all data driven applications. Workarounds in JSON would be possible, especially with Rust, but versioning makes it so, so simple. I am nothing close to a novice or junior - despite how naive I was here - so it really goes to show that you must always remain a student. Seven years of building all types of systems, and yet that means nothing when it comes to things I haven't done. I've only ever built systems and thrown them away when they didn't make money. Running something long term requires maintenance which is something I NEVER thought about. Wow. This is why Prime is great. Everyone else gives useless tips and tricks to people learning JavaShit. Nobody else is out here helping programmers who are already competent and capable. THANK YOU THANK YOU THANK YOU!!!!

Well, if you're making your own binary format you ought to have looked at how other people have done it and you always see a version first thing which should tip you off

It is an excelent thing, but it also has nothing to do with being "self-taught" or not.

Personally, I would put version information higher up anyway. For performance reasons, you don't want to mix v1, v2, v3, etc data in the same stream. You want a stream of v1 data, a stream of v2 data, etc. That way your server can have distinct forks in its code to process each version as optimally as possible. When a customer needs new functionality you make them a v2 node with the additional data they need.

Versioning binary data is for pussies. I've seen how the biggest russian social network live-migrated 4.8 trillion messages while being on high-load, and it's just work of three PHP programmers.

How does one make a log file?

A MAJOR part of this that went unmentioned, they didn't try to get all their data in there either, just the specific domain that they operate in so they're still clearly using aregular DB for anything else and that's why the version field is a lot less important for their use case, it's well known data theyve been dealing with for a while and its a specific subset of the data they use

@@TurtleKwitty Yep, like writing parts of python in rust, just the ones that hit performance the most in python.

@@TurtleKwitty Less important doesnt mean unimportant. They will have to change this at one point, and this will cause the worst headache in the universe.

​@@domogdeilig Depending on the wrapper, they might not. If you change a device ID on update, then you can just point to the old ID for archive retrieval and store what version the device is using. Possibly even encoding it into the ID itself. You can even run a different database backend for v1 and v2 since I doubt you would make a breaking change very often. Then you just query the right DB.

100% this is literally just kafka but shittier. Kafka is crazy fast because it uses DMA and can move data from network card buffers to disk without copy and without CPU involvement. This article honestly reads like some engineers with too much ego to realize that the optimal solution here is to write to kafka and dump that to static storage as an intermediate step, or just right to a data warehouse.

Kafka and then writing into Delta was my first thought.

would you still have to aggregate out of kafka into something else for reporting

@@themichaelw I understand , kafka seems nice though I don't understand what kafka does :/

@@guptadagger896 why? Kafka is an event db. It supports SQL style queries.

that would be true if the package has a fixed lenght, but since you can skip sections (hence the has xxx flags in the header) the lenght is variable and they can't just go to specifyc bytes via indexing.

The normal purpose of a version as the first field in the header allows everything, including the header, to change. The article (and the video) both discuss indexing on bytes in the header which are always there and not part of the variable capabilities.

Why would you need a version field in every database record? One for the whole database is enough. Or did you think this was a network protocol? As I understand it, it's a file format.

Or just leave it in Kafka.

They're not logging anything they don't already need for querying so if they materialized to a DB it would just be the same throughput problem with extra steps. I don't know whether or not they could have made Kafka work performantly for them on the read end for their GIS queries

@@Sonsequence not really. the aggregate view might be buffered, flatmapped and updated every 5 seconds or so. the main thing they want is being able to have the data for whatever reason they might need in the future.

@@Sonsequence if they need every data point yes. But, given they are only keeping 30GB of data they could hold that in RAM using one of the in-memory databases, and let Kafka tiered storage hold the history. No custom serialization required, and a full query engine.

@@michaellatta yeah, just going with in-memory for the recent might be a good option but I don't think there's a GIS DB for that. Would still have to be custom.

The whole company could be 2 developers in a garage. In which case custom solution that saves OPEX may just be necessary thing to make the company profitable for year or two. I agree it is not very robust solution but the whole thing is so simple you could migrate it to any alternative storage backend (like Kafka, improved storage format, etc.) under a week.

And there might be a header for the file. They only showed a single data entry.

@@steffenbendel6031 That's what I thought. Repeating version in each entry would cost extra few % of storage. It would make more sense in case of the broadcasting data between nodes idea.

What about optimizing the tracking by not tracking too often when the car is on a straight road with no intersections? Or when the speed is 0 track less?

You should at least use timescaledb with postgres. It's just an extension and it will significantly improve performance. But also if your use case is very analytics heavy, then u should use clickhouse, tdengine, victoriametrics etc.. which are also better for a distributed setting compared to postgres.

@@SandraWantsCoke standing time is valuable data

If you are not using timescaledb, make sure to use BRIN indexes.

@@muaathasali4509Those other database engines you suggest are all OLAP which are very, very bad at many TPS. Op is better served by something like Timescale or InfluxDB.

Yes, they say so in the article^^

Thank you! Thought exactly the same while watching the video.

Yeah, him saying this caught me off guard. Pretty sure the only reason we have versions in things like internet packets is because there is physical hardware in the loop that has static circuits based on the version of the packets.

I agree.

But if you mainly just want to write a file, just write the file. Like Elon would say, it is not that complicated. Does Netflix puts their movies into normal databases? (Well certainly not into a DB hosted with someone else)

Edit: now that I reread the article, I’ve noticed I was thrown off by the claim that they need extremely high write performance. In reality they have high writes, but they don’t seem to need that data instantly and they don’t need consistency. So now I think just put a Kafka cluster in front of the writers and index every X minutes into ClickHouse, then let your app read from CH. 30k/s for CH might be high, but by batching the inserts from Kafka it doesn’t sound that big. I don’t think Clickhouse would be a good candidate. It’s extremely optimized for read queries on massive amounts of data. What these guys seem to require is a DB with a high rate of transactions per second.

+1 same idea

merica

@@alexsherzhukov6747 Huh?

@@darkwoodmovies entry level 10k/mo? there is one single place on earth where that could be happening

@@alexsherzhukov6747 Ooh yeah, true

entry level software engineer are 3k/mo

They might be able to get away with assuming the version from the time too? Similar to flags it'd be better at the start of the header though if used for version; could get away with it if production is only expected to read the latest version and not older versions too.

It’s simpler than that really. Prime is confused here: this is not a wire protocol. It’s an on disk format. You put the version in the file header. When you deploy a new version it starts to write to a new file. But can still read the old files.

@@tsx7878 I know what flags are, I suggestes using them for adding the versioning because that's the easiest way to check what type of object was handed to them without modifying the original object. The lack of the appropriate flag says it's original object, anything else is a newer object. The header can then be modified to expose a new function that excepts a void pointer instead of a predefined type. The old functions source can be renamed to this new function and slightly modified to be a wrapper to the new function, thus retaining the "one root function" rule for stable behaviour without breaking the existing ABI

@@tsx7878 Thank you. I had this exact thought.

Very true. I'm actually a little confused about why they needed to do this at all. Their main problems were that they needed extremely high write performance (up to 30k locations per sec), and they want the data to take up as small a size as possible. To solve this, they developed a binary encoding scheme, only stored the full object state every 200 writes (and between updates just stored the changed fields), and they batched their updates to one write per second per node. They also moved everything above 30gb in their new scheme to AWS Glacier because they don't need speed for data that old. Like you said, I feel like they could've stopped after the encoding scheme part? I feel like they would've saved enough money to the point where the rest was achievable without writing their own database. Even if they did nothing at all, 10k a month isn't bad at all? That's basically the salary for a single database engineer, and now they need to handle maintaining and updating the software, backups, routing, monitoring, and disk management, etc. And that's not talking about how much it costs to pay the engineers to build and validate this system originally. Maybe after 10 years it'll be viable (not accounting for the fact AWS and other options will likely be cheaper then), but I feel like I'm missing something because this doesn't seem like a good solution at all?

You can Google "Tigrebeetle prime". I'll try to post the video in the next comment, but if you don't see it, it'll be filtered.

They're going to need to pay for a dedicated team of people to manage and work on this, and I'm pretty sure it'll come out to more than 10k a month...

Well, I would say it is mainly very simple binary file. They not even did some tricky compression. (I once did a binary file for storing exchange data, that used arithmetic compression on the diffs of the values. Would also fit the requirements since compressing ist faster than decompressing)

Yeah, we were doing 20K transactions per node at an adtech company 10 years ago, so 60-70K should be cake today.

Yeah... I don't really get it. A cheap PC can handle 100k+ writes per second with batching

per database node?

I was thinking the same thing. There load is not that high. It sounds more like poor indexing or a bad set of hardware for the on premise clients.

Transaction != Transaction. You can't just compare incrementing a view counter to whatever GIS magic is going on here; 30k/s might be easy, it might be impossible.

Mainframe databases now are mainly DB2, which is a relational database, VSAM (just an indexed file) and IMS, a hierarchical database. Great thing about mainframe is not really software, but hardware. Not sure that would be a solution though as these were really expensive when I used to work with that. Always been something everybody wanted to migrate from, but never could because it was too hard