It would be cool to see what the Ntex framework (Rust) can do. It’s what the original creator of Actix moved-on to after giving Actix over to the community. It seems like it performs really well on the benchmarks I’ve seen it in, but it would be cool to see it in this format here against the other rust frameworks!
Honestly the fact that you took feedback and went back to make further comparisons that incorporated the feedback from different communities may make it it one of the best videos of this kind around.
Thank you for correcting your mistake by rerunning the tests with optimisations! Mistakes are completely normal, but seeing corrections is surprisingly rare...
@@CoolestPossibleName there is some truth to that though. not that it doesn't have benefits over c, but i don't know if the benefits are strong enough for most companies to justify it over c. odin is kind of in that same place. in my opinion, c3 has more compelling arguments going for it.
Zig has only a safe mode to detect faults at runtime, in real word application you would release optimised for space or speed without the runtime failsafe
Release Safe does not ensure (full) memory safety. It adds stuff like UB checks (some of it is missing in Rust), but no "full" memory safety like in rust.
As a Rust dev, its really cool to see Zig and Rust trade blows. considering picking up Zig where i would normally reach for C due to the dev environment.
@@ionutale1950Depends wholly on the application. In-mem caching can help a lot with db hits. If you’re using a global state you can batch appends and read state once per server on ticks and it’d stay pretty light. If you’re displaying tables too big for cache for numerous reports and clients then yeah you’ll have some DB pains. But you likely wouldn’t get that kind of volume for that kind of thing.
even after those optimization performance and compile command change, rust still on the top. its so satisfying how the developer and man behind rust and actix can achive such performance.... they are the GOAT
Python/Nodejs for fast prototyping. Golang for production 98% of the time. Rust less than 2% of the time for certain super low latency requirements like HFT.
@@Z3U5.0g Yep. Its cool that its so fast, but I would expect a systems language to have a worse DX than Go. But I think choosing feature complete framework in any language (e.g. Laravel) is also a very good solution. No "stack" hopping
@@NabekenProG87DX comes down to preference. I found golang panicked too easily and the error handling was pretty bad. Rust's pattern matching, union types (enums) and macros make it a joy to work with if you can push through the steep learning curve. But it comes down to preference in the end
rust is a low level lang that doesn't feel low level at all, also it will not allow you to use pointers like crazy, and forces you to write everything thread safe, u can only make shitty code if u use "unsafe" keyword. also "thread_local" macro feels like magic, its wonderful.
@@Cuca-hn3md I wouldn't call Rust a low level language. It is a compiled language that allows you to write extremely efficient code and provides memory safety despite manual memory management. Those are indeed two highly desirable traits in system software, but that doesn't make Rust a low level language. Go, with its relatively low expressiveness, is a low level language - most of what you can write in other languages will take at least one and a half as many lines of code in Go. Rust, however, doesn't have this problem. With Rust, you can write a database driver or a Web application equally well, without writing much more code than if you'd use languages traditionally used for each job. In fact, if you compete against plain C in the database driver, chances are the Rust code will be smaller. I'd venture to say Rust be about on par with Python for the Web application.
@@a0flj0 "level" could also mean how much control you have over what will actually happen in the computer. The fact that you can write inline assembly in Rust makes it IMO as low level as C.
Note on the previous benchmark: Zig was in 'debug' mode, which is the default, and I didn’t know that. I found a "production-ready" Dockerfile and thought it was properly compiled, so I’m not the only one who made that mistake.
I'd like to see a comparison between the same web app on Rust Actix-Web and Elixir Phoenix. It might look like apples and oranges but it'll be interesting to see the memory utilization of the two apps. Elixir "processes" are light threads. Actix implements its own light threads but I understand those still use more memory than Elixir (BEAM) processes
Actix-web stopped using the Actix actors, tho. And yes, Rust actors cannot be as efficient as BEAM actors, because BEAMs are preemptible (need assembly for that) and all state in BEAM is immutable.
@@LtdJorge Thank you. I wasn't aware of the Actix-web change. I still think the comparison would be interesting to see though. It'll put specific numbers to the memory usage
Hi @AntonPutra I've been following your channel for a while now and I absolutely love your content on comparing tests and benchmarks of different programming languages and frameworks. I was wondering if you could consider doing a similar comparison between in-memory databases like Redis and RabbitMQ. I'm particularly interested in seeing how they perform under various workloads and use cases. Thanks for all the great content!
Thank you! Actually, the second benchmark should have been Redis vs. Memcache, but every time I get distracted by reading all the comments. I'll get there eventually 😂
Rust can actually be FASTER than Zig, C or C++ in some cases because their ownership and aliasing rules for &T and &mut T allow for more aggressive optimization techniques. Although the downside to Rust lifetimes is that it restricts which programs the compiler will mark as semantically correct, that is also one of its benefits. Since Rust programs are restricted to a smaller subset, the compiler can also make more aggressive assumptions/optimizations of those valid programs.
Unfortunately there are a lot of cases where the assembly generated by LLVM is suboptimal, like additional memory/register copies. It would be very interesting to see a Rust compiler that is built from the ground up to take advantages of all available guarantees.
You are talking about avoiding unnecessary copy operations? Does Rust actually have an advantage over Zig here? Zig does seem to do rather aggressive optimizations in this department already. Since Zig is going to have its own backend, it might be able to benefit more from Zig specific optimizations in the future. For the time being, most effort in Zig seems to go into optimizing the compiler performance (incremental compilation, etc.). I'm curious to see what happens when the focus shifts to code generation and optimization later on.
@@michaelutech4786 If Zig uses pointers, it should be the exact same. I suspect iterators are the main reason why. The Zig/C implementation probably does bounds and null checks every iteration of a loop. Rust's iterators don't have to.
150k with 2 cores that is a lot! i did some similar tests, but on my local machine with NestJs and Go Fibber, not even close: - NestJs 400 req/s - Go Fibber around 10k dam, rust is fast. i am wondering if there is anything faster than Rust, like a server is C or C++
I mean they would roughly be the same performance, since they are "manage your own memory" languages. what differs is the different HTTP framework implementation which actix seems to have the edge
Zig is something like 0.6 version lang m8. Id say that was pretty fucking impressive how zig ran that C library. Only small margin behind fully fledged rust. Id say that Zig will literally eat C
@@ionutale1950I would normally argue for it just being a bad zig implementation, but with the weird behavior we see, I suspect that most Zig server frameworks are just C copies, and a lot of C frameworks are pretty stable and reliable, but outdated when it comes to algorithms and implementations.
I was thinking of running this benchmark on a standalone VM, and I even have everything ready, including the systemctl service files. However, the problem is that when you run something without limits, it starts to affect all other services on that VM. For example, I use node-exporter to collect basic metrics, so without limits, those exporters and agents begin to degrade as well, causing gaps in your monitoring system. You can set cgroup limits using systemctl, but it would be similar to how Kubernetes handles it.
Hello Anton, I saw that a PR for setting the zap and zig stdlib implementation to 2 workers. Is that PR already included in this benchmark video? I really appreciate the effort you have put in these benchmark video. Thank you!
@@jiinyeongoh7458 no actually i just cloned that lesson including this PR which uses only 1 worker and run with it - github.com/antonputra/tutorials/pull/271/files i got pr to increase the number of workers later.
Yep, have only just started on this Will need to duplicate your test setup and iterate through config options to get it scaling properly for starters Current implementation that caps out like you saw uses only 1 worker thread :) So need to have some playtime with these AWS boxes to see how they behave under load
Thank you for this comparison, can you take into account the content of the queries? Actix sends more Data ( Http headers ) so you need Bytes/sec metrics. Also it would be really interesting to change Serde by Sonic-rs which is more performant, which will make Rust win even more points. And why not use Hyper directly instead of Actix? Thanks again 🙏🏻
It's the difference between caching a lot and no caching at all. Caching means availability remains constant as long as you can keep caching requests at the cost of using more memory. Actix is almost completely functional; no caching.
What was the request output? Could you make it more "real world". Like an json response of 1kb, 10kb, 100kb. And then another test that compiles some zig to html or rust to html (using thr same 1kb, 10kb, and 100kb html outputs). Could you also add golang to these ?
@sighupcmd A proposal for a more practical benchmark: add two dimensions, JSON/API and HTML/Templating. This moves away from pure theory towards real-world use cases. Research into optimal engines for each will be needed, trading some purity for more relevant metrics.
@@Matty0187 again, you propose to bench libs responsible for json/html (where are a dozens of them, btw). What's the point of this here? Here author used "minimal" required stack to make service respond. That's the correct way. Also, there's only one real use case: yours (your service's).
@@sighupcmd errors, rps, and latency while great metrics, a response of "hello world" from a server is a naive. We should measure bytes or packets out as an additional dimension. I still wouls love to see how common template and JSON responses effect the rps, latency, and error rates cross the servers
You also have to remember that Rust has hardened a lot of small optimization into std and some core libraries, this test would be more fair result in 10 years to give Zig more time to mature. But I'm team rust, even if it's slower I'll have less bugs with the safety features.
the pure zig implementation is another league on memory usage, it's heavily optimized to avoid heap allocations as much as possible at the expense of using the heap for parallelism with that memory usage it can scale an order of magnitude better than both zap and actix actix is an overall winner here, it has a mature and more featured api, memory usage & req/s better than zap zap is little immature as a project, it will benefit from both code optimization & from zig future versions
I don't agree with the "only use Rust if you need the performance" since I prefer the DX of Rust over Golang. Once I was used to the ownership concept and wrote some async code, I don't feel like Rust is slowing me down at all. Although the problem that if one function becomes async and the async keyword has to propagate to all caller functions is pretty annoying. Learning Rust was probably easier for me since I come from a C/C++ background, I guess it's much harder to learn Rust if you only used garbage collected languages before.
Cool to see that actix performance is basically the same as what is essentially a C implementation. Also you managed to show clearly what the effect is of setting cpu limits in kubernetes. When it starts throttling the stdlib one you see the latency jump up and the amount of requests it can handle plummet. Would be interesting to see what would happen if you run the exact same test but on a m7a.large without cpu limits. I would expect it to then not have such a sudden breaking point but just gradually increase latency with sustained max requests per second
I was thinking of running this benchmark on a standalone VM, and I even have everything ready, including the systemctl service files. However, the problem is that when you run something without limits, it starts to affect all other services on that VM. For example, I use node-exporter to collect basic metrics, so without limits, those exporters and agents begin to degrade as well, causing gaps in your monitoring system. You can set cgroup limits using systemctl, but it would be similar to how Kubernetes handles it.
@@AntonPutra Ah I see what you mean, cgroups have cpu shares which is what requests map to (after calculations). Just running in k8s without limits should already give the insights since all nodes will have the same kubelet overhead
Exactly what sort of hardware does m7a.large provide? Are they quoting real cores, or are they including hyperthreaded cores ? It does make a bit of a difference depending on what the app does Would be interesting as well to compare x86 vs Arm offerings on AWS
Probably a fair result then. The “std” zig implementation there is using only 1 thread, so 80k shows us where the hardware is getting saturated The flatline from there on is (likely) kubernetes throttling that maxed out single CPU Not surprisingly- all 3 implementations that are using the same llvm optimizer here perform very close up to that point. @anton - send me some links plz to how to duplicate your test setup so we can properly tune this for the environment it’s running on. Need to iterate the config to find a suitable balance here Fun project! Working with low level code (be it C / Rust / Zig whatever) is like tuning a car to match the course it’s racing on :) At least the audience is enjoying the show :)
haha, sure I'm using EKS which is a managed Kubernetes cluster in AWS. If you have never used it before it can be problematic and expensive. You can use minikube locally but you would never be able to reproduce the benchmark locally. But at least you can test it and compare it. So minikube - minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download Kubernetes deployments - github.com/antonputra/tutorials/tree/main/lessons/208/deploy you can apply with kubectl apply -f ... also i have clients as well - github.com/antonputra/tutorials/tree/main/lessons/208/1-test also kubectl apply -f ... all images including clients are compiled for both arm and amd, so you can run them on mac as well if you need any help please let me know
@@AntonPutra Cheers thanks. Yeah, I will go back and review some of your playlists, and try out those deploy scripts. Managed EKS is probs the way to go then. I dont mind spending a $bit to get it right .. just have to remember to turn it off when im done :) Interested in finding the code for your load-tester container - is that in a lesson somewhere ?
@@steveoc64 i have older version written in go with prometheus metrics - github.com/antonputra/tutorials/tree/main/lessons/201/client but it is really outdated but can give you an idea. i recently rewrote that client in rust but it is not ready for release yet. with go client i used m7a.8xlarge instances with new rust client i only need 4xlarge
@@steveoc64 Also, make sure when you run those tests in the cloud that your application and client are deployed in the same availability zone. AWS has data transfer charges between regions and between availability zones, but inside the same zone, it is free. If you start running load tests between zones, it could cost thousands of dollars in data transfer fees, so be careful.
Would be cool to see this in comparison to older languages like C(++) and java, that might not be using the newer techniques, but do have years of optimizations
Honestly with how much more knowledge we have now, it would be very surprising if you can "optimize away" the difference. It will still be C at the top unless there is a paradigm shift (like the functional style of Actix).
How do you time-lapse thru the results so it looks like they are going really fast? At first I thought it was real time and then I realized you are replaying them somehow on grafana
As a Rust dev, I was expecting more from zig on all fronts in this test, as I keep hearing that zig is more performant and has better memory control, while being somewhat memory safe. For it to not beat rust in any of these fields really showed me that any low system programming will accumulate a die hard, cult like, core, that will try to make a sell for things the language cannot even still do. In this case, zig is still far too young to compete with a powerhouse like Actix, and once mature, probably will match it more than beat it. Thats fine. The true selling point of the language is to be c like to ease dev in it and to improve over c. Rust is at the extreme opposite and you might fit in its targeted audience or not, but regardless, it delivers on the core premises of memory safety at no performance cost. In this regard, taking sides seems pointless. As a dev, you are not your tool, you just use the tools that you have at your disposal and are the most expert with even if its not the greatest one for the task (I look at you nodeJs and Flask). So learn both, become an expert on the one you prefer and respect those who prefered the other option.
this hit rock bottom. golang for fast prototyping and changing requirement. rust for rewriting core business or legacy system which cost is too expensive
Python/Nodejs for fast prototyping. Golang for production 98% of the time. Rust less than 2% of the time for certain super low latency requirements like HFT.
@@Z3U5.0g sadly, i am currently working in a company with very little care of prototyping. once it's implemented in python / node, it will stay that way. make sure your company can afford rewrite. mine is not that friendly toward this. that's why prototyping in golang is better, consider that we are only talking about web server in this video. python is a trap for complex web server. nodejs is better but we know "javascript"
@@AntonPutra Kubernetes solves the problem of inefficient and complex server infrastructure by creating a replacement system of equally inefficient and complex container infrastructure.
yes, zig was in 'debug' mode, which is the default, and i didn’t know that. i found a production-ready dockerfile, and i thought it was properly compiled, so i'm not the only one who made that mistake.
The Zig standard library sucks. And its documentation sucks even more. That's not, unfortunately, something new. The standard lib has always been an afterthought and will still be the case as Andrew Kelley doesn't seem to care about it.
well, I can only help if someone provides a well-optimized version, but again, performance is not the main criteria for why people choose one language over another
not really, i start with 20 independent client instances that generate load, then i increase the number of concurrent users for each instance by 1 every 30 seconds. so i start with 20 truly independent users and by the end of the test i run 4000
Zig is a pleasant language that is a joy to program in. It feels proper language, its approach to memory safety is more correct than Rust making it way simpler. I still not a big fan shorthand syntax but I can live with it. Zigg feels like a sensible refinement of C language for system programming. I like it, but clearly work is still needed on its stdlib for better implementations, being native compiled they should not see differences unless there is some inefficiencies in the library implementation .
I think the most impressive thing is that the Zig implementation were written by very small team if any, in a very short amount of time, in a language that's still very new. This to me feels like the real achievement of Zig, being fairly simple to use and write good code in. Rust is amazing, but it really takes some serious time to become productive in it, it takes even more time to understand how to write efficient Rust code. Early one I used to delay the difficulty by .clone() everything and .? everything or using RC everywhere. I think this confirm my intuition, Zig is going to be, if it keep on becoming more and more popular, the go to language for high performance software where safety is not necessarily the absolute main concern. Rust is going to remain the best choice to rewrite critical software in. I like this diversity, I think Rust is one of the best language to "Rewrite stuff" but not really a good language to explore, so that makes perfect sense.
It uses LLVM just like Rust, performance limit is pretty much the same until in-line assembly. I’d definitely expect performance hits to be a result from copying outdated C implementations vs creating custom implementations, which is what I expect the weird memory behavior is coming from too.
We should compare 0.6 version rust from 2016 to that 0.6 version Zig. Never ever i have written not a single line of Zig, but from halfway done lang compared fully fledged battle tested lang that was very impressive how fast Zig ran that C library. Id say Zig will literally eat C
