Your methodology is excellent. I much prefer this to the gamified mess that techempower has become. Startup time/cost, container size, memory/cpu usage, baseline memory/cpu usage are extremely valuable to us.
Cool test actually. A few possible improvements except trying the AOT: 1. Try enabling a full PGO (tiered PGO is enabled in .NET 8): # Full PGO ENV DOTNET_TieredPGO=1 ENV DOTNET_TC_QuickJitForLoops=1 ENV DOTNET_ReadyToRun=0 2. Use slim app configuration: WebApplication.CreateBuilder() vs WebApplication.CreateSlimBuilder() 3. Try increasing the minimal number of threads: ThreadPool.GetMinThreads(out _, out int completionPortThreads); ThreadPool.SetMinThreads(100, completionPortThreads);
also the two async operations (await cmd.ExecuteNonQueryAsync() and await amazonS3.Upload(appConfig.S3Bucket, image.ObjKey, appConfig.S3ImgPath)) are running sequentially, while they could be started both together and awaited after they both have been started; sure, it's a small to really small improvement, but this is the point of knowing well the language and doing a test well: it's not easy *at all*
@@AntonPutra I actually think you tested it fine. AOT is only for startup time. Tiered PGO is the default, so it makes sense to keep it here. My one complaint is that if you want this to be a "real world" test case your code should use Entity Framework as that's the most standard thing. I took a look at the code and you're doing it with Npgsql's built-in datasource. I don't know anyone who would do this in a real world application. There's all sorts of caching that you're missing from Entity Framework. Note that I think the way you did it is actually faster. EF has overhead. But I would still very much like to see that comparison instead.
None of these are needed, with vanilla configuration .NET is strictly better than Go. Anyone who ever looked at Go's compiler output and GC throughput knows that both are underwhelming. Note that for *small* deployments like these using NativeAOT is a good idea. .NET 9 also enables DATAS by defvault which is a new GC mode for more dynamic scaling of active heaps and their sizes based on load profile. This massively improves low heapsize scenarios like this one. Generally speaking you have to understad that for high throughput workloads 256mi is anemic. Bump it up to 512mi and you will see a completely different picture. Same applies to more core which .NET's threadpool and task system can utilize much more effectively than fixed runtime threads setup in Go. Lastly, I suspect the culprit here is S3 SDK, as it has quite room for improvemet and Amazon posted like a week ago that the have new preview version which is faster and has much less wasted allocations. The open question is also about Go using connection pool while .NET seemingly avoids opting into that, which would impact the behavior significantly and make it apples to oranges comparison.
@@AntonPutra startup time will increase by a miniscule amount as it needs to decompress before running. For this use case, using UPX is a good move as it will reducesdownload time.
@@ArnabAnimeshDas got it. unfortunately, it's hard to measure boot time in Kubernetes because the minimal interval is 1 second, and I want to focus these benchmarks on Kubernetes.
Also the images are already compressed by the container registry and runtime. So double compression is not all that effective in general. Something like the trimming and Native AOT in .NET does help a bunch though. Go already does a fairly okay job.
The idle memory usage difference has its reasons. By default dotnet runtime just doesnt release its reserved memory unless there's memory pressure. If you want it to release it more aggressively, you should run it as a desktop workload instead of a server workload (default).
Oooo reasons, reasons, reasons, come one, face it, Go lang with simpler approach without OOP studpidity rulset has shown it can perform. We kewn all a long, that is a case. Face it, Go lang is the new "C#" as new popular language for beginners. Modern C# is just what C++ was in the 80s. Even C# author said, C# is more C++ replacement that Java replacement.
@@Sam-gd4xp oh, I like golang as well and I've used it alot ❤️ But personally I wouldn't have courage to start developing a large scale enterprise-y system with it. Everything being so simple and imperative means there are more statements you'll have to manage, which means you'll have more state to manage, which means there'll be more chance for bugs. Golang's range requires much more statements, state and ceremony than C#'s Select(...).Where(...).GroupBy(...).Join(...).OrderBy(...)... I know there are some more functional slice utils, but still a lot is missing. C#'s LINQ (with and without EF), expressions and way superior generics is something I personally couldn't develop a large scale system without... But I do love to use golang for more algorithmic stuff and I'm also super interested in how golang's generics and itetators will turn out and improve 😋
@@deado7282 it is an ok default if you deploy to piss weak anemic container configurations and then have to scale with replica count and nodes, because go cant optimally do otherwise, instead of scaling up individual container resources, the ecosystem is in a different place now anyway
@@deado7282 Fiber also limits you. That would be far more fair. Not to mention that there's something bad with the C# build, because my projects are way bigger in code but way smaller when built. He did something bad.
Once again a spectacularly well thought out test. We can say that C# wins a proud silver medal in this head to head :D Would you be interested in a fresh comparison of some of the different javascript runtimes, like node.js, deno and bun?
At 8:01 you mention that theres a significant higher latency for C#. It looks worse than it actually is, since the graph doesn't start at zero (unlike the other graphs in this view). The difference is still significant, but being mindful about these things adds clarity for the viewer and potentially saves you from a wrong conclusion in the future :)
This is the first test video I've seen of yours and I love how thorough you are, from methodology and tools, to the results. However, as someone who is color deficient, I cannot tell the difference between low-contrast colors easily or at all. For your next test, please use high-contrast colors for all test subjects.
C# is only faster when you start using different types of objects like structs and ref structs. In a minimal API with all settings set for speed and memory efficiency, you can produce an api with complex business logic that never allocates any heap memory. This is something that is, in practise, done frequently when performance starts to matter. C# will beat Java that way. But there are many things that Java is just faster at. Datastreaming is one of them. Although the dotnet C# team has recently implemented very efficient ways to do so, Java doesn't just use Java under the hood. It interopts with C and C++ libraries when speed is really something that starts to be a problem. And even the most efficient C# code can't beat that. C# as a core language is in most cases faster when all language features are utilized. But most programmers won't go there. And imo as a C# guy, Java is in many common cases just as fast, if not faster. C# wins in most cases due to the simpel fact that it gives you more control over memory. Thats it. But sinse not a lot of devs will use that advantage, Java will be faster. And even if a C# dev optimizes their codebase to limit allocations, the second a Java dev starts using native libraries it's game over. C# as a language is faster due to its nature. Java will, in general, be faster due to it's optimized libraries that don't even have to use Java.
My thoughts exactly. Something in the setup of both versions are bad, neither go and c# seems to utitlize resources nowhere near 100%, yet they all start failing request. Maybe they can't see the resources they have available or similar and they are throtthling themselves by using less thread/connection/something that they could. Go (fiber) ofc will win anyways, but based on my very similar tests, the difference is only 10-15% in requests per second and this was with net6, not net8 (which can be significantly faster).
@@metaltyphoon yes, which means the margin should be even smaller for him compared to what I measured in net6 era, since net8 is faster then net6. My tests were capping cpu, but here we can see both frameworks failing without using the available resources, so something seems off.
Very informative as always, thanks also just a thought, maybe revisit apache vs nginx. mpm_event module for apache with default config sounds pretty strong I'd like to see how they would compare now keep up the good work
no vanila kubernetes cluster with containerd runtime. 2cpu and 256mb memory - github.com/antonputra/tutorials/blob/main/lessons/202/deploy/cs-app/deployment.yaml#L26-L32 have you tried it in k8s or just standalone? default file descriptors 1024 could play a role as well
It seems you didn't read any of the comments from the Go/Java comparison, the Go binary can be further reduced by removing debugging symbols with the linker options `-s -w`. You are also using an outdated AWS SDK.
it is in this video, and the difference is only 6 MB - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-56TUfwejKfo.html I'll update sdk in the next lesson, but it does not affect performance it's just a wrapper around rest api
@@VoroninPavel Well. Memory consumption. This interpreted vs. machine code discussion has been for 25+ years since Java became popular. JIT code can be fast, but in performance critical applications, machine code wins always. But is the pain of machine code worth of the sometimes minor gains is completely another thing. This was just pure performance. In practice the question is broader techno-economic question.
Which memory usage do you measure? Reserved or Committed? .NET GC eagerly reserves a lot of memory upfront, but does not commit it until necessary. in .net 8 they introduced Dynamic Adaptive GC mode, but it's on by default only for AOT application. DOTNET_gcServer=1 DOTNET_GCDynamicAdaptationMode=1 Or System.GC.DynamicAdaptationMode
well the same as kubernetes uses to oomkill my applications :) to be more precises container_memory_working_set_bytes / container_spec_memory_limit_bytes
I had a look at the source code a and configuration after the last video and I wondered why you used minIO. Thanks for giving the explanation. I did not know it supports (partially) the S3 API. At work, we have an application that uses it but more as a cloud agnostic object storage. The use cases here are more realistic than the tech empower benchmarks ones. The none fixed size of the DB pool for DotNet impacts the latency, opening a connection to the DB (especially if TLS was used) is expensive.
thanks! minio can also be used as a replacement for hadoop for some data lake stuff, etc. It uses significantly less disk space while supporting the same replication, but it's harder to scale. If you are on-prem, it's a viable alternative. I agree with the pool size, just the defaults, no more, no less.
The comparison seem kind of fair, but they're room for improvement, but i think that a good general "default" setup representation. But small precision (with a little history): It's not dotnet api or dotnet core, it's the aspnetcore framework just like gin/fiber are web framework that you're testing against, they may exist other web framework for .NET, but i agree that is most popular and used one. Nowadays it's just .NET (or dotnet) no "Framework", when we say .NET framework, it mean the old .NET Stack that was only working/designed to work on windows (yeah.. yeah...I know about MONO, but that out of the scope for this simple history) dotnet core was/is the rewrite of the .NET stack from scratch. Microsoft choose to call it dotnet core, because at first it was only a subset of library (the core part), and apsnet core was also the rewrite from scratch of the web framework stack that only supported a subset of the aspnet MVC api, why they didn't change aspnetcore to aspnet, because it's already exist (they were (as far as i know) 3 Microsoft's web framework: ASPNET, ASPNET WebForm, ASPNET MVC (with 3 version of that framework is recall correctly)). So why did they change .NET core to .NET ? i don't know the real reason, but i have some idea like stopping people to not use the new improve version of .net because in their head .NET framework is the supported version of Microsoft versus .NET Core was not as feature complete and not maintainted as good by microsoft because it's now not part of the automatic windows update. Fun fact that most people don't know, but a certain point in time, you were able to start a apsnetcore project using the .NET framework and this was not a hack, it was a valid use case, but today the required version of .NET for aspnercore is too high and .NET framework support was dropped, but switching from aspnetcore using .NET framework to aspnetcore using .NET Core (now know as .NET) was really easy.
Thanks for the clarification. I didn't have any experience with .NET, but someone suggested trying it out with Minimal API, so I did. I'm learning a lot from the feedback I get, lol.
Well, I used the available documentation that most people would use to compile and build Docker images. I'm pretty sure you can optimize both Go and C# further. But I get your point, and next time I'll test with AOT as well. If you have anything that could help me optimize .NET, please share it, and I'll definitely use it!
@@AntonPutra unless you change the algorithm you uses, there is not a lot of options that can be used to optimize your Go app, you only have the PGO (which also can be used with C#). In C# you just have to compile the program like Go to get a better performance.
@@AntonPutra luckily when you create a .NET solution you have the choice of selecting web api with AOT, which is a very good starting point, i strongly recommend you to check it out as it has a huge performance difference and the boot time is reduced, the only downside i would say is the image size. I made a POC for my company, it was a stress test of 1000 concurrent users making 1000 requests each and the memory stayed at 30mb with cpu to 1.5%
Sounds reasonable as a whole. .Net isn't particularly efficient unless you do a LOT of massaging. Then again, like ThePrimagen usually puts it, do you have more microservices than users? If I'm not doing high performance, I'd probably sacrifice efficiency for "ease of use and support", especially in a MS centric environment. If I am, then "who cares about your experience, fps for life" (old competitive Quake joke, where you'd sacrifice ANYTHING for more fps because max_speed was tied to fps).
Автор плодит холивары. Почему-то он взял AspNetCore старой версии, не запублишил(то есть нет компайла в AOT нейтив), без тримминга без ничего и сравнивает производительность с Go(который по умолчанию AOT), ближайшим аналогом которого для тестирования подобных задач больше подошел бы Blazor из мира Dotnet.
@@akknaodinden странно. Ютьюб как-то через раз ответы отправляет. Писал, что интересно еще сравнить производительность, когда не CreateBuilder, а CreateSlimBuilder
Hey, great job! One comment I have is: a lot will come down to how good the Amazon S3 and Postgres are optimized, and in DOTNET MS SQL and Azure Blobs will probably be a priority. Could you do a dedicated test for streaming back local file by id (from disk)? That would show how much HTTP connections specifically each app can handle.
interesting, looks like they integrate cloud sdks to the standard library.. so it sort of wrapper around all clouds? how well it is supported? I'll take a look...
Basically, if you want something that works and easy, go with Java, C#, Node. If you want improved performance and efficiency without too much work, go with Go. If you want the most performance, speed and efficiency you can get, with the downside of complexity, go with C, C++, Rust
Rust is not ready for the cloud, with poorly maintained SDKs, and many of them are just slower than Go implementations. But in theory, Rust should be second after C.
I have few tutorials that can help 1. github.com/antonputra/tutorials/tree/main/lessons/135 2. github.com/antonputra/tutorials/tree/main/lessons/136 3. github.com/antonputra/tutorials/tree/main/lessons/137
@@AntonPutra while at it. I see more an more job postings for python django, fast api, flask. I would expect these to perform "bad", but so many people choose them. I wonder if they are good or not
@@jozsab1 It's a very reliable and well-tested framework. We've been running Django as the main API gateway at the company where I work for the last few years with no issues at all. However, most of our other microservices are written in Go :)
Uh... There's a problem with your C# compilation because i have an API + Blazor + MAUI project which is 107mb. If you are only using one API it shouldn't even be half of this size.
Your videos are very interesting ... Keep doing it. I have one question. Is it possible for you to create videos for gin, fiber and echo frameworks? thank you
well i have all source code, including terraform, helm charts everything even dashboards in the repo, link in the description - github.com/antonputra/tutorials/tree/main/lessons/202
Yeah. I think it's no surprise that Golang is faster - but .NET is well established and really powerful. And in real use case, it's still REALLY fast. Performance should only be the deciding factor when you REALLY need it. Majority of projects don't. In most cases, there's more to it - from ecosystem to team/personal experience and preference.
@@Sam-gd4xp You're either trolling or you simply never comprehended the idea of "different tools for different jobs". I've never seen anyone mad that Golang is faster. Everyone knows it, everyone accepts it, and people happily pick it when they need it. Both languages are tools, and only immatures do these kind of "performance wars" and call each other mad over it.
@@TehGM if you're the head of small startup and small firm, and you learn basic accounting and project management properly, you will see. That's why Google push so hard for kotlin multiplatform bro. But will it come? After so much Java VS kotlin legal battle
@@AntonPutra Apologies, let me clarify. Would it be possible to conduct a test that includes django-ninja, Django REST Framework, fastApi and a Golang service? Just a reminder: people, please don't take these tests too seriously. Even if Python turns out to be five times slower, it's often better to use languages with faster development times. In 80% of cases, development speed is more important than performance.
@@mikemoore667 agreed. i like using Python as well, you can quickly create poc. a few years ago, someone in our company used django to create a poc for the main api gateway, and we're still using it 5 years later. :) no one wants to rewrite it in go cause it just works
@@AntonPutra because Dart serverpod is new framework, and we wanna see how it compares against popular alternative like Gin. Comparing it with Fiber or std I think is useless, because we already know that they are too fast :).
4 CPUs and 8 GB of memory. From now on, I'm only testing apps in production-ready EKS clusters to make it similar to what we actually run in production environments. I use m6a.2xlarge EC2 instances for the nodes...
Is it fair to compare a web framework specifically designed for maximum performance with many limitations and unimplemented web features to a standard enterprise application framework that has everything you need? At least this doesn't exactly match the title of the video.
I'm curious too and am trying to add more metrics for the next tests, maybe even open file descriptors, etc. - github.com/google/cadvisor/blob/master/docs/storage/prometheus.md
Awesome as always thank you. What do you think of including database traces? I guess for this you will need a little bit complex database queries. Maybe you can include some database triggers to achieve that. There is not so much info about database tracing 😢
well i have golang tracing tutorial on my chanel with open telemetry and grafana - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-ZIN7H00ulQw.html but it does not help to measure performance, rather then just help to debug
11 years with .NET at my main jobs (I'm a solution architect), 6 years with Golang (side projects for other clients' companies). I think .NET is more suitable for fast development, quick adaptation to changes, and Enterprise-focused products. Golang is better for cost-efficient projects, but requires more maintenance.
It's really not surprising that go would be faster and use less memory than .Net which has a big runtime. Enterprises use .Net over things like golang to cut development time. Two things I'd like to see .Net AOT vs .Net (no AOT) and .Net (AOT and not) vs Java. Also with AOT it should be possible to use something like an alpine linux image with no .net runtime installed.
AOT will not help you with performance. LOL And that is quite optimisation, so Go lang can also use more optimisation switches. And.... Go lang would be faster ;) Well we all knew that C# is fazing out, OOP and Functional rulset is to much of a heavy load that is slowing down .Net execution. It is becoming relic of the past. Did you knew that C# is trying to solve boiler plate coding. And because of that, language has over 100 reserved language words and compiler and compiler tokenizer needs to work hard and be compicated. Java and Go lang use around 50 reserved language keywords and have more solid priciples that .Net (C#). Sometimes C# can solve simple value addresing in 5 to 10 syntactical ways. Java and Go lang only in 2 to 3 syntactical ways, which is enough.
@@AntonPutra I'd watch it if you did make that video. It would be very interesting to see how much overhead or not is removed by compiling the whole app ahead of time and eliminating the runtime. Java has jaotc but I don't know if it ever left the experimental phase. Android has had ART for several years, which is also an ahead of time compiler. AOT must do something significant given where it is used today.
@@AntonPutra youtube is acting weird, don't see my reply. basically hperf, or check awesome-swoole to find popular frameworks. supposedly fomo is the fastest.
@@bibahbibah5108 Golang is not like Rust, its a very easy language to learn. Even easier than C#. So if ease of use, development speed are metrics for choosing a backend language between C# and Go, Go wins in my opinion without even considering performance.
Sorry, but your tests are still far from being a real application, even considering "Test 2". The test apps do not have business logic and even simple logging (not to say structural), which can reduce performance several times. Also, configurations in the .NET app are not "real", as well as working with the database and AWS services. Usually, it would be services registered in the DI container and used in other services or directly in controllers or methods from minimal API. If for some reason you drastically need the lowest startup time, you need to use Native AOT compilation for .NET apps.
i try to keep it as simple as possible, i'll include simple etl or just kafka consumer/producer in the following benchmarks, but thanks for the feedback!
Don't be pisst off because you bet with your knowledge on .Net. We knew all a long that .Net was a scam. Even Google knew it! Why would Google bet all backend code to Go lang? It oop was really good and far a head as oop programers are saying, then Goolge would have bought Java and not pass it to Oracle for expensive licencing :D
@@AntonPutra Yes, it is still maintain and very popular in product base company and in my company, mostly services is written in scala using AKka Http and stream. some services/in house project, is in rust and go like realtime database for metrics is written in rust.
@@AntonPutra Why i am suggesting for comparison, as go uses green thread and same is like cats effect so battle for performace will going to very close in my understanding. for memory comparison will not matter as JVM uses all available memory with time so ignore that.
@@IvanRandomDude its not a lie, the configuration that microsoft did is clearly not as the one used in this video, like AOT and so on… so i would not call a runner better than an other if one of them dont have running shoes 😅
