MMO Architecture overview with Unreal Engine 

yes I actually keep Runescape in mind when developing the server :D as well as some other mmo's such as WoW

IK you already have a blog but have you considered opening a discord server?? would love to chat with you since yyou already have a lot of resources i will most likely be following in the near future! i am trying to build something similar to oldschool runescape as far as point and click, many skills to train, and player freedom. but the setting and story is completely different. i already made a bunch of 3d assets, and have a musician friend who made some banger tracks, and we even have 2d concept art as well! but as we have no lead dev im gonna try and put it together haha@@unreal-dev-with-yaz

Hey :) This approach is not actually serverless, that’s something else too :D This is a horizontally scaled system (i.e. multiple server nodes can be spun up to cater for load) Vs vertical scaling, where you add more resources to single server instances Kafka is a key to connecting relevant nodes When you have more players join, you will have the option to scale Kafka by adding additional brokers to split the load and cater for it, it scales very well. I drew Kafka as one line, but underneath, its broken into multiple brokers If interested, you can learn more on Kafka cluster scaling here for example: developer.confluent.io/courses/architecture/cluster-elasticity/

I'm not sure about the UE servers running on docker and kubernetes, but all the other services will be able to :) definitely would be great if it was possible, may have to check that up. for the UE deployment in general, this will likely have its own separate pipeline and again will depend on what tools you use to deploy it, e.g. with PlayFab or AWS etc

@@unreal-dev-with-yaz I see thanks, do you think using docker would be even helpful or probably same effort than just using PlayFab and the other solutions?

@@unreal-dev-with-yaz Thanks, How would playFab play a role in this architecture? I'm currently working on the systems design of a MMO architecture and it's been interesting. Do you have a discord group of the likes?

@@devaereo Hey :) PlayFab can be used to host the UE server, such as the ones controlling mobs I may also consider using PlayFab authentication too, to handle player login, rather than building that layer in my custom server. They have other nice services & APIs that I will potentially explore

hey! :) well, for this one in particular, I think would be difficult to suggest 1 thing for me personally, it came from years of experience working in several different companies and getting involved with software architecture quite a few of my friends are also developers and we generally just chat about the new and upcoming technologies software architecture changes quite quickly so going and viewing tech conferences could be a great way of getting exposed to some of the new techniques and staying up to date there was a nice book on tech architecture that I read in the past, I believe it was this one: Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems but it's not specifically for game development then its about looking at topic specific things and seeing how it can all fit together, for instance see Networking for Physics Programmers: www.gdcvault.com/play/1022195/Physics-for-Game-Programmers-Networking this suggests that for motion networking I should use UDP, however for inventory management, I may still consider TCP for its reliability

​@@unreal-dev-with-yaz you are the man! sorry for the late reply but you went quite in depth! That sounds like the main thing I am lacking is experience I guess I'll start by just prototyping co-op networking games to understand and successfully implement this on a smaller scale before I fiddle around with Kafka etc.. I'll give myself a project and see how far i can get. UDC for motion and TCP for inventory definetly makes sense.. I didn't even think about different application of network protocols yet Anyways thanks again and I'll check out the link next

hey :) not yet, perhaps I will look to push something simple to AWS/Playfab - it can be quite complex subject :D

so unfortunately not out of the box. it would be possible to manually handle this and it can work, but it would require a bit of effort. potentially a feature that could be done in future :)

@@unreal-dev-with-yaz I'm looking forward for it. The GAS is so flexible and powerful, it would be insane replacing it with own code.

in long term I actually plan to introduce UDP also So the game will be split into several communication channels 1. before game session begins, i.e. login, character create/select screen. These will all be handled via HTTP this can also be useful when displaying stats on website for example as you can leverage same endpoints 2. most in-game logic I will leverage using websockets, so inventory control, starting skills and interactions, etc. 3. actor motion is currently using websockets, but it will become a bit heavy. I plan to extract and put it into its own service running using UDP for better performance inventory using HTTP/Websockets - the reason websockets are better is because I'd be leveraging the sessions notify functionality, rather than client polling for updates. (for example when item is dropped on map, or when you equip item and you need to notify other users to update the mesh) this is much more performant

Epic is releasing a new network model called Iris (still experimental) that will improve the number of clients that can connect to a UE dedicated server comfortably, however I have not seen official numbers as to how much of an improvement.

Hey :) So when it comes to players, the custom server will need to have authority. There would be 2 main scenarios: Client says: my new motion is Server responds: OK, passes this motion to listening players Or Client says: my new motion is Server says: NO, your new motion is , passes this motion to listening clients if different (either pure reject (last good motion) or acceptable motion relative to the requested motion) This validation is not yet implemented by the way. In terms of latency compensation This can be applied to motion as they are bound values. i.e. players cannot move faster than X units per second (configurable) - we know the timestamp of last (accepted) update, hence we know what’s acceptable range with new update, even if there’s frames dropped or latency. Bear in mind, skills that affect motion (e.g. teleport) will undergo separate, independent validations. Since this is something that’s not yet implemented, I’m also open to ideas, but I think this is a relatively easy and scalable way of handling motion updates

@unreal-dev-with-yaz This sounds promising and I'm very curious about how it performs once all that is done. I come from a similar dev background, you could swap Kafka with redis too just saying. I'm wondering tho if with this approach the latency is more than expected compared to a traditional setup. Have you tested latency even in a local ideal setup?

@@0x_Anakin Sounds good :) Btw redis not quite optimal choice against Kafka as you’d like a solid queue implementation (SQS queue for example should be better, I used Redis as queue before, its not bad but not quite as rich in features :)) I have in fact added redis as a shared cache in my latest PR though, so will be utilising it more in near future :) As I am implementing combat logic, I need to cross validate against other actors’ motion and redis will be great for sharing that context In terms of latency, of course it has to be higher than if you didn’t split the load with other servers, after all, there are more steps involved (you refer to Kafka overhead I assume?) However it doesn’t add much (benefits outweigh this significantly), perhaps I will create a video providing the latencies between msg sent on client1 to update received on client2 👍

Hey :) Do you mean setting this up in deployment? I was planning to do this closer to when an MVP is done (which is still quite far :D) But perhaps I can try do this after finishing the basics of the combat feature, as I expect I added most of the necessary components already

Thanks! It is definitely something that I’d like to cover particularly when looking at actual deployment (which may take some time to get to though :D) It’s quite common to try draw up the architecture diagram referencing those components to help understand which connections need to be made and it can also help with ‘seeing’ where the potential bugs may come from :)

hey :) there's some complexities in evaluating these things and will highly depend on your game I'm assuming you're referring to native UE replication each map can be run on an independent server, you can also split map into zones which can be run on other UE servers for more flexibility (e.g. check OWS: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-yRXzbaNT6_k.htmlsi=GITiGXNxtg9D5BaZ) the server does not load UI, so low poly etc won't have impact on the network load I'm not sure how you deploy but you can check cost calculators to get an idea of requirements and costs: playfab.com/mps-calculator/

@@unreal-dev-with-yaz Thanks!

hey :) I'm not going to have a complex debate in comments section, however if you think this approach will struggle with 100 players, then you've probably misunderstood the approach :D perhaps you will want to check this doc about how such an approach has been scaled to work with millions of concurrent users, albeit its not an 'mmo game' environment, but those are slow to adapt to new web technologies (this doc is referenced in the post also). www.confluent.io/blog/real-time-gaming-infrastructure-kafka-ksqldb-websockets/

​@@unreal-dev-with-yaz for what it's worth, I think the solution will scale to a few thousand depending on details around gameplay and how clustered players are. There are a lot of similarities to how Rift was set-up. However, Kafka is a peculiar routing choice and the JVM is going to have challenges with GC causing server hitches. My concern would be the smoothness and response of the simulation given all the work we had to do in Rift to reduce and speed up RPCs. With a huge pile of custom C++ a similar architecture got to the low thousands in 2010, with better hardware I would expect it to do quite a bit better depending on how you reign in N^2 from entitles within vision of each other. Also with current hardware you probably could manage to put more processes on the same machine to get traffic off the network card trading off performance for a larger blast radius when hardware fails.