Another double edged sword of microservices: Teams own services. So, changing your own service is pretty easy, but changing other people's service's can be very hard (if there isn't an explicit push for a collaborative culture). In a monolith, everyone feels like they softly own parts of the code, but you can change anyplace (say, during a refactoring). Try refactoring some global stuff in a business using microservices... There are two ways: sending MR for every repo yourself (which is not trivial, because you don't even have access to them), and people can and will ignore you. Beg people to do it, which is such a failing strategy because they don't have any incentives to do it. So you probably need some strong company process / culture that enables global changes (I've never seen it in real life).
The way we do it at my work is that anyone can make a change to any service but it has to have tests and it has to be reviewed by the owning team. If e.g. someone creates a pull request without tests, you just reject it until there are tests. If they don't understand the consequences of their change you talk to them and discuss a better solution. Most of the time changes are simple enough to be accepted though. Consumer driven contract testing is also useful. If you have to submit a PR/MR for every repo then you probably need to rethink why there are things repeated in every repo e.g. could they be packages or yaml templates etc.
I work in a monolith and I don't feel like I own any part of it. even the parts of it I built from scratch have to play ball with the (in this case extremely stupid) architecture someone else made up years ago since a total rewrite isn't in the cards rn.
yeah making MRs for other people's stuff isn't going to happen. The only way for a global change to happen is serious carrot/stick stuff. either there is some shared facility people want to use where they have to upgrade in order to make use of it (like if you want your logs to show up in splunk they're going to have to adapt to the prescribed format), or like with vulnerabilities or cloud migration or whatever, there is a big dashboard that shows who is out of compliance and the management is ready to mete out punishment.
if your team owns the monolith, and it grows and you now grow to 3 teams to own it all, how do you transfer ownership of various parts in a clean way? if you need to make changes with microservices and other teams own that stuff, these are external dependencies you should get buy-in (and coding partnership, if needed) for upfront, not at the last minute. I have seen situations where monoliths are owned jointly by a bunch of different teams... and all MRs get blocked until some integration tests get fixed. Most people sit around waiting, and the second it's finally fixed (which can take weeks), everyone jams their MRs through (after resolving merge conflicts) and the tests break again, and MRs get blocked for another couple weeks. We even set some things up where one specific dev had to approve every single PR (100+ total devs) for a while to try to stabilize things. It was a huge mess. That said, the places I've worked at that used microservices had much better devs and much better design/code/testing quality than places I've worked at that used monoliths, so it's not a perfect comparison.
This made me think of Conway's law - the structure of a system reflects the structure of the organization that makes it. They had to split into micro-services because their organization became too large.
We don't even do Microservices the way they are supposed to be done, just like OO programming got out of hand with wanting trivial classes that just serve one call over their entire existence
say a group of functions in total processes >5 inputs and outputs >5 outputs - creating a class to manage that, and then getting the outputs by calling getOutput1(), getOutput2() is a cleaner way to do that in my opinion
This is a cool video! I feel like everyone has read that blog (independent of your channel) or a blog similar to it, and having someone come back and reflect on it is really valuable.
A really nice point mentioned : there has to something in between, you can’t only have two options : monolith or a microservice. This is probably done by a lot of teams where they carefully break their overall app. They just don’t go say each module needs to be a separate service, rather how close are these two modules to each other and can they perform better together. If there would be tool for this, that would be great.
That would be the same as having a service separate to each other but shares the same db. Like there’s a separation of logic but not of data retrieval. Otherwise you have a monolith if you share functions writhing the same project or services if they have to do a rpc’s for function communication.
Idk if microservices are “better” overall but they are painful to develop: Setting up all the individual repos as submodules in one parent repo, dealing with all the messaging, individual deployment etc
Yeah, they're not meant for a team of a few devs, they're meant for very large teams as discussed in the video, and it makes sense for those large teams because the overhead you're talking about is affordable for them. But it does make it hard to learn them on your own without being in a company using them as you have to deal with that overhead 😅
I'm actually a proponent of the monorepo with many services. The trade off is you need something like bazel for build and unit testing which has to be maintained but if you have a lot of services could be a net benefit
@@yassinesafraoui no I think building and testing can be harder on microservices because if you have a large dependent graph getting all those spun up for integration testing is harder than a single binary or coping all your interpreted files to a docker container or server and running a single interpreter over them. The monorepo is just a comment on how you organize your microservices. E.g. separate services doesn't have to mean separate repos. Assuming your company has the people or expertise to maintain a build tool like bazel you can have your services in a single repo and run build and test together. This saves some overhead of having to maintain N repos where N is the number of microservices. I've only had bad experiences with submodules and subtrees so I stay away from those like the plague.
Simplicity is always the best, stick with monolith until you actually face performance issues. Complicate things up only when it is absolutely necessary. In terms of OOP only create hierarchy when you absolutely need that inheritance otherwise keep everything as dumb as possible.
I agree with almost everything being said here. I think there needs to be a reaction to the mid 2010s of microservice everything which caused this distributed monolith problem which is net worse than a regular monolith. Ultimately you have to be critical about the workflow and if you always need service B for service A and service B isnt being used by something else then they should definately be the same service. I think rather than counting total microservices i think people should count dependent microservice graph. If you need to call 100 services todo a single workflow that should be a big red flag. I dont agree with what he is saying about testing but get the sentiment but i think it comes down to internal engineering culture. IMO assertion quality check is called peer review. If people are writing or approving code that is blatantly not asserted on correctly that is a problem. What allows microservices to scale is contracts what helps ensure your contracts are you tests. So that can contribute to the distributed monolith if you dont have strong contracts because you dont have strong tests things can degrade fast. Also contract testing for every team i have been on has been a game changer in terms of being able to effectively "integration" tests on a service gives the developer or development team a higher degree of confidence in their change before it goes into a pre prod env for full integration testing. I think the TLDR of this is in SW engineering theres no free lunch. With microservices you gain deployment velocity and workflow isolation but that opens up a bunch of other considerations around how you think about persistence, curcuit breaking and tooling.
Microservices should be Edgecase, but this somehow becomes a pissing contest, adding Complexity for the Sake of proving Intelligence. Im guessing thats why Shipping Products has become almost impossible
The comment towards the beginning, about "no one" taking down a system to deploy anymore, shows how insulated folks like this are to how many enterprises run even today. One moral to this story is, understand who's telling you how you should architect a system, and what your constraints actually are, before following their advice.
That is a fair callout. All of the systems that I’ve worked on in my career have been internet connected and expected to be online always. But there are plenty of systems that are not like this, and for those you can make different tradeoffs.
Mutation testing can measure assertion quality! It has its limitations, and can be slow (especially if your tests are slow), but it's really powerful in certain contexts! See e.g. Stryker Mutator (supports JS, C#, Scala).
Apart from the techincal knowledge, we really see how to communicate well with respect when talking to a senior. Which can be applied in an interview as well. Thanks a lot!
Correct me if I am wrong, but basically: - Normal Monolith = Drinking Coffee in the morning and eating mostly healthy - Microservices = Taking Meth and eating somewhat healthy - Distributed Monolith = Taking Meth and being very unhealthy - Modular Monolith = Eating Healthy and staying fit through discipline for natural energy
Interesting talk. It is always important to keep in mind that principles and what is taught is always going to be ideals, and often always have idealized solutions and issues as examples. So, in the end it is always just guidelies that can help you think about how to design/architect or develop, and if there are good reasons not to follow those ideals then absolutely you are justified in doing something else, which is what happens as soon as any ideal meets the real world, at least to some extent, depending on the resources you have.
Awesome video, amazing to get a follow up to your video with the guy who left a comment noting he was a lead engineer at Door Dash. Really interesting hearing his take on things.
We are an isv for telco industry and our customers explicitly require us to have microservice architecture in their rfps... So it got to a point where their enterprise architects who probably has not developed anything for decades are trying to architect our products. This is a totally crazy situation. It got to the point that microservice is a synonym of a best of breed architecture even though it does not work for some problems at all.
I’m having a problem where some microservice owners take too long to finish work. Not every programmer develops at the same speed. I provided that person help and he still remains quietly working slowly.
Please continue this type of videos !!!! I hope you turn it into aserie where you bring distinguished engineers and talk all things engineering and management, for example one interesting topic is did they integrate ai in development workflows ? How ? What impact it did ?
I think it's missing some points: * Compartmentalization - one feature could fail, don't take down everything * Error handling - sure, everyone in the call chain is getting paged, but that call chain is normally for a subset of functions, vs everyone who works on the monolith It's also possible to create microservices without network calls. There are frameworks like vert.x which basically build a monolith up from individual smaller components (best of both world).
It seems that a lot of the issues caused in the software industry are by companies that needs to grow as big as possible as fast as possible and honestly warning signs are popping up all over the place that this is probably not the best way to do things if you want them to last.
What about being able to code up a service in say C#, another in Node, another in C, or Java.. That ability to switch is a freedom to fast but sure that also means change is owned by those teams that built it. It's similar to how our body's are built in a microscopic level. Each cell has it's area of responsibilities and that expands to organs to the collective organism to a society of collaborating organisms. To Countries and world politics. We are a real life implementation of Micro Services ;). Imagine if everyone spoke the same language, the same culture. The Diversity is what makes things work. Aka Micro-services :)
Reasonable but missing some considerations: Is functionality written in the same language? Are performance and security postures skewed heavily between parts of the monolith?
If you create micro-services to make teams more independent, why do you need 500 services? If you have 12 teams of 20 people each, can't you just create 12 services, each containing their own sets of modules?
On unit tests, I don't care about the number at all really. Of course bad coverage implies something is missing. But we have staff devs in my org just mocking and patching the hell out of everything, and doing "assert is not null" or "verify did call X function." Straight up heard these devs in meetings saying they do not care about software engineering principles. They have "99% coverage" but it's all fake nonsense.
Also lets be real i dont know the size of doordash. Some codebases are over 15 millions lines. Try building that and releasing it everyday without more insane tooling such as complex build systems like bazel
Do this one feature and you need to modify in 3 different services. you need to sequence the deployment of the different services. sharing databases. I dont want to be that guy, but this just seems like distributed monolith right?
Interviewed with Doordash recently. They were asking questions like "how did you help minority people at your workplace?". The systems design rout was really unpleasant as I was constantly interrupted lol
So many theories on how to scale software, OOP, microservices, etc... Literally all of them suck at scale. I don't think these theories are objectively terrible, I just think software sucks at scale.
Sound like too many microservices. You should avoid calls between microservices. Architect around that. Instead of making calls between services in some situations it’s better to replicate data as long as your fine with eventual consistency.
micro = 1/1000000 so theoretically it would be u need million microservices to make up one service i'm more of a fan of approximately deca services (patent pending :3) like 1/10 of a service seems like a good size 🤔
So I am solo dev building a system that is akin to ERP with bunch of features but on the small scale (car audio equipment for my uncles business). What benefit could I possibly get from microservices if I for example use a vertical slice architecture, properly separate concerns between features and then extract common reused code into generic interfaces/classes/methods that can later be turned into a NuGet package? Its much easier to deal with monolithic codebase on this scale, where at most, EVER, we might get like 200k requests PER DAY.
If you don't have a large team, I don't think micro services are for you, that's what I understood from the video. But still you need some structure in your codebase since it's a large project
Sound like too many micro-services. Instead of making calls between services in some situations it’s better to replicate data as long as your fine with eventual consistency.
From where I came from, microservices and serverless are the two most frequently quoted thing by fake paper architects to impress clueless bosses armed with gartner quadrants
I do. I use .env to point to database name and FQDN. Don’t replicate DB tables to different databases. Keep it simple. Microservice SQL should specific to a domain. Copying domain data to another domain doesn’t make sense to me.
ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-zf5HH1BPN4o.htmlsi=mUZ9S7cyjY-tLKKi and christophermeiklejohn.com/publications/cmeiklej_phd_s3d_2024.pdf
