DRY Software Patterns & Microservices 

"My general rule of thumb: don’t violate DRY within a microservice, but be relaxed about violating DRY across all services. The evils of too much coupling between services are far worse than the problems caused by code duplication." -- Sam Newman in "Building Microservices"

I liked this phrase so much that I had to register in the comments: "If I find my tests difficult to write, for whatever reason, I know I have a problem with the design of my code"

"Just because you write it in C it doesn't mean that you have to write crap code" -- my thought for this week

"DRY is a useful guideline and a rotten rule" You just put in a sentence all my feelings about DRY

"Just because you're writing in C it doesn't mean you have to write crap code" it's this kind of aside that makes me such a huge fan of Dave

Everytime I watch these videos I realise I'm a senior dev. It's been really good for overcoming my imposter syndrome. Which I think is largely due to how quickly trends change, and how aggressively people dwell on specific technology choices.

The subject is very important. In our company we've resolved the problem with templates. When we need a new microservice that needs common stuff like logging, metrics etc. we create it from a template. So when we change something common we change the template, and all next services will use the new one, with new features (and bugs), but all existing services aren't forced to be changed.

An absolute must watch is Ben Christensen's "Don't Build a Distributed Monolith" talk. He addresses this exact issue which gives you all the downsides of a distributed system AND a monolith but none of the benefits. 1. Don't put shared *business* logic in a library that is then pulled across your services, rather create a service that contains that logic. 2. Don't *force* consumers to use an "official" client to connect to your service. Expose contracts (not a lib) that can be consumed by any stack.

Very good treatment of this challenge. I’m the lead architect of a SW platform for deeply embedded systems and, if I translate the terminology of your examples from Web Microsystems to embedded platform modules, you’ve just described one of the fundamental challenges of my career. And, it has often been a bigger challenge to explain the trade offs made than it has actually been to make them. Many intelligent and skilled engineers wield design principles like DRY as a weapon, using it to attack any trade off (and it’s proponents) whenever it is inconvenient, without consideration for longer term effects. So, thank you for having the courage and intellectual honesty to challenge a design principles that is often taken as sacred. I may send people to his video when I’ve hit the limits of my own ability to explain it. :-)

Dave's t-shirts are always the icing on the cake

Keep shared code in an external library and simply follow the open/closed principle. The library is open for extensions, but closed to modifications. Updating a library should theoretically never break existing code, although this does happen entirely too often in the real world, which is why I am always hesitant to using third party libraries, but that is a different topic.

I wish this video existed 1.5 years ago. Spot on with the problems we had then. :)

While debugWarning() is a good *specific* example of when to discard DRY, it's a *bad* example because that type of function is a good case for DRY. Shared libraries are helpful when they provide common functionality without increasing coupling, where two services using different versions won't cause problems. Think of a JSON library. JSON has a fixed spec, and all implementations should implement that spec faithfully. If one service uses a buggy version that doesn't affect its corectness, that's fine. The key is that one service's JSON output must be parseable by other services. There's no reason to duplicate that code across services because it's stable. While DRY is binary, DRYness is a spectrum. Choose your level of DRYness based on the context. Within a service, keep DRY as close to 1 as possible, but slide closer to 0.5 or 0 if the costs outweigh the benefits between services. Also, the importance of DRYness decreases with the DRY code size. A shared library consisting of "const double = x => 2 * x" isn't going to win you any favors. 😆 Great talk!

I have been thinking a lot about this problem the last days and I am a little proud to have come up with the same conclusion

Good advice and essential for successful agile delivery of multi-team efforts. There is another adjacent problem with DRY; more a result of practice than concept. Dry is a concept that is so deeply ingrained into the psyche of developers that any whiff of similarity gets packed away into a DRY refactor. Sadly, this often leads to maintenance issues when DRY is the leading construct as the code must introduce more and more special cases to address differences, or forces awkwardness and, possibly, type safety issues as we torture the code into calling the DRY, refactored, code. Like so many skills, mastery is difficult. Knowing when to use DRY as a best practice and when to break the rules and repeat yourself is a key quality to remaining most agile while delivering quality, maintainable code.

I treat each microservice as if it were written by another company, on another planet, in another language and the only way to communicate is over the network.

Awesome content! You really discuss the most important subjects everytime

Excellent video, searched for something like this and ended up with a Dave Farley video of course. It is why I always say "the code may seem the same, but it might not always mean the same".

The clarity and quality of these videos is over the top!

Clear and valuable presentation. Thank you!

I think the biggest problem with DRY is that people don't understand what "repeating" means in context. It's _not_ about the crass mechanics of text looking the same in more than one place. It is about what should happen when you change the code: Do you need to do it in more than one place, then you are repeating yourself. If each place should change independently, then you aren't repeating yourself, even if you have the same piece of code repeated four hundred times. Taking your "debugWarning" example: are two identical implementations of "debugWarning" in different services "repeating yourself"? The anwer is the same as to the question: "Should adding logging to debugWarning in service A automatically add logging to service B?".

Excellent video. Everyone who thinks they have deployed Microservices should understand this.

Great stuff! Just came into this DRY vs coupling in my project.

Well in the context of microservices, you can still DRY by putting the "duplicated" behavior in a library that is linked by the various microservices that need to duplicate. So the coupling is not between the microservices, but only a compilation-time dependency.

Great video, really good reminder that context is key not just applying principles blindly.

Sidecars as a pattern are an important tool to have in your toolbox for DRY, since they are independently deployable. I would have liked for you to mention it and compare with the shared library approach. Everything you said is spot on, though. Keep up the good work.

Was literally just thinking on this. Great video!

Don't Repeat Yourself: For Team A the term yourself means Service A and so on. If service A and service B implement debugWarning, neither of them repeats itself. I quite like your lessons, Mr Farley!

One of the best channel for me...many thanks

Excellent video! thanks for sharing!

Dave I'm not sure I agree that treating the shared function as a versioned library makes the code not DRY, at least not in the sense of the problems that are typically created when code is not DRY. As you said, the big issue with having copies of the same behavior scattered around the code is that small changes to each copy add up fractally over time, but that is completely mitigated if any changes to that shared code happen in a linear, version-controlled fashion that can be adopted by the calling code at the latter's own pace. Of course it does create some overhead on the calling services to keep up with those changes over time, but that cost is already a given with proper external libraries (and the underlying technology) anyway so it still seems like the best of all worlds - especially since the shared code can use semantic versioning so that non-breaking changes can be adopted automatically by the calling services via configuration.

Wow, the bit about design and seams being where real skill in software engineering shows was great 😱 Thanks for another great video, Dave! I’ll make sure to keep these topics in mind on my projects 😤

Another great video, and personally feeling super lucky as this came out just when I was struggling with exactly the same issue in a project. Thank you so much for sharing your wisdom. 🚀🚀🚀

It's a fascinating problem. I often think about the data that travels between services, through those contracts, from one service to another. In this case, we're not talking about sharing implementation but a common language. It suffers from similar tradeoffs: how much is shared, how much is service-specific and hidden, who owns those contracts? It's another shared dependency. But with everything, it's finding that tradeoff.

One point not completely made is this: If you're extracting similar code, the question is: are they similar functionality, or similar implementation. If it's the first, then, extraction would make sense, and putting them in a library. If they're 'just' similar implementation, but the overall functionality is separate and different (or not related), then you're adding invalid coupling, since a change in functionality would then break the (other) coupled service. This will also show up if you want to give it a descriptive name. That name will either be fine in both (and then the functionality is similar), or clash since the functional name does not match up to the expectations of one of the components.

Awesome as always... Thank you very much Dave

I like the part of the video with the logo that goes after the introduction (I don't know what that part is called in English). It's short and the music is melodic and calming. I also like your logo

This was real food for thought, especially as I'm beginning to plan moving some of our online systems into microservices.

You are one of the best software youtubers. Thank you for your hard work.

Isn't the solution here to make sure that the dependencies are well-specified before they're used across services, and can thus be kept relatively backwards compatible? If you want to change the specification, then you can duplicate the code and change what you need to match the new specification.

Thank you for an excellent and insightful video. Thanks for also noting the lack of a simple solution or easy rule for some areas where you need skilled developers to create good designs.

Thanks for this. Thought provoking!

Thank you for the great video! It would be fascinating to watch some talk about technology obsession and overcomplicated design of modern software.

8:00 That's basically what I do with my projects. If I see some functionality, that might be useful for other programs as well, or when I need it for another program and don't want to rewrite it, I create a new repo, add this submodule of the existing program into that repo and let the other program use it. When I find a bug in my new repo or have ideas for improvement, or if I need to extend it to work with one of my programs depending on it, I have to upgrade the repo, but not all programs, which depend on it. Sometimes I do, sometimes I don't, most of the time I at least plan to upgrade to a newer version some day. Especially when using Rust, it's easy to have different programs using different versions of the same library.

I dont see the problem with making "debugWarning" its own repo and using fixed versions of it in different apps/services, especially if "debugWarning" was designed to be autonomous. We all do this on a daily basis by installing Npm and other packages... we use a specific version of a package, and as our application ages we might choose to lock it to an old version of the package or upgrade our app/service to use a new major version of the package. IMHO the technical debt, code quality and developer frustration of duplication and maintaining those instances is a much bigger problem and should be avoided at all cost... I would argue that if you are forced into anti-DRY then there are fundamental problems with your design.

In my case, between a service and a client (C# web-api and Blazor) I reduce coupling and repetition by using Interfaces (Contract between producer and consumer) also with default implementations of the intrinsic validations. Both sides share the Interace , and this forces both parts to have the same Entities/DTOs properties and comply with basic integrity.

I have recently discovered this channel, and I really enjoy the show. However, respectfully, I have another take on this subject, and I would like to put it out there for your consideration. Caveat: I have not worked too much on microservices myself. This makes sense if you have 2 microservices. But what if you have 100 microservices, and debugWarning() is 50 lines of code? If debugWarning never changes, then copy/paste is ok, but its a lot of work to install. Maybe you copy/paste some of them wrong if you get hasty. Its easy to make a mistake on tedious tasks. On the other hand, if debugWarning() changes in all of the microservices, now you have to change 100 microservices. If only 25% of the debugWarnings() change, thats still 25 microservices to handle. And any time you make changes to debugWarning(), you have to regression test the changes in all the affected microservices. Even if you only make a change to one debugWarning(), you have to ask yourself, why did only one instance of the method change? Did I make a mistake somewhere? Why didn't the other instances change? Good luck figuring that out. On the other hand, if you create a single microservice to encapsulate what varies, debugWarning(), yes there is coupling, but all the microservices are intercommunicating anyway across apis, so adding one more microservice doesnt add to the design complexity anymore than any other new microservice does. Make one microservice call to debugWarning() be the same for all other microservices. If microservice B needs a different version of debugWarning(), put a new version in your platform microservice and call it debugError(). debugError() will likely depend on code in debugWarning(), so you get code reuse locally and the benefits of that. And no one is making you use the platform version of the method. If one microservice needs a debugWarning() that is the same in name only to the platform version, then its free to write up a method locally. Versions of methods is handled, and so is minimizing coupling. It might be said that a downside of this is that a bug in the platform will cascade to all of the dependent microservices, due to coupling, but to be fair, so does the bug fix, saving you effort. It might also be said that having one microservice call per platform method call could have a lot of framework code to make it work, but its still better than having all the copied code in all the many microservices that are out there. But its a judgement call. If you have 2 microservices, copy/paste, but for 100, platform microservice it. At the end of the day, you want to make your microservices independent of other microservices, but according to my argument, you wouldn't achieve that completely. But realistically, if you did achieve this, you would never make a call to any microservice ever. EDIT: After a bit more thought, it occurs to me that the key issue with bad coupling caused by this approach would be that if the platform microservice goes down, then it brings down any other microservice that is using it, in a cascade effect. The solution is to spin up enough instances of the platform microservice to avoid this issue. Also, if the dependent microservices are composed well, then they will be fault-tolerant enough to carry on even if there is a platform outage.

Good piece of advice

Amazing, as always 💯

So right about interfaces of library vs microservice. I hear decoupling sometimes as reason behind using separate service. But in reality it might be a bit far from achieving that.

Good video. I've gone through dependency hell but have also heard about things like sidecar pattern to deal with reusable functions in microservices. Do you have a video that talks about that?

So few people are willing to talk about the cons of the divine DRY. 99% of devs think DRY is a non-negotiable and is a must in every situation. I would argue that DRY is NOT about code duplication but the underlying intent or knowledge of what the code does. So if two methods have the same lines of code in it but the intent is very different then you do NOT want DRY because you are coupling them together. If intent is different then you want the code to be separate so that they can evolve separately. The code may be the same now but wont be in the near future. This is where devs try to make complex solutions that can accommodate a million different possibilities that might happen in the future. Rather just leave it separate and simple and change the code when the future comes.

Even doing data analysis with MATLAB I recently decided to forego DRY for copies of the same functions in different folders for the very reasons mentioned herein

Very good video.

Does anyone know when the State of DevOps Report 2022 is going to come out?

An excellent explanation once again. Links to where to get the t-shirts would be a nice addition. ;-). I have a few but can always make space for more.

Pure and enlightened. Just dont write a manifesto to pester us for decades.

In the context of microservices, for me, DRY has a different meaning than in the context of monolithic code. In think having a library and defining the dependencies in the microservices is actually DRY in terms of code, because the code is technically in one place at design time and at development time. The code is only duplicated at runtime, and DRY, as I see it, applies to design and development times, not at runtime.

The question I asked myself in this case is who is responsible for changing this piece of code if both A and B are responsible for changing debugwarning I m not repeating myself because even if it's the same code, it's not the same reason. it's just the same code by coincidence and those requirements will change at their own pace. I feel like it's a mistake to see code duplication issue behind dry I think the idea of dry is to prevent duplication of responsibility. What do you think ?

Isn't the sidecar pattern meant to address this issue?

What do you think about architecturally DRY platforms with guaranteed backwards-ocmpatible µService APIs? Changes would only apply to newer releases.

One good example I've had is when you see two things that look the same but really aren't the same I tend to air on the side of caution and not make them the same. Trying to unpick it later is an absolute nightmare and likely one will break at some point

What determines of two pieces of code are duplicate code or not is not if the code looks the same to the naked eye. What determines duplication is potential sources of change. In other words if Service A could have a reason to change the code that would not apply to Service B then it is not duplicate code, it is merely code that looks similar.

"Discarding DRY for micro services" might be extreme. There are common services that you have to use. If your micro service offers a RESTful API and uses Node.js, it likely uses Express. If the service's state is stored in a database, it probably uses a vendor's product. Granted, these are external dependencies, but still are dependencies. No micro service lives in a vacuum. There is a fuzzy zone where DRY benefits equates costs, and navigating this is part of design indeed. The "best" choice may change with time, as products (internal or external) appear, change and die. Micro service approach moves the boundary significantly away from DRY, but it does not necessarily eliminate it. Sometime you can handle change by maintaining API compatibility, even if just adapter stubs. That's a design skill too, not easy but the more abstract the API is the easier it is. I agree with other comments: versioning is not full duplication. Versioning loosens the dependency. The rule is to not need all services to use the same version. That gives you temporary autonomy and buys you time. Eventually the old version goes away, unlike most real duplication. It is a tool in the box.. I have a love/hate relationship with the Node.js ecosystem. It is an extraordinary tool, but change is the name of the game. I reluctantly gave it up for personal projects due to the upgrade overhead. I don't consider external dependencies as different than internal ones: to "buy" or make is a design choice too, one that comes with a cost later.

I use to duplicate code and declare global variables when developing a certain f(). It's a bit faster. But once done, I immediately get rid of those issues. 4:00, or even a macro. 11:40, I think it's kind hard to write tests when you have classes holding data global to the tested f(). Those data are read/written inside the f(), but not come as parameters.

I insisted on DRYing microservices and it was not fun. Great video

What's the problem? Use a package manager, NuGet for instance, a package is both DRY and versioned

Another great video. I address DRY vs coupling by means of a nuget “subscriber” model with strong adherence to the Open/Closed principle. I create on-prem Nuget packages. Consumers of libraries can upgrade to new versions as they wish. “Breaking changes” are very vary seldom. DDD is also closely adhered to - especially with models and contracts. Again, great video!

I'm happy my two verticals I'm working on doesn't use DRY but I can't say for the other verticals my services communicates with. The maintenance is a pain and fixing. bugs Frontend wise we try to use public components so we don't DRY.

I prefer packaging apps with a package manager that uses semver with automatic updates for minor and patch releases which in semver should only be non-breaking feature additions, performance improvements, bug fixes. Major version releases need someone to check over and address possible breaking changes and push a button (by committing a major version bump and trigger a deploy) when it's ready.

Grandmaster, what is your feeling on a shared library of utilities that do simple things to reduce boilerplate? Things like clamping a number to a range + log a warning. Or defining a RGB color that can convert to HSV? The team is only 4 people in the mobile app space, but it has the potential to touch every part of the code, often in unexpected places in modules that have nothing to do with each other. i.e. the UI module, or the login service, etc etc.

Having non-dry code isn't necessarily a bad thing - I have the same error handler function copied 4 times in my game engine. Bad? Perhaps, but I know there won't be coupling between different parts of the codebase. BTW, the solution would be "closed for modification, open for extension".

Why is having versions of a library violation of DRY?

The presentor missing the point Deploy-Service does not equal Deploy-Code. Nothing wrong if a service is not aware (and so never touches) of some parts of the code. debugWarning function should exist in preceding shared-code (back reference Consumable) to make it callable from any service comes after. Well constructed (or bootstrapped) code is DRY, otherwise the cow is mad.

3:50 Only three of the lines are repeated. The last one is always different. It might be useful to only put the three same lines into a new function.

If you have 100+ micro services to use that `debugWarning`, do you really want to have 100+ copies of `debugWarning`, and then if there is a change required in `debugWarning` to be applied to all micro services, do you really want to update all those 100+ copies to apply the same change? What's bad about versioning the common-service-lib which accommodates the `debugWarning`? Each micro service gets to decide whether it wants to get the latest version of common-service-lib or stay with the currently installed version, depending on the major and minor version updates. When versioning done properly, the benefits of not having to update 100+ copies in 100+ micro service repos, outweighs the overhead of managing versioning of common-service-lib.

I still don't see how treating infra as 3rd party libraries could cause any harmful coupling. Should I not use Entity Framework in two of my services and implement my own ORM each time? I think the problems here would not come from coupling but poor API design and maintenance.

DRY converted to us to configurable and customizable microservices. A video about difference of configuration and customization will be helpful. Regarding this video - debug warning has to be a microservice as well.

In my opinion DRY can ruin mental abilities. Its much easier to reason about code when you know its isolated. Having repeat code is not a problem in itself, tje colpiler cetainly doesnt care, the prpbkem is human memory. I actually propose toolsets that can track duplicate code across your version control system so if you DID need to make a change you could easily find everything anyway. In fact i have designed a visual studio plugin that can do this, but in have not implemented it yet Also, another thing people forget about DRY is many times you will eventually come up with a "special case" well now you have to always remember who has the special case...and burden the code with it.p or give the special case its own copy. Lets say you give the special case its own copy instead. Well now everyone shares it but the special case. Good luck rememnering that. Its much easiet to remember by just having everyone own their own copy.

Lambda Layers can be used as template in serverless lambda architecture

Dave I think to use concept in barinless way is a bad habbit. I recall the time some 30 eyars ago... Take the requiremet text. Mark all the werbs and subjects. Now You have all the object with methods of the code. Okay back to CI/CD. There was a company where jenkins pipelines have been built with the use of ctrl-c and ctrl-v. I did an outrage with introducing shared jenkins libraries. Yes, writing a pipeline became harder but the quality of the whole is got better.

I think this is one of your best videos thus far. This topic is can be hard to break down and described in such a concise terms but you did it. Would you consider a video where you sit down and deal with example of real world issue where DRY and close coupling clashes?

note that if you would use a service for debugWarning you would have to develop, probably, a client, which would become a dependency, kind of a library, to both projects A and B. Thus, you would have more to manage for what could be done using a linkage. In the end there is no definitive answer to this question.

Multiplying technical mechanisms is just a source of defects and complexity. Multiplying business rules representation is is all of the above plus a reckless harm to our clients' business. Multiplying something is not decoupling, is always a (horrible) logical coupling. Is not independent deployed if you need to change all the copies (and for business rules is mandatory). We can multiply with less problems only what is easy to identify and produce (easy to identify and replace copies). Recap: Having different versions/copies for the technical mechanism or business access forms is not as bad as multiplying the representation of business rules. We may have discretion over our design, but we cannot deliberately affect the representation of the business. I have seen "dead" products due to the destruction of the business through multiplication (multiple versions = a lot of wrong versions). Resuscitation was done by DRY applied on the part of the business representation

Yeah

Hi I would like to hear tips about unit testing classes that communicate with DB and if there are any good tools to simulate the communication through ORM or plain sql.

Well code is only duplicate if it changes for the same reason AND at the same time. The latter is hard to get right.

why not make "base" debugWarning(), then debugWarningA() and debugWarningB() to "decouple" services and keep it DRY? i mean, services are coupled by os kernel anyway and call the same copy of read() and write() when doing i/o. i think it's not very nice to run each service on a different kernel even if you can, probably not a good omen when you see something like that. perhaps DRY is generally always better than not DRY, or maybe service should be agnostic of such things.

i see no problem with putting many microservices into the same repository. they can still be independently deployable. but more importantly, "everything in the codebase" means i do not sabotage my IDE support and make code sharing easier. there can still be an internal structure ("submodules") to prevent cross-service code contamination. also: whoever changes "debugWarning" is forced to change all the calls, so they will not ruin other services.

I kind of hate these acronyms for programming principles. I always think "Oh, another weird business technique, I'm not interested in". And most of the time, when I find out, what one means, it's either common stuff, I do anyway, because not doing it that way would be stupid (for example DRY), or it does not apply to the languages I'm using (like SOLID mostly).

The point is this: Use dry when it makes sense, which is most of the times. But as usual there are exceptions. I started working in microservices embedded system where they duplicated fucking common enums...

there is a code talk named “WET codebases” from some nodejs guy… its tricky

Take the reasoning of two different versions of debugWarning to not be called DRY. Does that mean that using different versions of libraries from npm or from Apache within your system result in it being WET? Or should that code always be forked into Microservices? Using software repositories to run multiple versions of the same utilitarian library in a complex system is practically speaking DRY (enough.) The Wikipedia article on DRY mixes two different approaches. It starts with the very pragmatic "reducing repetition of software patterns" and then follows up with the aspirational "Every piece of knowledge...". The former describes DRY's effects and the latter describes the ideal state of being when writing code; These should not necessarily be coupled. The words "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system" seem to allow the discarding of the evolution of the very "knowledge". I'd say something like "Tomorrow's knowledge may be more refined and efforts will be taken to maintain backwards compatibility. Upon braking BC mitigation measures will be undertaken." If your conceptual approach is to accept that versions of the same API represent the evolution of the correct knowledge, then different versions of the same utilitarian library may be considered DRY (enough.) Minor remark: Not the VCS but a software repository should hold released code to be linked into the applications thereof. One should remember that applications (that use APIs) gain the most by DRY-ness.

"DRY enough" is what I'd say is THE pragmatic strategy.

The problem is that developers too often start with "everything has to be a microservice". There are use cases where microservices make sense, but my observation is that the pattern is overused. I would rather do the following: * Focus on well defined interfaces that tests can be written against. This allows the development team to change the insides of functions without breaking other functionality. You get the benefits of services without the overhead. * Use dependency management and semantic versioning. You can declare that one part of the system is using version 2.X of your function. All upgrades to the function can stay at version 2 as long as no breaking changes are introduced. Having multiple versions does not mean that you have broken DRY. * Automated testing is essential. You can test that the function still works as specified and can test that all the consumers of the function still works. Imaging if you implement the above example as a microservice and you do a really good job so that the response time is 10 milliseconds. Now you call it 30 times in the code and you have just added 300 milliseconds to your function execution time.

Is this topic in your book Dave?

hierarchy is spelt incorrectly throughout the C code duplication example. LOL.

"I don't know how to agree with the type system so I just copy-paste code."

DRY can be not violated even in a microservices architecture and very naturally.

In these cases the ancronym becomes WET we enjoy typing