CQRS pitfalls and patterns - Udi Dahan - NDC Oslo 2023 

i don't know his other presentations, but i disagree

@@xybersurfer I'm genuinely interested in what aspects of the presentation you disagree. A thoughtful dialogue, and exchange of ideas - is always welcome.

@@KelvinMeeks he never described the underlying details of the problem. it happens too often that people wildy varying interpretations of what the problem is. i was also not happy with his explanation of CQRS, where he makes it sound as if it's impossible to base a write on a read operation

Udi targets applications with a HUGE AMOUNT OF LOADS IN A CONTEXT OF ACTING ON SAME DATA, aka Collaboration contexts. "Huge amount" is the keyword here. Ticketing might not be the best example because it could lead to think to a little event with 1000 places, that is not the purpose of Udi in his talk/demonstration. In such a case, even with a queuing system, you will experience a very slow experience because you would apply a back pressure, and a sequential way (like if you would have implemented a pessimist locking) to treat commands with a poor UX experience (waiting in line). And if you are using CQRS (to optimise queries), don't forget that in the same time, the query updater should run. And you don't want to add an additional "process" that slows down further the whole process. I really think that Udi has enabled his creativity to strive providing the best handling of scalabilty without the detriment of the speed, nor a poor UX experience for the user. Ideally, a user doesn't want to "wait" too long for a result, while he's not still there. User doesn't want to "hope", but "being certain" their demand for a ticket is possible or not. With an appropriate design of the process, most users would get that "acceptation" in a matter of seconds, better isn't it? Queuing system as you suggest under a huge load won't be able to suggest a fast enough experience.

​@@wealcome_company then the presenter really should have used a better example, than a ticketing system. i'm concerned with people like you that already think they have a solution, to a problem that was never fully explained. his compromise of not being able to choose the seat, doesn't even work for all ticketing systems like when children and parents are involved. i would have expected him to allow the user to specify that the seats have to be adjacent. i would have at least expected him to explain what the hell he was doing in the critical section, that took so long. without a good detailed example, it's all speculation

@@xybersurferI agree that there are a lot of implicit ideas/contexts that are actually important details. The case for a « concert » (not Michael Jackson’s one) may not be the most appropriate to explain to « novices in CQRS » the true problematic that can occur. It might not be a « huge loads context ». Olympic games is a better fit. You might want to charge 3 tickets in a same form (or capture the exact intent), and the application in the background would try to find adjacent seats or suggest (and book) the best alternative if it can’t by email. That’s why in CQRS, commands return void ;) Note that Udi has well explained how he’s handling critical sections that « took so long » as you said. Placing the charge form upfront is a way to do so for instance.

@@wealcome_company I am sorry but you are 100% wrong!!! Just look at how millions of trades are executed on a daily basis for billions of dollars via global stock exchanges at ridiculous scale. This is all done via queueing and has been in use for many decades now. Udi's example of an overstressed Taylor Swift ticketing system happens nearly every day when trading stocks in the stock market and the backbone is high speed async ticket based queuing which is a known solution with many types of implementations for over 50 years now. Really disappointed that Udi di not use this type of solution which has been successfully proven to work at nearly any scale possible.

@@basilthomas790 Exactly, also the cart example does not make sense, redis is used to solve the cart and inventory problems. Redis is fast and also has incr and decr commands to increment/decrement inventory numbers for example. Incr/decr doesn't have a race condition like the sql update command has.

I think the point of this talk is a reaction to the fact that CQRS is a buzzword, often it is pushed as an architecture, and more specifically a best practice, ie a thing you should do in every single system that you build. The point of listing the different styles was to explain that there is more than one way to do things, and there are reasons why you might select one or the other. CQRS is Command Query Responsibility Segregation. As far as the bad use cases it goes hand it hand with the "best practice" that is often suggests that CQRS should b used everywhere. Multi-datastore CQRS by its nature implies eventual consistency. I believe he tries to illustrate the problems with taking that approach. The response to the 44-minute mark is at 50 minutes, and he says its not purely technical is because the business needs to decide what to do with negative inventory. That's not a decision a developer should make on their own. In many cases the business may want to offer the customer options of what to do when an item is not in stock. They also may want to be free to change the rules of what happens in that scenario. I hope this helps address some of your concerns. Another thing that may not have been explicitly stated is the CQRS style typically works well when there is high contention for the same data. The mention of the Harry Potter book, the Taylor Swift tickets are some good examples of those things. The commands have a different responsibility than the query. In both example the command's responsibility is to captures the customer's order (while ignoring availability), which is separate (or segregated) from the query's responsibility to gather the items that are for purchase.

@@awright18 To respond to your points: I don't think it's meaningful criticism to say that any design principle or pattern is a buzzword, or that it shouldn't be used universally. That's just true of any design pattern, or really anything. You could say the exact same of object-oriented programming, functional programming, cloud native programming, clean code, literally anything. The problem with the first couple minutes of the talk is that we don't get a clear sense of direction from the beginning. We even get a slide saying "The first pitfall is not knowing what CQRS is" and then he just forgets it and talks about random architectures and design patterns! And they're not even ones that CQRS was meant to solve. To me, it's baffling that a presentation on any topic wouldn't take a quick minute to define terms and summarize what the presentation is even covering, much less spend time on topics that aren't really related. "He says its not purely technical is because the business needs to decide what to do with negative inventory." The problem is that negative inventory was allowed to be created, which is the real issue at hand. There are technical solutions that would allow you to avoid this, and lo and behold, in the real world they work - as much as I hate ticket sale sites, they figured out how to avoid creating negative inventory. Perhaps it can be a business decision to make the tradeoff between allowing negative inventory and being fast, but if you prevent the problem from existing, you don't need to invent its solution. To me, the entire talk feels like something that went through very little revision, because it has an incredibly disorganizaed structure, and doesn't respond to what I would consider extremely obvious questions or counterpoints.

​@@KeremyJato Thanks for the reply. I was recently talking to a friend about inventory a real world inventory system that they worked on. Their were times when the inventory system had a certain count, but the actual physical count was different. In many systems with physical inventory, reconciliation is a normal process to correct the system when these types of situations happen. There are many reasons this might happen, the inventory is bad, broken, stolen, not delivered etc. These are all people errors that some inventory systems are designed to deal with, sometimes by allowing negative inventory until reconciliation occurs. If it's in a physical retail scenario and a customer brings a physical good to the register and the system thinks there are 0 for sale, should it allow the transaction? I know this is a different use case. In the online ticket or book scenario, I do agree intentional allowing of "overselling" is not a great customer experience.Making people wait for long periods of time in order to maybe have a chance to purchase a popular item is also not a great customer experience. In otherwords there is likely to be a trade-off and the business needs to choose for different scenarios what rules should be enforced or not.

I am currently using cqrs to build a content management system for a real estate platform. It has been good so far. Though on a previous project we used microservices which was and postgres and it was straightforward. In cqrs we use repositories instead of services and we are using mongodb. I guess the requirements of this project is unique.

Same here 🎉

I didn't know him but this talk is really good so I'll be doing this in future!

The talk was held in room 8

CRUD means what it means for single entity. CQRS tackles whole model. For instance - CRUD for worker means update single record but CQRS commnad "Fire employee" can mean: update record, remove time attendation card, revoke access to the building, cancel lunch every workday supply, cancel all meetings etc. Of course you can use CQRS for single type entity. And Command handler can act as regular service. CQRS in small examples looks like without sense, but in more complicated app can have sense.

Omg you have no idea how rare it is for someone to really understand race conditions.

When you want to buy something, the system has to lock the record until you finish buying it. The actual buying process would involve you entering your payment information and then validating that against a payment provider. All in all, that's a long time to lock a DB record while everyone is waiting for you to do that so they can order their copy. So in order to have a quick lock and release of the record, payment information is given and validated before you are given the ability to actually buy what you want. That's what he meant, or at least what I think he meant and what I took away from it. Essentially shuffling the purchasing steps around in order to prevent a technical problem.

When you have the option to pick a seat, you could be doing this in parallel with 100k users. So imagine that the ui must sow you the available seats left, this is public data that you and the other 100k users want to modify. If you just skip this whole process you din't have that public data and all that complexity

Can you share some better ones I'm interested.

@@abdelrahmandwedar try any presentation from Greg Young

Why this would be a poor explanation of CQRS? Could you elaborate?

@@andredesantacruz this sounds like a trick question, but i'll answer: he talks about command and query separation, but a command often requires a query. i think that he should have explained this point of contention better, so that it at least sounds feasible

@@xybersurfer in CQRS, a command doesn't need a query. In fact, it's pretty much the opposite. In CQRS, commands and queries are not related to each other at all. If your command is "often requiring a query", you are not doing CQRS.

@@xybersurfer aren't you confusing cqrs's query with any general data base query?

​@@andredesantacruz i think you are mixing up 2 different discussion topics: - was the presenter's explanation of CQRS good? - what is CQRS? i've only been talking about the first. when i answered your question, i was only using what the presenter has said. i did not make any claims about what CQRS is. you say that commands and queries are not related, but this does not make sense either. are you claiming that the output of the queries are not affected by the commands? if we continue this discussion you will probably provide a better explanation about what is CQRS (second topic) than the presenter, but my point (first topic) still stands.

There many things that are bad for society. Harry Potter books are just bad for developers.

It's amusing how all their presentations follows the same structure: 1. State problems. 2. Don't give solutions. 3. Optionally downtalk ES/CQRS between the lines 4. Wait until people will come for answers. 4. Sell NServiceBus

yeah he should have explained that better. maybe it somehow decreases what has to be done during a lock. who knows. it's a shame that he didn't go into more details about the actual underlying problem

It's a domain problem, not a concurrency problem. If they have a system to deal with allocation and release of locked seats, it never happens. It only proves that engineering gets ignored at large scale, too. More often than you think. One key point this talk is missing out, eventually consistency requires error correction logic specific to business domain. Your controller / service logic is the business logic in most cases, while in query mode, you're just reading state of the system which may or may not have updated at given time. Whomever is reading state and whoever is issuing commands needs to be aware of this trait and embed this error correction logic in order to handle the issues that comes with pattern, it can be locks, status columns, internal state services, pure business logic in your service to get state or accept or reject command depending on previous state. The transport system may be stateless, async or unaware of internal state, your logic shouldn't be at given point in execution and be fully aware of current state of data and make relative decisions!

The example was about preventing users from choosing seats. When people choose seats, it requires locking that seat for 15 mins or so until the user completes the transaction. If you don't give people a choice and just assign them random seats, there is no lock and it can be done in the background.

That solution allows to not let the connection pool dry up, because a charge does not concern a shared data that everyone acts upon. A charge like acting on « private » data. There’s no efficient solution to prevent selling more tickets than existing. Here, the compromise is to favor scalability/speed. A compensating process will occur to signal to those users a kind of auto-cancellation.

@wealcome_company true, the analogy is wrong here - they can say I wish to book tickets, so users know there's a chance they might not get it. Building a system saying complete the transaction and you will get ticker doesn't communicate well. Thus, it's a business domain problem. There's not much you can do with concurrency in many cases except be clear about tradeoffs. In eCommerce, flash sales are the same examples of limited inventory. Massiv ingress till you are able to process it sequentially and be sure not to oversell by sold out and refund notifications or wishlist / backorder solutions. In the case of Taylor Swift , they could have prevented bots with multi-level authorization system, in Olympics they could have a confirmation system just like we have in trains ticket. Solutions to these problems are already out there in other domains, if only experts can be free to look into it and management agrees to change their business model ( which they dont in name of financial and managerial reasons as it essentially renders many of them jobless by implementing automated systems that works too well! )