Thanks for this, I hadn't heard of your channel before this, but I really like the pragmatism of the topic and focused illustration of problem, design, implementation, and issues. Can't wait to comb through more of your videos!
I can attest that this is a bad pattern for high volume. Your reader has to deal with data contention for reading items and deleting. It is better to store on disk if it fails and have another thread/process handle the messages.
Hi Derek, I'm curious can an event store be used in place of an outbox table? For example, the relay service reads new events inserted into the **event store** and publishes them to the message broker, and no need to delete events after. It seems like this could be an option instead of having both an event store and an outbox table?
Meaning if you were event sourcing instead of storing current state? Sure that would work as long as you know which events you've published. Also, EventStoreDB can act as a broker itself so no need to use a separate broker per se.
Hello, when outbox messages are processed by background jobs and there are idempotency consumers implemented as well, do they together raise the same event twice? If so, how can we avoid this and can you please show a complete demonstration where both background job processing outbox messages (events) and idempotent consumers are implemented in the same solution and yet they process an event only once? Thanks!
Good question. If you were to do that, you just reversed the order of possible failure. Meaning that if you start a transaction, publish an event, then commit the transaction. The issue with that is if you publish the event and the commit transaction fails, you then published and event and never saved state.
Again awesome and practical content! Looking forward for your upcoming solution i.e. handling duplicate messages or handling db update failure scenario
You had another video about CQRS where you ran all commands in a transaction, could you setup the same with the message queue like you see doing here? Mostly to make sure that it is always committed within the transaction
Hi Derek, great video (as always), thanks ! I know I'm two years late but I've got a question for you : what kind of event do you save using the outbox pattern? Only private (domain) or public (integration event), or both ?
Both. Often you'd think of public/outside events because other boundaries are consuming them, but you're own boundary that's publishing it may also be a consumer for specific domain events.
If you rely heavily on outbox pattern you may end up having a hot table. Is there anyway to manage outbox table, so it does not end up becoming a bottleneck?
It's a tradeoff. You're going to have some performance hit by using your primary business db to also store outbound messages. However, there are performance optimizations to be made in terms of how/when you read from the outbox to publish messages. Eg, you can write the event, commit trx, publish the message to broker, mark event as published within the same process/thread. You don't need something separately reading the outbox table constantly to publish.
As far as i undestood cap library does a distributted transaction between rabbit and sql database.Is this ok,distributted transactions has scalabillty issues.Is not more suitable a separate background process which tries to publish the messages from ques table to the broker?
Well it doesn't do a distributed transaction. It does as you mentioned write to the DB, then push to the broker. It then writes back to the DB to mark the message as sent. So yes there's added latency potentially as well as overall load to the DB.
Thank you so much for this content! If the service crash before publishing the event I guess that when it comes back it will query the outbox table and sends the events, but what happens when the service is replicated, if the service has for example 10 replicas the event is going to be published 10 times, how the publisher know if the message was taken from another service, is it marked the record has taken?
The outbox will coordinate it often by locking the records it's handling at the DB level. Regardless, you want to handle duplicates at the consumer. Check out this video on idempotent consumers ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-xeBY8fCWfvU.html
I'm not a .NET developer, but i've been learning a lot with your videos! Regarding the outbox pattern, i have a small question. Can the publisher be a cronjob task that pulls the messages every X sec/min from the DB and publish it to the queue?
Biggest concern there is that they don't overlap. You wouldn't want a job to start while the other of still processing. That will cause duplicate publishing. Although consumers need to be idempotent regardless.
This is interesting! Is there a link to the duplicate handling?? I’m just looking into this type of functionality ? How about doing this in the Azure stack, using their bus & queues etc?
I have just only one doubt and is what happen if you're using a different persistent storage from the business logic, or you are required to use the same database for both for performing in the same transaction? By the way excellent video!
I got a suggestion to use the outbox pattern for domain events. So the app has a recurring job fetching stored outbox messages and emitting the domain events internally (MediatR). Atomicity preserved. My concern is what about scaling with replicas of the services when both will be running the recurring job.
Concurrency between recurring jobs basically? Avoid it basically. Depends how you're implementing the outbox. I'd prefer not to do it myself, but if you were, try and publish the message immediately and if successful remove it from db. That way what's really ever in the db is when there is indeed a failure. At that point you really only need a single recurring job without concurrency.
@@CodeOpinion Hmm. Yes. I could just enable that recurring in-process job in only one instance/replica. That would solve the concurrency issue - just having one message dispatcher. And accept failure. At least, events are stored and can be reprocessed once that instance has restarted. Unless I implement some locking mechanism with a flag on each message. That would mean more commands to the database. But pragmatism is about being good enough. Not overdoing something if the cost is greater than the benefit.
I have quite recently come across your channel, and all of your videos are just REALLY good - so thanks a lot Derek!! Regarding the outbox pattern, would you consider implementing a corresponding «inbox pattern» on the consumer-side as well? I mean, wouldn’t it be beneficial to read the events into a table in the «consumer-db» without any validation or business logic, and then process the events from the table instead of directly from the queue? By doing so you could quite easily identify any duplicate messages (through i.e. a messageid) and avoid processing them, and also mark the events when they are finally processed by the consumer?
Yes, there are different approaches you can take. Ultimately you're trying to make consumers idempotent. Here's an older video, I should probably create a newer version. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-xeBY8fCWfvU.html
Hi Derek, why is a second outbox table necessary, instead of just doing CDC on the primary table that gets updated? Is it because you're including extra information in the outbox table that isn't available in the primary table? Thanks for the great video.
Not sure what you mean by "second outbox table". There is only one. The outbox represents the messages/events that will be published to the broker. An event/message isn't the state changes made (eg, CDC). There are various types of events used in various ways. Check out this video if you haven't already, it might clear things up: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-53GsiAcKm9k.html
@@CodeOpinion Yeah, I meant just "outbox table". Thanks, I checked out that video. I think I was confused because I'm familiar with CDC to do pure data replication, e.g. Postgres -> Debezium -> Kafka -> Data Warehouse for data analytics. And I'm less familiar with microservice architecture. It seems like the benefit of the outbox pattern is that you can add more context to the messages/events so consumers (who might not be able to read the source service's database) can work with enriched data. Is that right?
Should the publisher not care who consumes it? Does it not break pubsub pattern if the publisher have to wait to see if the publishing to the queue was consumed and then do a dependent action by updating the database? 7:58
The publisher doesn't know if there are consumers. It stores messages locally in an ACID transaction in the same data store as your application state. Then there's a separate thread/process that's taking the messages from that datastore, then publishing them to the broker. You aren't checking if the message was consumed, just that once it's published to the broker, it can delete it's local copy of that message in the data store.
Wouldn’t it be smart to keep these events in case, let’s say, a new service is published and you’d like to populate the database of that service based on the events?
This is very useful video as it is solving frequent occurring problem with Message broker. I have following two questions kindly help #1 - Here BeginTransaction is an extension method used with _dbContext however can we use this with other ORM like Dapper or directly with ado.net #2 - Other than CAP what are the other libraries we can make use of or how can we use this with classic .net app and not the .net core
For both questions, it comes down to the messaging library you're using and if it implements the outbox pattern. NServiceBus, Brighter, CAP, MassTransit (although I don't think it's quite the same) all have support, check some of those out.
