What is the Transactional Outbox Pattern? | Designing Event-Driven Microservices

Подписаться 72 тыс.

Просмотров 13 тыс.

50% 1

► LEARN MORE: cnfl.io/microservices-101-mod...
The transactional outbox pattern leverages database transactions to update a microservice's state and an outbox table. Events in the outbox will be sent to an external messaging platform such as Apache Kafka.
Check out the Designing Event-Driven Microservices course on Confluent Developer for more details: cnfl.io/microservices-101-mod...
This technique is used to overcome the dual-write problem which occurs when you have to write data to two separate systems such as a database and Apache Kafka. The database transactions can be used to ensure atomic writes between the two tables. From there, a separate process can consume the outbox and update the external system as required.
RELATED RESOURCES
► Eliminating the Double Write Problem in Apache Kafka Using the Outbox Pattern: cnfl.io/3UhVbVC
► Incorporating Event Storage: cnfl.io/492wmS0
► Microservices: An Introduction cnfl.io/3ZMt3up
► Event-Driven Microservices Architecture: cnfl.io/48FSYbj
► Migrate from Monoliths to Event-Driven Microservices: cnfl.io/3tsqlhu
► Get Started on Confluent Developer: cnfl.io/48FnKRB
CHAPTERS
00:00 - Intro
00:23 - What is the dual-write problem?
00:51 - What is the transactional outbox pattern?
01:33 - How can we emit events to Apache Kafka?
01:53 - What tools can be used to process an outbox?
02:49 - What delivery guarantees does the outbox pattern provide?
03:34 - What are the problems with the outbox pattern?
04:40 - Closing
--
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion - designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#microservices #apachekafka #kafka #confluent

Наука

Опубликовано:

19 июн 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 42

@ConfluentDevXTeam 4 месяца назад

Wade here. I spent a lot of time trying to solve the dual-write problem in silly ways (such as writing the event first). Eventually, I settled on Event Sourcing as a good solution (which it is). But, while event sourcing is a great solution with a lot of advantages, it definitely has a learning curve. A few years back, when I discovered the Transactional Outbox pattern, it felt like a bit of a breath of fresh air. Here was a solution to the problem that was relatively simple to understand and implement. If you are building applications with a relational database model, you don't need to re-architect your entire system just to solve the dual-write problem. That put the transactional outbox high up on my list of solutions.

@thestroke82 Месяц назад

Another great advantage is that it can be used almost anywhere, not only when architechting green field solutions. Consider, for instance, the shift from a traditional to an event-driven architecture in a legacy application fraught with complex and tipically messy business logic. In such cases TOP complements the strangler pattern exceptionally well

@schallereqo 4 месяца назад

Hi Wade, Thank you for the great video. This new series of videos is amazing. I am a customer of Confluent and use the Event Router product to implement the Outbox pattern. The issue is that this product feels incomplete compared to some of your other well-established connectors. We have made a few attempts to contact Confluent regarding fixing minor things and adding Protobuf support to this connector but with no luck. I'm glad you are aware of the Outbox pattern but wish you gave such a product more emphasis to help your customers make use of this solution.

@ConfluentDevXTeam 4 месяца назад

Wade here. I'm glad you are enjoying the videos. Unfortunately, I can't speak specifically to the issues you are having with Event Routers, but I will pass on the feedback.

@Pentatonic_Hardcore 3 месяца назад

didn't understant will try to find another video, thanks.

@ConfluentDevXTeam 3 месяца назад

Wade here. If you have specific questions, feel free to drop them in the comments. I'll do my best to answer.

@zhoulingyu 3 месяца назад

Great series. I have some experience with streaming. I find this very easy to understand.

@ConfluentDevXTeam 3 месяца назад

Wade here. Experience with topics such as event streaming would definitely make it easier to understand some of these concepts. I came from an Event Sourcing background so when I learned about this, it made a lot of sense.

@mehrdadk.6816 3 месяца назад

I've been using Outbox pattern, and it works nicely. I also used Kafka transactions and it works even better. Kafka transactions are available to link to database transactions.

@ConfluentDevXTeam 3 месяца назад

Wade here. The purpose of Kafka transactions isn't to link with database transactions. It is to ensure that if we write multiple events to Kafka, either they all get written, or none of them do. It solves the dual-write problem for the specific case where all of the writes are to Kafka. It makes no guarantees about writing to external databases. The problem is that there is no built-in way to commit a Kafka transaction and a database transaction in an atomic fashion. You have to commit them separately, one and then the other. As long as you do that, there is a chance that a failure leaves one committed and the other uncommitted, and now you have hit the dual-write problem. You might want to double-check that your logic isn't suffering from the dual-write problem. You can find more information in this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-FpLXCBr7ucA.html.

@prakharsahu7145 29 дней назад

Thanks a lot.

@ConfluentDevXTeam 29 дней назад

Wade here. You are most welcome. I'm glad you enjoyed it.

@JamesHarrisonHaribo 4 месяца назад

This sounds great! My problem is I have microservices that use both Postgres and Mongo, depending on the kind of state being changed. Of course, rarely does it write to both on the same command, but can happen. So in those cases, I don't feel like I can rely on a transaction inside one database, or an outbox that lives in one of those databases. I need an external transaction, that's where it gets a little messier.

@ConfluentDevXTeam 4 месяца назад

Wade here. It's definitely a tricky problem. Obviously, I can't speak to your exact situation. There will be plenty of nuances that I don't understand. But in those kinds of situations, I start to question whether I have drawn my boundaries correctly. Maybe the data that lives in two separate databases should actually live in one database. However, assuming that the boundaries are drawn correctly, the next thing I start to look at is whether these things are truly transactional in nature. Is it that either one could fail and that needs to trigger the other one to fail? Or is it that one causes the other. If one causes the other, then a transactional outbox might be a good solution. You implement the outbox in the "cause" and then use that as an event to trigger the "effect". And failing all of that, you might have to start looking at alternative solutions such as the Saga Pattern.

@paulcalinovici8808 4 месяца назад

I am not really sure that I understand how the data is stored in the outbox table. Is it some kind of event store from CQRS pattern, where we store events in a serialized format(json as example)?

@ConfluentDevXTeam 4 месяца назад

Wade here. To some extent, it's up to you. A serialized record would certainly be a valid option. Alternatively, if it makes sense, you could break the data into a columnar format. It will depend on what type of data you need to put into that outbox table. If you have a single event type, with a limited number of fields then columns might make sense. But if you have multiple event types, or data with many complex fields, then serializing it might be simpler. The other factor to consider is that you probably don't really need to query this table in any complex way. In that regard, keeping things as columns might not matter all that much.

@asifanwar7942 4 месяца назад

Would a suitable solution to dual write problem be - setting appropriate unique constraints on the DB then either creating or updating or doing nothing to a record ? E.g. Use if the record already exists on the DB as a way to de-duplicate when writing to the database

@ConfluentDevXTeam 4 месяца назад

Wade here. If I understand your question correctly, then this wouldn't be a solution to the dual write problem. Database constraints can certainly be used to deduplicate messages in the database and ensure it stays consistent. However, the dual-write problem isn't really about message duplication. It's about a lack of message consistency between disconnected systems. In that case, database constraints aren't going to provide any benefit when the second system is something like Apache Kafka, or an email server. Meanwhile, if you are only writing to the database, and not to a separate separate system, then a database transaction is the right solution. To solve the dual-write problem, you need to break apart the process. You need to separate it so that your database write, and your Apache Kafka write, are not happening in the same process. With the transactional outbox, that means writing first to the database, then having an async process that can retry the Kafka write until it succeeds.

@kevinding0218 Месяц назад

Thanks a lot for the explanation of the Outbox Pattern; I think I grasp the workflow, but I'm trying to get a better understanding about what benefit it brings. For benefit of outbox pattern, compare to if the original microservice updates the state in the database first and, only upon success, proceeds to publish the event with a retry mechanism. I sense that a benefit of using the Outbox Pattern might be 1 1) Un-blocking the original service while having to let it hanging there perform the retry if event produce failed. 2) Record the event in a different place so it won't be lost if original service goes down after only persisting the state in DB 3) Isolate the event produce with at-least-delivery manner while having to respect eventual-consistency is there anything missed by using the Outbox Pattern?

@ConfluentDevXTeam Месяц назад

Wade here. You suggest using a retry mechanism to publish the event. But how do you know what events need to be retried? You'd have to save them somewhere, like perhaps in the database, because if they are only in memory, they are prone to getting lost. You need to save them in a transactional fashion because otherwise, you encounter the dual-write problem. Essentially, you've now implemented the Transactional Outbox pattern while working on retry mechanism.

@AmanNidhi 7 дней назад

My understanding is same as @kevin's. @ConfluentDevXTeam Do you agree with the 3 pointer he has mentioned?

@mhetresachin717 29 дней назад

Hi Wade, Thank you for the informative video. I have a few questions: 1. Why is this pattern necessary when we already have CDC and DB connectors? 2. If I'm manually handling the Kafka publishing by reading data from the outbox table and publishing it to Kafka, how should I manage the scenario where the Kafka publish is successful but deleting the entry from the outbox table fails?

@ConfluentDevXTeam 29 дней назад

Wade here. 1. You've made an assumption that you have CDC and DB connectors. What if you don't? Now, for the sake of argument, let's say you do. What is the CDC process emitting? In a traditional architecture, you don't save events to the database. You save database/domain objects. Your CDC connector could certainly emit every change to a particular table, but that's not actually the same thing as an event. The event itself could span many different records in the database and may contain information in a different format than how it is stored in the database. It might filter out information, or include extra information such as more detailed context about the change. Now, CDC + Outbox is a pretty handy combination. 2. If deleting the entry (or marking it complete) fails, then you have to retry the entire process. And yes, that means you will get a duplicate event. Duplicates are always a risk when you are dealing with at-least-once delivery guarantees (Note: the same risk exists if you use CDC).

@nithinnambiar 4 месяца назад

I prefer to keep it simple, in the scenario where the application receives the message from the broker and processes it and then got to write to the db and kafka, the app wouldn't return till it successfully does both the writes. If the db commit is successful and kafka publishing fails the message would be processes again but it wouldn't write to the db since an entry already exists, it will then proceed with kafka publishing.

@ConfluentDevXTeam 4 месяца назад

Wade here. The simple case you describe is actually quite complex because it has a lot of assumptions built into it. You are assuming that the incoming message comes from Kafka. But what if it doesn't? You are also assuming that it is acceptable to retry the entire operation multiple times, but what if it isn't? You are also assuming that the destination for the messages is Kafka, but what if it isn't? The advantage to the Transactional Outbox pattern is that you can use it even in cases where Kafka isn't involved, or where multiple database write attempts might be problematic.

@dinhthaile8648 2 месяца назад

Hi Wade. Thanks for the great series. I have a question. What if the event has been emitted, but the deletion in outbox table has been failed? Once again, we still want it works in a transactional fashion, and we can't ensure the consistency.

@ConfluentDevXTeam 2 месяца назад

Wade here. The only part of the Transactional Outbox pattern that is actually transactional is the recording of the event and the updating of the data. Those two parts are done as a transaction to guarantee that either both happen, or neither does. However, once that's done, the rest isn't transactional. If you emit the event then fail to record that it has been emitted (either by deleting, or with a flag of some kind), then you will end up emitting the event again. However, distributed systems deal with duplicate events all the time. When you consume an event from the topic, it's possible for you to fully process it, but then fail to send the update to indicate that it was fully processed. At that point, you are going to get the same event a second time. As a result, event-driven systems typically implement deduplication techniques, or they ensure that the events are idempotent so that receiving them a second time won't matter. The delivery guarantees for the transactional outbox pattern are at-least-once.

@dinhthaile8648 2 месяца назад

Thank you. That's clear for me

@deependrachansoliya106 3 месяца назад

Hi Wade, Outbox pattern may cause performance issues for high volume?let's say system at peak hours can handle 16K QPS but after implementation of outbox, database performance degrade. What's better to use a JDBC connector or CDC like debezium in my case?

@ConfluentDevXTeam 3 месяца назад

Wade here. Yes, in some implementations, the Outbox pattern may cause issues at higher volumes. The example I give in the video involves a specific database I worked with in the past that had issues with Tombstones, but there definitely could be other issues. As for JDBC vs Debezium, either one is a valid option. What works for your specific case is a little outside the scope of what we can answer in a RU-vid comment.

@Andron4iKTV 2 месяца назад

@@ConfluentDevXTeam , as I understand it, implementing a method that uses, for example, debezium (any kafka connector) does not guarantee consistency with, for example, postgres (because it uses wal). If kafka connector and kafka is down wal can go ahead and miss some data . Do I understand this correctly?

@ConfluentDevXTeam 2 месяца назад

@@Andron4iKTV Depending on your choice of technologies, there definitely could be issues you need to watch out for. Best to keep an eye on the official documentation or drop into the community forums for those specific tools.

@combatLaCarie 3 месяца назад

I'm a bit confused. If my code wants to write to the db and then kafka it's already the case that I wouldn't write to Kafka if the db entry fails. ``` success = write_to_db_with_transaction() if success: write_to_kafka() ``` Maybe in this video we're assuming the db write is async?

@ConfluentDevXTeam 3 месяца назад

Wade here. No, we aren't assuming an async DB write. You might want to watch the video on the Dual-Write problem to understand the issue better. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-FpLXCBr7ucA.html

@patronovski 4 месяца назад

How different is KIP-892 with this?

@ConfluentDevXTeam 4 месяца назад

Hello, Wade here. The Transactional Outbox Pattern is a design pattern that can be used in a variety of situations, including situations where Apache Kafka is not involved. For example, rather than publishing an Event to Kafka, you could instead be sending an email. The Transactional Outbox pattern would still apply in that case. Meanwhile, as I understand it, KIP-892 is dealing with how Kafka Streams writes to local state stores such as RocksDB in a consistent fashion. It is very limited in scope and uses a different approach from the Transactional Outbox Pattern. The Transactional Outbox Pattern is a solution to the Dual Write problem which is a general distributed systems issue. Meanwhile, KIP-892 is a solution to a different, Kafka specific, issue. For more details on the dual write problem, check out this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-FpLXCBr7ucA.html

@sohaagarwal7336 3 месяца назад

What if write to outbox table fails?

@ConfluentDevXTeam 3 месяца назад

Wade here. If the write to the Outbox table fails, then the entire transaction fails. At that point, you are just dealing with normal failures of your commands. You still have to potentially deal with the failure, but you won't leave yourself with the inconsistency caused by the dual-write problem.

@WagnerBianchi 2 месяца назад

Yet, you have the data already delivered and consumed to Kafka, and you can start over from the point it has failed.

@ConfluentDevXTeam 2 месяца назад

@@WagnerBianchi Wade here. If the write the outbox table fails, then the data never gets to Kafka. The write to Kafka requires the initial write to the outbox table to be successful.

@larryhall1772 2 месяца назад

"Promo sm" 😒

@ConfluentDevXTeam 2 месяца назад

Wade here. I can't deny that I work for Confluent, and that this is solving a problem that might be encountered by our users. Having said that, the technique outlined in this video is applicable across a broad range of technologies. I'd be more than happy to hear that someone took this approach and applied it to any use case, regardless of whether it specifically touches a Confluent product or not.