Your videos are good. I'm in the industry for about 9 years, but the cases and points of view presented in this channel really help and get me to re-evaluate some of my own practices and solutions. Thank you for all the great work!
Thanks Derek, I am retooling our event driven content for our developers and these pitfalls are really valuable. These are great examples of how our industry tends to miss out on practice when on-boarding a technology or design pattern. Event driven design can be a real blessing when we’ll executed, but education, time pressure, or something else often get in the way.
Thanks, Derek, for this video :) Good idea to explain the engineering blog post which is complex mostly. Netflix, Twitter, and Uber blogs are also great.
A challenge with Outbox Pattern is it greatly increases load and storage in your database. Each service will now have its own event storage, for a wide varity of events, plus the publisher polling and updating events. It's fine architecturally but it can really increase cost as it requires higher provisioning of throughput.
Agree, when persisting to DB, you immediately try and publish, then delete the record you just inserted. Yes increased load for sure, but if you need to reliably publish, that's the tradeoff of that method. Another option is something like Temporal where each activity is guaranteed to execute.
@@CodeOpinion I asked Natan why they didn't use the traditional outbox (and if it was due to Kafka vs. traditional messaging patterns) and his response was: "I think that for most cases resiliency offered by Greyhound Producer was "good enough", and otherwise CDC via Debezium will be used which is a more generalized solution"
@@CodeOpinion "Agree, when persisting to DB, you immediately try and publish, then delete the record you just inserted" Huh? That defeats the whole purpose. After the publish has succeeded the main tx can still fail. If it does fail, all state changes are rolled back but the event was still published. If you don't want corruption there is no way around the fact that you must store the row in a transaction and pay for it, doesn't matter how soon you delete it after commited creation.
Great video. Another issue with generic CRUD events beyond needing to infer intent, is that it makes it much, much harder to determine whether an update to that message's structure will effect downstream consumers. By being more specific, we now know exactly how our events are consumed and exactly what they expect. It also allows us to evolve ideas more granularly without effecting all consumers. It almost feels like CRUD + CDC pattern (which feels like it's tied to Kafka specifically) is the worst of both worlds.
Well it sounds like they do CDC but within a logical boundary then transform those into something else that is consumed by other services (integration). This would allow you to version your events but the use case for CRUDy events to me often smells of data propagation and that always has me concerned.
I'd never known the language CDC, so thanks for that. Someone at work suggested using debezium. I Think we can have a better conversation now you've highlighted some of the pitfalls, and the fact it's just propagating state in a different way.
CDC is one of those patterns that become a necessary evil in some cases and the East Button in other cases. We tend to use CDC when first integrating legacy systems and evolve to something more purposeful as time and ROI permit
These debezium solutions have a history load feature, which actually pumps the existing data, which is not getting modified in any way....to the brokers...so that it helps in onboarding a tenant. Say, not everyone wants to move to Kafka at once, but, some tenants want to, they get the point of the new tech stack, so with the history load, they can be on boarded and the other tenants are not even aware of this. One debezium solution, switched on for one tenant, isolation(success/failure). Please do compare this approach with the outbox pattern(this is new to me)
How would you implement the outbox pattern without something like debezium? Is it a separate producer process which is polling on scheduling mechanism?
You don't have to poll so frequently as you'd expect. If you are indeed publishing and have a means to identify failures (metrics), you can trigger the process to start polling when a failure occurs.
For point 5, how about micro batch processing, and picking only the latest data by avoiding multiple hits to the same object. Like, say objectid 420, appears 10 times in a collection of 500 objects,so, instead of doing all 10 interactions with the dB at the sink connector side, what if we merge the changes as one and do just one dB interaction for objectid 420. Have seen people do this, but, want to hear your "codeOpinion" on it
One question around meaningful events like "InventoryAdjusted" vs entity state change "ItemQuantityUpdated": if several of your consumers need to keep track of item inventory counts, doesn't it leak abstraction(s)/business domain when a consumer has to be aware of every single reason an item's inventory might be updated (inventory adjustment, order fulfilled/sent to customer, return received/returned from customer, vender order received at warehouse, etc.)? When something like maintaining item inventory counts for several consumers/services becomes common throughout your system, how do you avoid the maintenance headache of process changes or added complexity to said business domain?
"if several of your consumers need to keep track of item inventory counts" would be the issue in question. It would be a headache keeping track of all the events that could adjust the inventory, but that pain should be indicating of the issue, which is why would other services need that data?
Given how much Event Store sponsors your content, could you do a video on how their product actually works and an example of how to apply it. I know you say to click on the link in the description but, meh, rather you just do a video. Perhaps the Event Store people could sponsor you to make such a video. Looking at you Greg Young!
On the topic of idempotent consumers you mention that you would use a message id related to the workflow as opposed to an entity related version. Did I understand correctly that you would make the message id something like a GUID? If yes: then how would you solve the problem if the message broker you're using does not guarantee ordering, but the order in which messages are processed is important? (e.g. we're doing event-carried state transfer and we don't want to overwrite newer state with older state because the messages arrived out of order). Are there any alternatives for using an entity-related version in that case?
If you're doing event-carried state transfer, then yes it's problematic. I'm generally not a fan for it for any transactional data, only reference data that likely is pretty static. Check out this video on ordering: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-ILEb5LsSf5w.html
