In addition, duplication may happen even from the system design of the broker as it requires to be highly available. Moreover, a replication strategy between data held among shards is required to be implemented, leading into duplication of messages regardless our acknowledgment mechanism.
I really love watching your videos so far. but as a beginner like me where I could possibly start implementing this kind of pattern. Do you have sample projects for references.. appreciate some advise. thank you sir. ☺️
Gotcha🙂, question on DLQ ..if your consumer is calling I/O service to make state changes then there is always a great chance that newer message gets successful than the old message in DLQ. How do we handle this ?
Depends what your consumer is doing. This one is hard to explain via a comment and I should probably create a video about ordering and expectations. The short answer is don't create event consumers that try and use event state to maintain some type of their own state. Expect to consume out of order.
Regarding handling failures- you're suggesting retrials might have an impact on the overall processing times of other messages by the consumer. But, aren't there buses and brokers that will reintroduce the failed message into the back of the queue, so that other messages will be handled before the failed one will get the retry? In some cases you get this by default, no?
At the end I'm suggesting each individual http call to the external service be driven by an individual message in a queue. In that case it will not have an impact on any other work. How it was implemented in the example would have an impact because all the http calls were made within the same process. If one causes an exception is would fail the message. If it was retried, it would end up making duplicate calls that could have worked the first time.
Love your videos! At 4:30 state update and message sending is in two separate transactions. I know you're familiar with the outbox pattern (you have a brilliant video about it) - dont you think these two operations _always_ need to be done in one transaction? If not, you have 'at most once delivery'. You wont even have eventual consistency - in a large production system you're basically guaranteed inconsistency?
Correct, if you aren't using the outbox pattern then you could fail to publish the event to the broker but still have saved state. If you are using the outbox pattern, then you can also introduce publishing duplicate messages if the publish occurs to the broker, but fails to remove from the outbox.
@@CodeOpinion Well I actually think blazor is great. I was a bit skeptic at the beginning, but I found out that you can build good looking apps with little effort and almost no js. The performence is pretty good aswell. There are improvements to be made, like better hot reload, but as of now I think it is totally production ready and a real game changer
