Event Sourcing You are doing it wrong by David Schmitz

Подписаться 156 тыс.

Просмотров 87 тыс.

50% 1

Please subscribe to our RU-vid channel @ bit.ly/devoxx-youtube
Like us on Facebook @ / devoxxcom
Follow us on Twitter @ / devoxx
"Every microservice get's its own database and then use Kafka" is a typical and naive advise, when reading about eventsourcing. If you approach this architectural style this way, you will probably have a really awful time ahead.
Eventsourcing and CQRS are two very useful and popular patterns when dealing with data and microservices. We often find in our customer's projects, that both have a severe impact on your future options and the maintainability of your architecture. Presentations and articles on both topics are often superficial and do not tackle real world problems like security and compliance requirements.
This combination of half-knowledge and technical confusion leads to many projects that either refactor back to a 'non-eventsourced' architecture or reduce eventsourcing to a message queue.
In this talk, I will summarize our experience while applying eventsourcing and CQRS accros multiple large financial and insurance companies over the last 5 years. We will cover the Good, the Not so Good, and the 'oh my god...all abandon ships!' when doing eventsourcing in the real world...and see how we solved these issues.
David Schmitz
From Senacor Technologies
Principal architect at Senacor Technologies with a history of + 16 years of working in various projects using a bunch of different stacks and environments.
Current focus is on migrating architectures and organizations to cloud and serverless platforms.

Наука

Опубликовано:

14 ноя 2018

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 85

@DodaGarcia 3 года назад

9:25 "So, end of talk. Maybe not" 😂😂😂 I love how deadpan he is

@DodaGarcia 3 года назад

Also the footnote on the banking slide at 14:00 is appreciated, comrade David

@cadessi2263 5 лет назад

Amazing talk!

@mohammadajafri6307 5 лет назад

Really excellent talk

@MerrionGT6 5 лет назад

One stream per aggregate is, of course, the way to go! Also - both logically and physically distinct.

@LonliLokli 4 года назад

I really like your talk! The only thing I miss is a discussion on after-party.

@charli3br0wn 5 лет назад

Love this guy's sarcasm :)

@ArunShankartheRealOne 3 года назад

german humour!

@tyrotoxin 5 лет назад

First part is about Partial Ordering of events which guarantees Causal Consistency model in distributed systems. Don't think that it comes for free though. The ability to specify an offset on emitting events to Kafka (aka "Lamport Clocks") creates a "lock" per partition what impacts performance as you have many concurrent producers. Besides, you need to emit events for the same Aggregate into the same partition - that reduces the Availability. Those are the right things to do though if you want to trade the Performance and Availability in favor of Consistency. And most likely you want to do so instead of putting that complexity into the application layer. Just be aware that the approach has its price.

@tiagodagostini 4 года назад

Sorry but solving a 100 event stream in 66 ms is NOT fast. it is ABSURDLY slow when you figure the total expected data troughput of a modern computer. it would need to be 66 Microsseconds to be reasonably fast.

@illyam689 4 года назад

agreed

@derNoaa270 3 года назад

Also the reason why I instantly cringed when the presenter mentioned it being _fast_. But I guess one could argue if this kind of "speed" might still be enouhg for your usecase. But if you only have such little streams of data maybe you don't need to go for distributed systems afteral? Feels like everything is a dristributed system nowadays just for the sake of being a distributed system.

@amando96 3 года назад

@@derNoaa270 Yes! Half the people doing this would be fine with a monolith but it's like engineering departments have to justify their cost by making things more complicated than they have to be

@AlgoristHQ 3 месяца назад

@@amando96often times, the distributed system is created for the developer. If you can break software down into its domains and subdomains, it makes it easier to design for resiliency and maintainability.

@b7otato Год назад

Great talk

@MykolaKlymyuk 3 года назад

How do you solve 'Set validation' problem?

@igrai 5 лет назад

What can I say - excellent talk!

@nimrodsadeh9616 2 года назад

Also, in the part about query aggregates you said to just put all the streams that correspond to your query in a bigger stream (e.g., users > 18 years old). This is a good example because it changes all the time; your users grow up and become 18. Wouldn't the logic to handle moving streams around like this become a bit cumbersome?

@michaelb.2924 4 года назад

What are the trade-offs?

@nimrodsadeh9616 2 года назад

Aren't event handlers that read from persistent storage CPU intensive since they deserialize every event?

@rum81 4 года назад

whatever he says "super easy" i find it difficult to follow

@devproco8063 3 года назад

Can you explain what `emitEvents` does? - does it _apply_ events to an aggregate? - does it save events generated by an aggregate (eg `executeBusinessLogic()`)? Is the `Account.lastEventNumber !== Stream.lastEventNumber` supposed to occur in the storage implementation? For example, inside of `Stream.append()`?

3 года назад

Can you build an "event sourced" system iteratively? It's very seldom you actually know your domain well enough to specify it in advance, and even if you do, it will most likely change. That would be an interesting topic to discuss and compare between different architectures.

@KoflerDavid 2 года назад

Personally, I think that's even the correct approach to do it. There is no chance that you will get the system exactly right if you build it in one go. Therefore, you will have to deal with change anyways. The biggest problem is that you will have to live with old versions of events for essentially forever. You really have to think it through and practice a few times how to handle migrations in event-driven systems. This talk shows a few techniques to do so.

@VictorMartinez-zf6dt Год назад

Yes, you can add events and commands to aggregates as you go. But I'd argue that you don't build event sourced system, but event driven systems. Event sourcing is a pattern of storing transactional data of single aggregates.

@krellin 2 года назад

the guy hates frameworks but uses eventstore which practically decides for you everything from what language you code in to how you do high availability... there are good points about version (or not doing versioning) though...

@codematrix 5 лет назад

Here's my take. The current industry state of event sourcing is still not well defined and there seems to be too many flavors on what is right and what is wrong. Like all greenfield approaches to solve a problem, I tend not go into it so quickly until a consistent well defined approach becomes the leading defacto. My suggestion, don't build your own homegrown ES because it will be a hodgepodge colossal mess and a maintenance nightmare. I plan to wait until AWS or Azure, come with a viable solution and the let them worry about the implantation and maintenance while I worry about the meat of my microservices.

@illyam689 4 года назад

I think we have to wait for a long time..

@codematrix 4 года назад

@Andreas Berger - I agree with the lack of progress as a possibility. If a team were to tackle this endeavor, make sure it's a new, small to medium size project. I would never convert an existing project from non-ES to ES unless the architecture lends itself already to a clean architecture, hopefully with CQRS on the read side of the equation.

@javier.alvarez764 2 года назад

True. For now CQRS and Event sourcing seems to be just a bunch of theories. No clear definition, and actual implementation.

@RikiPoon Год назад

6:26 take away the eventstore and the whole thing still works. To me it seems to be depicting event-driven architecture rather than event sourcing, as we make no use of the eventstore at this picture

@dmsanz_youtube 4 года назад

I liked it, but saying that read models is overrated could be very confusing especially considering that 80%-90% of the interactions with our web applications are "read" operations and also considering that one of the most powerful advantages of event sourcing with CQRS is the ability to replay the events in a stream in order to rebuild a read model (e.g: new geodistributed infinite cache) or in order to create a complete new one (e.g: reports). In my opinion read models are actually underrated and having a good event versioning and replaying strategy is crucial.

@lukasz_sarnacki 3 года назад

I think you missed the point. He just said that you may not need to persist and instead dynamically fold event streams each time.

@placidchat7532 5 лет назад

If eventstore documentation is awful but the community is great, how would you choose this library vs every other one out there?

@jeffcrow1481 4 года назад

They have support and offer for business

@DavidCumps 5 лет назад

How does GDPR deal with the requirement of storing data for fraud detection as a bank? I cannot image a customer can have his bank history forgotten to be hard deleted? Doesn't the government require an audit trail in case you are charged in court?

@dmsanz_youtube 4 года назад

Encrypt the personal identifiable info of the events and store the specific key for that subject in a separate storage. If required by the user, just soft-delete the key so that the info cannot be decrypted. If there is a government/court requiring the info, simply undo the soft delete and provide a good reason to do so. That's at least how I would do it. Whether you hard delete the info or not I think it's irrelevant for the GDPR as long as the personal info cannot be recovered. So the security to undo the soft-delete must be strong..

@bluejanis5317 4 года назад

@@dmsanz_youtube If you can undo a soft-delete, then you can recover the personal info. That is not allowed with the GDPR & DSGVO.

@RonArts 4 года назад

@@bluejanis5317 There are often multiple laws at work. Financial data may need to be kept for 7 years for example, but other data may just 2. So you create multiple keys with each user, and encrypt financial data with the financial key etc. Then you can just delete the appropriate key.

@guibirow 3 года назад

If the data is kept to comply with regulatory purposes, the company cannot delete the data, otherwise they would be infringing other laws. But they must keep only the data needed for regulatory requirements, any other data must be deleted

@JeremyAndersonBoise 5 лет назад

One of the most hilarious presenters in tech. Love him.

@Spiritusp 5 лет назад

37:30 Love the Monthy Python reference!

@SashaArsic 3 года назад

My favorite colour :p

@bluejanis5317 4 года назад

What about commands? Your talk never mentioned them. CQRS = Command-Query-Responsibility-Segregation

@aneshas 3 года назад

Hi, what do you want to know about commands in this context?

@jinhanchoi1 4 года назад

one stream means a topic for one user?

@aneshas 3 года назад

Yes, one topic / stream per aggregate "instance" not aggregate "type"

@jmilkiewicz 5 лет назад

a messy mix of event sourcing and event driven architecture.

@boot-strapper 3 года назад

Luckily we dont have to go to ops to ask for anything now because AWS offers managed kafka.

@TheApeMachine 3 года назад

Yeah, until the bill scales exponentially :p Or someone requires you to move to another cloud or on-prem... But you may not run into that in your use-case of course...

@boot-strapper 3 года назад

@@TheApeMachine mature companies arent switching cloud providers. Also the billing is linear...

@TheApeMachine 3 года назад

@@boot-strapper recently spoke to a mature company who's clients at time demand certain providers or on premise systems. And linear pricing doesn't make something cheap. Like i said, depends on the use case

@kphamcao Год назад

Apache Flink renders this talk obsolete.

@richardwang3438 3 года назад

works like a charm, super easy, oh let me take a drink

@user-xb5lq6ve3n 2 года назад

「あなたの動画はとても良いですし、メッセージがた

@humanyoda 4 года назад

So, in order for my bank to tell me my current account balance, it would have to process thousands of my transactions all the way to my initial deposit?! Wouldn't that slow down the system to a crawl?

@dmsanz_youtube 4 года назад

yes, but with CQRS your query would not be made against the "write" side but against the "read" side. The read side replays all the events in order to infer the balance and creates a read model. You can think of it as an infinite cache. So whenever you want to query info such as your balance, the operation will be lightning fast as the read model will be just a view ready to serve the info in the same way the info is required, without transformation of data on this specific operation and of course without join query on a database. In other words, CQRS will take care of this as long as you're willing to tolerate a bit of eventual consistency (i.e: the gap between the write transaction ends and the read model that projects the resulting event is ready)

@a0flj0 3 года назад

@@dmsanz_youtube I think a better answer would be that CQRS/event sourcing isn't necessarily the best solution for any problem. IME, whenever you need read optimization on real-time data, the best solution is a mix - you pre-aggregate old data.

@salvatoreshiggerino6810 5 лет назад

Optimistic Concurrency Control is nice in a lot of places, but in this case, isn't it easier to keep the state of the whole system (or at least the subsystem that needs to be consistent) in a single thread that you submit commands to that get validated and turned into events in a fixed order that is guaranteed to be valid? As he said, it's usually neither I/O nor CPU intensive, so that single thread is going to be fast enough to deal with an awful lot of traffic.

@oreoluwapeace510 4 года назад

That would fail in a load balanced system. multiple commands can still be processed on different instances of the service.

@a0flj0 3 года назад

Low reliability (you run on a single machine), limited scalability.

@salvatoreshiggerino6810 3 года назад

@@a0flj0 Not unreliable at all. You start a new process with the same events and you end up with the same state. If recovery from failure needs to be instant, use a hot standby. In fact not having nearly as many moving parts makes it much more reliable. As for limited scalability, that's true. But anything up to a few hundred thousand transactions per second is perfectly feasible on one thread. And if it's a problem you can't scale with sharding, you're going to struggle with scaling no matter what architecture you use.

@a0flj0 3 года назад

@@salvatoreshiggerino6810 You need to put some transactional mechanism around event/message delivery, or else you risk loosing data, with a single thread. That's what I meant with not reliable. With such a mechanism in place, I doubt that a large number of events per second on a single thread are still feasible. The thread can easily handle the load, but most likely the network can't. Event sourcing without network involvement makes no sense to me. If you spread load among multiple instances, time and causal ordering are no longer simple issues, which makes the problem even bigger. But you are right, sharding can help a lot - if you can do reasonable sharding considering causal ordering.

@salvatoreshiggerino6810 3 года назад

@@a0flj0 Those are already solved problems. Look up the LMAX architecture.

@alexd7466 4 года назад

he is making it more complicated than needed

@flolu 4 года назад

What does he over-complicate? I would be very interested which points can be simplified :)

@heiko3169 4 года назад

3:51 is basically a mis-conception of microservices at all

@vorandrew 2 года назад

Must prohibit drinking during presentation

@socialcatalyst2608 2 года назад

English?)) No)) Engldeutsch )))) 👍

@nexus888 5 лет назад

mate, you are really thirsty! :)

@lucioluciolucio 4 года назад

Kinda gross him gulping the water with the mic on

@zahirjacobs716 3 года назад

probably a soothing behaviour.

@CosasCotidianas 3 года назад

The explanation is GOLD, but please let that bottle take a rest man.

@No1Melman 3 года назад

Not just me then 😅 did he go in super dehydrated or something 😅 Quality talk though!

@mati1979b 5 лет назад

This is so super easy that I am not sure why there is even a presentation? :)

@matrixlnmi169 5 лет назад

Suggestion- Next time avoid drinking water style showcase please , I was using headphone and yours water sound through your throat was annoying. BTW very good presentation