The sad reality is to the initially question is that sometimes we DO have time to do it twice. Why? Because projects set unrealistic expectations for code freezes and release but build in obscene “testing” cycles in which you can essentially rewrite the entire Feature as long as you’ve said you finished the first time. I’ve come across quite a few of these types of companies. As much as one pushes to do it right the first time, it’s not always possible given the project dynamics.
The problem with "do it right the first time" approach is that its an all or nothing mindset. You sometimes dont have the time or knowledge to do it right the first time so you do a far from perfect solution that is still enough to drive the business forward and thats all that matters. The second problem i have with his database approach is that resolving data inconsistencies via compensating actions can quickly become a major source of data inconsistency bugs. Its a lot harder to maintain data consistency without the help a relational database provides in terms of transaction support.
Great talk, but one has to have a great ROI to choose Micro Services.. Sometimes or most of the time it is a showoff from the point of architect, trying to force
What you could do is say the transaction rollback is happening from service D -> C -> B -> A. For example if there was a failure when D sent the event to C. You could have a journal of events. D would send the event once and maybe receive a confirmation whether the event was consumed or not. If D didn't receive a confirmation, it could retry sending the event again. On service C side, let's say it received the event, it could store it in its local journal and try to consume it. If for any reason the event consumption failed, it could retry consuming failed events later or maybe a patch fix could be sent to service C after identifying the bug. Since the failed event is still present in its local journal. It would again try consuming it and maybe succeed now. After succeeding it would pass the event down to event B and so forth.
@@zpengh What if the action you take when the rollback fails also fails? What if the code that handles that failure also fails? There's no real guarantee on consistency if you accept the fact that any part of your system can fail, but having good testing and resilience of your systems combined with a good set of tools for your operations teams is a solid way of keeping on top of things. Good alerting, tracing, and logging will be essential to this, along with making sure that your systems/services are written defensively to notice malformed or inconsistent data and fail gracefully.
Around 38:41, for materializing the view , he says bunch of items and feedback about each item is a many to many relationship. How ? 1 item can have N feedbacks. Can 1 feedback to associated with M items ? ( Understandable if it's order feedback and each order feedback consists of multiple item feedbacks)
Not clear abt sagas - how you do "rollback" (compensating) if process broke somewhere in between ? who initializes rollback process? Will that compensation event go to both direction - to A and to D services?....
what does it mean to have read-only, non authoritative cache? The fulfilment service doesn't store customer's data in it's database right? where is it stored?
Wait, back up, that last part didn't make sense. Seriously though, great talk. This is a topic that is much harder in practice than the theoretical ideals make it sound, and the speaker clearly understands that and highlights the key aspects that teams must keep in mind throughout their design and implementations. At times it can make you question the ultimate benefits of such an architecture vs the new challenges it creates. And I think that the key is the understanding the problem domain, resources, and true requirements.
For those thinking to replace transactions with Sagas: don't! ACID was mentioned in the talk but Sagas are NOT ACID! en.wikipedia.org/wiki/ACID_(computer_science). Sagas don't deal with isolation, the I in ACID! Atomicity would also be hard to guarantee because systems can fail on rollback.
transactions have existed before computers and RDBs. you can get transactions by having data structures that are amenable to everybody converging on the same state. (ie: compensating transactions). when you are mutating existing records in a DB (which you dont have to do), you are creating a problem that requires the transactions in the first place; namely, undoing updates made to records. more abstractly: if you use a sharpie marker to draw out immutable functional data structures, you have not lost information when you add more structures (ie: extending a linked list where new nodes backpoint to old nodes). if you number new nodes with the current transaction, you can see that if you can't complete the transaction by adding the final node, then you just delete (or ignore) the new objects you were adding. if you use immutable append-only structs, "rollback" is trivial. of course if you have an actual side-effect (ie: launch the missles), then you can't roll back. but an RDB won't solve that problem either. if you want a consistent and mutable database, then you are saying that it's ok for all updates to stop once two halves of the network become isolated, and a vote on what the next state change is cannot pass.
if you don't start with microservices, you are in reality falling prey to "you don't have time to do it right ? Do you have time to do it twice?". Migrating to microservices is far more complex and time consuming than just doing it right to first time. This is totally avoidable tech debt.
So basically you are sacrificing all the built in features of relational databases such as transactions and implementing your own version of transactions - to me seems like lots of extra work and architectural burden to maintain.
Muhammad Kamran, if you listen to the speaker he is saying that you do this when you need to scale. It doesn't make sense if you don't need to scale. So, this isn't a problem for most companies and shouldn't be implemented at most companies.
I think Muhammad has a point here, though, because what seems silly here as that the primary recommendation is to split core tables to 'service-per-table', which is likely to end up being pretty naive for most monolithic databases. Considering bounded contexts and domains may be more effective _and_ more efficient.
dovh49 seems like there are other ways to scale without throwing out decades of database technology and research (clustering? sharding?), but maybe that's just me
Unfortunately those scaling technologies cost a lot of money and effort to run. Thankfully we have Amazon RDS that does quite a bit for us and even if it is costly, at least they provide that service that can scale up much better than an in-house database team for the most part.
Archimedes Trajano reinventing the wheel is surely going to be more costly in the long run than leasing someone else's working wheel. When you are in APPLICATION development, you should be caring about business application logic, not about conundrums of distributed systems design.
hi, i just want to know how to handle real time data that requires join. If we use a some event to propagate data from services to maintain cache in some other service. How we maintain consistency
You have to change the way you think. Leave SQL behind when you go distributed. You have to reuse the old architecture of hierarchy that was the norm before SQL (Think foxPRO, dBase). So basically key value pairs, mongoDB and friends. The "re-creating" of the join mechanism seems stupid a bit.
where is the link to this new open source tech which can check the architectural efficiency of the microservices based systems...this DTMS he mentions? DTMS(Does This Make Sense) Jokes apart...great insight. thanks.
Nope: 4 months to deliver a perfect system; How to do that right? Software is never complete, it’s just a version... in reality we do not have time to do it right in all cases...
I am not quite sure that every microservice project must used its own database and did all that problematic corrections to have consistence data. If you project is big and you want to scale or you need performance out of db, then you can do that.
The point is that microservices should be individually scalable. If you have a shared database, thats a single point of failure and also a shared dependency. You can't scale the database without affecting all the connected services. That breaks the rules of microservices being independent. Take this example: You use STRIPE for your payments processing. You share the database with Stripe because "it makes everything easier", then STRIPE decides it needs to take their database offline to update it or scale it or even change it to another technology. That now affects YOU. That is the fundamental point of microservices - you bring all those dependencies into a single place that relates to one service and you communicate with the other services in your application by decoupled methods
By the time this company completes moving all of their entities from the monolithic shared database he references into separate databases, microservices will no longer be in style.
I don't usually comment on tech talks but I'm a bit upset I wasted an hour or so. Long exposition, contrived examples, zero useful information. The only way this whole talk really makes sense (pun intended) is if you consider it to be a show-off for potential investors.
So instead of managing one database (availability, backup and recovery, administration [capacity,performance,deployment,upgrades]) you manage "N" databases. "Makes sense". Makes even more sense with sharded databases - then you multiply "N" by "M" (number of shards)
When I watched some early presentations about this shit, I thought they were contriving examples to make them more approachable, but they actually seem serious about it. One database per table?!...
@Yahor Sinkevich For the great majority of applications scalability of this kind is not an issue at all. The pain in the ass caused by implementing caching and consistency management not to mention simple joins with distributed data without ACID properties is, on the other hand, quite certain. And what is a "service", how do you defined which data belongs to one service and which to another - when it's most likely that you will need to join it at one time or another (else you would have two different applications).
"Don't use 2PC for managing ACID transactions - bad for scalability" and then proceeds to re-invent a distributed transaction management system with what he calls "SAGA" and events and state machines! What if the compensating "rollbacks" themselves fail??
Stitch Fix has a pretty meh rating of 2.5 stars out of 5 on consumer affairs. 100 data scientists and 100 software engineers can't do anything but polish a turd. Typical wall of greedy VC money in desperate (failed) search for an idea that will make a ton of cash.
