"Consistency without consensus in production systems" by Peter Bourgon

Подписаться 83 тыс.

Просмотров 24 тыс.

50% 1

When software developers think of distributed systems, they typically think in terms of consistency via consensus. That is, enabling a heterogeneous group of systems to agree on facts, while remaining robust in the face of failure. But, as any distributed systems engineer can attest, it's harder than it sounds. Failure happens in myriad, byzantine ways, and failure modes interact unpredictably. Reliable distributed systems need more than competent engineering: they need a robust theoretical foundation.
This talk describes the state of the art in consensus-free consistency: the CALM theorem, "ACID 2.0" semantics, and CRDTs. Roshi-a production CRDT system built at SoundCloud to serve high-volume time-series data-serves as an example. The goal is to provide engineers in industry with a set of powerful building blocks, which will enable them build and improve their own distributed data systems.
Peter Bourgon
SoundCloud
@peterbourgon
Peter Bourgon is a distributed system engineer who has seen things. He works at SoundCloud, building and improving their data systems and infrastructure.

Наука

Опубликовано:

5 окт 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 11

@rangelspasov 10 лет назад

Great talk for anyone interested in CAP, ACID, CRDTs, and distributed databases/systems in general.

@pengdu7751 2 года назад

excellent talk

@gabrielgiussi4422 2 года назад

My two cents, CRDTs give you causal consistency which is much more powerful than eventual consistency.

@Manish10napstar 2 года назад

While a read query in the proposed system is returning from the single pool, and listeners doing read repair in the background, won't the system return inconsistent data, as it is returning the value read from the single pool ?

@KacperPerschke 3 года назад

I'm just referring to corba. From what I understand, Plan 9 Filesystem Protocol ( en.wikipedia.org/wiki/9P_(protocol) ) was already, promised less and had working impelmentation. Rob Pike still hopes for a successful follow-up ( ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-ENLWEfi0Tkg.html ).

@pm71241 9 лет назад

regarding "ACID 2.0". The argument is correct but the slides are flawed. It's not because 1U1=1 that it's Idempotent. It's because 1U1U1=1 too.

@zantrua 8 лет назад

+Peter Mogensen 1U1=1 => (1U1)U1=1

@ruslanfadeev3113 5 лет назад

@@zantrua 1U2U2={1,2} is also necessary for idempotence, but does not follow from 1U2={1,2}

@lootic 4 года назад

@@ruslanfadeev3113 It does follow from the rule stated though: 2U2=2 1U2={1,2} 1U(2U2)={1,2} (replacement 2=2U2)

@carlosmiguelsoto3853 Год назад

An element e is idempotent iff e*e = e (for some operation *). In this case you only need 1 U 1 = 1 for it to be idempotent. You're thinking of the function f : Set -> Set, f(s) = s U 1. To prove that f is idempotent you need to check that f o f = f, which is the same as f(f(s)) = f(s) for all s, which is the same as s U 1 U 1 = s U 1, which is what you're saying.

@houcebr 5 лет назад

Interesting like any talk given by Peter A question (that won’t never be answered here) as a user in my stream how do I get the aggregated data for all artists across all partitions ? And how do they deal with pagination and so on ?