"Systems that run forever self-heal and scale" by Joe Armstrong (2013)

Подписаться 82 тыс.

Просмотров 73 тыс.

50% 1

How can we build large self-healing scalable systems?
In this talk I will outline the architectural principles needed for building scalable fault-tolerant systems. I'll talk about building systems from small isolated parallel components which communicate though well-defined protocols.
Programs will have errors in them and will fail so I'll talk about detecting and correcting errors at run-time. Programs will evolve with time, so I'll talk about how they be changed while they are running. I'll talk about Erlang and how it relates to these architectural principles.
Joe Armstrong ( / joeerl )
Joe Armstrong is one of the inventors of Erlang. When at the Ericsson computer science lab in 1986, he was part of the team who designed and implemented the first version of Erlang. He has written several Erlang books including Programming Erlang Software for a Concurrent World. Joe held the first ever Erlang course and has taught Erlang to hundreds of programmers and held many lectures and keynotes describing the technology.
Joe has a PhD in computer science from the Royal Institute of Technology in Stockholm, Sweden and is an expert in the construction of fault tolerant systems. Joe was the chief software architect of the project which produced the Erlang OTP system. He has worked as an entrepreneur in one of the first Erlang startups (Bluetail) and has worked for 30 years in industry and research.

Наука

Опубликовано:

5 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 27

@wuschelthepuschel 3 года назад

I upvoted this before seeing it because there is no such thing as a bad Joe Armstrong talk.

@user-uq5qw1fk3d 3 года назад

I loved Joe's The Mess We're In talk and have rewatched it multiple times over the years and am looking forward to watching this. Thanks for uploading all the older content recently!

@AFerreiraV 2 года назад

I never get tired of reviewing Joe's past talks. I didn't know him personally and I didn't always agree with him on some non-technical opinions but his disappearance was something I felt personally.

@linkernick5379 3 года назад

I had the first enlightenment in my way of grasping CS at the moment of the end of reading Joe Armstrong's thesis.

@udirt Год назад

I love the all sequential programming languages get error handling wrong, this would better explain the communication barrier admins face reporting structural errors to devs than any other explanation ever thought of.

@mahdi2761 3 года назад

RIP Joe

3 года назад

"Scaling down" sounds good, if you know what system you're building before you start. If you don't, which is always the case, you're basically building your best guess of what the system should be and then iterating and changing it. Doing this with an already scaled up system seems unnecessarily complex to me. You first need to find your problems, fix them, and then worry about scaling those fixes up.

@LeviRamsey 3 года назад

The problem with that approach is that it's amazingly easy to (especially via premature optimization: it's fairly common that algorithms that are slow (even Big-O slow) don't parallelize well, especially when parallelism entails network communication) fix problems at small scales in ways that will require a top-down rewrite to work at larger scales.

@Kenbomp 3 года назад

Brilliant.

@csbnikhil 2 года назад

I feel happy listening to him.

@laughingvampire7555 11 месяцев назад

sensei 🙇🏻

@a0um 3 года назад

He glossed over runtime upgrades… are they really working smoothly? Any pointer for further study would be greatly appreciated.

@davidjohnston4240 Год назад

That's Erlang's best party trick. You can find videos of examples on youtube.

@Oktokolo Год назад

Runtime upgrades definitely do work, but you probably don't need them. If you have multiple machines, you can always just stop a machine from accepting more work, wait, till it is idle, shut it down, update and start it again. It probably was more useful in the mainframe era, where a single machine represented a huge chunk of the combined processing power and you didn't want to restart them for a minor bug fix.

@lepidoptera9337 Год назад

@@Oktokolo The limitations on old hardware were IO based. Unloading and re-loading memory was expensive. Today a well architected SSD array is close to saturating main memory bandwidth. A reliable architecture has to be able to freeze state anyway.

@AlexRodriguez-gb9ez Месяц назад

Makes me want to learn Elixir... Any starter project ideas？

@owenimholte3835 8 месяцев назад

Whoah. He called the WhatsApp acquisition perfectly! 1:02:00

@triularity 8 месяцев назад

@39:37 - You still need backups. It doesn't matter how reliable your storage is if the data gets corrupted (intentionally or unintentionally) and then gets propagated through out your network. Too late to realize this after your long running system has accumulated month, years, or decades of data, which is now useless due to it being invalidated. Imagine in today's world having someone hack into such a banking system and change everyone's transaction/balance history. Even if you you could detect the data was altered, you still wouldn't know what it is suppose to be. He does mention using snapshots.. but seemingly in the context of short-term restarting, not "I can go back 2 years and do a full data audit", which long term [protected] backups can provide.

@mortenbjoernsvik6550 Год назад

kubernetes done right 🙂

@2brazy4ubitch 9 месяцев назад

RIP

@SneedsFeeduckAndSeeduck Год назад

Distributed computing is easy… until you stop assuming every actor in the system is honest.

@a0um 3 года назад

There seems to be an error at 33:30 where it’s written “World is concurrent“, shouldn’t that be “parallel”? Concurrency is when processes are interleaved on a single executor, parallel is when they run on independent executor, or not?

@justin883 3 года назад

The world is at least concurrent and it might be parallel. Maybe we don't have access to the hardware it runs on to check.

@efraimcardona8452 2 года назад

Concurrency is processes being executed independently taking care at nodes of possibly shared resources. Parallelism means executing all processes at the same time while coordinating them globally with as many processors as needed. That is to say parallelism is a "synchronization" of concurrency