Kafka in 100 Seconds 

The thumbnail is unexplainably aesthetic, please dont stop this series , always learning something new from these

Hey man, love your videos, but a small correction for what you mention at 1:45 "Kafka guarantees that any consumer of a given topic will always read the event in the exact same order" That's true but at partition level. A topic is made of partitions, and a topic with 1 partition will guarantee that. But as soon as you need parallel consumption you need more partitions (one per concurrent consumer). When having multiple partitions, a single, non-concurrent consumer will get the events of any partition in the proper order but events from different partitions may be consumed with different interleaving This is why it's so important to put a proper key for your messages, as it's the default value used for sending your message to different partitions (different events with the same key will always end up in the same partition) Again, love your content, you explain super great. But I thought that that small point had to be addressed 😄

Totally not here while searching for kafka Honkai star rail..😂😂😂

hey this is not the kafka I am searching for, but thank you fireship

Everyone is gangsta until you have to handle reset flow

HERE'S TO EVERYONE WINNING THEIR 50/50 RAHHHHHHHHH 🗣‼🔥💯

There's a different Kafka that's even better 👀

Good time to be a coder and a Honkai star rail fan.

I'd love a 100 seconds on a CI tool like Jenkins, your videos are always so informative :)

I'd love to see a quick into to Event Sourcing on this channel!

Please make comprehensive videos on topic like Kafka, I think 100s videos are good but they are getting boring now

This series never fails to teach me about things I didn't know I needed to know about.

Please, do a video in beyond fireship about kafka :) Just to you know, nice job

The Brazilian Central Bank uses Kafka in its PIX system, that makes money transactions between individuals or entities in real time between bank accounts in any Brazilian bank. If anything, that show how powerful Kafka can be (2 billion transactions per month, with almost R$1 billion - which is roughly US$200 million - moved between accounts in that period).

0:10 yesss, i was hoping that you will make this reference, and you did! yaay also great video as always

I have a theory about why Kafka the messaging system is named Kafka after Franz Kafka the writer. 'Das Schloss'(The Castle) is a Novel written by Franz Kafka in which he describes a castle having occupants structured as the ultimate bureaucracy. Members of the Castle does not know each other and communicate via a paper based messaging system. You don't want to deal with such a bureaucracy, but you certainly would want to design large complex systems like so: Independent services not knowing about each other communicationg via an asynchronous messaging system. I think that is why Kafka is named Kafka, although just a personal theory...

One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin.

You should make a "Qt in 100 seconds" :3 its a very very popular native Desktop Development framework that can compile for nearly any platform and is written in C++ Many of your production software like earlier iterations of Photo Shop or After Effects, currenr Davinci Resolve, OBS, Krita and many more are written with Qt and still holds strong as one of the best native GUI solutions (native, aka. no web related stuff)

What do you get from searching Kafka? Honkai kafka Franz kafka Kafkaqesue Apache kafka

Thought this was meme compilation about Kafka from Honkai Star Rail for a sec

I love honkai star rail 😨

2:20 in java, since you’re implementing single-method interfaces you can just make it a lambda and can omit the argument types

I always learn something new from these vids they're truly great

I thought this was about the author. :(

This feels like ROS, which could be an interesting 100 seconds video.

As someone who works for Confluent and has worked on Kafka for some time now, I absolutely love your video. 🔥

help i thought that you meant honkai: starrail kafka

1:50 it is a little disturbing how fireship replaces the cute cats for a single word "Consume"

Imagine a Kafka topic is like a multi-lane highway, where each lane (partition) has cars (messages) driving in a single direction. *Order Guarantee:* Within each lane (partition), cars (messages) drive one after the other in a strict sequence. If you're observing just one lane, you'll always see cars in the exact order they entered that lane. *Multiple Lanes (Partitions):* If you're watching the whole highway (the entire topic with multiple partitions), cars from different lanes might cross you at slightly different times. So, while cars within each lane are in order, across lanes, they might appear mixed up. *Why Lanes (Partitions) Matter:* More lanes mean more cars can drive simultaneously, allowing for faster traffic flow. In Kafka terms, this means more consumers can read messages concurrently, leading to faster data processing. *Choosing the Right Lane with a Key:* When a car (message) enters the highway (topic), it needs to pick a lane (partition). This choice is based on the car's license plate number (message key). Cars with the same license plate number will always choose the same lane. In Kafka, messages with the same key always go to the same partition, ensuring they're read in order. In summary, while Kafka keeps the order of messages within each partition, when you have multiple partitions, the order of messages across them can be mixed. Choosing the right key for your messages ensures they land in the expected partition.

omg this is exactly what I needed lol my new job requires Kafka. Thanks!

well, time to say bye. boom.

can u do AutoHotkey in 100 seconds? i think that would be cool. the release of v2 changes a lot of the syntax from v1.1

As someone who uses Kafka at work, it is so shit and I wouldnt wish it on any dev

Kafka rerun in 2.0❤❤🔥🔥🔥 i wanna get her lc

Was expecting a philosophical analysis

I’m honestly tired with all these frameworks

Not sure about the developer but it's a good channel for a Product manager who can look at multiple techs from the top without going deep i to weeds

Great video! I would love to see a video of "Scala in 100 seconds" :)

Do you actually know Java? Because it would explain the lack of sanity

Much awaited, thanks!

I was expecting a video on Franz Kafka.

I didn't think I'd be interested in Kafka but I'm glad I watched. You're like a tech magazine, broadening my horizon.

Fireship in 100 Seconds

The statement about guaranteed order of reading the messages from a topic is misleading at best, as it only applies only to single partition topics. And topic partitioning is a big part of what makes kafka scalable.

What’s more is you can use consumer groups to split the workload if you have massive data to compute. And what is more is you can use different strategies to reassign partitions to new workers to increase ingestion throughput

In an Iron Man movie, Kafka could have been useful in a scene where Tony Stark (Iron Man) needs to process and manage a large volume of real-time data or communications. Here's a hypothetical scenario where Kafka could play a role: Scene: Tony Stark is in his high-tech lab, and he's remotely controlling his Iron Man suit, which is deployed in a distant location to handle a crisis. He needs to receive and process real-time data from various sensors on the suit, such as vital signs, telemetry data, and external environmental data, while also receiving live video feeds. How Kafka could be useful: 1. **Real-time Data Ingestion**: Kafka could be used to ingest data from these sensors and video feeds in real-time. Each type of data (vital signs, telemetry, video) could be treated as a separate Kafka topic. 2. **Data Processing**: Tony needs to process this data for real-time decision-making. Kafka Streams, a component of Kafka, could be used to perform real-time data processing, such as analyzing vital signs for signs of distress, stabilizing the suit's functions, and identifying threats in the video feed. 3. **Reliability**: In a high-stakes situation like this, Kafka's reliability ensures that no data is lost. If there are network interruptions or delays, Kafka can buffer and replay messages, ensuring that Tony has access to all the critical data. 4. **Scalability**: If the crisis intensifies and more data needs to be processed, Kafka can scale horizontally by adding more Kafka brokers, allowing Tony to handle the increased data flow without performance issues. 5. **Monitoring**: Kafka provides extensive monitoring capabilities, which could be depicted in the movie as Tony monitoring the health of the data pipeline in real-time, ensuring that he has a clear view of the suit's status. In this scenario, Kafka would enable Tony Stark to efficiently manage and respond to real-time data, enhancing his ability to control the Iron Man suit and handle the crisis effectively. It would add a layer of realism to the technological aspects of the movie.

not the kafka i was hoping for

You read my mind, I’ve been hoping you’d cover Kafka for months now!

Request for GTK(specifically GTK4 and libadwaita). Its really hard to understand for absolute beginners. And also how to read its documentation and how the documentation are relevant for PyGTK, or Relm or GTK-rs. For example, its really confusing for devs who started by learning web technologies first and also its quite different from XAML based winapps. So it will be really be helpful if you guide us.

I swear your worse than Alexa. I was just talking to a buddy yesterday about kafka and here's the video... 🤣🤣Great video like always!!!!

Pretty sure more people are making new web libraries than actually building websites that people actually use at this point.

Please do hot wire (/howired) as an alternative to regular web architecture

It also used with frameworks as Apache Flink or Apache Spark combined together for distributed event system such as streaming , application to alert ⚠ on some event and now with many machine learning Apps. Great explanation consise love u're 100s @fireship

Dude slow down. You lost me at 00:01

Love this explanation!

Umm FYI you don't need zookeeper anymore? Also another FYI no one needs Kafka stop. Your startup doesn't need google scale. Unless you are already making 100's of millions of dollars please don't use Kafka!

1:09 Rest in Peace party guy from that Radio Disney song from my childhood.

I always thought it was called Kakfa because modern web development is a bewildering and occasionally infuriating jumble of nonsense

I've watched this before doing the quickstart on apache kafka website, to have a general idea of what it was, and ended up with a really abstract idea of it. Then after doing the quickstart, and playing with it a little bit, I re watched this video and now it felt like the perfect summary, all became crystal clear! I'll encourage anyone who feels like it is too abstract or is bit confused to follow similar steps :)

But why.?! @Fireship I think topics like this should have a comparison on the why to use this VERSUS a simpler concept say a SQL database. Kafka good for data immutability and distributed scaling?, SQL for data integrity and little data? So SQL good for a single user todo app, Kafka for a collab project-mgmt with real time updates on todo completion... _Stupid hard to tl;dr something like this_. If you don't know what "Message Queueing for real-time distributed streaming data pipelines.. " means then even helpful introductory vids like this sound like the video for the Rockwell Retro Encabulator, or describing a monad. Good for inoculation though. As we get deeper into development concepts and specialist niche software the WHY gets lost easily and dogma enters; such as assuming something like kafka can outright replace a database (aka "Streams Everywhere" makes everything look like a nail). Along with the worse dev sin: Docs written for those who know so if you know you know thus the docs are pointless, if you don't know the docs make no sense. Ight Imma Head Out To The Mountains But why? ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-zWH_9VRWn8Y.html

My dear Jeff, we wait prolog in 100 seconds, we want it NOW.

Wow. I started working on Kafka and had no clue what it was. After months got a basic idea but this video makes it so helpful

Please make a Kafka for Haters video where you mention people overusing Kafka like using it as a relational database replacement. Would be funny to show at my work. I feel like people start using Kafka for more than they should just because it's easy to put up a streams applications and do some filters. Not even a small company, bmw. And they have different pods calling each other to know who has what information because the data is partitioned across pods according to the Kafka partitions

Bro you should make a video about delphi

To use Kafka do we have to learn Java? If you only know Python or Go do you have to use like Azure event hubs instead?

Can you make a video about Windows driver development? Beeing able to write programs to execute in the kernel sounds very powerful but there's not realy mouch documentation exept of Microsoft's own. I found this topic quite hard to get started with. A fireship video would be amazing.

I love how he didn't use lambdas just to make java appear worst (as if java doesn't have bad syntax already)

Also come here from the wrong Kafka. but great re-cap video after all.

1:42 are you my mummy?

As Fireship awoke one morning from uneasy dreams he found himself transformed in his bed into a gigantic insect

So kafka has more throughput than RabbitMQ? Thought it was the other way

excellent timing! Had kafka come up in a meeting at work last week and didn’t know anything about it.

First! Also as my request, please do Love2D in 100 seconds next!

Thanks for info! Can you please do a full video to teach js ?

I thought this was the algorithm

Can we get a longer like 10 min long video, that explains it more.

Suggestion: OpenSSL in 100 seconds

Day 1 Billion of asking what's the name of the background song 🤒😶‍🌫️

how about a comparison between apache kafka and redpanda data?

NATS is another great tool that does sort of the same thing as Kafka. it's JetStream module is amazing!

but what's the point of events? what need isbeing metbybeingable to reliably stream sequential data to multiple parties? can't this be done much more simply? or is scalability really that important?

watching this channel has made me think creating languages is just a walk in the park lol

please make video on node server less Frame work

I just started at a new position and needed to understand Kafka. Wow, Fireship keeps reading my mind

what is the app name that you use to start kafka and zookeper?

Content of videos on this channel should be in first paragraph of projects readme, just because it short and complete.

Been waiting for this video for a while maybe do couple deep dives on kafka?

Hadoop in 100 seconds

1:09 aaron carter came out of the blue

Can you also make a video about V programming

this video was very Kafka-esque

Ordering is guaranteed only per Topic Partition.

I'm shocked that I've been studying now 6 years of CS and I have almost no idea what was said in the video. Welp, better prepare for that class that I will never use in my life if I wanna pass🤷‍♂

i was the crazy developer about 3 years ago for wanting to introduce Kafka in a project which was having load problems, great now Kafka is famous.

Understood nothing... I didn't get an overview of what is the problem that kafka is trying to solve.

In Hyperledger Fabric, which I use for our Thesis, it uses Kafka for consensus. Like, leader election and storing of data.

I worry that it's harmful to suggest running ZooKeeper in CURRENT_YEAR. Specifying CLUSTER_ID and NODE_ID instead would be better.

An extra detail. RabbitMQ has launched a Streams last year, improving massively its throughput

Sadly, no mention about partitions, keys, and group_ids

I felt this is the ideal speed (or rate of speech) for your videos, otherwise usually they're so sped up that it gets hard to follow along with what you're saying. Besides, the visuals are (sometimes) can be distracting. By the time I focus my attention on the concept that you're explaining, you're already light years ahead. Love to watch your videos/content, but this was a common obstacle 😵‍💫