Kafka Streams 101: Getting Started (2023)

Подписаться 71 тыс.

Просмотров 102 тыс.

50% 1

► TRY THIS YOURSELF: cnfl.io/kafka-streams-101-mod...
In this course, Sophie Blee-Goldman (Apache Kafka® Committer and Software Engineer) gets you started with Kafka Streams.
Kafka Streams is declarative, so you state what you want to do, rather than how to do it.
Imagine that you have a topic, from which you'd like to filter all records marked with the color "red." You could accomplish this with plain Kafka, but the equivalent Kafka Streams code would only take a third of the lines.
For a complete IMMERSIVE HANDS-ON EXPERIENCE: cnfl.io/kafka-streams-101-mod...
--
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion - designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#kafka #kafkastreams #streamprocessing #apachekafka #confluent

Наука

Опубликовано:

28 май 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 34

@elbowdashizzle Месяц назад

The subscribe button in RU-vid lit up when you said “subscribe to the topic” 😂

@sonnyps413 2 года назад

4:50 Is when she starts explaining about Kafka Stream

@johnkelty4001 2 года назад

Thanks, this is very helpful info in a short amount of time!

@tomknud 2 года назад

Good talk, thanks

@mehrdadk.6816 Год назад

That was a great introduction

@krishnasharma-nr3yg Год назад

Thanks, very helpful

@ankitlakum1 2 года назад

Good info 😊

@jasper5016 8 месяцев назад

Fantastic tutorial. Thanks so much Sophie!

@kartikprabakara3126 2 года назад

Excellent

@komiljonaliev3415 Год назад

Thank you for giving helpful information

@SzymekCRX 4 месяца назад

Thank You!

@ighsight Год назад

Excellent. Wondering can I use Kafka Streams with the KafkaJS library for Node JS?

@adityanjsg99 9 месяцев назад

you may.. i am using with python

@avimehenwal Год назад

Does kafkajs in nodejs provide support for kafka streaming API ?

@KarthikeyanSkarthikeyanmadurai Год назад

okay my long pending doubt is cleared now 🙏🏽

@javadahmadzadeh7129 Год назад

Thanks. it was really helpful but for me as a non-native English speaker, the speed of talking was so fast, and I had to use subtitles for better understanding

@sapientum8 2 года назад

@stream2learn 2 года назад

Can we use Kafka Streams and Spark Streaming in the same project ?

@ConfluentDevXTeam Год назад

Hi, I'm Gilles Philippart from Confluent. I don't think there's anything that would prevent you from doing so, Kafka Streams is just another library that you pull into your application dependencies after all. I'm not sure there would be a good reason to do that though, as it might prove harder to tune the performance, scale, secure, troubleshoot, etc. Have you considered creating another microservice that would consume or produce to a topic to interact with the Spark Streaming app?

@shyammohan2007 Год назад

If each topic is a file, how is about a partition? Does it means the aforesaid file will be divided into a subset of file again where each sub-file is a partition?

@ConfluentDevXTeam Год назад

Hi Shyam, I'm Gilles Philippart from Confluent. Sophie says things like "This log is a file and it's a special kind of file" then later on "the topic itself really is just a name that is given to the log. So on the broker, they store each log and directory names by that topic". I agree that this part is a bit confusing as we normally associate a " Log" to a "Partition" not a topic. Sophie glossed over the details of how the data is stored physically on disk because that's not really the purpose of the course, but let me get this right: 1. A "topic" is a logical stream of records, and it is the primary unit of data organization in Kafka. 2. A "topic partition" is a logical unit that represents a subset of the data within a Kafka topic. Each partition is an ordered and immutable sequence of records. 3. Now, Kafka stores the data for each topic partition in a set of "log segments". Each log segment is a file on disk and represents a sequential chunk of data. The size of each log segment is configurable and typically ranges from a few kilobytes to several gigabytes. As new records are appended to a partition, they are written to the active log segment. For example, the directory structure of the first partition of the "account-deposits" topic would look like this: /var/lib/kafka/data/account-deposits-1 00000000000047926734.log 00000000000047926734.index ... 00000000000052497535.log 00000000000052497535.index If you want to learn about Kafka internals, check out developer.confluent.io/learn-kafka/architecture/broker/ Hope this helps!

@sghsghdk Год назад

So kafka streams is just a convinience client api? will the network traffic be equivalent?

@gillesphilippart5986 Год назад

Kafka Streams is a streaming processing library to join streams (topics) and simplify the building of stateful applications (windowing, aggregations). If might consume more network traffic due to the fact that most of the local state is also stored in Kafka for backup, but that's usually a good thing as it make your application more reliable!