Create a Bank Account System with Kafka Streams - Stateful Processing

Подписаться 2,9 тыс.

Просмотров 11 тыс.

50% 1

Kafka Streams is well prepared to handle stateful processing of streaming data. It has abstractions, operations and heavy lifting to help us in this difficult task.
In this tutorial we are going to build a Bank Account system to process live transactions and update the balance of each user.
---
Follow me:
Twitter: / progwithmati
LinkedIn: / matiasdsalerno
GitHub: github.com/Programming-with-M...
---
Sections:
00:00 - Intro
00:28 - Why stateful processing?
01:32 - State Stores in Kafka Streams
03-01 - KTable
03:32 - Stream/Table Duality
04:14 - Introducing our tutorial
06:01 - Browsing the source code
09:25 - Implementing the Topology
10:39 - Topology: Aggregate Operation
17:01 - Implementing the Process Transaction method
22:52 - Run application
25:50 - Outro, next episode...
---
Links:
GitHub repository: github.com/Programming-with-M...
Previous tutorials can be found in the GitHub page: programming-with-mati.github....

Наука

Опубликовано:

3 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 17

@TriggerrSupernova 7 месяцев назад

As a beginner to Kafka streams I found the explanation really clear. Great explanation Mati Thanks.

@twodots1990 Год назад

Awesome explanation 👏

@rupeshjha02 2 года назад

Good explanation and well organized videos. Looking forward for more such videos.

@programmingwithmati 2 года назад

Thanks for the feedback! Really appreciated! More videos are on the way!

@pedrocorreia5857 Год назад

Thank you for the information, there really isn't much quality info about streams online so once again thank you and keep up the good work!!!

@programmingwithmati Год назад

Thanks Pedro! I'm glad it was helpful to you

@ayyoubtelmoudy574 Год назад

I miss the words to describe you ,so thank you so much ❤

@programmingwithmati 11 месяцев назад

You are welcome! Thanks for the kind comment!

@thesadanand6599 Год назад

Great Work, There are very few doing playlists on kafka-streams end-to-end. would have been great, if you could have picked some easier short examples to help out begineers.

@programmingwithmati Год назад

Thanks for your kind comments and your feedback!

@iliapetrov1743 2 года назад

Hi Mati! Thanks once again for your videos! Could you please create any example of using Kafka as an interaction system among micro services if possible?)

@programmingwithmati 2 года назад

I will! Currently I’m preparing a video on how to scale a stateful application, which is taking me longer than I expected! Thanks for the feedback!

@pritamshirbhate6801 2 года назад

nice explanation. how can we store the bank balances into the redis db using state store, so that we can fetch it whenever required?

@programmingwithmati 2 года назад

Hi Pritam, thanks for the comment! I haven’t seen an implementation of stateStore with Redis. So, as far as I know, you’ll have to create your own implementation, which you can absolutely do! You can probably look in the internet if someone created an implementation already.

@wangfred Год назад

Hi Mati, great content! I really appreciated your work. I have three questions: 1. noticed in BankBalanceTopology when filter by state==rejected, looks like the stream is's split to two branches, can you explain why we don't nee split().branch() here? 2. when you declare the rejected stream, how does it know which point of the stream it plugs in? this case, look like the order to declare the balance topic and rejected topic doesn't matter. 3. in the second test case, I understand the first balance is 0 and second one is 3000. But for the third one, it should be 3500, why it is 500?

@marquardtsnell7916 8 месяцев назад

Oh boy, a test! I think I got this: 1. split().branch() would create an either/or condition. We still want to send the rejected transactions to the "bank-balances" topic, it's used for the **latestTransaction** field. So, we always send to "bank-balances" topic, but if it's rejected, we *also* send to "rejected-transactions" topic. 2. because of the first answer, you're correct. Once we have enough information to determine the value of the state (after deserialization), we can filter and send in any order. Think of sending to the "bank-balances" topic and filter/send to the "rejected-transactions" as parallel operations. You can do them in either order. 3. Notice the balanceIDs. In the first (-500) and third (+500) transactions, the balance Id is 1. In the second transaction (+3000) the balance Id is 2. The balance Ids are like account numbers, that's how we keep everyone's accounts separate. The transition happens in BankBalance, the BankBalance.id = bankTransaction.balanceId, then in the topology, the transactions are .groupByKey() and aggregated *by those keys*. Keeping balanceId#2's account safely away from balanceId#1!