Stream your PostgreSQL changes into Kafka with Debezium 

Thank you! I will start posting again soon, so please let me know what type of content interests you the most.

Thanks for watching!

You are welcome!

Glad : )

Glad you found it helpful : )

Thank you! I will start posting again soon, so please let me know what type of content interests you the most.

Ahh I see! That’s a good callout. I swear it used to work before without the semicolon :/ Glad you found it helpful regardless : )

So nice of you

Hi! Unfortunately, I don't! I do plan on getting to it at some point though. Meanwhile, I am doing most of these data pipeline stuff using AWS. So if you are interested, please check out those videos.

Hi! That's a very good observation. I was actually running into that intermittently. Like 8/10 updates were working properly, but for 2/10 the before was NULL for some reason. Resetting the WAL of the database fixed it every time, but I am not sure why it happens in the first place.

@@irtizahafiz Hi! Before is null because you missed semicolon ';' after ALTER TABLE REPLICA IDENTITY FULL

I don't think I follow the question. If you mean the CDC data from kafka, you can route it to whatever you want to - a different DB, application logic, cache, etc.

I would recommend restarting Postgres after you change its configuration.

@@irtizahafiz Thanks you Sir... Now it is working as expected....

Hi! I can share as much as my iterm command history allows me. If you need anything in particular, let me know. curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" 127.0.0.1:8083/connectors/ --data "@debezium.json" docker run --tty --network postgres_debezium_cdc_default confluentinc/cp-kafkacat kafkacat -b kafka:9092 \ -C -s key=s -s value=avro -r schema-registry:8081 -t postgres.public.student

same question

Hi! Thank you for bringing this up. I think I answered this in another comment. I was running into this bug intermittently. I think it had to do with some misconfiguration of the postgres cluster.

I am actually not familiar with Outbox :(

Typically the outbox pattern is to guarantee once delivery of the message, it's not to perform cdc (imagine a critical service that you must guarantee message delivery). Your application does not send messages to a kafka topic but instead send to a db table. The outbox service then is responsible to see which messages in your table (outbox) have been successfully published to kafka and retries for the ones who haven't.There is a chance that the outbox publishes a event more than once, so your consumers must be idempotent. From the consumers side you may have what would be called an inbox pattern, which blocks the same message to be consumer more than once, or they are okay with receiving the same message more than once. What you may have is CDC with Debezium but instead of publishing directly to kafka it publishes the events to a DB table via outbox.

For consuming data into a target database, you should be able to do it with a Sink Connector of your respective database. Almost all Sink Connectors should be able to read from a Kafka topic. As for consuming Kafka using Python, I have an end to end example of that here in this video. I think you will find it helpful: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-qi7uR3ItaOY.html

Glad it was helpful. I added the idea to my “future videos” list. The next few weeks I will be focusing on the System Design course, so unfortunately it will take some time :(

Unfortunately, I do not. Maybe something in the future.

Many companies use it at scale.

Hi! It's difficult to help without more details.

I can consider making future videos on these. Thank you!

Hi! Unfortunately, I don't think I can be too helpful when it comes to debezium specifics.

I don’t have any experience with it :( So unfortunately I won’t be of much help here sorry.

I dont think so, the process of doing a major version upgrade of a database typically requires some level of downtime. But if you can setup a replica and divert the traffic there for the time being it might be a solution

docker run --tty --network bin_default confluentinc/cp-kafkacat kafkacat -b kafka:9092 -C -s key=s -s value=avro -r schema-registry:8081 -t postgres.public.student this command works for me now.

I do have a few SQL videos coming up soon.

Hi! Sorry it's difficult for me to help without some more context.

I don’t have a video yet, but I do plan on making one soon. : )

@@irtizahafiz Did you happen to create one?

Yes. It supports delete too.

That was happening intermittently. I believe there are some discussions about this in the comments section.

Hi! Yes it is possible! I might be able to demonstrate that some time in the future, but it’s currently not in my plans unfortunately.

It should be similar as long as there's a connector available.

I think you can do a lot with Airflow, but not sure about the specifics.

Potentially. I know that the Kafkacat command has lots of "quality of life improvements" baked in.

Unfortunately I don't have the code for this anymore :(

Make sure you define the network in the docker-compose file.

That has to do with some kind of Postgres configuration. I remember fixing it immediately after the video, but can't remember :(

@@irtizahafiz It is working after below command, because in video it was not executed because you've missed semicolon. ALTER TABLE public.student REPLICA IDENTITY FULL;

Not sure without more details. Can you provide with examples of your payload?

Sorry about that!

Depends on the app you are building. I tried keeping things generic here. The ranking can be based on some score generated by a machine learning model, etc.

Yes definitely!

Yup for sure! Products like AirFlow make data engineering super easy by abstracting away all these.

Are you sure the Kafka topic was created successfully? I would recommend listing the available topics in the broker first.

@@irtizahafiz I'm not sure that kafka topic created successfully because I followed all the instructions in your video :D, how to listing the available topics in the broker? thank you

You can run something like this in your CLI to list the Kafka topics: ./usr/bin/kafka-topics --zookeeper zookeeper:2181 --list

@@irtizahafiz i have the same problem and the topic postgres.public.student exists

it works!! It is sufficient runs the command two or threee times

Hi! I do plan on doing that in the near future. After the system design videos that is.

Thank you for the correction! I will start posting again soon, so please let me know what type of content interests you the most.

I know :( I was getting null every now and then. I believe it was because my Postgres docker container wasn't retaining the WAL setting every time I exited out of it.

I have faced the same issue in windows command prompt, but it works in powershell.

Are you trying it out in Windows or Mac or Linux?

@@prasadkintali6560 i am having same issue in both command prompt and powershell.i am using windows machine. Can anyone help

Hi! I am not really familiar with the project. Just make sure both the Go project and your kafka containers are running on the same network.

Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "-b": executable file not found in $PATH: unknown

Hi! I can look at this later on in the week when I have some time.

Have you created the Kafka topics successfully? You have to register the topic first, before you can consume from it.

Maybe try resetting Docker if it's running already?

please help me .. --network postgres_debezium_cdc_default denotes what?

please i need help...

That's because the network has another name. Print in the command line "docker network ls" and find the network in the list. In my case it was "debezium-default".

@@iuriivdovin731 thanks bro

Thanks for helping out! : )