Тёмный

Source MySQL table data to Kafka | Build JDBC Source Connector | Confluent Connector | Kafka Connect 

The Java Tech Learning
Подписаться 3,6 тыс.
Просмотров 28 тыс.
50% 1

The Kafka Connect JDBC Source connector allows you to import data from any relational database with a JDBC driver into an Apache Kafka® topic.
This video will build JDBC Source Connector to source or stream MySql Table data into Kafka Topic on real time basis. It will explain creating source connector config and then deploy it into Confluent Platform using Control Center Gui.
Like | Subscribe | Share

Опубликовано:

 

4 окт 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 65   
@oyeyemirafiuowolabi2347
@oyeyemirafiuowolabi2347 2 года назад
Dear Vishal, Thank you for your effort & time to produce & upload these tutorials.
@javatechlearning
@javatechlearning 2 года назад
Thanks a million
@manindersingh6371
@manindersingh6371 Год назад
Nice video bro. It worked for me. One quick question for you: How to run query like this: select id,CUST_LN_NBR FROM flex_activity limit 1; In this query I am using limit option in due to that it is getting failed. If I use simple query without limit then it works fine: Error I am getting: org.apache.kafka.connect.errors.ConnectException: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE `ID` > -1 ORDER BY `ID` ASC' at line 1 This is not only with limit option, this is for all whenever you have to apply any filter.
@ttc744
@ttc744 Год назад
Very helpful video. I see the 'timestamp.column.name' should have been TXN_DATE (as flashed up on video) otherwise update wont work. It would have been nice to see this working at the end in addition to adding new records
@amankumar-f4n5r
@amankumar-f4n5r Год назад
AWESOME !!.Can you also upload video on sink connector to oracle database or any rdbms
@resam32
@resam32 11 месяцев назад
Thank for the great tutorial. Did you have to create the avro schema or it gets generated?
@resam32
@resam32 11 месяцев назад
nvm, I found the ans from the video.
@Rajeshkumar-hs5hi
@Rajeshkumar-hs5hi 2 года назад
Hi Vishal, I am facing issue Invalid connector configuration: There are 2 fields that require your attention while connecting to mysql db ..is this required mysql jdbc configuration as well ? if yes how we can do that ?
@ibnu0muhammad
@ibnu0muhammad Год назад
Really cool bro!
@krushnapardeshi3694
@krushnapardeshi3694 2 года назад
I'm beginner on kafka and DB concepts. What is dialect exactly in this case ?
@jinraven77
@jinraven77 2 года назад
Could you please do a Video loading tables into their respective topics for each tables , using SQL SERVER as a source??.... so everytime each table suffers a DML action the new event s sent to the Confluent plataform..
@MohdMahebub
@MohdMahebub Год назад
Hi, Vishal , Could you please help me in pushing confluent topic data into scylladb table...
@mirzauzair6854
@mirzauzair6854 2 года назад
Great video! Can you help with connecting to snowflake as source?
@ravirty8962
@ravirty8962 Год назад
Hi Mirza did you find the answer for "snowflake as source connector"??..if yes please let me know the info.
@Владимир-в1в5ш
@Владимир-в1в5ш 2 года назад
wtf , you are good
@nareshkyv
@nareshkyv Год назад
Hi Vishal thanks for sharing videos. Please let me know where can i check the errors/logs when connectors failed? also logs/errors if any issues in processing data. Please share me paths
@javatechlearning
@javatechlearning Год назад
Where is your confluent server running? On local or confluent cloud?
@carlaguelpa4574
@carlaguelpa4574 2 года назад
Hi Vishal, thank you very much for this! :) I have 1 question, how do you installed the connector plugin for sink ?
@javatechlearning
@javatechlearning 2 года назад
Hi, thank you very for your positive feedback. I have installed it using confluent hub command which you can use from the place where I downloaded confluent bundle
@ravirty8962
@ravirty8962 Год назад
Hi Vishal, Could you please help me with Snowflake Source connector via (JDBC) to unload the data to Confluent Kafka. If there's any sample code please share me the github details.
@oyeyemirafiuowolabi2347
@oyeyemirafiuowolabi2347 2 года назад
Please, what is the best method or connector available to pull or ingest data that normally get updated or changed. For example bank accounts statement. Thanks.
@javatechlearning
@javatechlearning 2 года назад
It depends where the data is present. Kafka provides range of connector to source data. In special cases we could implement our own connector too
@oyeyemirafiuowolabi2347
@oyeyemirafiuowolabi2347 2 года назад
@@javatechlearning If the data is present in a legacy application table that has no column for timestamp. In this case, how can we ingest the records anytime there is update to the already ingested record. Thanks.
@shuchikumari8031
@shuchikumari8031 2 года назад
Sir nice video, but atleast you can upload the code link.
@javatechlearning
@javatechlearning 2 года назад
Sure will provide you
@javatechlearning
@javatechlearning 2 года назад
Please find code link - github.com/vishaluplanchwar/KafkaTutorials
@pavankumarmantha
@pavankumarmantha Год назад
Hey, i tried to connect to postgreSQL and was finally landing with below error in logs Caused by: org.apache.kafka.common.errors.SerializationException: Error serializing Avro message Caused by: org.apache.kafka.connect.errors.DataException: Failed to serialize Avro data from topic any help ?
@javatechlearning
@javatechlearning Год назад
It seems your not using right avro schema to read events from topic. Try comparing topic data samples with your schema
@muralidharan3756
@muralidharan3756 Год назад
hi by using your code connector is got connected but status is got failed and no data is fetched from db can i know what was the reason
@javatechlearning
@javatechlearning Год назад
Did you check connector logs? Any error coming up there ?
@mdicopriatama6380
@mdicopriatama6380 Год назад
hi Vishal, when i want to upload the connector i faced the issue "invalid connector class" i need advice. thanks
@javatechlearning
@javatechlearning Год назад
I thiy you might have not installed connector plugin before uploading
@mdicopriatama6380
@mdicopriatama6380 Год назад
@@javatechlearning yes correct, thank you Vishal, now my another problem with jar jdbc
@mdicopriatama6380
@mdicopriatama6380 Год назад
@@javatechlearning Hai Vishal, i have facing issue no suitable driver found for jdbc mysql, I already put jar file on /kafka/share.. please i need advice. Thank You
@hamzaaitoumghar9359
@hamzaaitoumghar9359 Год назад
can you give us the link for the config file ?
@ankanchanda8272
@ankanchanda8272 2 года назад
Thanks for the video , Vishal. I wanted to know that , we can also source Oracle DB using JDBC Source Connector too, isn't it? And where did you deploy the connector config file? In the Confluent cloud?
@javatechlearning
@javatechlearning 2 года назад
Source Oracle db to Kafka topic is done by jdbc sink connector. Answer to second question I deploy using control center on local machine itself. Watch my installation video
@ankanchanda8272
@ankanchanda8272 2 года назад
@@javatechlearning Ok. Then when is the jdbc source connector used?
@javatechlearning
@javatechlearning 2 года назад
Source topic data to any relational database
@ankanchanda8272
@ankanchanda8272 2 года назад
@@javatechlearning Can you tell whether the confluent platform, if installed in my local machine, is it chargeable? How to continue with confluent platform in my local machine, if you could instruct?
@javatechlearning
@javatechlearning 2 года назад
Confluent platform for single node machine is free. You can download and setup on ur machine.
@akshatkhandelwal6384
@akshatkhandelwal6384 2 года назад
Can you please guide me how we do this in local instead of confluent cloud? What I want to do is like to connect database with kafka connect and the data gets inserted into kafka cluster
@javatechlearning
@javatechlearning 2 года назад
I did it in local itself. I'm not using confluent cloud
@akshatkhandelwal6384
@akshatkhandelwal6384 2 года назад
@@javatechlearning I meant like you used confluent centre right.....What I am trying to do is run zookeeper, kafka from command line then I am stucked like how we can connect database to kafka, I saw some tuto they all are confusing do I need system registry or anything?
@javatechlearning
@javatechlearning 2 года назад
Sorry I couldn't follow you much. Confluent center gui is only for monitoring purpose. Main servics are connect, schema registry, Kafka, zookeeper etc. They all should be started in order to connect database
@akshatkhandelwal6384
@akshatkhandelwal6384 2 года назад
@@javatechlearning okay got it, sorry I got confued....I thought the GUI is of cloud. Thanks Man!
@sandeepravitej4583
@sandeepravitej4583 2 года назад
My Confluent installation dont have JDBC source connector by default , Please help me how to install ?
@javatechlearning
@javatechlearning 2 года назад
Hello, you need to install jdbc connect plugin using confluent hub
@javatechlearning
@javatechlearning 2 года назад
You can find command in my video
@sandeepravitej4583
@sandeepravitej4583 2 года назад
@@javatechlearning Thanks for your time figured it out , my source connector is up now ..would it be possible to make a video on how to read from mysql and write into mongodb ? or you can share any known sources which speaks about it
@javatechlearning
@javatechlearning 2 года назад
You first need to build jdbc source connector to read MySQL data and then one sink connector to sink topic data to nosql db
@nareshkyv
@nareshkyv Год назад
@@javatechlearning not able to find command, please let me know the steps to install jdbc connect plugin
@sahilmakandar9623
@sahilmakandar9623 2 года назад
Sir Can you provide the jdbc driver link
@kjvinodkumar
@kjvinodkumar 2 года назад
how to download kafka connect JDBC COnnector
@kjvinodkumar
@kjvinodkumar 2 года назад
Could you please provide me the link
@javatechlearning
@javatechlearning 2 года назад
github.com/vishaluplanchwar/KafkaTutorials
@Devendrakr63
@Devendrakr63 2 года назад
Can anyone would paste the config properties in comment.
@omerimam8256
@omerimam8256 Год назад
thanks i need your email please
@vishnutiwari9640
@vishnutiwari9640 2 года назад
from oracle, it is unable to fetch data. Any license issue?
@vishnutiwari9640
@vishnutiwari9640 2 года назад
Thank you for your efforts and time in making this wonderful video.
@javatechlearning
@javatechlearning 2 года назад
Thanks for your feedback 🙂
Далее
КОГДА НАКРОШИЛ НА ПОЛ #shorts
00:19
Тренд Котик по очереди
00:10
Просмотров 254 тыс.
Data Pipelines: Using CDC to Ingest Data into Kafka
7:20
From Zero to Hero with Kafka Connect
33:49
Просмотров 28 тыс.
Introduction to Kafka Connect
17:28
Просмотров 7 тыс.
Writing My Own Database From Scratch
42:00
Просмотров 232 тыс.
КОГДА НАКРОШИЛ НА ПОЛ #shorts
00:19