Realtime Kafka Message To ElasticSearch Using Apache Spark|ELK+Apache Spark+Kafka on Windows 10

Подписаться 3,1 тыс.

Просмотров 12 тыс.

50% 1

In this video, I am going to walk you through some very basics of Apache Kafka and how to make topic in Kafka. Then we will see how Spark Streaming can consume the data produced by Kafka. Finally we will see how Spark Streaming Process can feed this data to Elastic Search.
🙏🏻SUPPORT THE CHANNEL🙏🏻
Buy me a coffee: ko-fi.com/bigtechtalk
Subscribe: bit.ly/2A2h6sJ
Facebook: / bigtechtalk
Telegram: t.me/bigtechtalk
#bigtechtalk
#logstash #kibana #elasticsearch
#Spark
#Kafka
#ELK

Опубликовано:

8 дек 2019

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 73

@rashmibansal1966 4 года назад

thanks for connecting all the 3 hot technology

@bigtechtalk 4 года назад

Welcome

@praxtopher Год назад

Very informative Video, It would be really good if you can also provide the link to the code and data files with the commands as well.

@bigtechtalk Год назад

Hi, You can get the code from below link. github.com/BigTechTalk/Spark-ElasticSearch/blob/master/src/com/spark/elasticsearch/writeKafkaToElastic.scala

@iamvij 4 года назад

Good one. Easily explained with neat pics bro.

@bigtechtalk 4 года назад

Thanks Vijeesh

@singhsengar1 4 года назад

@@bigtechtalk can you plz share the code used in above demo

@bigtechtalk 4 года назад

Hi aaa, Will soon upload the code in GitHub and will share the link with you.

@singhsengar1 4 года назад

sorry for pushing, but this is really good and eager to see the detailed project for hands-on, please inform as soon as it uploaded.

@bigtechtalk 4 года назад

Plz give me your email is so that i can send you the code.

@singhsengar1 4 года назад

Nice video, if you have shared the build.sbt file for your scala project as well it would be very helpful

@bigtechtalk 4 года назад

This is a maven project. Can share pom file with you.

@madhavkondapalli785 4 года назад

Thanks for creating such a wonderful content Can you please give github link for the source code. That will be really helpful for me. Thanks in advance.

@bigtechtalk 4 года назад

Will upload soon

@CarlosEduardoOchoa 4 года назад

Hi there! I'm new of this and the video is great! Thanks for that. At this moment I have no installed nothing, you got a video tutorial to install kafka, spark and elasticsearch to test the code you wrote in this video?

@bigtechtalk 4 года назад

Hi Carlos, Below is the list of video for installation of spark,kafka and elasticsearch Install Spark on windows: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-cYL42BBL3Fo.html Install kafka on windows: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-kS-wBI96OJc.html Install ELK(Elasticsearch,Logstash,Kibana) on windows: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-WGNHLcG_OrQ.html

@CarlosEduardoOchoa 4 года назад

@@bigtechtalk Thanks a lot @Big Tech Talk . You got your script in github?

@CarlosEduardoOchoa 4 года назад

Hi @Big Tech Talk could you send me an email how to install the scala dependecies pls? I do not understand this part

@adammejdi9643 4 года назад

good job man

@bigtechtalk 4 года назад

Thanks Adam

@adammejdi9643 4 года назад

@@bigtechtalk hi , how to push data into kafka from web api ? please that s urgent

@bigtechtalk 4 года назад

Hi Adam, You can push data from any source if you are writing code in java,scala and python. Add the kafka lib to your source code and then create Producer object and call producer.send(yourmessage,topic) to push msg into kakfa topic.

@adammejdi9643 4 года назад

@@bigtechtalk please can u make a video or a small tutto of code ; from aapi or u just send me an exemple that s be great mejdiadam@gmail.com thank u man

@MrMaragwa 4 года назад

Great content..make video on kafka and Apache spark installation

@MrMaragwa 4 года назад

Also how to stream such data from a database using same method

@bigtechtalk 4 года назад

Thanks joseph Kafka Installation video : ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-kS-wBI96OJc.html Will make a video on spark installation.

@MrMaragwa 4 года назад

Thank you...How do you create same pipeline with a database

@bigtechtalk 4 года назад

Spark has sql context which can be used to configure connection with any database and then create dataframe and perform your logic.

@bigtechtalk 4 года назад

I am planning to make a video on spark with mysql . Stay connected for more videos.

@fakirrabih2860 3 года назад

Thanks again for your usual great videos.. can you please explain how you created the index in elasticsearch ? i could not get the streaming correctly nothing sent.. thank you .

@bigtechtalk 3 года назад

Hi Fakir, You can create index in many ways. You can use kibana dev tools to create. PUT Or you can directly push the data to elasticsearch and a index will be created. Then you need to go to kibana and add the index to kibana UI.

@fakirrabih2860 3 года назад

@@bigtechtalk Thank you for your reply, the data was pushed correctly but nothing is created in elasticsearch, i think i have problem with the library : Dependencies += "org.elasticsearch" %% "elasticsearch-spark-20" % "6.2.4" don't know why it is not known as artifact NOT resolved or indexed .. any idea?

@bigtechtalk 3 года назад

Hi Fakir Below is the dependency which i have used. org.elasticsearch elasticsearch-spark-20_2.11 7.0.0 Hope this might help

@user-hk1ow5ek6y Год назад

very helpfull video can you share the link of code

@bigtechtalk Год назад

Hi you can get the code from below link. github.com/BigTechTalk/Spark-ElasticSearch/blob/master/src/com/spark/elasticsearch/writeKafkaToElastic.scala If this was helpful kindly subscribe to my channel. It will help me a lot.

@pankajupadhayay6370 3 года назад

@Big Tech Talk video is awesome and very useful ,can you provide git link

@bigtechtalk 3 года назад

Hi Pankaj, You can find the code github.com/Abhishek009/BigData/tree/master/Spark_Scala/StreamingWithSparkAndKafka Subscribe to my channel for more such videos.

@AyushiGupta-hb5gf 3 года назад

Thanks for the video, its quite easy to understand with much clarity. However, Is it possible to implement the same activity without using Scala. I mean only Kafka, Elastic Search and plain Java 8 ?

@bigtechtalk 3 года назад

Hi Ayushi, Yes it is possible java instead of scala.

@AyushiGupta-hb5gf 3 года назад

@@bigtechtalk Could you please help me with that? Please share the link of such video if you have created or may be some other link for reference. Thanks.

@kasenreign2340 2 года назад

sorry to be so off topic but does anyone know a method to get back into an Instagram account?? I stupidly forgot my password. I appreciate any help you can give me

@darwineddie760 2 года назад

@Kasen Reign Instablaster ;)

@kasenreign2340 2 года назад

@Darwin Eddie thanks for your reply. I got to the site thru google and Im trying it out now. Seems to take quite some time so I will get back to you later with my results.

@manikandan-wz7tx 4 года назад

Hi.Very Eager to see the code and do hands on.Can i get the code?

@achyutvyas 3 года назад

We can sink Kafka topics in ES using kafka-connect, so how this approach is better than Kafka-connect.

@bigtechtalk 3 года назад

Hi Achyut, ES has a Kafka-connect to directly sink the data from kafka topc to ES. But you cannot do anything else with that for ex. let's say you have your merchant data in kafka and merchant location in hbase or mysql. In this scenerio u can use spark streaming to connect to kafka and get the merchant name and merchant location at the same time and push it to ES.

@MrMaragwa 4 года назад

import com.scala.spark.kafka.StreamingLogConfig there is an error on this line,object scala not found

@bigtechtalk 4 года назад

Hi Joseph, You can ignore this import. Actually this is a custom package for log compaction. It is not being used in the code.

@MrMaragwa 4 года назад

@@bigtechtalk thanks, no errors now

@bigtechtalk 4 года назад

@@MrMaragwa that's grt. Enjoy Coding and let me know for any help

@CarlosEduardoOchoa 4 года назад

@@MrMaragwa can you send me your pom file pls?

@MrMaragwa 4 года назад

Carlos Eduardo Ochoa your email?