Create a Kafka Cluster Using AWS MSK And Stream Data - Full Coding Demo 

Thanks for watching and the comment Quivun.

Thanks very much.

No problem Ajay.

Thanks very much.

Thanks for watching Matheus.

Thanks for watching Jim.

Thank you very much

Yes, and absolutely advisable. I tend to make everything as open as possible on these demos, just so there are less chances of something going wrong for those following along. However, in the real world. Lock it down with security groups and Nacls.

Thanks for watching.

Thanks adriano

Thanks for watching Mohamed.

Thanks for watching Prithvi. First person to mention the audio. I usually edit using headphones so it wasnt noticeable to me, but I'll keep it in mine for future vids.

@dante thanks for watching and following along.

I am not overly familiar with FileBeat, but there is a config file which let's you specify Kafka brokers and suspect this needs updated to reflect the MSK information. Being inside the same VPC should make this easier. www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html

Johnny Chivers thank you. Will try it out.

I don't at the moment unfortunately. It is on my future lesson plans list for a series on EMR using both Spark and Flink streaming, but they won't be completed for another couple of months as it stands. Presently, there is a lot of Kinesis on the channel - but in future I will be addressing this imbalance with other streaming technologies.

Hi Nozzom, Thanks for watching. What version of the CLI are you using? There was an issue with this exact thing on 1.16 and going to 1.17.9 or higher seems to solve it.

@@JohnnyChivers I have not checked the version. Thank you for the suggestion.

Did you get any leads? I am also looking to connect to MSK using spring boot application running locally

The smallest MSK broker instance is a T3 small which isn’t in the free tier unfortunately. The pub/sub EC2 I spin up could be a micro instance in the free tier. It will just take slightly longer for the updates and installs than this demo. I had the larger instances in this demo spun up for 6 hours and it cost me 5 USD, so by using the free tier EC2 for pub/sub plus the smallest broker instance it shouldn’t be anymore that a few dollars. In terms of glue I have a getting started video on this channel ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-w54eIfIjJRw.html

​@@JohnnyChivers Thank you very much! Im trying to do some jobs using MSK and Glue ETL Streaming jobs into S3 (Parquet format). This will come in handy!

Thanks for watching Diego.

Hi Yovel, thanks for watching. At what point are you running into the issue? Is it after the cluster has become available on the console?

​@@JohnnyChivers hey man thanks for the response, well i got that error when i tried to use the aws kafka command to get the MSK ARN in the terminal after the cluster became available. i managed to just use the information displayed in the aws console to complete the video though

Hi John, It could be a couple of things. Firstly double check that the zoo keeper connection string is correct. It will time out if it can't reach zoo keeper. If the connection string is correct then it's probably a security group or ACL issue preventing the connection. In the demo i open up the security group to all traffic. This is a bit excessive but my aim is to make there less issues when following along. So, ensure you can reach zoo keeper and you're not being blocked by in bound rules in a security group. After that, you may have an ACL issue depending on how your aws account is configured. There is a great article on stacked overflow about it here. stackoverflow.com/questions/58173102/aws-msk-timeout-when-creating-kafka-topic-with-acl-turned-on Hope this helps. Johnny

@@JohnnyChivers hey Johnny. Turns out you were right about the security group - I was trying to adjust the main SG I was using to accommodate my local machine IP. Then I added in an inbound rule for said SG and it worked. Thanks!

No problem. Good news that it is working. Thanks for watching.

Sounds like your minimum sync replicas have gone below the limit for the topic. It could be a configuration issue, but if your working to the exact settings I have? Then the cluster maybe unhealthy due to disc space running out. There is some debugging information about disc space and sync issues here docs.aws.amazon.com/msk/latest/developerguide/troubleshooting.html#troubleshooting-offlinepartitions-outofsyncreplicas

@@JohnnyChivers All brokers have enough disc space, still it is throwing same error

It may be one of those annoying cases where deleting the MSK cluster and rebuilding from scratch could be the best solution. I've had a couple of issues with EMR today where deprovisioning and reprovisioning solved the issue.

Thank you Mharon. Yes, you can use kafka connect. Unfortunately, there is no MSK managed service for this. There is however a great stack overflow post on this exact topic and a link to a video that is really helpful on the best answer post. stackoverflow.com/questions/59194786/kafka-connect-with-amazon-msk

@@JohnnyChivers Okay Thank you for this Do you suggest using Amazon MSK When you want your spring boot app to produce and as well as consume using the @KafkaListener annotation in the Spring boot app? from the amazon msk cluster?

It certainly can be done. It's really a personal thing. I've always found that third party offerings of kafka remove the headache of managing your own cluster. The great thing about MSK is the abstraction. However, the trade off is you don't have as much control compared to building your own cluster from scratch. But, building one from scratch and then maintaining it is not a simple task.

Yes. I'll add it to the list of future requested videos. I hope you found this one useful for getting started.

@@JohnnyChivers yes it was, managed to do it

It's one of the desktops on Mac Dark Mode. If you Google "dark mode mac ocean" the background is available as a wallpaper on the 5/6th hit on image search. You can download it and use that as a desktop on Windows or Linux.

Yes, just be careful with latency.

@@JohnnyChivers Which way do you suggest doing ?

It's a bit of a hard one without more detail. If you had your zoo keeper server up, and the client connected it maybe a case you need to restart your zookeeper server and reconnect the client. Make sure you do this from separate terminal windows.

@@JohnnyChivers thank u, will try

@@JohnnyChivers can't find commands to stop zookeeper on windows

Are not using msk and shhing into it? Same as the tutorial? If so, you'll be on Linux even if your using a Windows machine.

@@JohnnyChivers yeah i am following the steps, but getting WARN Client session timed out, have not heard from server in 10026ms for sessionid 0x0

Thanks

Thanks for watching Naresh. I’ll increase the font size.

yes, you could have maximized your ec2 putty windows , audio could have been little louder, also windows flipping was little faster, btw great content all in all!!!