Creating a Serverless Apache Kafka(MSK) publisher using AWS Lambda

Подписаться 27 тыс.

Просмотров 10 тыс.

50% 1

Kafka is a top-notch industry platform for streaming data processing at scale. No surprise that first-class citizens of Kafka world are 24/7-running producer/consumer applications (e.g. classical servers, k8s-pods, etc.).
But what about the rapidly rising world of AWS Serverless ecosystem?
In this video ,let’s explore how to publish messages in MSK Kafka Topic from Serverless computing platform, AWS Lambda.
Prerequisites:
----------------------
Learn AWS VPC , Subnet , Route Table, Internet Gateway with Hands-on Demo
• Learn AWS VPC , Subnet...
VPC with Public and Private Subnets in-depth intuition with Hands On
• VPC with Public and Pr...
AWS VPC NAT Gateway In-depth intuition
• AWS VPC NAT Gateway In...
AWS VPC ENDPOINT in depth intuition & hands on
• AWS VPC ENDPOINT in de...
Using Amazon VPC Endpoints to Access DynamoDB in-depth
• Using Amazon VPC Endpo...
Getting Started with AWS Managed Streaming for Kafka with in-depth service setup
• Getting Started with A...
Steps for the Lab:
-------------------------------
github.com/SatadruMukherjee/D...
You can download the Lambda Layer zip from here :
------------------------------------------------------------------------------------------------
github.com/SatadruMukherjee/D...
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Apache Kafka form scratch
• Apache Kafka for Pytho...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.com/37466271/d/h/...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY RU-vid CHANNEL

Наука

Опубликовано:

12 янв 2023

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 9

@rushikeshkarkhanis7915 Год назад

Great tutorial, you explained with Security group configuration. Most of the videos on youtube have public access, which can be vulnerable. Thanks for sharing knowledge.

@KnowledgeAmplifier1 Год назад

You are welcome Rushikesh Karkhanis! Happy Learning

@ATHARVA89 4 месяца назад

hi @knowledge amplifier do you have any video related to testing in data engineering? if not could you suggest some resources

@KnowledgeAmplifier1 4 месяца назад

Hi @ATHARVA89, you can refer this video -- ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-EVJm3aL7ftY.htmlsi=ATOAHV5JO34xSO-C , for more testing related resources , you can refer medium.com/@samzamany/unit-testing-in-data-engineering-a-practical-guide-91196afdf32a

@user-kb8fe5uw7j 11 месяцев назад

you r running loop for Provisioned in lambda can i do with serverless cluster for same i have tried by adding client property's and all but facing some issues (Address family not supported by protocol) and some more pls help me out on this And special thanks for this great video

@ay9523 Год назад

I am not sure why it is coming "errorMessage": "Unable to import module 'lambda_function': No module named 'kafka'" ? I have tried with S3 and direct uploading .zip file to lambda layer

@vipinsharma-jt3qt Год назад

Nice tutorial, it would be really helpfull if you can create some tutorial on mutilple zookeeper and kafka broker using docker compose with scheme registy with avro in Python. Done research online but didn't find any solution, If you can do that reallly helpfull. keep creating wonderfull content @KnowledgeAmplifier1.

@prabhuofficial6072 4 месяца назад

i creted msk cluster and how to use it in nodejs

@KhalidKhan-kr5it 10 месяцев назад

Nice video..I tried but getting error... "errorMessage": "NoBrokersAvailable", "errorType": "NoBrokersAvailable",when i run the producer code from cmd it works fine but not with lambda function