Creating a serverless Apache Kafka publisher using Flask & AWS Fargate

Подписаться 28 тыс.

50% 1

Streaming data and event-driven architectures are becoming more popular for many modern systems. The range of use cases includes web tracking and other logs, industrial IoT, in-game player activity, and the ingestion of data for modern analytics architecture.
One of the most popular technologies in this space is Apache Kafka. This is an open-source distributed event streaming platform used by many customers for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
This example walks you through how to build a serverless real-time stream producer application using AWS Fargate and Flask.
Prerequisite:
===========
Master Containerization with AWS : Docker, ECR, ECS & Fargate
• Master Containerizatio...
🐳 Automating Docker Image Build, Push to AWS ECR, and AWS Lambda Deployment via GitHub Actions🔄
• 🐳 Automating Docker Im...
Code:
=====
github.com/Sat...
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Apache Kafka form scratch
• Apache Kafka for Pytho...
Messaging Made Easy: AWS SQS Playlist
• Messaging Made Easy: A...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.co...
Explore our vlog channel:
/ @funwithourfam
Your Queries:
===========
A Demo project for Apache Kafka on Flask
Event Driven Applications with Flask and Kafka
Event-Driven Microservices with Python and Apache Kafka
Event-Driven Microservices with Flask and Apache Kafka
Building a Simple Distributed System with Python,Flask and Apache Kafka
Power your Kafka Streams application with Flask & AWS Fargate
AWS Fargate & Kafka
flask-kafka
Stream Processing with Python and Apache Kafka
Mastering Real-time Data Streaming with Kafka and Python
#kafka #fargate #aws #flask

Опубликовано:

1 окт 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 6

@penumurumohith247 Месяц назад

Hi Bro As you mentioned previous approach aws api gateway and lambda is also a serverless approach and most cost effective right

@KnowledgeAmplifier1 Месяц назад

@penumurumohith247, cost effectiveness depends on multiple factors, for example, Prime Video switched from AWS Lambda based architecture to Container based systems (reference blog: www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90), also when handling large traffic, lambda concurrency might be a blocker in long run, but when the system is in it's initial phase, i.e. when the team or the traffic is not too big, then lambda can be a good choice wrt cost & maintenance , eventually, we can migrate to container based approach with little changes (as long as the main business logic is written in modular manner)

@venugopal-nc3nz Месяц назад

Can you suggest one data Engineer project where we can see practical implementation of CI/CD?

@KnowledgeAmplifier1 Месяц назад

@venugopal-nc3nz, you can refer this video where practical implementation of CI/CD is covered for all major Cloud Services used frequently by Data Engineers (AWS EC2, Lambda, Glue, Snowflake etc.) -- ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-N5Z_M1HWTMA.htmlsi=LIgpnHzhpUunZdCC

@penumurumohith247 Месяц назад

Bro It's not working Bro everything deployment and all things done fine but while sending request not able get the response

@KnowledgeAmplifier1 Месяц назад

Hi @penumurumohith247, requestign you to check the ECS Tasks logs, that will help you to resolve the issue in better way, also the security groups in VPC might cause some issues..