No video :(

How to use AWS Batch (Fargate) and Lambda for long-running processes in a Serverless app

Подписаться 28 тыс.

Просмотров 3,1 тыс.

50% 1

In this video , we are going to implement an Event-Driven serverless Data Pipeline using Fargate over AWS Batch, s3 & Lambda Function.
Prerequisite:
----------------------
Master Containerization with AWS : Docker, ECR, ECS & Fargate
• Master Containerizatio...
Introduction to AWS Batch: Easy and efficient batch computing | in-depth intuition
• Introduction to AWS Ba...
Code:
----------
github.com/Sat...
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Apache Kafka form scratch
• Apache Kafka for Pytho...
Messaging Made Easy: AWS SQS Playlist
• Messaging Made Easy: A...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.co...
Explore our vlog channel:
www.youtube.co...
Your Queries -
=============
AWS Batch with Lambda
Building Event-Driven Batch Analytics on AWS
Orchestrating an application process with AWS Batch
Batch-Based Architecture - High Performance Computing
Invoking AWS Batch Service via AWS Lambda
My experience with AWS Batch Processing
Batch Processing- The Serverless Way
AWS Batch with AWS Lambda
Designing a modern serverless application with AWS Lambda and AWS Fargate
Converting an AWS Lambda application to Fargate
Run event-driven and scheduled workloads at scale with AWS Fargate
AWS Fargate
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY RU-vid CHANNEL

Опубликовано:

5 сен 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 19

@pvshark 4 месяца назад

Amazing class! It was exactly what I needed. Thanks for you time and work!

@KnowledgeAmplifier1 4 месяца назад

Glad to hear this @pvshark! Happy Learning

@alagappan2214 Месяц назад

Thanks mate!

@KnowledgeAmplifier1 Месяц назад

You're welcome @alagappan2214! Happy Learning :-)

@user-jd4kt5kc7p 5 месяцев назад

Great video, was really helpful. Just curious we can also use Glue instead of the aws batch right? as we wont need any kind of docker/ec2 setup here and glue jobs runs more than 15 minutes.

@ShreyaSingh-so6jg Месяц назад

could you pls share the files you have uploaded on the S3 bucket. The one which I'm uploaded seems to throw error somehow.

@KnowledgeAmplifier1 Месяц назад

@ShreyaSingh-so6jg, FYR -- github.com/SatadruMukherjee/Dataset/blob/main/Setosa.csv

@ShreyaSingh-so6jg Месяц назад

@@KnowledgeAmplifier1 Thankyou

@KnowledgeAmplifier1 Месяц назад

You are welcome @@ShreyaSingh-so6jg ! Happy Learning

@kalpatarusahoo844 5 месяцев назад

Kindly make a video of AWS kinesis

@KnowledgeAmplifier1 5 месяцев назад

Hello @kalpatarusahoo844, have already created multiple videos on different concepts related to AWS Kinesis Family , you can refer this playlist -- ru-vid.com/group/PLjfRmoYoxpNrI6CuZt486uDjSvhhODleP&si=p0pIhlOAAq-Xl4ft Stay tuned for more videos

@kalpatarusahoo844 5 месяцев назад

@@KnowledgeAmplifier1 I will check. thanks

@KnowledgeAmplifier1 5 месяцев назад

No problem@@kalpatarusahoo844! Happy Learning

@douglashenrique6766 3 месяца назад

When to use lambda+batch and when to use event bridge pipe + batch? In this case, wouldn't pipe solve the problem?

@mhaesh 5 месяцев назад

@KnowledgeAmplifier1 is there any chance in future will see apache flink tutorial . I am looking for this tutorial . Can you please let me know any good documents or videos are available for this course. Thanks advance

@techaisolution 2 месяца назад

Hi, this setup spike my billing very high, The setup was to build lambda function to read the latest file from the s3 dir and make transformation then finally to s3 target dir, So this all setup with the python script has to run once the s3 notification to lambda function that an file just came to s3. But it went into a loop and made the s3 and lambda billing spike

@techaisolution 2 месяца назад

Let me knew what is the issue I didn't not noticed first in this setup while running my python script.

@KnowledgeAmplifier1 2 месяца назад

The issue you're experiencing is likely due to an infinite loop caused by your Lambda function writing an object back to the same S3 bucket, which then triggers the Lambda function again via a put event. This continuous cycle can significantly increase your S3 and Lambda usage, leading to higher billing. To resolve this, please ensure the following: Check the Bucket Configuration: Verify if your Lambda function is writing to the same S3 bucket that triggers the function on a put event. If it is, this is likely causing the infinite loop. Separate Buckets or Folders: Different Buckets: Consider writing the transformed data to a different S3 bucket. Different Folders in the Same Bucket: Alternatively, you can configure two separate folders within the same bucket. Configure the S3 event notification to trigger the Lambda function only when a file is added to the specific source folder. The Lambda function should then write the transformed data to a different target folder. This can be set by specifying the folder name in the bucket prefix in the S3 event setup. By implementing one of these solutions, you can prevent the infinite loop and avoid unnecessary costs.

@techaisolution 2 месяца назад

@@KnowledgeAmplifier1 mostly the setup is same as you explain, the source s3 dir read the latest .gz file and then save to another s3 dir the transformed with truncate if exited so this new transformed csv then merger to the master data csv to anothe s3 dir data file, the only one thing i would have missed here is to assign the prefix and suffix setup. i would have tested while it run and stop, but when i assign the s3 notification to tigger the target destination lambda fucn to execute on a particular time, so here after this does not stop till i delete all my resources.