AWS Lambda BATCH Processing From SQS | Powertools for AWS Lambda | .NET ON AWS | Serverless | Amazon

Подписаться 32 тыс.

Просмотров 1,1 тыс.

50% 1

Happy New Year 👋
The AWS Lambda PowerTools Batch processing utility makes it easy to process a batch of messages from Amazon SQS, Amazon Kinesis Data Streams, and Amazon DynamoDB Streams.
It provides a simple interface to process each batch record, enables parallel processing, reports batch item failures to reduce the number of retries, and also supports customizations to meet your business needs and functionality.
In this video, let’s learn how to use the Lambda Powertools Batch processing utility to process messages in batches from Amazon SQS.
This video is sponsored by AWS.
00:00 Introduction
00:49 Native Batch Processing From SQS
02:04 Report Batch Item Failure
03:15 Deploying Resources
04:27 Bulk Record Processing
06:04 Powertools Batch
06:32 Add Nuget
07:15 BatchProcessor Attribute
10:19 Batch Utility in Action
11:45 Dynamo & Kinesis
12:20 Error Handling Policy
12:55 Parallel Processing
13:48 BatchProcessor on Any Function
16:01 Conclusion
Additional Watching
📹 Lambda Powertools - • AWS Lambda Powertools
📹 Lambda Powertool Idempotency - • IDEMPOTENCY - Powertoo...
📹 Lambda Powertool Logging - • LOGGING - Powertools f...
📹 Lambda Powertool Tracing - • TRACING - Powertools f...
📹 Lambda Powetool Metrics - • METRICS - Powertools f...
📹Lambda Annotations Framework - • Lambda Annotations Fra...
📹Lambda Annotations CRUD API - • SERVERLESS CRUD API | ...
🔗Lambda Lifecycle - www.rahulpnath.com/blog/lambd...
📹 .NET Dependency Injection - • DEPENDENCY INJECTION i...
📹AWS Lambda - • AWS LAMBDA For The .NE...
📹Lambda Udemy Course - bit.ly/aws-lambda-udemy
📹Lambda Development Environment - • SETUP AWS FREE Tier Ac...
📹AWS Credential Management - • AWS CREDENTIALS: How T...
📹 ASP Series - bit.ly/asp-net-core-series
📹AWS Series - bit.ly/aws-net-series
Come say hi! ✋
🌍 Blog - rahulpnath.com/
✉ Subscribe to my Newsletter - www.rahulpnath.com/subscribe
🐦Twitter - / rahulpnath
📸Instagram - / rahulpnath

Хобби

Опубликовано:

19 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 11

@MarianoGomezBidondo 6 месяцев назад

Hi Rahul! I'll tell you something about your "subscriptions", in Argentina they appear with the values 20 and 50 Argentine pesos, which are 2 and 5 cents of a dollar, I wanted to let you know in case you want to update them because they are too low.

@RahulNath 6 месяцев назад

Thank you Mariano for supporting me this long!! I guess that's youtube doing the conversion automatically, but thank you for letting me know! Hope you are liking the videos.

@lodoss118 Месяц назад

What is maximum we can use foe the batch size

@chiragpatel908 6 месяцев назад

I want to know about how millions of records retrieved from Dynamo and execute long processing within the limits of lambda net core c# and AWS lambda using steps function.

@RahulNath 6 месяцев назад

Hey Chirag - What are the issues you are facing with it currently? Planning to do a video on step functions soon

@chiragpatel908 6 месяцев назад

@@RahulNath I'm trying to export more than 2 millions records from Dynamo. So after retrieving we have a real time calculation of verifying and creating missing values inside another Dynamo and then exporting in CSV. So that real time calculation process takes time and gives error timeout. How can I divide the whole process into 3 retrieving ,calculation processing, exporting lambdas and design step function architecture to pass that data through one lambda to another. So I can sure not hit lambda limits of 15min. Thank for your prompt response 🙏

@RahulNath 6 месяцев назад

@@chiragpatel908 is it the calculation process alone that is throwing the timeout? Is one lambda trying to do the whole 2 million records? I would look at batching them up based on your partition key or whatever makes sense.

@chiragpatel908 6 месяцев назад

@@RahulNath Yes. I'm now thinking about the divide into above-mentioned but the problem is how can I pass a large amount of data from one lamba to another and process all records parallelly to reduce time and prepare for the final result for export using step function.

@RahulNath 6 месяцев назад

@@chiragpatel908I would store the data in a temporary location (dynamodb, s3 etc ) depending on the nature of data and how the final export is. You could then pass around the identification to that (batch id partition id etc) to the other lambda function. Hard to tell exactly which without knowing more details.