AWS SQS to Lambda Tutorial in NodeJS | Step by Step

Подписаться 222 тыс.

Просмотров 111 тыс.

50% 1

SQS integration with Lambda functions are a new offering from AWS that solve many of the problems of SNS + Lambda. In this video, I show you exactly how to set up a SQS queue and consume events from a Lambda function. I start out by talking about some of the basic concepts of SQS and Lambda integration. From there, I walk you through the AWS console and show you how to create a SQS queue and a Lambda function. Additionally, I show you which permissions you need add to your Lambda IAM role. Finally, I test the setup by manually pushing messages to the Queue and verify they are being processed in the Lambda function.
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
🎉SUPPORT BE A BETTER DEV🎉
Become a Patron: / beabetterdev
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driver Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
Resources:
SNS to SQS Setup: • AWS SNS to SQS Tutoria...
AWS Lambda Introduction: • AWS Lambda Introduction
Getting started with AWS: • Introduction to AWS | ...
☁Topics covered include:Lambda Concurrency Gotchas
- Lambda SQS trigger
- Lambda DLQ
- Lambda Concurrency
- Lambda / SQS Permissions
- SQS Visibility Timeout
- SQS Message Retention Period
- SQS Receive Message Wait Time (aka Long Polling timeout)
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
#Serverless
#SQS
#SimpleQueueService
#Lambda
#AWS

Опубликовано:

7 авг 2019

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 119

@sharpcode_ 4 года назад

Great video! Also no need to apologize when you make a mistake, love seeing videos that include the mistakes in them. It is actually helpful!

@BeABetterDev 4 года назад

Thanks Paul! I can't help saying sorry sometimes :)

@balajink2461 4 года назад

@@BeABetterDev You are Awesome guy

@bivensrk 4 года назад

I love videos like this. HOURS of lecture about SQS and I don't get "long polling." You explain it in about 5-10 seconds. "Ask less frequently, 'Hey, you have anything?'" The object being to get fewer instances of "Nope, nothing" from the queue, which also means the Lambda is firing less often -> less $ spent for little gain. This rocks!

@BeABetterDev 4 года назад

So glad you found it helpful! Welcome to the channel!

@veera421 3 года назад

but the only challenge I see there is a concurrency rate would be decreased?

@antonarbus Год назад

I have been doing only frontend tasks so far and now my manager told to start touching backend also. At first it looked very complicated and frightening, but your videos slowly sort everything out. You channel is the gem, thanks a lot. You have very good explanation skills.

@markemerson98 3 года назад

perfect to the point walkthrough. exactly what I was looking for...

@BeABetterDev 3 года назад

Thanks Mark! Glad you enjoyed :)

@mitlandir5761 3 года назад

Such an amazingly clean explanation! Liked and subbed!

@BeABetterDev 3 года назад

Thanks Mitlandir! Welcome :)

@sudhakarammula3534 3 года назад

you got me when you forgot to save the lambda, nice video, thanks.

@DanPanuMisailescu Год назад

Thank you very much for this video Daniel! It was very helpful to me!

@kalpeshmalviya118 10 месяцев назад

Great video and a lovely explanation.

@sanbaldo 4 года назад

thanks for the tutorial! very clear

@BeABetterDev 4 года назад

very welcome!

@veera421 3 года назад

Like the way of explanation! Thank you!

@BeABetterDev 3 года назад

Glad it was helpful!!

@RafaganAbreu 4 года назад

Awesome tutorial! Very very nice!

@BeABetterDev 4 года назад

You're very welcome Rafagan!

@indiawoods7738 4 года назад

Very nicely presented, Thanks

@Danielbfr3 4 года назад

Awesome tutorial!

@BeABetterDev 4 года назад

Thanks Daniel! Glad you enjoyed.

@DavinderSingh-cg7zy 4 года назад

Very nice presentation. Thanks for sharing.

@BeABetterDev 4 года назад

Thanks for your comment!

@mondayemmanuel191 2 года назад

I enjoyed the tutorial.

@awsnewborn8335 3 года назад

Great video, Thanks !!!

@BeABetterDev 3 года назад

You're very welcome!

@veronicanisenbaum4266 4 года назад

Hey there... Awesome and simple video!! Do you know by any chance if there is a good wrapper in node to push messages to SQS?

@BeABetterDev 4 года назад

Hi Veronica, Thank you! I did a quick search and found that the aws-sdk npm repository which contains a client wrapper you can use with SQS. The node repo is here: www.npmjs.com/package/aws-sdk?S_TACT=105AGX63 And an example on how to use it to push to SQS is here: docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/sqs-examples-send-receive-messages.html Hope this helps and thanks for watching!

@andrewmcburney4425 Год назад

Great video. thanks

@BeABetterDev Год назад

Very welcome

@ChanceMinus Год назад

Thank you.

@quewealawea 3 года назад

Awesome video!! It worked smoothly for me. One doubt though: in minute 23:32 when you have your lambda ready, you have two destinations: Amazon SQS and Amazon CloudWatch Logs. I have none of those on my Lambda. How do you set that???

@kumargaurav5223 2 года назад

Hi Nice explanation I am facing one issue.my lambda is invoking only for the first message pushed to the queue but for second time it does not invoke.I have used batchSize 1 .Please suggest where I am doing wrong.Thanks

@biggbosstelugu8252 3 года назад

Great video

@BeABetterDev 3 года назад

Glad you enjoyed it

@EvangelistaBoccadoro 4 года назад

Very Intersting tutorial thanks! Regarding batch messages apparently if one fails the entire batch then has to retry and is not clear which message has failed so generally I prefer to set that parameter to 1.

@BeABetterDev 4 года назад

Thanks Gregorio! I share the same sentiment regarding batch processing. In my view there isn't enough benefit to performing batch processing that outweighs the cost of partial failure scenarios.

@rezataheri1358 3 года назад

@@BeABetterDev And worthy to note the maximum batch size is 10 and I think higher the batch value translate to less lambda cost because lambda function will pull 10 messages with a single action. so less spin up more message receives.

@maYYidtS 2 года назад

well explained... can I use SQS in my use case...like I have a requirement to auto start an EC2 when a user got uploaded file in S3. for this I have created Lambd function and add a S3 event notification trigger in lambda...it is working fine, and auto starting as well and with a basic function I'm stoping EC2 when the instance process got finished. But the problem is...before stoping EC2 if user upload other file in s3...I want notification sould hold till EC2 stop, once the EC2 stoped second notification will pass to lambda then EC2 starts again...

@abhaythakur8572 2 года назад

Awesome

@aniketkokate3563 4 года назад

you are a saviour

@BeABetterDev 4 года назад

Thanks Aniket!

@chandanhasija8863 3 года назад

Hi, thanks for the great video. One thing i wanted to know that when tried sending the message using SNS to SQS. I also attached a lambda function to read data from SQS. In the monitoring tab where the graph is shown, i saw that the graph of 'Number Of Messages Sent' started before the graph of 'Number Of Messages Received'. I thought message should be received first and then sent. Correct me please if i am wrong and please explain.

@BeABetterDev 3 года назад

Hi Chandan, I found this article that may be of help to you explaining the differences between these metrics. Take a look: www.bluematador.com/blog/how-to-monitor-amazon-sqs-with-cloudwatch

@rihamfayez1568 3 года назад

Thank you so much for the video. Does the Message get automatically deleted from Queue without having to this manually from Lambda after it has been triggered?

@BeABetterDev 3 года назад

Hi Riham, it automatically gets deleted as long as your function call returns successfully. Hope this helps.

@tishamahuja 24 дня назад

Hello, Dan! First of all thanks for posting this tutorial. In the section about ‘Default Visibility Timeout,’ you stated that the consumer should manually delete the messages. However, in the example provided, the messages are being automatically deleted. Our consumer (in this case, a Lambda function) is not handling message deletion. I’m confused.

@thiagoaugusto300 3 года назад

Great...just one question. Is it possible to get a specific event from SQS, like deletion for example?

@jesha1995 3 года назад

You should use a SNS queue instead for that and subscribe to topics

@Vishal-wr6cc 3 года назад

Very nicely explained.. Waiting for SES+Lambda..if possible customize template.. Thanks for sharing your valuable knowledge..

@BeABetterDev 3 года назад

Thanks Vishal!

@sandipansarkar9211 Год назад

finished watching

@muhammadkholilurrohman3449 2 года назад

new subscriber here :) great video!!

@BeABetterDev 2 года назад

Thanks so much Muhammad and welcome to the channel!

@CarlosSanchez18 2 года назад

Hey brother, excellent tutorial, I'm very thanksfull with you!!! I have a question about lambda Poller, What happens if at end of day my application dosen't send any messages to SQS but mi lambda poller continue asking for any messages. Does it have any cost? ... thanks on advance!

@harryspooner Год назад

Does AWS charge for the concurrency run-time? Seeing that it could be set at a specific limit.

@jamescoffey5282 2 года назад

Do you have to use a Lambda function to read and delete messages from SQS or is there a step function defined to do this? I already use step functions in my solution and currently do not need to use a Lambda function. I see there are SQS support in step functions but I don't see any documentation of video explaining how to use them. Thank you.

@mhoskins555 4 года назад

I would like to limit the SQS to invoke a lambda at a certain rate, a maximum 500 times per second. Anyone have an idea how to do that? Does SQS have a setting to slow down? : )

@vishaldindalkop2952 Год назад

Please do on batch response failure and Dead-letter queue

@newbie2226 Год назад

Can you tell how to trigger a lambda using sqs , all in the cloudformation template?

@mahadevanh 2 года назад

in the lambda function, I don't see you delete the message. How did it get deleted? is there a setting to delete after reading the message?

@manideepnaama5711 2 года назад

What's the purpose of using Boto3 here.Cannot the other lambda be directly called from the invoker as both of them are on AWS?

@ravirathour1872 Год назад

Hi, I am new in aws . i have some question which mention below 1. I am using sqs fifo queue to receiving the data into the node app. Without lambda function i can't trigger the receving message continuously into the node application. 2. If we need lambda function for trigger the queue and receving the reposone in to the node application. 3 . How can i continuously consume the sqs queue in to the node application thanks

@glennadams7047 4 года назад

Well done. (If I had a dollar every time I forgot SAVE... )

@BeABetterDev 4 года назад

Thanks Glenn!

@heenaahuja651 3 года назад

I am facing issue with the api gateway integration with SQS. can you please help me with it. I want to write the serverless.yml mentioning the integration of api gateway with SQS. (in this case while creating the API GATEWAY THE CODE SHOULD BE SUCH WHICH SHOULD HAVE INTEGRATION TYPE AS SQS). Please let me know how this can be done. ?????

@sumac3631 2 года назад

❣

@paulmiller9430 4 года назад

Hi great video. I am hoping you can comment on my use case which is similar but not quite. I have a client that pushes message to SQS, I have Alexa trigger a lamba which I want to poll the queue and then it will speak the message. So the main difference is ASK is the trigger for the lambda not SQS and the lamda performs a poll at the time it is triggered. Cheers

@BeABetterDev 4 года назад

Hi Paul. This sounds like a reasonable application of SQS + Lambda. Sounds like a neat project!

@matteogioioso896 4 года назад

How is it possible to do the same thing with FIFO queues? Thanks

@khalidskiod 4 года назад

aws lambda not supporting FIFO queues for now check their docs

@BeABetterDev 4 года назад

khalid is correct - FIFO queues are not supported for AWS Lambda Triggers. See documentation here: docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-lambda-function-trigger.html "Note: FIFO queues don't support Lambda function triggers."

@maximeroma2723 4 года назад

Fifo is now officially supported for lambda!

@whats-better 3 года назад

I was able to receive message ID but I am not able to view the contents and attributes of the message in PYTHON. can you pls help?

@samikshasharma3544 8 месяцев назад

I am not able to understand what does the term "lambda vacation" mean? Could you please help to understand the term

@tonycavanagh1929 2 года назад

Thanks this is a great vid. But I hope someone can help me with the following problem. The Lambda will process the message and then wants to pass the message onto another AWS service. At the moment I have to hard code the properties needed to call the other service. It can be any service S3 bucket or. ElasticSearch. But I would rather have those details set up in a separate config file, similar to properties or a yml file, on a server based application. Is there a way to do this. Thanks.

@BeABetterDev 2 года назад

Hey Tony, You can use something called Lambda Environment Variables. These can be fed in dynamically as config values for your function. Alternatively, if you're looking for a full-fledged config service, there's AWS AppConfig. Hope this helps Daniel

@tonycavanagh1929 2 года назад

@@BeABetterDev Thank you, I did actually find AWS AppConfig in my research. I will do a review on AWS AppConfig vs Lambda Environment Variables, and will see which is best for my needs. Again thank you.

@avindersingh3390 3 года назад

Hey can I send message from AWS lambda to fifo sqs?

@BeABetterDev 3 года назад

Hi Ajay, absolutely you can!

@simplyviki 3 года назад

I am looking for lambda code in python, where it accepts an sns topic via sqs and then triggers a step function, any suggestion where I can find such thing in python ? One thing to mention, I have to do everything by python code and can't do anything via console. I will be creating resources via terraform

@BeABetterDev 3 года назад

Hi there, I don't have a specific resource for this kind of functionality. However you may be able to piece together some of my smaller videos to deliver this functionality. Good luck!

@AmeerulIslam 2 года назад

where do I visualize that the lambda is polling SQS for new message? I can't see any log for that.

@BeABetterDev 2 года назад

Thank you so much!

@mikebarrett393 2 года назад

Hi, This is a great video. But I hope someone could answer the following question: My AWS lambda needs to establish a database connection and serve the requests. And if there's a big spike of traffic then AWS lambda might spins up 1000 containers to serve the requests(Ex: If there are 20000-30000 sqs messages), but if database only allows, say, 100 connections then most of the requests won’t be able to connect to the DB. Then what should be the batch size and reserved concurrency be initialized to ??

@BeABetterDev 2 года назад

Hi Mike, Do you happen to be using Aurora? If so you can use RDS Proxy which manages the connection pool for you. If not, you'll probably need to limit the reserved concurrency to the maximum number of concurrent connections you can support. Hope this helps

@mikebarrett393 Год назад

@@BeABetterDev Thanks for your reply!

@AlvinC-sz3li 3 года назад

Could you please give a video of the SQS to Lambda scaling issue? We are facing the issue that if SQS has a spike, and there isn't enough Lambda, after several retries, messages from SQS will be sent to DLQ.

@BeABetterDev 3 года назад

Hey G O - can you tell me more about your problem? Are you seeing throttling on your Lambda function?

@AlvinC-sz3li 3 года назад

@@BeABetterDev yes. Basically the root cause is, the SQS to Lambda trigger works in a way that there are workers pull messages from SQS regardless of whether Lambda is throttled. If Lambda is throttle, then the message just go back to the SQS after the visibility. If the same message is picked up more than a certain time, it will be throw to the DLQ.

@AlvinC-sz3li 3 года назад

@@BeABetterDev There are ways to mitigate this issue, like increase Lambda quota, but no permanent fix. The only way to fix it is to not use the trigger, but write our own logic to pull message from Lambda. However it is hard to tell how many lambda I should run.

@BeABetterDev 3 года назад

Hi G O, This is a very interesting problem that only occurs at the edges of scale. Have you tried looking into using Provisioned Concurrency with an AutoScaling policy that tracks the # of messages in your queue? This way, you can ensure that you always have enough capacity to handle the workload and can control the amount of processing power you have through a managed policy. Hope this helps!

@AlvinC-sz3li 3 года назад

@@BeABetterDev Hi, thanks for the reply. After the burst, Lambda can only scale up at the rate of 500 more instances per minute. If request rate is more than this, the throttle can still happen. So long as there is throttle, there is chance that message will be received multiple times and end up in being sent to DLQ.

@TomerBenDavid 4 года назад

Splendid 👌 could you provide less step by step and more theoretical lectures on AWS services

@BeABetterDev 4 года назад

Thanks for the feedback Tomer. I'm currently trying to mix it up but expect a new overview lecture this weekend on Cloudwatch Anomaly Detection!

@programmer7449 2 года назад

It's very good video everything is detailed but I am following this video and not getting SQS inside this lamda function timestamp: 22:00

@MalrickEQ2 3 года назад

Can I send food in a queue? So like, when I'm hungry, I just request a sandwich and it pops out?

@BeABetterDev 3 года назад

Haha, I wish it was that simple!

@UOUOssas 3 года назад

Your fridge is kind of like that, right?

@wgpubs 3 года назад

Is there a way to configure the lambda so doesn't do anything with the message when its triggered? I'd like to set something up so that the lambda gets notified that there is a message in the queue, and then spins up an EC2 instance to actually pull and process the message. Thanks!

@BeABetterDev 3 года назад

Hi Wayde, You can certainly just ignore the contents of the message and spin up your EC2. Another way of doing this is to set up an Autoscaling group that triggers based on the SQS metric - you can set it up so that it will automatically spin up an instance when queue size > 1. Hope this helps and thanks for watching!

@johnbalvin5401 4 года назад

shouldn't you delete the messge?

@BeABetterDev 4 года назад

Hi John, If you return out of your lambda function without throwing an exception, Lambda handles deleting the message for you!

@manish2simpl 3 года назад

If you can trigger a Lambda by SQS, why you need to poll SQS. I think you need to publish new video :P AWS is changing

@waldosoto4353 Год назад

"message": "Endpoint request timed out" :(

@BeABetterDev Год назад

Is your Lambda function performing a job that takes longer than 3 seconds? My default 3s is the Lambda timeout duration. You can adjust this in the configuration section of your function.

@tomknud 2 года назад

Not sure what you or I can do about it but the fuzziness of the aws window text is head aching, my only complaint about your otherwise great videos.

@BeABetterDev 2 года назад

Thanks Tom! This video was one of my first on RU-vid. Since then, I've made changes to the way I record. If you check out my more recent videos the quality (both video and audio) is much better. Thanks again for watching and your feedback

@ShahzadKhan1234 Год назад

I said it "u didn't press save" then I realized it was a recorded video from years ago..

@TariqASheikh 4 года назад

One of the simplest tutorial I have ever seen. 2 min is actual execution time , rest is all explanations. I was hoping for more advanced implementation or a practical demo using an application flow logic.

@BeABetterDev 4 года назад

Not sure if this is a compliment or criticism, but thank you for the comment. In the future I will include more advanced implementation videos. Thanks.

@devottam2485 4 года назад

The Tutorial should be in Python and you have used NodeJS for Lambda Creation. its confusing!!!

@BeABetterDev 4 года назад

Apologies for the confusion - I updated the title accordingly.

@mahadevanh 2 года назад

in the lambda function, I don't see you delete the message. How did it get deleted? is there a setting to delete after reading the message?

@BeABetterDev 2 года назад

Hi Mahadevan, Deletion of the message happens automatically by Lambda on successful process of the message. That is, if your Lambda does not throw an exception and returns normally, the message is considered successfully processed and followed by a SQS deletion. Hope this helps clarify, Daniel