Read JSON file from S3 With AWS Lambda in python with Amazon EventBridge Rule

Подписаться 19 тыс.

Просмотров 15 тыс.

50% 1

This video is a step-by-step guide on how to configure an EventBridge Rule to trigger a lambda function and read in a JSON file that was uploaded into S3. It also covers how to write a pandas file to read in a JSON file into a Pandas dataframe in python so data transformation could then be prepared on it.
#aws #eventbridge #awsdatawrangler
timeline
00:00 video overview
00:37 Amazon EventBridge configuration
02:19 Lamba configuration
04:14 lambda python code
09:45 end to end test
Useful resources:
Install AWS Data Wrangler on a lambda function: • How to Install AWS SDK...
AWS Data Wranlger Overview: • AWS Data Wrangler Over...

Опубликовано:

19 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 15

@markthomas9641 2 года назад

Great video, thank you. DeltaJSON is really useful for devops working with JSON in AWS, it does compare, merge and graft via API service.

@DataEngUncomplicated 2 года назад

Cool I'll check it out, thanks mark!

@EagleHunt 2 года назад

Hi, I am trying to follow along but at 3:18 I only have the CloudWatch Logs option. I cannot select s3, what should I do? Any help would be greatly appreciated!

@DataEngUncomplicated 2 года назад

Hi, it sounds like you are missing the required iam permissions for your lambda role. Make sure to update it for the permissions you need to s3

@tekkemganesh9424 Год назад

Can we read image from s3 bucket and integrate it with Amazon eventbridge and send it to the lambda function to trigger can any one helpme??

@DataEngUncomplicated Год назад

Yes, when you set up Eventbridge, the key and bucket should be in the metadata, you can write a lambda function which can then read that image into your lambda function

@georgelza Год назад

Personal request, do you have an example where the JSON is published onto a Kafka topic ?

@DataEngUncomplicated Год назад

Hi George, no unfortunately I haven't worked with Kafka

@georgelza Год назад

@@DataEngUncomplicated bummer... it's such an important building block these days in data streaming, and Lambda is such a nice "publisher" from the AWS ecosystem, thanks. back to Google searching.

@georgelza Год назад

@@DataEngUncomplicated sorry, another question, I got a potential inbound stream, .json.gz that i first need to uncompress. , the resulting .json file could then have up to 1million records.... comment from viewpoint of awswrangler handling this and Lambda handling this ? At any time i might have ,multiple of these files being dropped into the S3 bucket.

@DataEngUncomplicated Год назад

I have handled more than 1 million records using aws wrangler. You will need to increase the memory from the default to handle it though. Keep in mind, if your lambda functions is doing complex transformations and creating many data frames in memory, it will eat up more memory and you could run out of memory.

@georgelza Год назад

@@DataEngUncomplicated all that i need to do is loop through the doc and post them as individual messages onto a kafka topic, so little processing.