Batch Processing Example using Amazon Data Pipeline | S3 to DynamoDB using Amazon EMR | Tech Primers 

⏱ Chapter Timestamps =================== 0:00 - Introduction 00:28 - Architecture Flow 2:31 - Pre-requisite: Amazon S3 Buckets Creation 3:44 - Pre-requisite: Amazon DynamoDB Table Creation 5:15 - Pre-requisite: Amazon S3 file format Creation 8:20 - Pre-requisite: Amazon Data Pipeline Creation 12:40 - Data Format Configuration in Data Pipeline 14:14 - Activate Data Pipeline 17:52 - Data Loaded into DynamoDB 19:03 - Serverless Data Pipeline Trigger using EventBridge and Lambda 20:06 - Summary

Interviewer was asked me about Batch processing but I explained him Cron Job using Spring scheduler, but now I come to know what's batch processing from in detail, thanks a lot and it means a lot for me...

Wow! Great bro... I love ur dedication.. great job 🔥🙏

I love your dedication brother....late night doing great job for the dev community....👍👍👍 @Tech Primer

Thanks for the video, this really helps. Could you please suggest what are the options we could use to concatenate two file based on common column from S3. For this should I use Lambda or some other services may be Data Pipeline / Glue etc?

Nice one.One question.Next time when another file is uploaded in S3 and we activate pipeline then will it try to upload the first file as well..if yes how to avert that?

Guys if you're stuck save the file as .txt and change data type to small "s", "n", "l".

There are much cheaper way such as athena CTAS or even Glue Nice video concept is good

Great video and congrats on being the AWS Community Builder! AWS DataPipeline looks it's a competitor of Airflow, the architect UI is similar to Airflow DAGs, interesting!. Also, curious to know what you use for presentations Keynote or PowerPoint?

Great video.. Thumbs up..

Great video Ajay once again :). Is the data ingestion continuous or done in intervals?

Nice demo, useful.

Brother, how do you know these many topics? Really great 👍👍

Could you please give some clarity on EC2 instance role creation at the time of data pipeline creation?

Is it possible to do update item? Or it will only do put item in dynamo db?

can you build a scalable pipeline working on real time data and use services like kinesis

Great job...👍👍👍

Nice video, is is possible to run multiple instances of a single lambda in parallel to perform serverless processing?

please upload airflow cluster setup and data pipeline

As always, Amazing and Awesome. Very informative & insightful. Thanks, @techprimers

There is developer can do here