Can you comment on getting data from SaaS applications which only provide APIs, and something getting data of large volume may take 5-6hours. Is glue the right approach?
Thank you, content is quite helpful and would like to request you please share video with step by step setup of data ingestion pipeline of Glue/Lamda/DMS
There is no license cost. Being cloud, it is pay as you go model. You can see pricing of the services online. For instance, here is the lambda and glue pricing. aws.amazon.com/lambda/pricing/ aws.amazon.com/glue/pricing/
EMR can process streaming data. But the ingestion is done using services like Kinesis. So Kinesis ingests data and then hands over to EMR for processing using Kinesis Firehose.
Thank you, it is extremely informative, and it implies data ingestion decision making. As an observation, the suggested services in the tutorial are AWS native services. Would you consider AWS MSK or AWS managed RabbitMQ if the data sources are social media and web traffic/front-end client event data?
You are right. The scope of tutorial was aws native services and S3 as the data location. However, since AWS Lake House architecture, support multiple databases registered with various data types of data sources, it is perfectly ok if you use MSK or RebbitMQ for the ingestion,
in DATA SOURCE- file system, we can also use direct file transfer from source location to s3 using programmatic way (aws cli, python boto3 ) OR user longing in s3 console .
Technically you can do. Think about security especially if you are using AWS Access Key / Secret Key. Also think about scalability, performance when handling big files especially when using code.
Thank you, great content! I love your video and you are excellent to synthesize the information. If I use AWS Glue, can I ingest multiple tables in one job?
@@adityamishra7438 You can pass comma separated tables names in job parameter a the time of job execution. Then loop in the tables and do ingestion one by one.
