Amazon Athena and AWS Glue for S3 Data Query

Подписаться 14 тыс.

Просмотров 11 тыс.

50% 1

===================================================================
1. SUBSCRIBE FOR MORE LEARNING :
/ @cloudquicklabs
===================================================================
2. CLOUD QUICK LABS - CHANNEL MEMBERSHIP FOR MORE BENEFITS :
/ @cloudquicklabs
===================================================================
3. BUY ME A COFFEE AS A TOKEN OF APPRECIATION :
www.buymeacoffee.com/cloudqui...
===================================================================
This video shows how to set up AWS glue crawlers to extract the raw data from AWS S3 bucket and put it across in AWS Athena service to query the raw data.
It has step-by-step explanation of all set up that has to be put in place to extract the raw data from AWS S3 bucket ( of format .csv) and put in Amazon Athena so that Business analytics can be performed there.
#awsglue #awsathena #query #s3 #aws #cloudquickpocs

Наука

Опубликовано:

4 ноя 2021

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 14

@thewisearchitect Год назад

Great tutorial. Thanks.

@cloudquicklabs Год назад

Thank you for watching my videos. Thank you so much for the encouragements. Keep watching and keep learning , Happy learning.

@danieljust295 Год назад

The most important part is missing. This is how the crawler assigns the type to the columns and how to modify the type of the column when clawer finished its job.

@cloudquicklabs Год назад

Thank you for watching my videos. Could you please elaborate the question here, is it 'how to modify the type of the column when crawler finishes its job?'

@danieljust295 Год назад

@@cloudquicklabs Notice, that Glue reads the file with data and decides the type for each column (string, integer, float etc.) - this is also crawler’s job and this is why crawler can create schema. This step should be always verified because we don’t know if crawler mapped data types properly to each column from data file. What’s more - I guess - Glue’s crawler uses ML algorithm to determine the type of data found the column.

@cloudquicklabs Год назад

Thank you very much coming back on this. You have made very good observations, may be it should the same way that you guessed. AWS Glue crawlers should be Intelligent enough to indetify the data type while extracting data from source

@nivas143srinivas 2 года назад

Excellent work sir , followed all the instructions but I'm getting 0 records when I query the table from Aetna even though there is data in my S3 bucket files. Any suggestions would be appreciated.

@cloudquicklabs 2 года назад

Thank you for watching my videos. This could be happening because of Glue crawlers has imported the successful. Please validate the crawler configuration again. And also don't forget to create new work group at Athens Query.

@jettsalvador8688 Год назад

You may also try to check the policy configured in the iam role used by the glue crawler, you might need to add the resource arn of the S3 bucket that you want to be crawled.

@ndipsamuel475 11 месяцев назад

@@jettsalvador8688 Thank you. i was blocked for two days and didn't know why. This solved my problem.

@riyajain6713 2 года назад

what if i applied a schedule to it and i wanted only the new data in athena to perform logics

@cloudquicklabs 2 года назад

Thank you for watching my videos. In this case you need to configure from S3 bucket and AWS Glue side to extract only the required data by archiving not required data.