Hi, excellent video about iceberg. I have a question, i have a datalake with many parquet files and i want to use iceberg tables. what is the correct way to deals with this parquet files, do i read all parquet files and insert data into iceberg table? or is there any to link iceberg table to existing parque files without copy then into iceberg table?
Thanks for this very clear presentation on more of the details of Iceberg. Whilst there are lot of talks about Iceberg they gloss over the details which are quite important for those who need them.
When you add a new column for instance, it create a new snapshot and you can query the snapshot you want. But how performant is it ? Let's say our team use iceberg and over a year, 1000 snapshots were created, with some time the create of a new column added or the deletion of another. If the snapshots store the transactions, does it means that when we are going to query the first snapshot, it reapply all the 1000 modifications done, and then query this version of the table ? Or does it create new data file each time that copy our table with the modification ?
I created iceberg tables inside an EMR notebook, and while they do show up in Athena, the columns do not load. When I went to view the table in Glue, well the columns are also not there. Why does this happen? I can only interact with the table within the Spark session, but in Athena or in Glue, it's just an empty table with the name but no columns nor the data.
Sorry about this inconvenience you've faced here. I recommend reaching out via our re:Post forum and posting your question there for more visibility & insight from our tech community. You can do that via this link: go.aws/aws-repost. ^BG
