When you start talking about 10TB+ datasets yo run into issues on whether database can hold the dataset and performantly. Also different purposes need different tools so you need your data in a way that be used by different teams with different tools.

Also with data lakehouse tables there doesn’t have to be any running database server when no one is querying the dataset since they are just files in storage while traditional database tables need a persistently running environment.

@@Dremio wow! Now I have got full clarity. Thank you so much for your response.

@@Dremio cost saving. Thanks for the tip 😀.

I think I got it now. Storage layer does not have this concept of branches, so in the waraehouse/data/ directory, it stores parquet files both lesson2 branch and main branches. I can tell this because there are files with different timestamp associated with my sql operations in each branch.

Dremio Cloud has the ability to automate these types of operations

If you want pm me (Alex Merced) your spark configs. Usually it’s a typo or an update that needs to be made the spark configs. Spark can be very touchy on he config side which is one reason using Dremio for a lot of iceberg operations is so nice (much easier).

It’s mainly based on the engine, most engines will use position delete but streaming platforms like Flink will use equality deleted to keep write latency to a minimum

That means the table is in Nessie and it needs to know which branch your using so it would be AT BRANCH main

@@AlexMercedCoder Thanks Alex. This would seem to be a limitation of the 'Save as View' dialogue, as it doesn't allow me to do this and it doesn't default to the branch you're in the context of currently.

If your following this tutorial sometimes Spark has some weird dns issues with the docker network. The solution is to use the ip address of the Nessie container which you can find by inspecting the network in the docker desktop ui or inspecting the network using the docker CLI to find the ip address of the Nessie container. If you run into a "Unknown Host" issue using minio:9000 then there may be an issue with the DNS in your Docker network that watches the name minio with the ip address of the image on the docker network. In this situation replace minio with the containers ip address. You can look up the ip address of the container with docker inspect minio and look for the ip address in the network section and update the STORAGE_URI variable for example STORAGE_URI = "172.18.0.6:9000"

This tutorial does the same thing without spark www.dremio.com/blog/intro-to-dremio-nessie-and-apache-iceberg-on-your-laptop/

The CSV file is not part of the iceberg table, in this example we taking a CSV file and adding the content of it to an Iceberg table but new parquet files are being written and a new metadata snapshot being created.

Thank you

You can use update on iceberg tables, this could be a spark or iceberg version issue. What version of both are you using?

@@Dremio apache spark 3

Thank you

Dremio is particularly designed for structured and semi structured data. Although in the future different AI tools can help turn unstructured data into structured data for analytics.

Yes, dbf was just demonstrated to show the integration but it isn’t required for using Dremio. You can do everything via the UI. Here is an exercise to show you just that -> bit.ly/am-sqlserver-dashboard

In this video we are using AWS Glue studio but a docker container with a notebook server and am configuring the environmental variables in the docker run command. AWS glue is just being ci figured as the catalog in the Spark session. In AWS glue you should be able to specify env variables on the job settings page. Find examples here: github.com/developer-advocacy-dremio/quick-guides-from-dremio

There is a text to Sql feature you can use express filters like this us using plain text.

@@Dremio Thank you but could not find text to SQL on the local dremio client, is it exclusive to cloud? I feel like dropdown headers could easily generate SQL.

@@emonymph6911 yes, text-to-sql is an exclusive to cloud feature. Both cloud and software have no-code features that can be accessed by clicking on a column to generate calculates columns, joins, data type changes and others.

@@Dremio Thank you. My only feedback is that Excel style filters for unique names that are part of your column headers would be really convenience and nice to see in a future release. Apart from this I think the software is amazing, lots of respect for the team.

SQL is fine you can use airflow to send SQL to Dremio to insert records into the desired table. In this tutorial I give an example of doing an an append only insert. bit.ly/am-sqlserver-dashboard

Shoot a message to Alex Merced on linked in so he can also some follow up questions for the best recommendation.

www.dremio.com/blog/exploring-the-architecture-of-apache-iceberg-delta-lake-and-apache-hudi/

@@Dremio It's exactly that article which made me ask the question. ^.^ Don't get me wrong I'm trying Dremio right now in local docker looks amazing. But I still thought Hudi with timeline is more suitable for BI considering dates ties well together with graphs, event streams and Data Vault methodology as well. Going to watch the Xtable presentation at subsurface, looking forward to it! PS: Alex your customer care videos and docs are the best in the world for a software application. I like how you guys go at a moderate pace and cover terminologies in the tutorial before showing the ropes. Makes it an easy barrier of entry. Please keep that up. 10/10 waves!

@@emonymph6911 I think this may be answering the opposite question but this article may be helpful too: www.dremio.com/blog/dremios-commitment-to-being-the-ideal-platform-for-apache-iceberg-data-lakehouses/ I do think there is a tremendous benefit to the reusability of Iceberg's metadata structure along it's partitioning evolution and hidden partitioning features which are unique to the format.

www.linkedin.com/posts/learniceberg_pyiceberg-write-support-is-here-links-in-activity-7165735714381312002-21-1?

In this blog you can see the sql in more detail, but essentially I turn the json string into an object and access the properties via the keys. www.dremio.com/blog/how-to-create-a-lakehouse-with-airbyte-s3-apache-iceberg-and-dremio/

Yeah thanks@@Dremio. I would like to ask a question. what if we want to use Project nessie as a catalogue for iceberg tables directly Is there any option for this !

Mike in the demo is running Enterprise Edition. In Dremio Cloud, many of the security features are available on the free tier.

both links still seem to be working for me.

Nessie has the ability in its rest api to force merges or ignore certain objects. These aspects of its features should be coming very soon to its SQL support. In future iterations it will become more context aware to auto reconcile such conflicts further down the road,

Is there a way to message me on linked the code your using in a git repo so I can inspect it, just message Alex Merced on linked in.

I think it was just my particular env at the time i kept running into an issue with the host name is my docker env so i just used the ip to be safe.

If using COPY INTO or INSERT INTO to populate iceberg tables, new data files are written and the added on new snapshot on the iceberg table.

Is scanning the QR codes on the phone not working, both videos should also be searchable on Dremio you tube channel. I think the next video in the first QR code is ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-bvXj4ANMy10.htmlsi=KrthtZQr_Dve9Ter

I'd have to see the whole log output and catalog settings to determine the issue. If you want message me on LinkedIn and I can examine further. - Alex Merced

1. dbt-dremio needs to be installed in the context your dbt-models exist which is usually not on the same system Dremio is running on (you wouldn’t want both processes fighting over resources) 2. The virtual environment is to isolate dependencies from other projects like web apps, so that way I can more easily make the environment more replicatable.