I was literally going to research on ci/cd and I randomly enter youtube to watch a Dallas Mavs podcast and I see your video. Watched and Understood. Thank you
We're in the process of setting much of this up. We want our end workflow to be something like * Clone PROD database (Using Snowflake ZCC) * dbt build against that Clone * Perform a data-diff between PROD and Clone * Report results of the dbt build AND any data-diff variations for review * Bring down the Clone We have a bit of background work to do until we're ready for that, so for now we just do a `dbt compile` step - it doesn't catch everything but does at least catch simple syntax issues, or stuff like invalid docs syntax etc
Awesome Content, any chances you'd be able to do a video on DBT cloud CI? My team is using DBT cloud and we're definitely gonna be implementing the slim CI jobs.
Great video! Just been wanting to learn ci/cd with dbt but how does it test functionality of dbt models, doesn't it need actual data on which dbt models operate? I thought GitHub actions run on some ephemeral machine for GitHub ci/cd (which doesn't have access to my data), or I'm wrong?
Great question. I purposely avoided getting too far into the dbt-specific requirements b/c it can get a little confusing but since you asked.. here is how it works: First: When talking about dbt specifically, it's important to remember that it's effectively just compiling your code and sending it to your database to be run. Here's how it is able to still connect through a triggered workflow every time, even from a virtual runner on Github: 1. Configure a new profiles.yml in your project's root directory (can be elsewhere, but be sure to align with step 4) 2. Set username/password credentials in profiles.yml to use environment (ENV) variables 3. Set the ENV variables during the workflow by passing in SECRETS (so it's secure - this is set in the Github UI outside of this video) 4. In your dbt command, set the --PROFILES-DIR=. flag so that it uses this root profiles.yml file instead of the default 5. Now, when the CI/CD workflow runs, it spins up a hosted runner, the profiles.yml values are passed the credentials (secret > env var > profiles) and it can then successfully connect to your DB and run the dbt queries against your data & do everything as normal. This was real confusing for me at first too but hopefully that clears it up a bit. I have a few other videos on using Secrets & Creating Actions in my GitHub playlist if you want to see more!
i guess this all a bit too advanced for me: where are you doing all these changes in the beginning? VS Code? I cannot see the full window so difficult to follow