I think you just saved me weeks worth of work. I was just put in charge of flattening dozens of tables with multiple levels totaling 100s of columns and that macro is perfect.
I started off by asking myself, "this is cool but how could I convince my team to use this vs flattening via ADF". By the end seeing how reusable it was totally sold me, so much quicker than spinning up a new ADF pipeline, creating source etc (assuming the extract isn't taking place in ADF). Im curious if you think there are any more benefits using DBT/Snowflake to flatten vs ADF?
hey there! this video is awesome! do you have any tips on how to turn a variable into a string? I'm trying to take this concept and apply it to Postgres, but i need the column names to be strings to make it work. not sure how to get quotes around the variable name!
Hey Jason - There very well could be but I'm not familiar off the top of my head. Seems like a common scenario so I wouldn't be surprised if it exists somewhere.
@@KahanDataSolutions Thanks for the kind response. Might I ask, for example, let's say birth_name is another JSON. And, I simply want to reuse the marco again. intermediate_model as ( {{ flatten_json( model_name = 'source_model', json_column = 'birth_name' )}} ), how can I do it here by passing the source_model into the macro again?
That is a bit more complex but it is doable. Here is an example function in Snowflake that you can use with JSON data to get you there - docs.snowflake.com/en/sql-reference/functions/typeof.html
That really depends on the individual. Like anything else, the advanced components will take time/experience to fully learn. But if you know SQL you should be able to start contributing to dbt projects pretty quickly once you learn the basic concepts.
