More Complex Environment Structure

In-Depth Discussions
1

New DBT user here… Our Snowflake environment is structured such that we have three environments for our development lifecycle: Dev, Integration and Production, but we also have another facet to our environment which is “consumer” and “core” data. We intend to maintain consumer (i.e. customer) data in a separate database and schema structure to minimize unauthorized access. So effectively we have 6 Snowflake databases: DEV/INT/PROD x CONSUMER/CORE.

Would it be a better practice in DBT to have 6 targets in the profiles.yml file, or to override any of the database/schema information at the model level? At this point I’m thinking we have consumer and core as subdirectories of our model folder in our project.

Looking forward to the discussion - thanks!

2

I would think of profiles.yml as Authenticator only and structure models folder so that each folder is a db then override db in dbt_project.yml.

That beings said, why not have a single db with multiple schemas? You can control permissions the same way at the schema level.