Any help you can give on the below would be gratefully received.
We have the following challenge:
- We have many customers for a platform we’re building that ingests data using dbt from various sources
- The bulk of the logic in the dbt flow that process this incoming data is the same for all customers
- For some models however, a different approach needs to be configured per customer. Eg. Both customers use a table called “Accounts” to hold customer data, but customer A uses a standard field “Industry” to hold the customer segment, whilst customer B uses a custom field “Segment_C”.
- We break the logic for how to capture “customer_segment” into two separate models (one for customer A and one for customer B) and then ask dbt to use the correct model when it gets to that part of the flow.
- The “which is the correct model” config lives in a table in our data warehouse. We want to have dbt read the configuration for which model to correctly include for customers A and B whilst retaining the rest of the flow identical for both customers
– dbt won’t let us dynamically insert “–depends on comments” in models, which breaks our approach
- Is there a way to add --depends on comments for models in some kind of pre-hook to get around this?
- Is there a better way to deal with dynamic model requirements?