Error when same variable is used in target of 2 different models

Help
1

Hello folks, I am facing a weird issue. I’m trying to set the target table via a var. So my model has something like this:

{% set target_schema = var('target_schema') %}
{% set target_table = var('target_table') %}

config(
schema = target_schema,
alias = target_table
)

I passed the variables via command line. It worked fine when there was only one model. As soon as I replicated the same code to another model, dbt started giving me the below error:

Compilation Error
  dbt found two resources with the database representation "`project_id`.`dataset_id`.`table_name`".
  dbt cannot create two resources with identical database representations. To fix this,
  change the configuration of one of these resources:

I am passing different table names via var to both models, but still dbt is giving me this error. I believe this should not occur as same variable name across models doesn’t mean same target name. Is there any solution to this?

2

If I understand correctly, this is happening because when dbt compiles your project, the same variable value is fed into two different models, so when it compiles the information about your project as a whole it finds two database representations that are the same.

The critical information here is that even if you dbt run -s model_one, dbt compiles your whole project and so even though you’re intending to pass info to one model at a time, dbt feeds the same variable to every model with that var()

It might help to step back and understand what you’re trying to do and why, and then we can see if there might be a more dbtonic solution to your goals.

3

@jaypeedevlin Thanks for the reply.

I am trying to setup airflow dags to be used with dbt. For each dbt model, there will be a dag in airflow which would trigger the model as per the schedule requirements. I am keeping variables like source_schema, source_table, target_schema, target_table in airflow variables, so that changes don’t require editing the file which would also warrant version control.

I made a list config var in airflow with json object for each model. Of course, the key names are the same for each model. But due to this limitation, I am not able to keep the key names same which is where I am stuck.

4

I would think your only solution given this setup would be to give each model it’s own set of vars.

so that changes don’t require editing the file which would also warrant version control

It’s worth keeping in mind that by thinking this way you’re trying to circumvent something that’s at the core of dbt’s philosophy (analytics should be version controlled), so it’s not totally surprising that it’s cumbersome to do in the tool.

1 Like