I’m having a problem with dbt lineage… When I run locally this command dbt build --select +model_x
it builds 25 models with version 1.7.4 when I updated my dbt to 1.8.2 the dbt build failed and it generated 58 models and tests
So if we ignore the issue that different dbt versions are behaving differently!!!
My main problem is, in this command I ran, I’m only selecting the models and tests related to +model_x the tests which are failing are related to a completely different model !! which should not be run, since I’m selecting specific model lineage. I tried to verify the lineage using dbt docs and the tests which are failing related to models are not included in the lineage, so why am I getting errors for them??? It’s driving me crazy ! Anyone has any idea?
Thanks in advance
The context of why I’m trying to do this
Work related. I want to build only part of my project.
What I’ve already tried
I tried dbt build --select +model_x
I was expecting to build models that model_x depends on, and run the tests of these models and the tests of model_x
Some example code or error messages
I'm getting error for 3 tests that are generated from one generic test.. which are related to models are not part of the lineage.
It’s going to be very difficult to give specific advice without seeing your project. There are differences between 1.7 and 1.8 and while they shouldn’t cause you problems like this, they could be the cause.
You can do dbt ls --select +model_x --exclude resource_type:test to get a list of sources & models that are upstream of model X in dbt’s DAG. That list may give you some insights into why it’s different between versions.
My intuition would be that you’ve done something like shimming the ref macro and that’s no longer working correctly, but it could be any number of things. Creating a minimal reprex would help you figure out the root cause
From what you are explaining check for a reference to a model that doesn’t exist. Seems the parser runs over the other models.
If you change a model name referenced elsewhere it breaks the whole project
Thank you for your reply…
I cannot share all the details of the project because it’s work related…
But I ran the command you shared dbt ls --select +model_x --exclude resource_type:test … it gave me a list of models, and none of them are related to the tests failing…
I ran then this command dbt ls --select +model_x and it shows the tests that are failing!! but why? they are for models that are not in the lineage!
what does this mean shimming the ref macro ?
The tests which are failing are generated from this generic test:
{% test data_latency(model, column_name, unit, num_units) %}
{{ config(severity = 'warn' if target.database == 'DEV_EDW_DB'
and model == ref('int_iterable_events')
else 'error') }}
select
max({{ column_name }}) as max_timestamp
from {{ model }}
having max_timestamp < date_trunc('{{ unit }}', dateadd('{{ unit }}',-{{ num_units }},current_timestamp()))
{% endtest %}
I finally found the issue. I will post the answer here in case someone was having similar issue. I was getting error for 3 tests that are generated from one generic test… which are related to models are not part of the lineage. The generic test code was as following:
{% test data_latency(model, column_name, unit, num_units) %}
{{ config(severity = 'warn' if target.database == 'DEV_EDW_DB'
and model == ref('int_iterable_events')
else 'error') }}
select
max({{ column_name }}) as max_timestamp
from {{ model }}
having max_timestamp < date_trunc('{{ unit }}', dateadd('{{ unit }}',-{{ num_units }},current_timestamp()))
{% endtest %}
That ref in the beginning of the test was making the mess… and breaking the lineage. when I removed it, it was solved.