The problem I’m having
Rerun of failed models doesn’t fully work with incremental model using the “-select 1+result:error+” option
The context of why I’m trying to do this
I want a way to just rerun all failed incremental models. Also fine is rerunning all failed jobs for each day.
What I’ve already tried
If incremental models fail over multi days the rerun using this command only will rerun one day:
dbt run --vars ‘{“snapshot_date”:“2024-12-03”}’ --select 1+result:error+ --defer --state ./target
Assuming the above was the first one I reran it will work. However, the below rerun will not work as the model is now marked as working. The below run will do nothing. Models are either in an error or failed state without regard to the incremental state.
dbt run --vars ‘{“snapshot_date”:“2024-12-02”}’ --select 1+result:error+ --defer --state ./target
Sample code, which works fine on a daily basis if a rerun isn’t required. We run this way as we only load the one day at a time due to large data volumes.
{{ config(materialized='incremental',
unique_key='snapshot_date',
incremental_strategy="delete+insert",
pre_hook = ["commit;
ALTER TABLE dbt_test_external.dbt_test
ADD IF NOT EXISTS PARTITION (snapshot_day = '{{ var('snapshot_date') }}' )
LOCATION 's3://dbt-test/dbt-test-external/simple/snapshot_date={{ var('snapshot_date') }}' "]) }}
with source_data as (
select *
from {{ source ('dbt_test_external', 'dbt_test') }} test
)
select *
from source_data