The problem I’m having
Looking to create more of an event based build to reduce wasteful runs on data. I came across this functionality
Command step order
dbt source freshness
dbt build --select “source_status:fresher+”
but when I run dbt build --select “source_status:fresher+” I’m met with an error :
" No previous state comparison freshness results in sources.json"
From my understanding, each run of ‘dbt source freshness’ overwrites sources.json and so it has no other copy to compare itself to.
Looking to get some help on either implementing this section below since I feel this is what I’m missing.
You can also set the dbt_ARTIFACT_STATE_PATH environment variable instead of the --state flag.
dbt source freshness # must be run again to compare current to previous state
dbt build --select “source_status:fresher+” --state path/to/prod/artifacts
The context of why I’m trying to do this
By running this on a 15 minute interval for example, we can consistently check for fresh data sources and only run those models.
What I’ve already tried
I’ve tried pointing to an older version of sources.json for it to compare itself against but the same error returns.
Some example code or error messages
Internal Error
No previous state comparison freshness results in sources.json