I love how dbt allows me to test schema and data , but there’s one thing I haven’t figured out - it’s testing of incremental models.
My typical CI test run looks like this
dbt deps --profiles-dir $DBT_PROFILE_DIR dbt seed --profiles-dir $DBT_PROFILE_DIR dbt run --profiles-dir $DBT_PROFILE_DIR dbt test --profiles-dir $DBT_PROFILE_DIR
That works well for the
--full-refresh case, which is basically what CI does, since it always run in a clean environment and DB. But the
--full-refresh case is rare in production and what runs every day is processing incremental changes, which is left untested.
Not sure if it’s relevant, but I use static data in my seeds to cover all edge cases. So the increment isn’t always “today”, but rather some static date in the past.
What would be a good approach to correctly test full + incremental transformations?