Dbt run + test interwoven

Archive
1

Hi,
I am interested in following workflow and how this can be organized in dbt.
I really love the dag functionality where data lineage is guarenteed but would like to enhance this with testing and data correctness.
The workflow I would wish is :

  • run a model in a temporary environment
  • test this model
  • if correct move the model to the normal accessible environment

At this moment we do 1 dbt run (which runs of a lot of models) followed by 1 dbt test of all models. I would prefer run and test to be interwoven to catch errors immediately. After each run of a model in a temporary environment the corresponding test should be executed and the model shoud only then be made available for further processing.

Is this currently possible in dbt?
Could you imagine a command like dbt runtestmove?

Regards,
Marc

2

Hey Marc,

The way we do this at better.com is through an Airflow dag. For your case it could look like:

dbt_run_staging >> dbt_test_staging >> dbt_run_prod

Where dbt_run_staging is configured such that it writes to a custom hidden schema which is also where the tests run.

You can also add other stages to the dag e.g. source freshness tests upstream of dbt_run_staging.

Hope that helps~

3

Hey Kenny,

thx for the answer. One question : is your dbt_run_prod the same as your dbt_run_staging but only another schema or are it completely separate models which must be maintained.

Regards,
Marc