I am interested in following workflow and how this can be organized in dbt.
I really love the dag functionality where data lineage is guarenteed but would like to enhance this with testing and data correctness.
The workflow I would wish is :
- run a model in a temporary environment
- test this model
- if correct move the model to the normal accessible environment
At this moment we do 1 dbt run (which runs of a lot of models) followed by 1 dbt test of all models. I would prefer run and test to be interwoven to catch errors immediately. After each run of a model in a temporary environment the corresponding test should be executed and the model shoud only then be made available for further processing.
Is this currently possible in dbt?
Could you imagine a command like dbt runtestmove?