This is makes sense as default behavior, but we’re kind of stumped how best to continue our deployment at the moment without an optional --no-compile dbt run parameter. Our idea at the outset was to use Prefect for orchestration of all of our data products in a just-in-time ELT method where 1 master DAG would dbt compile once at the beginning, execute/await source data ingestions, kick off individual dbt models as soon as each model’s source dependencies are met, then kick off each single downstream ML and BI workloads as soon as each exposure’s models have completed their runs and tests.
Since this end-to-end plus just-in-time orchestration is a top requirement of our internal customers for our data + analytics engineering overhaul (and what we sold to them as the only acceptable modern way of running things), is waiting a few minutes for the entire project to compile for each model run our only option? Our models are well written incremental loads, so most actually individually run much faster than the compile step.