Why does one Snowflake WH in a DAG "idle" while the other WH is processing?

Help
,
1

The problem I’m having

I have 4 Models in a DAG as follows:

  • CF_1 (runs on LARGE WH)
  • CF_2 (runs on LARGE WH)
  • CF_3 (runs on X-LARGE WH)
  • CF_4 (runs on LARGE WH, transforms CF_3 and joins on CF_1 and CF_2)

When I execute dbt run --select +CF_4, I see that CF_1 and CF_2 execute quickly on the LARGE WH, but while CF_3 is running on the XL WH, the LARGE WH is “idling” and not running any queries. Of course, once CF_3 completes on the XL WH, CF_4 is run on the LARGE WH and the WHs are suspended within the configured time.

  1. Why isn’t the LARGE WH suspending while the XL WH runs CF_3?
  2. If I set the final Model of the DAG to also run on the XL WH, will the LARGE WH suspend because it is done with all of its processing after CF_2?

The context of why I’m trying to do this

We have 1900+ tables that run fine on a LARGE WH, and 1 massive table that requires a larger WH. We don’t want to pay for the LARGE WH to “idle” while the XL WH is running, but dbt is doing something that keeps the LARGE WH from suspending while the XL WH is processing. Testing this out is an expensive process, so I’m hoping someone in the community can help shed some light on this process.

2

For anyone curious, I moved CF_4 to the XL WH. From that point onward, I saw that the LARGE WH ran CF_1 and CF_2, then suspended while the XL WH ran the other CF Models. Problem solved, but I’m still unsure if this is what I’ll have to do in the future to keep the WHs from idling.