dbt cloud + Airflow

In-Depth Discussions
1

Hi.
Firstly, apologies if this has been asked - I couldn’t find a definite answer.

We are currently using dbt cloud and adding Airflow into the mix as an orchestration tool.
I see 2 options that we can approach this

  1. Airflow trigger dbt jobs via API (this is well documented)
  2. Airflow triggering dbt commands (bypassing jobs)

Is Option 2 something that’s recommend (or is it even possible) ? Doing this seem more relevant for dbt-core, but can this be done with dbt cloud?

Thank you!

2

Hey! Thank you for posting the question!

The first option is absolutely how users use Airflow to orchestrate dbt Cloud jobs. For more flexibility, you can use the steps_override parameter to change the dbt command (assuming you’re dynamically changing this). You will need to keep in mind how our scheduler queues jobs though, as you can’t run the same job ID in dbt Cloud in parallel, but you can run as many unique jobs in parallel as you wish.

As for option 2, you’d be executing dbt-core, thereby losing many of the advantages you get by running a job in dbt Cloud: job artifacts management (to drive CI/Merge job state comparison), Explore page metadata population, inability to make cross-project references (if you’re doing this) and more! I’m not saying you can’t do this and make it work, but on the long term going with option 1 will be a better experience.