What is the best practice for deploying Airflow together with dbt?

In-Depth Discussions
,
1

Hello : )

My objective is to deploy airflow and dbt on Kubernetes (more precisely, I am deploying on EKS and using a Helm Airflow chart and using Celery executor). I am trying to grasp, what are the best practices for deploying airflow together with dbt?

  • Should I include both airflow and dbt in the same container? And do I have to?
  • Is there a best practice in terms of where to keep the dbt models? (I am curious if I can only install dbt in the container but keep the dbt models outside the container the same way I plan to do with the airflow dags).

I am happy to provide any details if needed.

2

A solid implementation used by many community members is to use Docker, K8 Operator and Airflow.

At a high level, the steps to do so are:

  1. Build, tag and push an image with the most recent version of your dbt project on each of your deployments
  2. Use the Kubernetes Operator for all tasks in Airflow that run dbt
  3. Have the K8 Operator pull and run the latest tagged image, with the relevant dbt command
2 Likes