Hi, I would like to run dbt through Cloud Composer (which is the GCP’s managed service for Apache Airflow) and I am struggling to find a proper and secure way in order to authenticate dbt so that it can perform operations on Google Cloud BigQuery. So here’s my profile in profiles.yaml file: my-pr…

type: bigquery method: oauth dbt will then automatically use the credentials of the service account of the node pool. I have written a series of articles on how to run dbt at best on Google Cloud. Here is the link of part 1 if you are interested: [image] DBT at scale on Google Cloud — P…

Hi Giorgos, We run dbt via “GKEStartPodOperator” on Cloud Composer. We can then leverage workload identity of the service account running the node pool. This has the benefit of separating the scheduling of Airlfow from the execution of dbt that will be on a separate node pool. And also no need anym…

Thanks a lot for your reply Charles! May I also ask what method/keyfile do you specify in your profiles.yml file when using workload identity?

How to run dbt on Cloud Composer and authenticate the service account

Help

Topic		Replies	Views
Can DBT connect to BigQuery using Workload Identity? Help bigquery	1	1785	April 26, 2024
Running dbt in Composer using a KubernetesPodOperator Show and Tell airflow , orchestration-and-deployment , gcp-cloud-composer	7	18117	April 30, 2024
Running dbt in Airflow using GKEStartPodOperator Help gcp-cloud-composer	0	174	December 5, 2024
dbt cli on gcp cloudshell Help	12	2849	November 3, 2023
Dbt extremely slow on Airflow (Cloud Composer) Archive	3	3589	August 12, 2022

How to run dbt on Cloud Composer and authenticate the service account

Related topics