How to run dbt on Cloud Composer and authenticate the service account

gmyrianthous
Fri, 22 Jul 2022 16:44:35 +0000
Help

Hi,

I would like to run dbt through Cloud Composer (which is the GCP’s managed service for Apache Airflow) and I am struggling to find a proper and secure way in order to authenticate dbt so that it can perform operations on Google Cloud BigQuery.

So here’s my profile in profiles.yaml file:

my-profile:
  outputs:
    dev:
      dataset: my-bigquery-dataset
      job_execution_timeout_seconds: 300
      job_retries: 1
      keyfile: /path/to/keyfile.json
      location: EU
      method: service-account
      priority: interactive
      project: my-gcp-project
      threads: 4
      type: bigquery
  target: dev

Where am I supposed to store the service account JSON key in order to be visible to dbt when running it through Cloud Composer?

One option is to place the JSON key file on Cloud Storage, under gs://<cloud-composer-bucket>/data which is mounted into every Airflow worker and simply use the /home/airflow/gcs/service-account-key.json for keyfile on dbt’s profile but I don’t think this is a secure enough approach (i.e. storing the service account key on an object storage such as GCS).

PS: Note that I am not using Secret Manager on GCP, but self-hosted HashiCorp Vault instead.

Thanks in advance.
Giorgos

6 posts - 4 participants