I would like to run dbt through Cloud Composer (which is the GCP’s managed service for Apache Airflow) and I am struggling to find a proper and secure way in order to authenticate dbt so that it can perform operations on Google Cloud BigQuery.
So here’s my profile in
my-profile: outputs: dev: dataset: my-bigquery-dataset job_execution_timeout_seconds: 300 job_retries: 1 keyfile: /path/to/keyfile.json location: EU method: service-account priority: interactive project: my-gcp-project threads: 4 type: bigquery target: dev
Where am I supposed to store the service account JSON key in order to be visible to
dbt when running it through Cloud Composer?
One option is to place the JSON key file on Cloud Storage, under
gs://<cloud-composer-bucket>/data which is mounted into every Airflow worker and simply use the
dbt’s profile but I don’t think this is a secure enough approach (i.e. storing the service account key on an object storage such as GCS).
PS: Note that I am not using Secret Manager on GCP, but self-hosted HashiCorp Vault instead.
Thanks in advance.