Where does the dbt CLI take the project name from? I used it with a project “xyz” but then I configured dbt (profiles.yml) with project “abc” but for some reason when I run dbt run it tells me that user doesn’t have privileges to project “xyz”.
When I look at gcloud config list I see that my default project is indeed “abc”.
I also ran dbt debug and it shows the right project in BigQuery Adapter: abc line but when it makes a call it’s trying to connect to “xyz”… Anybody has any solutions for this?
Your dbt project might be overriding the GCP project that it’s trying to write the tables to. This can be overridden if <https://docs.getdbt.com/docs/build/custom-databases|custom databases> have been set (dbt uses the term “database” and “GCP project” interchangeably).
If you do dbt compile and inspect the resulting manifest.json, it will tell you where it’s going to write the subsequent models to. Usually the database is pulled from profiles.yml but the linked page explains what other behaviour can happen.
Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.
The manifest shows the right project “abc” but actual connection is made to a project that is not mentioned in any configuration anymore
Provide the exact error message you’re getting (this is a good idea any time you’re asking about an error you’re seeing)
Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.
gcloud config list’s default project has nothing to do with where dbt will write stuff. It always full-qualifies with the project name.
Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.
Caller does not have required permission to use project xyz that’s the line after Connection Test even though in the BigQuery Adapter line with the full URL the project is “abc”
And even at the beginning of the dbt debug the line of connection details contains the right project but somehow the actual connection takes an old value that doesn’t exist in profiles.yml anymore
is profiles.yml using the execution_project config? if not, the next thing I would check is the query logs in BigQuery to see what exactly is happening. It’s also a good idea to check that dbt is looking at the correct profiles.yml, which you can check in dbt debug
Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.
How does dbt know which google account to use for connection? Because I think it still uses previous account that is indeed connected to xyz project.
From docs:
Not only I specified it in profiles.yml but also confirmed that my
gcloud config list indeed says
project = abc
–edit
Looks like I had to run gcloud auth application-default login and now it works. I would still like to understand how gcloud interacts with dbt. What if I need to switch between google accounts and dbt projects?
It depends on which connection setting (method in profiles.yml) you’re using
Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.
I have oauth as my method
then it’s using your application default settings and/or the currently active account and the default project set in gcloud config
Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.
the docs page above explains
Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.
it’s better to explicitly set the execution project than rely on gcloud to get it right, in my opinion
Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.
