# dbt cli on gcp cloudshell

**URL:** <https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137>\
**Category:** Help\
**Created:** [March 6, 2023, 2:23pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137 "2023-03-06T14:23:09Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![dbt\_dive](https://avatars.discourse-cdn.com/v4/letter/d/a5b964/32.png) [@dbt\_dive](https://discourse.getdbt.com/u/dbt_dive)\
**Post date:** [March 6, 2023, 2:23pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/1 "2023-03-06T14:23:09Z")

</div>

## The problem I’m having

I’ve installed dbt cli bigquery on GCP cloud shell and cloned my github repo. Now when I’m trying to run a dbt model it throws error as -

```auto
Encountered an error:
Database Error
  expected str, bytes or os.PathLike object, not NoneType

```

I’ve created profile.yml in ~/.dbt folder with the following structure -

```auto
default:
  target: dev
  outputs:
    dev:
      dataset: my_dataset
      job_execution_timeout_seconds: 1000
      job_retries: 1
      location: EU
      method: service-account
      priority: interactive
      project: my_project
      threads: 4
      type: bigquery
      keyfile_json:
        type: service_account
        project_id: XXXXXXX
        private_key_id: vasljasgjaspogjsapodgjasaklgjsagjs
        private_key: -----BEGIN PRIVATE KEY-----\nYYYYYYY\n-----END PRIVATE KEY-----\n
        client_email: XXXXXXXXX.iam.gserviceaccount.com
        client_id: 9577623947510934752394752
        auth_uri: https://accounts.google.com/o/oauth2/auth
        token_uri: https://oauth2.googleapis.com/token
        auth_provider_x509_cert_url: https://www.googleapis.com/oauth2/v1/certs
        client_x509_cert_url: https://www.googleapis.com/EWPOTEOIGH;LSGJJ;LJgcpsecuritykey-gbl-ww-pd.iam.gserviceaccount.com

```

The dbt\_profile.yml file -

```auto
# Name your project!
name: 'my_new_project'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'default'

# These configurations specify where dbt should look for different types of files.
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  
.......
......
.......

```

Run command → dbt run --select model\_name

Can anyone please help me point what I’m doing wrong?

---

<div class="post-metadata">

**Author:** ![brunoszdl](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/brunoszdl/32/4113_2.png) [@brunoszdl](https://discourse.getdbt.com/u/brunoszdl)\
**Post date:** [March 6, 2023, 3:10pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/2 "2023-03-06T15:10:49Z")

</div>

When you run `dbt debug` is everything fine? Just to check if there is really some problem with the profiles and the project file.

---

<div class="post-metadata">

**Author:** ![dbt\_dive](https://avatars.discourse-cdn.com/v4/letter/d/a5b964/32.png) [@dbt\_dive](https://discourse.getdbt.com/u/dbt_dive)\
**Post date:** [March 6, 2023, 5:29pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/3 "2023-03-06T17:29:06Z")

</div>

Yes brunoszdl, you are right the dbt debug gives the error message -

```auto
Running with dbt=1.4.4
dbt version: 1.4.4
python version: 3.9.2
python path: /home/dbt_dive/dbt/dbt/bin/python
os info: Linux-5.15.65+-x86_64-with-glibc2.31
Using profiles.yml file at /home/dbt_dive/.dbt/profiles.yml
Using dbt_project.yml file at /home/dbt_dive/git_repo/dbt_project.yml

Configuration:
  profiles.yml file [OK found and valid]
  dbt_project.yml file [OK found and valid]

Required dependencies:
 - git [OK found]

Connection:
  method: service-account
  database: gcp-project-name
  schema: dbt_dive
  location: EU
  priority: interactive
  timeout_seconds: 1000
  maximum_bytes_billed: None
  execution_project: gcp-project-name
  job_retry_deadline_seconds: None
  job_retries: 1
  job_creation_timeout_seconds: None
  job_execution_timeout_seconds: 1000
  gcs_bucket: None
  Connection test: [ERROR]

1 check failed:
dbt was unable to connect to the specified database.
The database returned the following error:

  >Database Error
  expected str, bytes or os.PathLike object, not NoneType

Check your database credentials and try again. For more information, visit:
https://docs.getdbt.com/docs/configure-your-profile

```

I don’t know what’s the problem here? The database & execution\_project are same as the GCP poject name “gcp-project-name”. Can anyone spot something wrong here? Please help.

---

<div class="post-metadata">

**Author:** ![brunoszdl](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/brunoszdl/32/4113_2.png) [@brunoszdl](https://discourse.getdbt.com/u/brunoszdl)\
**Post date:** [March 6, 2023, 5:33pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/4 "2023-03-06T17:33:51Z")

</div>

The ‘`method`’ key is incorrectly indented or it was just a copy-and-paste problem?

And the only thing your profiles differ from mine is that `client_id` and `private_key` are between single quotes `' '`on my side.

Try that and see if it works.

---

<div class="post-metadata">

**Author:** ![dbt\_dive](https://avatars.discourse-cdn.com/v4/letter/d/a5b964/32.png) [@dbt\_dive](https://discourse.getdbt.com/u/dbt_dive)\
**Post date:** [March 6, 2023, 5:46pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/5 "2023-03-06T17:46:11Z")

</div>

Hi brunoszdl, thanks for your quick response!

‘method’ key was a copy & paste error, I’ve corrected that.  
I tried wrapping `client_id` and `private_key` in ’ ’ single quotes but still it didn’t work. I really don’t know what I’m doing wrong?

---

<div class="post-metadata">

**Author:** ![brunoszdl](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/brunoszdl/32/4113_2.png) [@brunoszdl](https://discourse.getdbt.com/u/brunoszdl)\
**Post date:** [March 6, 2023, 5:51pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/6 "2023-03-06T17:51:17Z")

</div>

So, let’s take a step back to better analyze where the problem is. Configure your profile with just the keys needed, without the optional ones

and then try dbt debug. Then we can see if the credentials problem is in the optional ones or not.

```auto
my-bigquery-db:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: service-account-json
      project: [GCP project id]
      dataset: [the name of your dbt dataset]
      threads: [1 or more]

      # These fields come from the service account json keyfile
      keyfile_json:
        type: xxx
        project_id: xxx
        private_key_id: xxx
        private_key: xxx
        client_email: xxx
        client_id: xxx
        auth_uri: xxx
        token_uri: xxx
        auth_provider_x509_cert_url: xxx
        client_x509_cert_url: xxx

```

---

<div class="post-metadata">

**Author:** ![dbt\_dive](https://avatars.discourse-cdn.com/v4/letter/d/a5b964/32.png) [@dbt\_dive](https://discourse.getdbt.com/u/dbt_dive)\
**Post date:** [March 6, 2023, 6:08pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/7 "2023-03-06T18:08:38Z")

</div>

No, I think the problem is in my credentials as it’s still giving me the same error. But the same credentials works for dbt cloud!

---

<div class="post-metadata">

**Author:** ![brunoszdl](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/brunoszdl/32/4113_2.png) [@brunoszdl](https://discourse.getdbt.com/u/brunoszdl)\
**Post date:** [March 6, 2023, 6:37pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/8 "2023-03-06T18:37:16Z")

</div>

Let me see just one more thing, can you paste here the log from `log/dbt.log` related to the debt debug?  
It will be at the bottom

---

<div class="post-metadata">

**Author:** ![dbt\_dive](https://avatars.discourse-cdn.com/v4/letter/d/a5b964/32.png) [@dbt\_dive](https://discourse.getdbt.com/u/dbt_dive)\
**Post date:** [March 7, 2023, 4:58am UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/9 "2023-03-07T04:58:13Z")

</div>

Hi brunoszdl,

This is the logs generated, I’ve pasted the last few lines

```auto
05:16:00.543072 [debug] [MainThread]: Acquiring new bigquery connection 'master'
05:16:00.545125 [debug] [ThreadPool]: Acquiring new bigquery connection 'list_'
05:16:00.545828 [debug] [ThreadPool]: Opening a new connection, currently in state init
05:16:00.546292 [debug] [ThreadPool]: BigQuery adapter: Got an error when attempting to create a bigquery client: 'expected str, bytes or os.PathLike object, not NoneType'
05:16:00.547996 [debug] [MainThread]: Connection 'master' was properly closed.
05:16:00.548340 [debug] [MainThread]: Connection 'list_' was properly closed.
05:16:00.548646 [info] [MainThread]:
05:16:00.549211 [info] [MainThread]: Finished running in 0 hours 0 minutes and 0.01 seconds (0.01s).
05:16:00.550158 [error] [MainThread]: Encountered an error:
Database Error
  expected str, bytes or os.PathLike object, not NoneType
05:16:00.551455 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xxxxx0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xxxxx0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xxxxx0>]}
05:16:00.552013 [debug] [MainThread]: Flushing usage events

```

Output of dbt --version

```auto
Core:
  - installed: 1.4.4
  - latest: 1.4.4 - Up to date!

Plugins:
  - bigquery: 1.4.1 - Up to date!

```

---

<div class="post-metadata">

**Author:** ![dbt\_dive](https://avatars.discourse-cdn.com/v4/letter/d/a5b964/32.png) [@dbt\_dive](https://discourse.getdbt.com/u/dbt_dive)\
**Post date:** [March 7, 2023, 8:39am UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/10 "2023-03-07T08:39:48Z")

</div>

There one more thing I’ve discovered, if I write a simple python program to execute a query it works -

```auto
>>> from google.cloud import bigquery
>>> client = bigquery.Client ()
>>> QUERY = ("select col1,col2 from project.dataset.table")
>>> query_job = client.query(QUERY)

```

I checked the python path →

```auto
dbt_dive@cloudshell:~ (gcp-project)$ which python
/usr/bin/python

```

```auto
(dbt)dbt_dive@cloudshell:~$ which python
/home/dbt_dive/dbt/bin/python

```

The python paths are different. Could this be the reason? Please help in correcting this.

---

<div class="post-metadata">

**Author:** ![brunoszdl](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/brunoszdl/32/4113_2.png) [@brunoszdl](https://discourse.getdbt.com/u/brunoszdl)\
**Post date:** [March 7, 2023, 12:38pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/11 "2023-03-07T12:38:34Z")

</div>

Ok, so the problem is a python error while creating the BigQuery client.

Sorry, I don’t know how to proceed now ☹ , maybe @joellabes or another one who knows the dbt-bigquery connection well may help.

---

<div class="post-metadata">

**Author:** ![dbt\_koen](https://avatars.discourse-cdn.com/v4/letter/d/dec6dc/32.png) [@dbt\_koen](https://discourse.getdbt.com/u/dbt_koen)\
**Post date:** [May 23, 2023, 3:17pm UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/12 "2023-05-23T15:17:35Z")

</div>

Hey @dbt_dive ,

Not sure if you spotted this, but you have the wrong method in the file. Instead of service-account you should use service-account-json.

---

<div class="post-metadata">

**Author:** ![nsyifa](https://avatars.discourse-cdn.com/v4/letter/n/eada6e/32.png) [@nsyifa](https://discourse.getdbt.com/u/nsyifa)\
**Post date:** [November 3, 2023, 7:42am UTC](https://discourse.getdbt.com/t/dbt-cli-on-gcp-cloudshell/7137/13 "2023-11-03T07:42:38Z")

</div>

Waw that’s really works! Thanks a lot