"Invalid value for '--profiles-dir': Path '~/.dbt/' does not exist." error gets thrown despite this path existing

The problem I’m having

I used pip to upgrade multiple dependencies, including dbt-databricks and dbt-core so I could install data-diff.

After installing it, dbt run and other CLI commands throw this error: Error: Invalid value for '--profiles-dir': Path '~/.dbt/' does not exist.

However, this path exists. When I run dbt run --profiles-dir ~/.dbt/ the job completes without error.

I am pretty sure this is an issue with dbt-core however because this error is thrown whether running a databricks job or a bigquery job.

What could be causing this? :thinking:

MacOS Ventura 13.1
dbt-core vsn 1.5.0
dbt-databricks vsn 1.5.1
data-diff vsn 0.7.5
dbt-bigquery vsn 1.5.0a1

What I’ve already tried

I’ve confirmed that my ~/.zshrc contains export DBT_PROFILES_DIR="path/to/.dbt" and restarted Terminal.

This looks similar to (but not the same as) this one: [CT-2531] [Regression] v1.5.0 cannot run "dbt deps" in Dockerfile - "Error: Invalid value for '--profiles-dir': Path '/root/.dbt' does not exist." · Issue #7511 · dbt-labs/dbt-core · GitHub

Does this work or give an error?

dbt run --profiles-dir $DBT_PROFILES_DIR

What is the output of the following command for you? Is it a relative path or an absolute one?


If relative, does it work if you export an absolute path?

1 Like

dbt run --profiles-dir $DBT_PROFILES_DIR returns the same Error: Invalid value for '--profiles-dir': Path '~/.dbt/' does not exist.

echo $DBT_PROFILES_DIR returns ~/.dbt/ despite the absolute path being exported in my .zshrc :thinking:

When I explicitly export export DBT_PROFILES_DIR="/absolute/path/.dbt/" then dbt run` works :thinking:

I just realized this line is further down in my .zshrc:

export DBT_PROFILES_DIR="~/.dbt/"; I certainly don’t remember adding this.

Either way, this is no longer an issue in the sense that I can now start a new terminal session and run dbt CLI commands without error. The mystery still remains which account dbt-core is using that doesn’t have a relative ~/.dbt/ path :thinking:

Awesome @samgarfield !

I don’t understand what this means. Can you share more about the lingering mystery and we’ll see what we can do to help?

Thanks @doug.beatty!

In terms of the mystery, the folder ~/.dbt/ does exist, and contains a a valid profiles.yml file, so it’s a mystery to me why it throws Error: Invalid value for '--profiles-dir': Path '~/.dbt/'

The best explanation would be that dbt either doesn’t have the ability to parse ~/ as my home folder, or is being invoked by another user on my machine with a different home folder, both of which would be surprising to me.

With zsh, I would have expected the path in the error message to be fully expanded like:

Error: Invalid value for '--profiles-dir': Path '/Users/samgarfield/.dbt/' does not exist.

Something else you can try to see if your zsh shell is expanding the --profiles-dir:

dbt debug --profiles-dir ~/.dbt/

We want/expect the output to include something like this:

Using profiles.yml file at /Users/samgarfield/.dbt/profiles.yml

I’ve also been running into this with dbt version 1.5.0 and ZSH on my Mac.

@ianarmstrong_rl Would you be willing to open up a bug report here?