The problem I’m having
I used pip to upgrade multiple dependencies, including dbt-databricks and dbt-core so I could install data-diff.
After installing it, dbt run and other CLI commands throw this error: Error: Invalid value for '--profiles-dir': Path '~/.dbt/' does not exist.
However, this path exists. When I run dbt run --profiles-dir ~/.dbt/ the job completes without error.
I am pretty sure this is an issue with dbt-core however because this error is thrown whether running a databricks job or a bigquery job.
What could be causing this? 
MacOS Ventura 13.1
dbt-core vsn 1.5.0
dbt-databricks vsn 1.5.1
data-diff vsn 0.7.5
dbt-bigquery vsn 1.5.0a1
What I’ve already tried
I’ve confirmed that my ~/.zshrc contains export DBT_PROFILES_DIR="path/to/.dbt" and restarted Terminal.
This looks similar to (but not the same as) this one: [CT-2531] [Regression] v1.5.0 cannot run "dbt deps" in Dockerfile - "Error: Invalid value for '--profiles-dir': Path '/root/.dbt' does not exist." · Issue #7511 · dbt-labs/dbt-core · GitHub
Does this work or give an error?
dbt run --profiles-dir $DBT_PROFILES_DIR
What is the output of the following command for you? Is it a relative path or an absolute one?
echo $DBT_PROFILES_DIR
If relative, does it work if you export an absolute path?
1 Like
dbt run --profiles-dir $DBT_PROFILES_DIR returns the same Error: Invalid value for '--profiles-dir': Path '~/.dbt/' does not exist.
echo $DBT_PROFILES_DIR returns ~/.dbt/ despite the absolute path being exported in my .zshrc
When I explicitly export export DBT_PROFILES_DIR="/absolute/path/.dbt/" then dbt run` works
I just realized this line is further down in my .zshrc:
export DBT_PROFILES_DIR="~/.dbt/"; I certainly don’t remember adding this.
Either way, this is no longer an issue in the sense that I can now start a new terminal session and run dbt CLI commands without error. The mystery still remains which account dbt-core is using that doesn’t have a relative ~/.dbt/ path
Awesome @samgarfield !
I don’t understand what this means. Can you share more about the lingering mystery and we’ll see what we can do to help?
Thanks @doug.beatty!
In terms of the mystery, the folder ~/.dbt/ does exist, and contains a a valid profiles.yml file, so it’s a mystery to me why it throws Error: Invalid value for '--profiles-dir': Path '~/.dbt/'
The best explanation would be that dbt either doesn’t have the ability to parse ~/ as my home folder, or is being invoked by another user on my machine with a different home folder, both of which would be surprising to me.
With zsh, I would have expected the path in the error message to be fully expanded like:
Error: Invalid value for '--profiles-dir': Path '/Users/samgarfield/.dbt/' does not exist.
Something else you can try to see if your zsh shell is expanding the --profiles-dir:
dbt debug --profiles-dir ~/.dbt/
We want/expect the output to include something like this:
Using profiles.yml file at /Users/samgarfield/.dbt/profiles.yml
I’ve also been running into this with dbt version 1.5.0 and ZSH on my Mac.
@ianarmstrong_rl Would you be willing to open up a bug report here?