Using Pyenv to run multiple versions of dbt, per project

Archive
1

To run multiple versions of dbt on the same machine, for example to test one project against 0.14 while leaving others on 0.13, I’ve been using the following approach (Mac Only).

Install pyenv to manage Python versions:

Easiest install is typically via homebrew:

brew update

brew install pyenv

Pyenv allows you to run multiple versions of Python side by side without issues.
So, for example, we can install 3.7.1 and then configure our machine to use that globally:

pyenv install 3.7.1
pyenv global 3.7.1

python --version
Python 3.7.1

We can then install the current version of dbt (currently 0.13) into this fresh 3.7.1 install, via

pip install dbt

0.13 is now the default version for all of our projects.

In addition, you will want to be able to manage mutiple evironments of the same Python versions on the same machine, so that you can install different packages within in each environment.
For that, we install the pyenv-virtualenv plugin from

via

brew install pyenv-virtualenv

We can then create a new virtual environment using the previously installed 3.7.1 that we’ll call venv371_dbt_014, via

pyenv virtualenv venv371_dbt_014

pyenv virtualenvs will list the newly configured environment.

We can now use this new, separate 3.7.1 environment in one of our dbt projects ~/dev/my_dbt_project, by adding a file to the root of the dbt project, called .python-version. In this file, we simply add the name of the virtual environment we just created and want to use only for this project for now:

Contents of .python-version.:
venv371_dbt_014

Whenever we navigate to ~/dev/my_dbt_project now, pyenv will automatically switch to use the venv371_dbt_014 environment.

Running pip list in this folder will show that no packages are installed in this environment, so we’ll install the latest release candidate,
pip install dbt==0.14.0rc1

We can now test 0.14 in just this project, leaving our other projects untouched. If we’re testing 0.14 on a new git branch, we might even want to check the .python-version file into our repo until we’re done with testing.
To use the global version, 0.13 , simply remove .python-version from the project folder and reload your terminal.

1 Like
2

Awesome writeup!

The only thing I would add is that instead of fooling around with .python-version, use pyenv local venv371_dbt_014 to set the value, pyenv local --unset to set it.

And to temporarily switch to a new virtualenv, you can do pyenv shell venv371_dbt_014 which will only change the env for your current terminal session. I use that a lot more often than pyenv local these days.

3

Thanks, those are helpful!

I prefer using .python-version because it makes the env/version switching automatic when you navigate to the folder, but yes, you do have to remember it and it could be confusing to other users of the project.

4

Maybe I wasn’t as clear as I could have been! pyenv local actually sets that file for you:

jake@boromir ~/tmp $ ls -la
total 0
drwxr-xr-x   2 jake  staff    64 Jul  5 12:11 .
drwxr-xr-x+ 73 jake  staff  2336 Jul  5 12:11 ..
jake@boromir ~/tmp $ pyenv local dbt36
(dbt36) jake@boromir ~/tmp $ ls -la
total 8
drwxr-xr-x   3 jake  staff    96 Jul  5 12:11 .
drwxr-xr-x+ 73 jake  staff  2336 Jul  5 12:11 ..
-rw-r--r--   1 jake  staff     6 Jul  5 12:11 .python-version
(dbt36) jake@boromir ~/tmp $ cat .python-version
dbt36
(dbt36) jake@boromir ~/tmp $ pyenv local --unset
jake@boromir ~/tmp $ ls -la
total 0
drwxr-xr-x   2 jake  staff    64 Jul  5 12:11 .
drwxr-xr-x+ 73 jake  staff  2336 Jul  5 12:11 ..
1 Like