Get dbt up and running from the command line on a MacBook with an M1 chip

[roliepoly]

If you have a MacBook with an M1 chip and have tried to get dbt up and running from the command line, you may have run into the error below:

After some searching of the interwebs, I found a solution that worked for me, tucked away in a GitHub issue, in a comment by from Mike Lanza (thanks Mike! none of the other mentioned workarounds worked for me ).

Here’s how you do it

Step 1: Set up your Terminal to use Rosetta.
Go to Applications > Utilities > Terminal (or go to whatever other Terminal app you use), right click, select ‘Get Info’ and then select ‘Open using Rosetta’ under the General options. Close the Get Info window.

Step 2: Open your Terminal and uninstall any versions of dbt you may have tried to install before.
I’m team Homebrew - run brew uninstall dbt in the Terminal*. Then, if you run which dbt you should get dbt not found.

*Per @claire - you may also consider uninstalling Homebrew completely and reinstalling brew via Rosetta first.

Step 3: Create a python virtual environment and install dbt.
Full instructions here, but run these are the exact commands I ran in the Terminal:

python3 -m pip install --user --upgrade pip
python3 -m pip install --user virtualenv
python3 -m venv env
source env/bin/activate
pip3 install dbt==0.19.0
pip3 install --upgrade pip
dbt --version

Step 4: Change your working directory, if necessary.

Step 5: Do whatever you need to do in dbt!

Step 6: Deactivate your virtual environment.
Run deactivate in the Terminal.

And that’s it! Hope this saves some time for anyone struggling through the same situation

[claire]

Chiming in here with my suggestions! (Thanks for the tag @roliepoly!)

First of all, if dbt is the only command line program you are having issues with, use Olya’s approach! It’s great!

I personally was finding that I had M1 issues on other programs installed via homebrew (pyenv, ruby, everything, gah!) that a python env wouldn’t solve, so I had to do something a little more drastic!

Here’s what I did:

1. Completely uninstalled homebrew, and all brew-installed programs
   I can’t remember the exact instructions here, but think this was pretty google-able

2. Relauched iTerm with Rosetta.
   This is a way to use programs as though they are operating on an non-M1 Mac
   To do this: right-click on iTerm, click ‘Get Info’, and check the ‘Open using Rosetta’ option.

3. Reinstalled homebrew using my (Rosetta) iTerm
   This let me install dbt without issues, and all my other programs

Granted, using this approach unilaterally means that I lose some of the speedups from the M1 chip, and some programs are compatible with the M1 chip.

If you want to take advantage of the M1 software where possible, some solutions on the internet also suggest that it’s possible to run a dual setup, so that you can install M1-compatible software on the M1 architecture, and then have a backup way to install non-compatible software (like dbt).

There are two main approaches to a dual setup, neither of which I could get to work, but you may be able to! I’ve included a conceptual overview of both for context

Option 1: Using two versions of iTerm (one using the M1 architecture, one using Rosetta):
To create your second iTerm app, copy your iTerm app, name it “iTerm rosetta” version, and launch it using Rosetta.
Then install homebrew from the iTerm Rosetta app, and any incompatible programs.
Default to using your standard iTerm app when installing programs, and use the iTerm Rosetta app when installing programs that have compatibility issues.
I copied the app successfully, but couldn’t get the two versions of homebrew to coexist.

Option 2: Using one iTerm application, but two versions of homebrew
To install the second version of brew (using Rosetta) use arch -x86_64 as a prefix to the installation command (source).
Then use that prefix again when installing programs with the old version of homebrew: arch -x86_64 brew install <package>.
Default to using the standard brew command when installing programs, and use the prefixed brew command when installing programs that have compatibility issues.
Again, I couldn’t get this to play nicely, mostly due to pyenv issues.

The issues I hit were to do with $PATH variables, sourcing commands (esp. on pyenv, a tool I use a lot) and all kinds of things that I hate debugging. A more patient person than me could probably get a dual architecture working! For now, I’m sticking with my Rosetta-only approach.

[aathresh]

Hey folks! Adding in my solution which worked for dbt 0.20.1, Python 3.9.6 and M1 Mac.

This will require using Pycharm as the IDE for building and running dbt models.

Step 1 : Use Pycharm IDE ( latest version) for development & terminal emulator within the IDE for running dbt commands

Step 2: Create virtualenv with Python 3.9.* installed in it.

Step 3 . Add the location of above as the interpreter in PyCharm

Step 4. Install dbt inside the interpreter through PyCharm. Latest version of Pycharm has started supporting M1 ARM architecture without any explicit configurations, see the blog post here.

Step 5 . dbt run inside the terminal emulator might throw the error “command not found”. Solution to this is to alias ‘dbt’ in bashrc/zshrc to point to dbt installation location.

alias dbt=/Users/aathresh/.pyenv/versions/dbt-env/bin/dbt

[qbatten]

Chiming in with my experience dealing with this…

I ended up with the same solution as you, Claire. I tried the “both” thing (using your option 1, two versions of iTerm). Some of my projects were in the M1 setup and I switched them over to Rosetta as-needed if I ran into M1 compatibility issues. I was able to mostly get it to work although there were a lot of hiccups, especially around my .zshrc/.bashrc files.

After a few weeks I decided it was just too annoying/confusing/complex. I do everything w a Rosetta terminal now and basically just pretend I’m not on an M1. I have the base Macbook Air model and everything feels pretty snappy despite being in the emulator.

[cts]

Just a quick note on the original code in the article that tripped me up. dbt requires psycopg2 (or in the M1 case, psycopg2-binary) dependency. When following these steps, a critical error is to not upgrade pip AFTER you activate your virtual environment.

I spent a solid amount of time trying to figure out why nothing would load, thinking the pip upgrade outside of the virtual environment would propogate through - that is not the case.

source env/bin/activate
pip3 install --upgrade pip
pip3 install psycopg2-binary 
pip3 install dbt

Then you’re good to go

[clairetan]

Just to add on to the last comment.

pip3 install dbt is no longer valid

Use the below instead

     As of v1.0.0, `pip install dbt` is no longer supported.
      Instead, please use one of the following.
      
      **To use dbt with your specific database, platform, or query engine:**
      
          pip install dbt-<adapter>
      
          See full list: https://docs.getdbt.com/docs/available-adapters
      
      **For developers of integrations with dbt Core:**
      
          pip install dbt-core
      
          Be advised, dbt Cores python API is not yet stable or documented
          (https://docs.getdbt.com/docs/running-a-dbt-project/dbt-api)
      
      **For the previous behavior of `pip install dbt`:**
      
          pip install dbt-core dbt-postgres dbt-redshift dbt-snowflake dbt-bigquery
`