Report: Upgrading from 0.15.2 to 0.20.0

Archive
1

I just upgraded our DBT project from version 0.15.2 up to 0.20.0 and thought it might be useful to describe the process. We’ve been on this super-old version and one of the reasons we kept putting off the upgrade is that we were worried about the time commitment required for the upgrade process. I ran into one unexpected issue, but overall the whole thing was ridiculously easy!

TL;DR:

  1. Upgrading is easy, do it!
  2. When you upgrade from pre-0.17 to 0.17 or greater, make sure to check your docs and schema.yml descriptions for any Jinja code! The fix is to wrap it in a {% raw %} Jinja tag.

Part 1: Read the release notes, identify relevant changes

I went through all the release notes for versions between our current one and the newest version (see this folder on github). I copy-pasted all the breaking changes into a single markdown doc, separated by version.

Then, I went through from the earliest to latest version and, for each breaking change, checked our project to see whether we were affected. Mostly this just involved searching the repo— for example, in v0.16, fishtown deprecated the one-argument version of generate_schema_name. I just checked to see that we weren’t defining that macro w the deprecated call signature.

After removing all the changes that weren’t relevant to our project, I ended up with basically just a single change that needed to be made. I’m guessing many other people will find more than that, this is just how things shook out for us.

Part 2: Send it

It probably would’ve been better / more conservative to upgrade & test through each version one at a time, but there were so few changes to be made that I decided to just go for it all at once. Here are the main steps I took:

  1. In dbt_project.yml, I added config-version: 2, and changed require-dbt-version: from 0.15.2 to 0.20.0. (other things to check for when upgrading config-version to match the changes in 0.17 are listed here)
  2. Upgraded packages in packages.yml; we just use dbt-utils and are jumping from 0.3.0 to 0.7.0 . You of course have to do this with any other packages you’re using; hopefully their release notes mention which package version is compatible with which dbt version.
  3. Upgrade everything.
    1. First, I upgraded dbt-core and both adapters we use (dbt-snowflake and dbt-postgres). If you installed via pip, just pip install --upgrade dbt-core dbt-postgres dbt-snowflake; if you used brew, you can either upgrade it straight up or unlink dbt, install the newer version, and link that one (brew unlink dbt && brew install dbt@0.20.0 && brew link dbt@0.20.0).
    2. Next, I installed the upgraded dbt packages that I’d previously added in packages.yml (dbt deps).

After the upgrade I ran into an issue and had some trouble finding the root cause.

Part 3: Error – Macro that does not exist

When I first did dbt run after upgrading, I got a weird error. I had a lot of trouble tracking down the cause, but eventually found a previous conversation in DBT Slack that gave a solution (Thank you Zack and @coap!). I thought it might be helpful to add this issue/solution to Discourse as well, so I’ve described the problem in detail here, and given the solution that coap described, which worked for me.

Part 3.1: How the problem appeared

Upon upgrading all the way up from 0.15.2 to 0.20.0, I got this error, referring to a custom macro in our project:

Compilation Error
  'nr_host_filter_excluded_host_patterns' is undefined. This can happen when calling a macro that does not exist. Check for typos and/or install package dependencies with "dbt deps".

The macro that’s causing this error is in macros/standardization_macros/nr_host_filter_excluded_host_patterns.sql, and is configured in macros/standardization_macros/standardization_macros.yml. Macro-paths in my dbt_project.yml is ["macros"], which should be correct given those paths.

So… my macro-paths is correctly configured, and there are no typos in the macro title or where it was called. I had not run into this error when using dbt v0.15.2. The DBT log wasn’t very helpful, it just threw an error in the middle of compilation; anything else I tried to run threw the same error (since it couldn’t compile, it couldn’t really do anything with my project).

Part 3.2: Root cause of the problem

The schema.yml file that described the problematic macro had Jinja in the description field. I had included an example of how to call the macro:

  - name: nr_host_filter_excluded_host_patterns
    description: >
      ...
      Assuming there are no other items in where clause, usage is:
        `WHERE TRUE {{ nr_host_filter_excluded_host_patterns(COLUMNS) }}`

DBT was trying to render that Jinja, but it ended up trying to call the macro before it was done parsing it (or something like that).

I suspect that this error started happening due to the change in version 0.17 that enabled NativeEnvironment rendering for yaml fields.

Part 3.3: Solution

I just needed to prevent DBT from trying to render that Jinja code. The {% raw %} tag is useful here. Instead of the schema.yml code I have above, I ended up with

  - name: nr_host_filter_excluded_host_patterns
    description: >
    {% raw %}
      ...
      Assuming there are no other items in where clause, usage is:
        `WHERE TRUE {{ nr_host_filter_excluded_host_patterns(COLUMNS) }}`
    {% endraw %}

Part 4: Done :slight_smile: !

That’s it! Before PR-ing these changes and upgrading prod (this was all on my local), I ran a quick test to make sure nothing unexpected was happening: Using dbt 0.20, I

  1. Created a new development database that is a clone of prod,
  2. Ran a few models with full-refresh, and compared them to prod (full table equality)
  3. Waited a day for some new data to come in, then ran every model in my test database (incremental models non-full-refresh, since we have a few truly gigantic tables in there) and compared rowcount for every table & view between prod and the test database.

There are a few macros I wrote that help with the above tests; they were originally written to help us manage development environments. I plan to share them in a later post.

Meta-note: I wasn’t sure what the right topic was for this post; if anyone has an idea for a better place to put it, let me know!

5 Likes