Coming soon: v0.17.1

[drew]

dbt v0.17.1 is a bugfix release that fixes regressions and reverts significant functionality released in 0.17.0. The 0.17.1 release of dbt is now in Release Candidate mode, with a final release to come after community testing. We need some help with testing this prerelease to ensure that the included fixes and changes are working correctly.

Fixed regressions

The following regressions included in 0.17.0 have been fixed:

• dbt compile no longer creates schemas in the database
• dbt deps now respects the --project-dir flag
• Parallel RPC requests are now executed correctly
• Fixed a Windows FileNotFound error when dbt writes files with very long paths
• Native rendering has been disabled by default (see below)
• And more… see the changelog for complete information on the changes in this release.

Reverting Native Rendering

The last release of dbt, v0.17.0, included native rendering, a powerful feature which made it possible to supply non-string values to dbt configs. As an example, native rendering makes it possible to supply boolean values to the enabled config using environment variables:

# dbt_project.yml

models:
  my_project:
    reporting_rollups:
      +enabled: "{{ env_var('include_rollups') == '1' }}"

In previous versions of dbt, this enabled expression would evaluate to "True" which is a string. Because the enabled config requires that a boolean value is supplied, dbt would fail with a type validation error like got string but was expecting boolean. The code shown above looks reasonable, but did not work on older releases of dbt.

With native rendering (released in 0.17.0), dbt would detect that "True"(a string) looks like the string representation of the boolean value True. Accordingly, dbt would replace "True" with True before passing it on to the enabled config. This all worked well and unlocked new, exciting use cases.

We were eager to release this feature because it was a much-requested piece of functionality in dbt. In a lot of ways, this is a “magical” feature. In 0.16.1, the code above did not work. In 0.17.0, the code above does work. We liked that native rendering appeared to be Doing The Right Thing with minimal effort from end-users.

What we did not realize in pre-release testing was in just how many cases dbt would do totally the wrong thing. Here are some examples of code that worked in 0.16.1 that we errantly broke in 0.17.0:

• String values like "0000" were converted to integers with different representations (eg. 0)
• String values like "0001" were not converted to integers for an obscure and not at all obvious reason
• Empty strings ('') were converted to None
• Quotes in quoted strings were stripped (eg. "'abc'" → "abc"). This is a big problem for schema.yml specifications which make use of column-level quoting.

For an explanation of why these seemingly confusing behaviors occurred (and what exactly native rendering means), see the appendix at the bottom of this post.

What we’re doing about it

Simply put, we need to revert the change. There are just too many inscrutable and unexpected failure modes. Native rendering is a powerful feature, but it should absolutely not be the default behavior.

To this end, dbt v0.17.1’s Jinja rendering reverts back to the 0.16.x behavior by default. Users who wish to take advantage of native rendering can make use of jinja filters to opt-in to the functionality. This feels right to us: if you want to make dbt do something special and magical, you should just need a couple extra keystrokes. Here’s the same example from above, as you’d write it in 0.17.1:

# dbt_project.yml

# The "as_bool" filter marks the Jinja expression as a boolean

models:
  my_project:
    reporting_rollups:
      enabled: "{{ (env_var('include_rollups') == '1') | as_bool }}"

In this example, the as_bool filter applies a special marker to the preceding expression. This marker indicates to dbt that the expression (env_var('include_rollups') == '1') should be rendered as a boolean instead of as a string. This code works correctly in dbt v0.17.1, and the as_bool or as_integer Jinja filters will now be necessary to take advantage of native rendering in all future releases of dbt.

Breaking changes

The changes described above are breaking changes that serve as a fix for another set of previously released breaking changes. This is a pretty rough spot to put dbt users in, and it’s definitely a departure from our usual practices and intentions. When considering how to proceed here, we could either:

• Live with the curious and hard-to-reason-about behavior, improve documentation, and commit to supporting—basically forever—dbt users who continue to hit these problems
• Revert the change, ask folks to perform maybe another migration, and learn from the experience

We’ve decided to go with the second option here after seeing a ton of feedback about these failure modes from the dbt community. Here are our key takeaways:

• The #1 thing we can do to prevent this from happening again is getting more beta testers for prerelease versions of dbt. Please join the #prereleases channel in dbt Slack to be notified of new prereleases and to discuss the relevant changes contained within them.
• We need to improve the breadth and depth of our automated testing around some of the trickiest parts of dbt. We just didn’t anticipate some of the failure modes listed above, and automated testing is one of the things that will best assist us in catching these issues before they make it into a production release.

What’s next

We’re delaying the 0.17.1-final release a little longer to get feedback from dbt users. If 0.17.0 broke your project in a way that prevented you from upgrading, please try out the 0.17.1-rc4 prerelease and let us know how it goes. The #prereleases channel is a great place to give feedback and ask questions.

At the same time, we’re full-steam ahead on dbt v0.18.0, and we’ll be sure to keep everyone apprised of the new and compelling features coming in that release so we can get some help with testing them out :).

Appendix & Technical Details

• What is native rendering? This term comes from the Jinja library that dbt uses. The default Jinja Environment renders Jinja templates to strings. The NativeEnvironment renders Jinja templates to Python types.
• How does dbt/Jinja “pick” the Python type that Jinja templates are rendered to? Great question, glad you asked. dbt uses the ast.literal_eval function in Python to safely find the Python representation for a given string. This is the key feature uses in the native rendering environment, and it’s also the thing responsible for all of the bugs/confusing failure modes shared above.

An annotated Native Rendering example:

$ python
>>> import ast
>>> ast.literal_eval('1') # Interpreted as an int
1
>>> ast.literal_eval('"1"') # Interpreted as a string. Double quotes are stripped
'1'
>>> ast.literal_eval('True') # Interpreted as a boolean
True
>>> ast.literal_eval('"True"') # Interpreted as a string, Double quotes are stripped
'True'
>>> ast.literal_eval('000000') # Interpreted as an integer
0
>>> ast.literal_eval('000001') # Non-zero Python integers cannot start with zeroes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 46, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    000001
         ^
SyntaxError: invalid token

This last case is the most interesting and most terrible in my opinion. Python3 does not allow numbers to begin with leading zeroes (except for the actual literal number zero). In the type coercion from a string to an integer, dbt’s native rendering code would catch this error and return the original string instead, so:

key1: "{{ '0000' }}" # Evalues to 0
key1: "{{ '0001' }}" # Evalues to '0001'

Try explaining that one in #beginners!