Release: v0.18.0 (Marian Anderson)

Archive
1

Release: v0.18.0 (Marian Anderson)

:bell: Who is Marian Anderson? Check out the release notes for a biography of this famous Philadelphian :bell:

dbt v0.18.0 introduces several new beta features around complex node selection and Slim CI workflows, as well as more extensible tooling for authors and users of community-supported plugins. For a highlight of the biggest features, watch 0.18.0 Office Hours and check out the prerelease discussion:

Breaking Changes

These are relevant for authors of community-supported plugins and some packages. We donā€™t expect these to affect most projects.

  • adapter_macro is no longer a macro and will raise a deprecation warning. Use adapter.dispatch instead.
  • dbt only has access to adapter plugin macros from the currently-in-use adapter or one of its dependencies, rather than macros from all adapters installed in the namespace.
  • Data tests are written as CTEs instead of subqueries. Adapter plugins for databases that donā€™t support CTEs may need to override this behavior.

For the full set of changes implemented since 0.17, please consult the changelog and migration guide.

Installation notes

# with pip
pip install --upgrade dbt==0.18.0

# with homebrew
brew install dbt@0.18.0
brew link --overwrite dbt@0.18.0

Iā€™ve devoted plenty of time elsewhere to talk excitedly about the biggest new featuresā€”node selection, Slim CI, dispatched macrosā€”but thereā€™s even more worth mentioning in v0.18.0.

More flexible documentation site

Assets

Previously, the only way to include images in the dbt-docs site was by hosting them at publicly accessible URLs. Now, it is possible to include static assets (such as images) in auto-generated docs site. Add a path to your dbt_project.yml:

asset-paths: ["assets"]

Add an image file:

ā”œā”€ā”€ dbt_project.yml
ā””ā”€ā”€ assets
    ā””ā”€ā”€ payments_erd.png

This file will be compiled to target/assets/payments_erd.png. You can reference it using Markdown image syntax in a resource description:

version: 2

sources:
	- name: stripe
	  description: "![payments ERD](assets/payments_erd.png)"

Custom overviews

You can create a custom overview page for each project and installed package, to appear when a user clicks into the file tree, by defining a docs block __[project_name]__:

{% docs __overview__ %}
    This is an overview of **the whole documentation site**!
{% enddocs %}

{% docs __my_project__ %}
    This is an overview of _just_ the resources defined in my own dbt project.
{% enddocs %}

{% docs __dbt_utils__ %}
    An installed package of utility macros that we leverage in our project, namely:
    - `surrogate_key`
    - `test_equality`
{% enddocs %}

Improved resource search

Since the very beginning, dbt-docs shipped with site-level search. In larger projects, though, you needed to know what you were looking forā€”the search function would match a term against a resourceā€™s name, description, column names, raw SQL, or tags; it would then return the list of all matches in alphabetical order.

dbt v0.18.0 includes two improvements:

  • A relevance ranking that returns the best matches in descending order.
  • A set of toggles so that you can limit the search criteria (e.g. tags only, or model and column names but not the rest).

Database-specific improvements

Redshift: Specify an IAM profile when connecting to Redshift, enabling you to leverage credentials configured in a different AWS account from your Redshift cluster.

Snowflake: Configure a query_tag as part of your connection profile, with the option of overriding for specific models. These tags can be searched, filtered, and parsed in the QUERY_HISTORY view.

BigQuery:

  • All BigQuery statement types return the number of bytes processed, enabling you to keep an eye on costly queries.
  • Local dbt developers can authenticate via OAuth while impersonating a BigQuery service account, a nice balance of security and scalability.
  • Set time-to-live for BigQuery tables by setting the hours_to_expiration config.
  • Set policy_tags on BigQuery columns to tap into column-level security.

Coming soon

Stay tuned for more details about ā€œSlim CIā€ in dbt Cloud!

For the next minor version (v0.19.0), we are focusing on the artifacts produced by dbt invocations. This includes better versioning and some breaking changes that will make these more stable in the long term. If youā€™re someone who currently parses the manifest, catalog, run results, or source freshness results as part of your larger workflows, please take a moment to check out the relevant issues (#2493, #2670, #2671) and drop us a line in the dbt Slack #development channel.

Thanks to our contributors!

If youā€™re interested in working on a feature in the dbt backlog, read through the Contributing Guide, check out the good first issues, and pop a message in Slack. Weā€™re always here to help.

Thanks to the following contributors who submitted PRs for the v0.18.0 release:

1 Like
2

This is great news, and I had no idea it was coming!

Can you say any more about the relevance ranking? What sort of principles does it follow?

3

Itā€™s quite straightforward, and hopefully a big improvement over listing alphabetically. Matches are assigned point values: the most points if the search term is found in a resource name or tag, fewer points for each of resource description, raw SQL, and column name. Then the results are displaying in descending order by point value.

Check out the PR:

1 Like