Announcing DBT-Helper

Archive
1

In order to help with some common workflow tasks when using DBT, I’ve been working on a command-line tool that automates some common tasks I’ve encountered.

It’s called dbt-helper and you can install it using pip install dbt-helper. A few notes:

  • It only works on MacOS
  • It only works with dbt v 0.13.*
  • It only works with dbt installed via pip

Here are the commands we’ve added (thanks @claire!!) so far:

Usage

dbt-helper (currently) has five sub-commands:

  • compare: Compare the relations in your warehouse with those that dbt is managing. This is useful for identifying “stale” relations that are no longer being updated by dbt (like if, for example, you converted the model from materialized to ephemeral).
    • Note: dbt-helper compare will compare all schemas that are impacted by models in the models/ directory. There is (currently) no way to specify a single schema to compare.
  • bootstrap: Create starter “schema.yml” files for your project. This function helpfully generates boilerplate dbt-model files for you so you don’t have to go through the copy/paste when you’re developing a new model.
    • Note: this command will not over-write existing schema.yml files. It will default to printing templates to the console, but you can create new files by using the --write-files flag.
  • show_upstream: Inspect the dbt graph and show the relations that are “upstream” from (i.e., the “parents” of) the selected relation. Print to the terminal.
  • show_downstream: The same as show_upstream but in the other direction – show dependents ‘downstream’ from (i.e., the “children” of) the selected relation
  • open: Open the compiled .sql file for a model by providing the model name only. You can also open the source or run .sql files for a model by using the appropriate flag. Useful when working in large dbt projects and you want to open files quickly wihout having to navigate a file tree.

As one might hope, you can view the command line options directly from the tool by using the help functionality:

dbt-helper --help

compare 

$ dbt-helper compare

Comparing local models to the database catalog. Checking schemas:
- dev_downstream
- dev_example
Warning: The following relations do not match any models found in this project:
TABLE "dev_example"."b"
VIEW "dev_downstream"."d"

bootstrap 

$ dbt-helper bootstrap --schemas dev_example

Bootstrapping the following schemas:
- dev_example
--------------------
Design for relation: dev_example.my_first_dbt_model
--------------------
version: 2
models:
  name: my_first_dbt_model
- columns:
  - name: id
  description: 'TODO: Replace me'

--------------------
Design for relation: dev_example.b
--------------------
version: 2
models:
  name: b
- columns:
  - name: city
  - name: count
  description: 'TODO: Replace me'

show_upstream 

$ dbt-helper show_upstream d

--------------------------------------------------------------------------------
                                  downstream.d
--------------------------------------------------------------------------------
                                   example.b
--------------------------------------------------------------------------------

open 

# Open the compiled version of model
$ dbt-helper open my_model

# Same as above
$ dbt-helper open my_model --compiled

# Open the run version of the model
$ dbt-helper open my_model --run

# Open the source version of the model
$ dbt-helper open my_model --source

Hopefully you find this useful – if you have questions about this tool (or ideas for new features) please drop a github issue on the repository (not the core dbt repository).

4 Likes
2

And we just added the extremely-useful retry-failed failed subcommand.

retry-failed

$ dbt-helper retry-failed
dbt run --models my_failed_model my_skipped_model
Running with dbt=0.13.0
Found 8 models, 20 tests, 0 archives, 0 analyses, 113 macros, 0 operations, 3 seed files, 0 sources

17:40:11 | Concurrency: 1 threads (target='dev')
17:40:11 |
17:40:11 | 1 of 2 START view model dev_example.my_failed_model.................. [RUN]
17:40:12 | 1 of 2 OK created view model dev_example.my_failed_model............. [CREATE VIEW in 0.65s]
17:40:12 | 2 of 2 START table model dev_example.my_skipped_model................ [RUN]
17:40:29 | 2 of 2 OK created table model dev_example.my_skipped_model........... [SELECT in 16.86s]
17:41:31 |
17:41:31 | Finished running 1 view models, 1 table models in 80.96s.
1 Like
3

Looks awesome! I will look forward to a Windows version!

4

Unless you pick up the gauntlet yourself, I suspect you might be waiting quite a while :wink:

1 Like