Testing source data with user defined tests/macros

Help
1

Hi Everyone,

The problem I’m having

I’m trying to develop some tests/macros that can be specified in the project models .yml file. But dbt either returns a warning that there’s nothing to do or that the selection criteria does not match any nodes.

The context of why I’m trying to do this

Our source data is the culprit of most of our data issues. Identifying issues there is ideal. Although, our stage tables are just unions of the various source tables; testing their will suffice. But for my knowledge I’d like to know how to test on sources.

What I’ve already tried

I started with the basic built-in tests like unique on the models, then applied them to the sources. Running dbt test does not work, but that might be expected behavior if the sources are not in the DAG (ie called by source())? Some of the sources are in the DAG while others are not.

dbt test --select source:<source> results in the same issue, and actually source:* throws a zsh error.

So then I tried adding the sources as a model and attempted to test them as such. Same issue as above.

Obviously this does not get to the heart of the issue, that I’m trying to develop my own functions, but using a built-in test is a good starting point. One question related to the real issue: do all dbt tests consider having any rows in the final query a failure? Is there a way to specify a failure using a jinja variable, maybe a boolean?

Some example code or error messages

Testing as a source:

sources:
  - name: nm_trace
    description: Trace tables in snowflake
    tables:
      - name: bmsi_customers_raw
        columns:
          - name: customerid
            tests:
              - not_null

dbt test -s source:bmsi_customers_raw

The selection criterion 'source:bmsi_customers_raw' does not match any nodes
                                                                             
[WARNING]: Nothing to do. Try checking your model configs and model specific

Running it as a model:

models:
  - name: bmsi_customers_raw
    columns:
      - name: customerid
        tests:
          - not_null

dbt test -s bmsi_customers_raw

[WARNING]: Did not find matching node for patch with name 'bmsi_customers_ra
w' in the 'models' section of file 'models/bt_backend_data.yml'
[WARNING]: Nothing to do. Try checking your model configs and model specific
ation args
2

I’m also getting the same issue with zsh when trying to run commands like dbt test --select source:*. Other commands seem to work fine, and if I switch to a bash terminal then this command works too. Would love to know to know why it’s not working.

1 Like