How to deal with failing tests in development due to only having a subset of rows

In-Depth Discussions
1

Hi all,

I have been trying to get around a problem that it’s becoming annoying (especially for newcomers).

Let’s say I have a model “transactions” and a model “customers”. There is a test in “transactions” for the column customer_id referencing the column customer_id in “customers” (as any transaction must have a customer that exists in the system).

However, we are using a common pattern (Best practices | dbt Developer Hub) to limit the number of rows we use in development, preventing our data usage from exploding.

Basically we have

{% if target.name == 'dev' %}
limit {{ var("development_row_limit") }}
{% endif %}

in our staging models.

The problem is that tests like the one explained above fail, How can we deal with this? I currently just ignore those tests in development (but I shouldn’t be doing that).

2

I found a temporary solution to the problem.

In the dbt_project.yml file I added:

tests:
  <project_name>:
    intermediate:
      +severity: "{{ 'warn' if target.name == 'default' else 'error' }}"
    marts:
      +severity: "{{ 'warn' if target.name == 'default' else 'error' }}"

That way tests are going to be set to warning severity by default in the development environment for each user