Limiting the not_null test

[rob]

This question was posed in Slack in 2019:

When running schema tests, such as not_null, the SQL is executed over the entire table. As the table grows in size the cost of running such tests increases… Is there a way to limit the schema tests to the partition of data that was inserted on the current run ?

One proposed solution was “write a custom test that only filters on the latest [data].”

What’s the best practice?

[joellabes]

As of dbt 0.20.0, you can include a where clause in your test config! Here’s an example from the docs: where | dbt Docs

If you’re not able to upgrade yet, earlier versions of dbt-utils have a not_null_where test you can use instead.

[rob]

Great. Thanks @joellabes