Running the same freshness test for all models

[shayms8]

The problem I’m having

I have multiple tests that i want to run on all my models. My problem is that I need to write the same configuration for all my models one by one

The context of why I’m trying to do this

I want to be able to reduce amount of yml lines and create more generic testing

My Question

I want to set Is a freshness test for to all models in specific schema. Can it be done?

[joellabes]

Hey @shayms8, a couple of options:

For sources: define freshness tests for all sources in a schema at once

If all the source tables in a schema contain a consistent loaded_at_field and have consistent freshness rules, you can define them once under the sources key as described here: freshness | dbt Developer Hub

If you have different rules for individual tables, you will just need to override any variations from there. Here’s a worked example from the docs:

version: 2

sources:
  - name: jaffle_shop
    database: raw

    loaded_at_field: _etl_loaded_at
    freshness: # default freshness
      warn_after: {count: 12, period: hour}
      error_after: {count: 24, period: hour}

    tables:
      - name: customers # this will use the freshness defined above

      - name: orders
        freshness: # make this a little more strict
          warn_after: {count: 6, period: hour}
          error_after: {count: 12, period: hour}

For models: use anchors (or reconsider using freshness tests altogether)

Freshness tests are most relevant on source tables, i.e. the raw data that you’re ingesting from another system. Sometimes it can be useful to verify that data in downstream tables is being updated, in which case you can use the dbt_utils.recency test. To attach it to multiple models you can use YAML anchors.

However, I’d be surprised if you needed to run these tests against all of your models - they will be as up to date as your last dbt invocation and/or the freshness of your source data (depending whether you’re using views or tables), so you shouldn’t need to be doing those tests across the board. If this is what you’re trying to do, I’d be interested to hear your reasoning.