The problem I’m having
I wrote a custom generic test “not_blank.sql” (code below). When I do dbt compile, this warning shows up: 08:50:51 [WARNING]: Test 'test.[project name].not_blank_[model name]_[column name].0d6793aa36' (models\staging\schema.yml) depends on a node named 'not_blank_[model name]_[column name]' in package '' which was not found.
I tried putting “not_blank.sql” in tests/generic, macros/tests, macros/generic, or macros/, but none worked.
{% test not_blank(model, column_name) %}
{% set relation = ref(model.name) %}
{% set database = relation.database %}
{% set schema = relation.schema %}
{% set table = relation.identifier %}
{% set col_type_query %}
SELECT data_type
FROM {{ database }}.information_schema.columns
WHERE table_schema = '{{ schema }}'
AND table_name = '{{ table }}'
AND column_name = '{{ column_name.upper() }}'
{% endset %}
{% set results = run_query(col_type_query) %}
{% if execute %}
{% set col_type = results.columns[0].values()[0] %}
{% else %}
{% set col_type = 'UNKNOWN' %}
{% endif %}
SELECT *
FROM {{ relation }}
WHERE
{% if col_type in ['TEXT', 'STRING', 'VARCHAR', 'CHAR'] %}
{{ column_name }} IS NULL OR {{ column_name }} = ''
{% else %}
{{ column_name }} IS NULL
{% endif %}
{% endtest %}
The context of why I’m trying to do this
I’m trying to use this custom generic test on multiple columns across multiple models.
What I’ve already tried
- Putting the model in tests/generic, macro/tests, macro/generic, or macro/.
- dbt clean, dbt deps, dbt compile
Other relevant codes
Below are codes for dbt_project.yml and models/staging/schema.yml
dbt_project.yml:
# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: <project name>
version: '1.0.0'
config-version: 2
# This setting configures which "profile" dbt uses for this project.
profile: 'nss' # references the profile in your ~/.dbt/profiles.yml file
# These configurations specify where dbt should look for different types of files.
# The `source-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
- "target"
- "dbt_packages"
# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models
models:
<project name>:
staging:
materialized: view
schema: staging
analytics:
materialized: view
schema: analytics
business:
materialized: view
schema: business
# Add seed file definitions
seeds:
+database: landing
+schema: dbt_seeds
<project name>:
<table name>:
+column_types:
<column 1>: integer
<column 2>: varchar
<column 3>: varchar
models/staging/schema.yml:
version: 2
models:
- name: <model name 1>
columns:
- name: <column name 1>
tests:
- not_null
- unique
- name: <column name 2>
tests:
- not_blank
- unique
- name: <column name 3>
tests:
- not_blank
- unique
- name:<model name 2> ......