The problem I’m having
I added a source and model in a yml file and added a sql (or pyspark) file that uses the defined source. All very basic, but I keep getting this error:
Compilation Error
Model ‘model.governance_dbt.triple_a’ (models/bronze/breeding/triple_a/triple_a.py) depends on a source named ‘aeu.bull_triple_a’ which was not found
I can’t figure out what I am doing wrong. Other models in my project are similar and work fine. Location and directory structure are also similar. Both my files are in the same directory.
Any suggestions are welcome!
The context of why I’m trying to do this
Just want to add a simple model
What I’ve already tried
dbt parse(or dbt run) without the new model files works fine. But as soon as I add the model, it fails with the above mentioned error.
Instead of using pyspark, I tried sql, but same result (as expected)
The files for the model I am adding
file: bull_triple_a.py
import pyspark.sql.functions as f
def model(dbt, session):
df = dbt.source("aeu", "bull_triple_a").filter(f.col("expiry").isNull())
return df
file: bull_triple_a.yml
models:
- name: bull_triple_a
config:
submission_method: job_cluster
job_cluster_config: "{{ var('job_cluster_config_pool_breeding_small_prd') if target.name == 'prod' else var('job_cluster_config_pool_breeding_small_dev') }}"
create_notebook: true # Set to true to create a separate notebook for development and testing
sources:
- name: bull_triple_a
description: AEU triple A data
schema: aeu
tags:
- breeding
meta:
owner: '@BGF'
database: versioned_{{ 'prod' if target.name in ['prod','staging','lab'] else 'dev' }}
tables:
- name: bull_triple_a