The problem I’m having
I have source properties defined in a sources.yml file stored under a subfolder in the models folder that defines the source name, database, schema, and tables. However when I run my dbt model, it keeps defaulting to the information stored in the profile.yml file stored in a separate folder.
The context of why I’m trying to do this
I have a unique situation where the name of my source schema may change unexpectedly over time so I’m compensating by generating the sources.yml file with updated source information (schema name). So I’m trying to have my dbt models refer to the sources.yml file for that updated schema name using the source() jinja function.
What I’ve already tried
Example source.yml file code (stored in “/models/sources”)
version: 1
sources:
- name: my_source_name
database: my_source_db
schema: updated_schema_name123
tables:
- name: my_source_table
identifier: my_source_table123
Example profiles.yml code (stored in “/profiles”)
example_project:
outputs:
dev:
dbname: dev_database
host: localhost
pass: my_password
port: 5432
schema: default_schema
threads: 4
type: postgres
user: db_user
target: dev
Example dbt model code (model name “my_test_model.sql”)
{{ config(materialized='table') }}
SELECT
col1,
count(*)
FROM {{ source('my_source_name', 'my_source_table') }}
group by col1
dbt run command
dbt run --profiles-dir=/profiles --target dev
When running the dbt run command, the result is a new target schema created named “default_schema_test_dbt_schema” since “test_dbt_schema” is defined in dbt_project.yml. But my expectation is a new target schema named “updated_schema_name123_test_dbt_schema” according to this documentation where referring to the source name should automatically map to the database, schema, and table(s) defined in the sources.yml file. Does the “–profiles-dir” parameter override looking at the sources.yml file somehow.
Thanks,
Glen