The problem I’m having
In my dbt workflow, in steps like dbt seed, dbt ls, etc. Full reparse happens with message -
Unable to do partial parsing because profile has changed
.
We are fetching parameters in the profiles.yml file from the environment as they can change based on the specific use-cases. I believe this is the reason why partial parse is disabled for workflow runs, but is there a workaround for us using the same?
The context of why I’m trying to do this
Our dbt project has 500+ models, so a full reparse in the workflow steps makes it slow. So we are looking to utilize partial parsing for a speed-up.
Profiles.yml sample file -
default:
target: dev
outputs:
dev:
type: snowflake
account: "{{ env_var('DEV__ACCOUNT')}}"
user: "{{ env_var('DEV_USERNAME')}}"
password: "{{ env_var('DEV_PASSWORD')}}"
role: "{{ env_var('DEV_ROLE')}}"
database: "{{ env_var('DEV_DATABASE')}}"
warehouse: "{{ env_var('DEV_WAREHOUSE')}}"
schema: "{{ env_var('DEV_SCHEMA')}}"
threads: 25
prd:
type: snowflake
account: "{{ env_var('PRD_ACCOUNT')}}"
user: "{{ env_var('PRD_USERNAME')}}"
password: "{{ env_var('PRD_PASSWORD')}}"
role: "{{ env_var('PRD_ROLE')}}"
database: "{{ env_var('PRD_DATABASE')}}"
warehouse: "{{ env_var('PRD_WAREHOUSE')}}"
schema: "{{ env_var('PRD_SCHEMA')}}"
threads: 25