We are trying to generate schema name based on the target passed (–target) and the schema name provided in the model in profiles.yml. While running from dbt cli command through locally, it is working fine. ie schema name is derived based on the target name and schema name from model (using generate_schema_name). However when we run the same application in databricks using the job in workflows, it is picking up the schema name from seeds. Say the seeds has the schema name as “my_seed” and the target is qa, then the schema is derived as qa_my_seed.
We are running using the dbt version 1.4.1 and in workflow jobs it is 1.4.5.
1 Like
One thing to check is if there are any differences in the way the profiles.yml file is being read by the local dbt cli versus the Databricks job (like did them https://www.programmingassignment.net/). It’s possible that the Databricks job is not reading the profiles.yml file correctly or is using a different version of the file