Error when running dbt-dry-run command in Databricks workflow

Help
1

I’m encountering an error when trying to run the dbt-dry-run command within a Databricks workflow. The error message I’m receiving is as follows:

CalledProcessError: Command 'b'\nif cd "/tmp/tmp-dbt-run-833842462680805" ; then\n set -x\n dbt-dry-run returned non-zero exit status 1.

+ dbt deps
10:32:57  Running with dbt=1.7.9
10:32:57  Warning: No packages were found in packages.yml
10:32:57  Warning: No packages were found in packages.yml

+ dbt seed
10:33:01  Running with dbt=1.7.9
10:33:03  Registered adapter: databricks=1.7.9
10:33:03  Unable to do partial parsing because saved manifest not found. Starting full parse.
10:33:06  Found 5 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 538 macros, 0 groups, 0 semantic models
10:33:06  
10:33:06  Concurrency: 8 threads (target='databricks_cluster')
10:33:06  
10:33:06  1 of 3 START seed file dbt.raw_customers ....................................... [RUN]
10:33:06  2 of 3 START seed file dbt.raw_orders .......................................... [RUN]
10:33:06  3 of 3 START seed file dbt.raw_payments ........................................ [RUN]
10:33:17  1 of 3 OK loaded seed file dbt.raw_customers ................................... [INSERT 100 in 10.91s]
10:33:17  2 of 3 OK loaded seed file dbt.raw_orders ...................................... [INSERT 99 in 11.16s]
10:33:17  3 of 3 OK loaded seed file dbt.raw_payments .................................... [INSERT 113 in 11.21s]
10:33:17  
10:33:17  Finished running 3 seeds in 0 hours 0 minutes and 11.83 seconds (11.83s).
10:33:17  
10:33:17  Completed successfully
10:33:17  
10:33:17  Done. PASS=3 WARN=0 ERROR=0 SKIP=0 TOTAL=3

+ dbt-dry-run
Traceback (most recent call last):

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/dbt-dry-run", line 8, in <module>
    sys.exit(main())

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt_dry_run/__main__.py", line 5, in main
    exit(app())

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt_dry_run/cli.py", line 117, in run
    exit_code = dry_run(

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt_dry_run/cli.py", line 48, in dry_run
    project = ProjectService(args)

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt_dry_run/adapter/service.py", line 40, in __init__
    dbt_project, dbt_profile = RuntimeConfig.collect_parts(self._args)

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/config/runtime.py", line 251, in collect_parts
    profile = cls.get_profile(

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/config/runtime.py", line 106, in get_profile
    return load_profile(

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/config/runtime.py", line 70, in load_profile
    profile = Profile.render(

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/config/profile.py", line 436, in render
    return cls.from_raw_profiles(

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/dbt/config/profile.py", line 391, in from_raw_profiles
    raise DbtProjectError("Could not find profile named '{}'".format(profile_name))
dbt.exceptions.DbtProjectError: Runtime Error
  Could not find profile named 'jaffle_shop'

The command I have given:

image
image1043×561 23.8 KB

But when I tried this same project without a dry run in workflow command this was working

  1. Installation: I’ve installed the dbt-dry-run Python library on the Databricks workflow cluster.
  2. Environment Setup: The environment is configured to use Databricks, and I’m running the workflow within this environment.
  3. Project Configuration: The project configuration seems to be correct, as other dbt commands execute without errors.
  4. Profile Configuration: I suspect the issue might be related to the profile configuration. The error message indicates it can’t find a profile named ‘jaffle_shop’. However, the profile is properly configured and the project runs successfully without the dry run option.
  5. Cluster Libraries: Ensure that all necessary libraries and dependencies are properly installed and accessible within the Databricks cluster environment.

Any insights or suggestions on how to resolve this issue would be greatly appreciated.
Thanks in advance