dbt Core 1.9 - Intermittent error "OSError: [Errno 116] Stale file handle"

The problem I’m having

We are experiencing intermittent OSError: [Errno 116] Stale file handle errors when running dbt commands inside a Google Cloud Run Job. The error occurs during the dbt parsing phase, specifically when dbt is trying to read the project’s source files.

Our setup involves running dbt from the official dbt-labs/dbt-bigquery:1.9.0 public Docker image. The dbt project source code is stored in a Google Cloud Storage (GCS) bucket, which is mounted as a volume into the Cloud Run instance for each execution.

Crucially, we are already isolating our artifacts by providing a unique --target-path and --log-path for every single dbt execution, so the issue is not related to concurrent writes to the target directory.

The context of why I’m trying to do this

Environment

  • Image: dbt-labs/dbt-bigquery:1.9.0
  • OS: Debian 12 (Bookworm) (derived from the python:3.11-slim base image)
  • Python: 3.11
  • dbt: 1.9.0
    Which database adapter are you using with dbt?
    bigquery

Expected Behavior

dbt should reliably parse all project files from the GCS mounted volume without encountering a filesystem error. The process should be robust to high-concurrency environments, especially given that mutable artifacts (target/, logs/) are already isolated per run.

Some example code or error messages

11:02:24 Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb141fd6090>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb1423d6b90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb1424c1550>]}
11:02:24 Running with dbt=1.9.0
11:02:24 running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'version_check': 'True', 'fail_fast': 'False', 'log_path': '/mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64', 'debug': 'False', 'profiles_dir': '/mnt/gcs/dbt', 'use_colors': 'False', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'empty': 'None', 'log_format': 'json', 'static_parser': 'True', 'invocation_command': 'dbt --log-format-file=text --log-format=json --warn-error-options {"include": ["NothingToDo"]} --no-use-colors test --select source:my_first_source source:my_second_source --vars {"AIRFLOW_DAG": "my_example_dag"} --target-path /mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64 --log-path /mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64', 'warn_error_options': "WarnErrorOptions(include=['NothingToDo'], exclude=[])", 'target_path': '/mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64', 'introspect': 'True', 'send_anonymous_usage_stats': 'True'}
11:02:27 Sending event: {'category': 'dbt', 'action': 'project_id', 'label': 'ec4be0c9-09e6-46a7-934a-2b5de26cdda6', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb12c183ad0>]}
11:02:27 Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': 'ec4be0c9-09e6-46a7-934a-2b5de26cdda6', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb12c223950>]}
11:02:27 Registered adapter: bigquery=1.9.0
11:02:28 checksum: b46e93efb52f67a19bdb6e7abf8f24e437648676c7bd1d5db6ccdaa72cc84c4c, vars: {'AIRFLOW_DAG': 'my_example_dag'}, profile: , target: , version: 1.9.0
11:02:28 Unable to do partial parsing because saved manifest not found. Starting full parse.
11:02:28 Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': 'ec4be0c9-09e6-46a7-934a-2b5de26cdda6', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb127f3e150>]}
11:02:35 Encountered an error:
[Errno 116] Stale file handle
11:02:35 Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 153, in wrapper
  result, success = func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 103, in wrapper
  return func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 235, in wrapper
  return func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 264, in wrapper
  return func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 311, in wrapper
  return func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 327, in wrapper
  setup_manifest(ctx, write=write, write_perf_info=write_perf_info)
  File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 351, in setup_manifest
  ctx.obj["manifest"] = parse_manifest(
  ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/parser/manifest.py", line 2037, in parse_manifest
  manifest = ManifestLoader.get_full_manifest(
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/parser/manifest.py", line 310, in get_full_manifest
  manifest = loader.load()
  ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/parser/manifest.py", line 357, in load
  file_reader.read_files()
  File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 212, in read_files
  self.read_files_for_project(project, file_types)
  File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 219, in read_files_for_project
  project_files[file_type_info["parser"]] = read_files_for_parser(
  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 167, in read_files_for_parser
  source_files = get_source_files(
  ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 156, in get_source_files
  file = load_source_file(fp, parse_file_type, project.project_name, saved_files)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 84, in load_source_file
  file_contents = load_file_contents(path.absolute_path, strip=True)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dbt_common/clients/system.py", line 178, in load_file_contents
  to_return = handle.read().decode("utf-8")
  ^^^^^^^^^^^^^
  OSError: [Errno 116] Stale file handle
11:02:35 Resource report: {"command_name": "test", "command_success": false, "command_wall_clock_time": 11.851658, "process_in_blocks": "0", "process_kernel_time": 0.763955, "process_mem_max_rss": "197116", "process_out_blocks": "32", "process_user_time": 2.880488}
11:02:35 Command `dbt test` failed at 11:02:35.484429 after 11.86 seconds
11:02:35 Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb141fd5d90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb141fd5990>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb12c759590>]}
11:02:35 Flushing usage events
11:02:45 An error was encountered while trying to flush usage events