The problem I’m having
We are experiencing intermittent OSError: [Errno 116] Stale file handle errors when running dbt commands inside a Google Cloud Run Job. The error occurs during the dbt parsing phase, specifically when dbt is trying to read the project’s source files.
Our setup involves running dbt from the official dbt-labs/dbt-bigquery:1.9.0 public Docker image. The dbt project source code is stored in a Google Cloud Storage (GCS) bucket, which is mounted as a volume into the Cloud Run instance for each execution.
Crucially, we are already isolating our artifacts by providing a unique --target-path and --log-path for every single dbt execution, so the issue is not related to concurrent writes to the target directory.
The context of why I’m trying to do this
Environment
- Image: dbt-labs/dbt-bigquery:1.9.0
- OS: Debian 12 (Bookworm) (derived from the python:3.11-slim base image)
- Python: 3.11
- dbt: 1.9.0
Which database adapter are you using with dbt?
bigquery
Expected Behavior
dbt should reliably parse all project files from the GCS mounted volume without encountering a filesystem error. The process should be robust to high-concurrency environments, especially given that mutable artifacts (target/, logs/) are already isolated per run.
Some example code or error messages
11:02:24 Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb141fd6090>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb1423d6b90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb1424c1550>]}
11:02:24 Running with dbt=1.9.0
11:02:24 running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'version_check': 'True', 'fail_fast': 'False', 'log_path': '/mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64', 'debug': 'False', 'profiles_dir': '/mnt/gcs/dbt', 'use_colors': 'False', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'empty': 'None', 'log_format': 'json', 'static_parser': 'True', 'invocation_command': 'dbt --log-format-file=text --log-format=json --warn-error-options {"include": ["NothingToDo"]} --no-use-colors test --select source:my_first_source source:my_second_source --vars {"AIRFLOW_DAG": "my_example_dag"} --target-path /mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64 --log-path /mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64', 'warn_error_options': "WarnErrorOptions(include=['NothingToDo'], exclude=[])", 'target_path': '/mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64', 'introspect': 'True', 'send_anonymous_usage_stats': 'True'}
11:02:27 Sending event: {'category': 'dbt', 'action': 'project_id', 'label': 'ec4be0c9-09e6-46a7-934a-2b5de26cdda6', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb12c183ad0>]}
11:02:27 Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': 'ec4be0c9-09e6-46a7-934a-2b5de26cdda6', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb12c223950>]}
11:02:27 Registered adapter: bigquery=1.9.0
11:02:28 checksum: b46e93efb52f67a19bdb6e7abf8f24e437648676c7bd1d5db6ccdaa72cc84c4c, vars: {'AIRFLOW_DAG': 'my_example_dag'}, profile: , target: , version: 1.9.0
11:02:28 Unable to do partial parsing because saved manifest not found. Starting full parse.
11:02:28 Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': 'ec4be0c9-09e6-46a7-934a-2b5de26cdda6', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb127f3e150>]}
11:02:35 Encountered an error:
[Errno 116] Stale file handle
11:02:35 Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 153, in wrapper
result, success = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 103, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 235, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 264, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 311, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 327, in wrapper
setup_manifest(ctx, write=write, write_perf_info=write_perf_info)
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 351, in setup_manifest
ctx.obj["manifest"] = parse_manifest(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/manifest.py", line 2037, in parse_manifest
manifest = ManifestLoader.get_full_manifest(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/manifest.py", line 310, in get_full_manifest
manifest = loader.load()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/manifest.py", line 357, in load
file_reader.read_files()
File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 212, in read_files
self.read_files_for_project(project, file_types)
File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 219, in read_files_for_project
project_files[file_type_info["parser"]] = read_files_for_parser(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 167, in read_files_for_parser
source_files = get_source_files(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 156, in get_source_files
file = load_source_file(fp, parse_file_type, project.project_name, saved_files)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 84, in load_source_file
file_contents = load_file_contents(path.absolute_path, strip=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt_common/clients/system.py", line 178, in load_file_contents
to_return = handle.read().decode("utf-8")
^^^^^^^^^^^^^
OSError: [Errno 116] Stale file handle
11:02:35 Resource report: {"command_name": "test", "command_success": false, "command_wall_clock_time": 11.851658, "process_in_blocks": "0", "process_kernel_time": 0.763955, "process_mem_max_rss": "197116", "process_out_blocks": "32", "process_user_time": 2.880488}
11:02:35 Command `dbt test` failed at 11:02:35.484429 after 11.86 seconds
11:02:35 Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb141fd5d90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb141fd5990>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fb12c759590>]}
11:02:35 Flushing usage events
11:02:45 An error was encountered while trying to flush usage events