Hi,
I’m using dbt on an airflow setup to to create some tables (using models). I have a combination of airflow operators and python code behind that will generate around 10 airflow tasks, each is a bashoperator that execute a dbt run command and each command is about a specific model and each command has it own variables. All the tasks are running concurrently but they sometimes fail randomly with an error that seems to indicates that the partial_parse.msgpack file is unavailable.
The models i have, uses the same variables (key) but they can have different values from a model to another.
I’m wondering if this kind of dynamic workflow is a good fit for dbt and what conditions can cause the issue.
I must mention that i cannot reproduce the issue in local, it only occurs in production ( airflow instance deployed with google cloud platform (GCP)
[2023-01-26T08:20:27.267+0000] {subprocess.py:85} INFO - Output:
[2023-01-26T08:20:35.405+0000] {subprocess.py:92} INFO - [0m08:20:35 Running with dbt=1.3.1
[2023-01-26T08:20:36.852+0000] {subprocess.py:92} INFO - [0m08:20:36 Unable to do partial parsing because config vars, config profile, or config target have changed
[2023-01-26T08:21:15.901+0000] {subprocess.py:92} INFO - [0m08:21:15 Encountered an error:
[2023-01-26T08:21:15.902+0000] {subprocess.py:92} INFO - [Errno 5] Input/output error: 'target/partial_parse.msgpack'
[2023-01-26T08:21:15.903+0000] {subprocess.py:92} INFO - [0m08:21:15 Traceback (most recent call last):
[2023-01-26T08:21:15.904+0000] {subprocess.py:92} INFO - File "/opt/python3.8/lib/python3.8/site-packages/dbt/main.py", line 135, in main
[2023-01-26T08:21:15.904+0000] {subprocess.py:92} INFO - results, succeeded = handle_and_check(args)
[2023-01-26T08:21:15.905+0000] {subprocess.py:92} INFO - File "/opt/python3.8/lib/python3.8/site-packages/dbt/main.py", line 198, in handle_and_check
[2023-01-26T08:21:15.906+0000] {subprocess.py:92} INFO - task, res = run_from_args(parsed)
[2023-01-26T08:21:15.906+0000] {subprocess.py:92} INFO - File "/opt/python3.8/lib/python3.8/site-packages/dbt/main.py", line 245, in run_from_args
[2023-01-26T08:21:15.907+0000] {subprocess.py:92} INFO - results = task.run()
[2023-01-26T08:21:15.907+0000] {subprocess.py:92} INFO - File "/opt/python3.8/lib/python3.8/site-packages/dbt/task/runnable.py", line 453, in run
[2023-01-26T08:21:15.908+0000] {subprocess.py:92} INFO - self._runtime_initialize()
[2023-01-26T08:21:15.908+0000] {subprocess.py:92} INFO - File "/opt/python3.8/lib/python3.8/site-packages/dbt/task/runnable.py", line 161, in _runtime_initialize
[2023-01-26T08:21:15.909+0000] {subprocess.py:92} INFO - super()._runtime_initialize()
[2023-01-26T08:21:15.909+0000] {subprocess.py:92} INFO - File "/opt/python3.8/lib/python3.8/site-packages/dbt/task/runnable.py", line 94, in _runtime_initialize
[2023-01-26T08:21:15.910+0000] {subprocess.py:92} INFO - self.load_manifest()
[2023-01-26T08:21:15.910+0000] {subprocess.py:92} INFO - File "/opt/python3.8/lib/python3.8/site-packages/dbt/task/runnable.py", line 81, in load_manifest
[2023-01-26T08:21:15.911+0000] {subprocess.py:92} INFO - self.manifest = ManifestLoader.get_full_manifest(self.config)
[2023-01-26T08:21:15.917+0000] {subprocess.py:92} INFO - File "/opt/python3.8/lib/python3.8/site-packages/dbt/parser/manifest.py", line 221, in get_full_manifest
[2023-01-26T08:21:15.918+0000] {subprocess.py:92} INFO - manifest = loader.load()
[2023-01-26T08:21:15.919+0000] {subprocess.py:92} INFO - File "/opt/python3.8/lib/python3.8/site-packages/dbt/parser/manifest.py", line 410, in load
[2023-01-26T08:21:15.919+0000] {subprocess.py:92} INFO - self.write_manifest_for_partial_parse()
[2023-01-26T08:21:15.920+0000] {subprocess.py:92} INFO - File "/opt/python3.8/lib/python3.8/site-packages/dbt/parser/manifest.py", line 559, in write_manifest_for_partial_parse
[2023-01-26T08:21:15.920+0000] {subprocess.py:92} INFO - with open(path, "wb") as fp:
[2023-01-26T08:21:15.921+0000] {subprocess.py:92} INFO - OSError: [Errno 5] Input/output error: 'target/partial_parse.msgpack'
[2023-01-26T08:21:15.921+0000] {subprocess.py:92} INFO -
[2023-01-26T08:21:16.570+0000] {subprocess.py:96} INFO - Command exited with return code 2
[2023-01-26T08:21:16.725+0000] {taskinstance.py:1904} ERROR - Task failed with exception