The problem I’m having
I am running dags in airflow with dbt core 0.21.1 but its erroring out
The context of why I’m trying to do this
I have dbt code and we use airflow to schedule in aws
What I’ve already tried
I am trying to understand the error. Can someone please help me on this
Some example code or error messages
[2024-08-29, 02:52:31 UTC] {{taskinstance.py:1035}} INFO - Dependencies all met for <TaskInstance: MH_OB_dbt_0_21_1.bash_command manual__2024-08-29T02:52:29.719710+00:00 [queued]>
[2024-08-29, 02:52:31 UTC] {{taskinstance.py:1035}} INFO - Dependencies all met for <TaskInstance: MH_OB_dbt_0_21_1.bash_command manual__2024-08-29T02:52:29.719710+00:00 [queued]>
[2024-08-29, 02:52:31 UTC] {{taskinstance.py:1241}} INFO -
--------------------------------------------------------------------------------
[2024-08-29, 02:52:31 UTC] {{taskinstance.py:1242}} INFO - Starting attempt 1 of 1
[2024-08-29, 02:52:31 UTC] {{taskinstance.py:1243}} INFO -
--------------------------------------------------------------------------------
[2024-08-29, 02:52:31 UTC] {{taskinstance.py:1262}} INFO - Executing <Task(BashOperator): bash_command> on 2024-08-29 02:52:29.719710+00:00
[2024-08-29, 02:52:31 UTC] {{standard_task_runner.py:52}} INFO - Started process 395 to run task
[2024-08-29, 02:52:31 UTC] {{standard_task_runner.py:76}} INFO - Running: ['airflow', 'tasks', 'run', 'MH_OB_dbt_0_21_1', 'bash_command', 'manual__2024-08-29T02:52:29.719710+00:00', '--job-id', '1013', '--raw', '--subdir', 'DAGS_FOLDER/MH_OB_dbt_0_21_1.py', '--cfg-path', '/tmp/tmpbw5bh2ar', '--error-file', '/tmp/tmpr9c101ni']
[2024-08-29, 02:52:31 UTC] {{standard_task_runner.py:77}} INFO - Job 1013: Subtask bash_command
[2024-08-29, 02:52:32 UTC] {{logging_mixin.py:109}} INFO - Running <TaskInstance: MH_OB_dbt_0_21_1.bash_command manual__2024-08-29T02:52:29.719710+00:00 [running]> on host ip-10-133-216-202.ec2.internal
[2024-08-29, 02:52:32 UTC] {{taskinstance.py:1429}} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=MH_OB_dbt_0_21_1
AIRFLOW_CTX_TASK_ID=bash_command
AIRFLOW_CTX_EXECUTION_DATE=2024-08-29T02:52:29.719710+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2024-08-29T02:52:29.719710+00:00
[2024-08-29, 02:52:32 UTC] {{subprocess.py:62}} INFO - Tmp dir root location:
/tmp
[2024-08-29, 02:52:32 UTC] {{subprocess.py:74}} INFO - Running command: ['bash', '-c', 'cp -R /usr/local/airflow/dags/MH_OB_dbt_0_21_1 /tmp; cd /tmp/MH_OB_dbt_0_21_1; /usr/local/airflow/.local/bin/dbt run --project-dir /tmp/MH_OB_dbt_0_21_1/ --profiles-dir . ']
[2024-08-29, 02:52:32 UTC] {{subprocess.py:85}} INFO - Output:
[2024-08-29, 02:52:34 UTC] {{subprocess.py:89}} INFO - Running with dbt=0.21.1
[2024-08-29, 02:52:34 UTC] {{subprocess.py:89}} INFO - Encountered an error:
[2024-08-29, 02:52:34 UTC] {{subprocess.py:89}} INFO - argument of type 'NoneType' is not iterable
[2024-08-29, 02:52:34 UTC] {{subprocess.py:93}} INFO - Command exited with return code 2
[2024-08-29, 02:52:34 UTC] {{taskinstance.py:1703}} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
result = execute_callable(context=context)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/operators/bash.py", line 188, in execute
f'Bash command failed. The command returned a non-zero exit code {result.exit_code}.'