I am trying to do a poc on airflow + dbt, I need to be able to run dbt on airflow container without using pre built airflow+dbt image.
Below are the steps i am following. is there anything i am missing, any help is appreciated.
- I created a dbt project using dbt init and downloaded docker image for airflow
2 copied dbt folder inside airflow folder
- In airflow’s docker-compose.yml, i’ve added dbt directory as a volume for webserver
volumes:
- ./dbt:./
4 In airflow’s Dockerfile, install dbt and copied dbt code
COPY dbt/ .
RUN pip install dbt && pip install SQLAlchemy==1.3.23
I am not able to run the container with this image,getting below error after I hit docker run
INFO [alembic.runtime.migration] Running upgrade a56c9515abdc, 004c1210f153, 74effc47d867, b3b105409875 -> 08364691d074, Merge the four heads back together INFO [alembic.runtime.migration] Running upgrade 08364691d074 -> fe461863935f, increase_length_for_connection_password INFO [alembic.runtime.migration] Running upgrade fe461863935f -> 7939bcff74ba, Add DagTags table Done. ____________ _____________ ____ |__( )_________ __/__ /________ __ ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / / ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ ____________ _____________ ____ |__( )_________ __/__ /________ __ ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / / ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ [2021-04-29 17:55:14,319] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:55:14,332] {{scheduler_job.py:1344}} INFO - Starting the scheduler [2021-04-29 17:55:14,333] {{scheduler_job.py:1352}} INFO - Running execute loop for -1 seconds [2021-04-29 17:55:14,333] {{scheduler_job.py:1353}} INFO - Processing each file at most -1 times [2021-04-29 17:55:14,333] {{scheduler_job.py:1356}} INFO - Searching for files in /usr/local/airflow/dags [2021-04-29 17:55:14,333] {{scheduler_job.py:1358}} INFO - There are 0 files in /usr/local/airflow/dags [2021-04-29 17:55:14,334] {{scheduler_job.py:1409}} INFO - Resetting orphaned tasks for active dag runs [2021-04-29 17:55:14,357] {{dag_processing.py:556}} INFO - Launched DagFileProcessorManager with pid: 50 [2021-04-29 17:55:14,362] {{settings.py:54}} INFO - Configured default timezone <Timezone [UTC]> [2021-04-29 17:55:14,374] {{dag_processing.py:758}} WARNING - Because we cannot use more than 1 thread (max_threads = 2) when using sqlite. So we set parallelism to 1. [2021-04-29 17:55:14,422] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:55:14,422] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags Running the Gunicorn Server with: Workers: 4 sync Host: 0.0.0.0:8080 Timeout: 120 Logfiles: - - ================================================================= [2021-04-29 17:55:15 +0000] [51] [INFO] Starting gunicorn 19.10.0 [2021-04-29 17:55:15 +0000] [51] [INFO] Listening at: http://0.0.0.0:8080 (51) [2021-04-29 17:55:15 +0000] [51] [INFO] Using worker: sync [2021-04-29 17:55:15 +0000] [54] [INFO] Booting worker with pid: 54 [2021-04-29 17:55:15 +0000] [55] [INFO] Booting worker with pid: 55 [2021-04-29 17:55:15 +0000] [56] [INFO] Booting worker with pid: 56 [2021-04-29 17:55:15 +0000] [57] [INFO] Booting worker with pid: 57 [2021-04-29 17:55:15,746] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:55:15,746] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:55:15,842] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:55:15,843] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:55:15,848] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:55:15,848] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:55:15,869] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:55:15,869] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:55:46 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 17:55:46 +0000] [102] [INFO] Booting worker with pid: 102 [2021-04-29 17:55:46,744] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:55:46,744] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:55:47 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 17:55:47 +0000] [54] [INFO] Worker exiting (pid: 54) [2021-04-29 17:56:17 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 17:56:17 +0000] [114] [INFO] Booting worker with pid: 114 [2021-04-29 17:56:18,078] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:56:18,078] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:56:18 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 17:56:18 +0000] [55] [INFO] Worker exiting (pid: 55) [2021-04-29 17:56:49 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 17:56:49 +0000] [126] [INFO] Booting worker with pid: 126 [2021-04-29 17:56:49,413] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:56:49,413] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:56:50 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 17:56:50 +0000] [56] [INFO] Worker exiting (pid: 56) [2021-04-29 17:57:20 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 17:57:20 +0000] [138] [INFO] Booting worker with pid: 138 [2021-04-29 17:57:20,923] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:57:20,923] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:57:21 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 17:57:21 +0000] [57] [INFO] Worker exiting (pid: 57) [2021-04-29 17:57:51 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 17:57:51 +0000] [150] [INFO] Booting worker with pid: 150 [2021-04-29 17:57:52,111] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:57:52,111] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:57:52 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 17:57:52 +0000] [102] [INFO] Worker exiting (pid: 102) [2021-04-29 17:58:23 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 17:58:23 +0000] [162] [INFO] Booting worker with pid: 162 [2021-04-29 17:58:23,425] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:58:23,426] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:58:24 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 17:58:24 +0000] [114] [INFO] Worker exiting (pid: 114) [2021-04-29 17:58:54 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 17:58:54 +0000] [174] [INFO] Booting worker with pid: 174 [2021-04-29 17:58:54,787] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:58:54,788] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:58:55 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 17:58:55 +0000] [126] [INFO] Worker exiting (pid: 126) [2021-04-29 17:59:25 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 17:59:25 +0000] [186] [INFO] Booting worker with pid: 186 [2021-04-29 17:59:26,134] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:59:26,135] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:59:26 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 17:59:26 +0000] [138] [INFO] Worker exiting (pid: 138) [2021-04-29 17:59:57 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 17:59:57 +0000] [198] [INFO] Booting worker with pid: 198 [2021-04-29 17:59:57,449] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 17:59:57,449] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 17:59:58 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 17:59:58 +0000] [150] [INFO] Worker exiting (pid: 150) [2021-04-29 18:00:28 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 18:00:28 +0000] [210] [INFO] Booting worker with pid: 210 [2021-04-29 18:00:28,797] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 18:00:28,798] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 18:00:29 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 18:00:29 +0000] [162] [INFO] Worker exiting (pid: 162) [2021-04-29 18:00:59 +0000] [51] [INFO] Handling signal: ttin [2021-04-29 18:00:59 +0000] [222] [INFO] Booting worker with pid: 222 [2021-04-29 18:01:00,165] {{__init__.py:51}} INFO - Using executor SequentialExecutor [2021-04-29 18:01:00,165] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags [2021-04-29 18:01:00 +0000] [51] [INFO] Handling signal: ttou [2021-04-29 18:01:00 +0000] [174] [INFO] Worker exiting (pid: 174)