DBT on airflow docker container

CloudTechGirl · April 29, 2021, 6:08pm

I am trying to do a poc on airflow + dbt, I need to be able to run dbt on airflow container without using pre built airflow+dbt image.
Below are the steps i am following. is there anything i am missing, any help is appreciated.

I created a dbt project using dbt init and downloaded docker image for airflow

2 copied dbt folder inside airflow folder

In airflow’s docker-compose.yml, i’ve added dbt directory as a volume for webserver

volumes:

./dbt:./

4 In airflow’s Dockerfile, install dbt and copied dbt code
COPY dbt/ .
RUN pip install dbt && pip install SQLAlchemy==1.3.23

I am not able to run the container with this image,getting below error after I hit docker run

INFO  [alembic.runtime.migration] Running upgrade a56c9515abdc, 004c1210f153, 74effc47d867, b3b105409875 -> 08364691d074, Merge the four heads back together
INFO  [alembic.runtime.migration] Running upgrade 08364691d074 -> fe461863935f, increase_length_for_connection_password
INFO  [alembic.runtime.migration] Running upgrade fe461863935f -> 7939bcff74ba, Add DagTags table
Done.
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2021-04-29 17:55:14,319] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:55:14,332] {{scheduler_job.py:1344}} INFO - Starting the scheduler
[2021-04-29 17:55:14,333] {{scheduler_job.py:1352}} INFO - Running execute loop for -1 seconds
[2021-04-29 17:55:14,333] {{scheduler_job.py:1353}} INFO - Processing each file at most -1 times
[2021-04-29 17:55:14,333] {{scheduler_job.py:1356}} INFO - Searching for files in /usr/local/airflow/dags
[2021-04-29 17:55:14,333] {{scheduler_job.py:1358}} INFO - There are 0 files in /usr/local/airflow/dags
[2021-04-29 17:55:14,334] {{scheduler_job.py:1409}} INFO - Resetting orphaned tasks for active dag runs
[2021-04-29 17:55:14,357] {{dag_processing.py:556}} INFO - Launched DagFileProcessorManager with pid: 50
[2021-04-29 17:55:14,362] {{settings.py:54}} INFO - Configured default timezone <Timezone [UTC]>
[2021-04-29 17:55:14,374] {{dag_processing.py:758}} WARNING - Because we cannot use more than 1 thread (max_threads = 2) when using sqlite. So we set parallelism to 1.
[2021-04-29 17:55:14,422] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:55:14,422] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
[2021-04-29 17:55:15 +0000] [51] [INFO] Starting gunicorn 19.10.0
[2021-04-29 17:55:15 +0000] [51] [INFO] Listening at: http://0.0.0.0:8080 (51)
[2021-04-29 17:55:15 +0000] [51] [INFO] Using worker: sync
[2021-04-29 17:55:15 +0000] [54] [INFO] Booting worker with pid: 54
[2021-04-29 17:55:15 +0000] [55] [INFO] Booting worker with pid: 55
[2021-04-29 17:55:15 +0000] [56] [INFO] Booting worker with pid: 56
[2021-04-29 17:55:15 +0000] [57] [INFO] Booting worker with pid: 57
[2021-04-29 17:55:15,746] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:55:15,746] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:55:15,842] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:55:15,843] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:55:15,848] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:55:15,848] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:55:15,869] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:55:15,869] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:55:46 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 17:55:46 +0000] [102] [INFO] Booting worker with pid: 102
[2021-04-29 17:55:46,744] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:55:46,744] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:55:47 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 17:55:47 +0000] [54] [INFO] Worker exiting (pid: 54)
[2021-04-29 17:56:17 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 17:56:17 +0000] [114] [INFO] Booting worker with pid: 114
[2021-04-29 17:56:18,078] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:56:18,078] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:56:18 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 17:56:18 +0000] [55] [INFO] Worker exiting (pid: 55)
[2021-04-29 17:56:49 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 17:56:49 +0000] [126] [INFO] Booting worker with pid: 126
[2021-04-29 17:56:49,413] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:56:49,413] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:56:50 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 17:56:50 +0000] [56] [INFO] Worker exiting (pid: 56)
[2021-04-29 17:57:20 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 17:57:20 +0000] [138] [INFO] Booting worker with pid: 138
[2021-04-29 17:57:20,923] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:57:20,923] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:57:21 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 17:57:21 +0000] [57] [INFO] Worker exiting (pid: 57)
[2021-04-29 17:57:51 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 17:57:51 +0000] [150] [INFO] Booting worker with pid: 150
[2021-04-29 17:57:52,111] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:57:52,111] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:57:52 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 17:57:52 +0000] [102] [INFO] Worker exiting (pid: 102)
[2021-04-29 17:58:23 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 17:58:23 +0000] [162] [INFO] Booting worker with pid: 162
[2021-04-29 17:58:23,425] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:58:23,426] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:58:24 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 17:58:24 +0000] [114] [INFO] Worker exiting (pid: 114)
[2021-04-29 17:58:54 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 17:58:54 +0000] [174] [INFO] Booting worker with pid: 174
[2021-04-29 17:58:54,787] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:58:54,788] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:58:55 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 17:58:55 +0000] [126] [INFO] Worker exiting (pid: 126)
[2021-04-29 17:59:25 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 17:59:25 +0000] [186] [INFO] Booting worker with pid: 186
[2021-04-29 17:59:26,134] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:59:26,135] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:59:26 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 17:59:26 +0000] [138] [INFO] Worker exiting (pid: 138)
[2021-04-29 17:59:57 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 17:59:57 +0000] [198] [INFO] Booting worker with pid: 198
[2021-04-29 17:59:57,449] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 17:59:57,449] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 17:59:58 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 17:59:58 +0000] [150] [INFO] Worker exiting (pid: 150)
[2021-04-29 18:00:28 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 18:00:28 +0000] [210] [INFO] Booting worker with pid: 210
[2021-04-29 18:00:28,797] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 18:00:28,798] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 18:00:29 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 18:00:29 +0000] [162] [INFO] Worker exiting (pid: 162)
[2021-04-29 18:00:59 +0000] [51] [INFO] Handling signal: ttin
[2021-04-29 18:00:59 +0000] [222] [INFO] Booting worker with pid: 222
[2021-04-29 18:01:00,165] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2021-04-29 18:01:00,165] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2021-04-29 18:01:00 +0000] [51] [INFO] Handling signal: ttou
[2021-04-29 18:01:00 +0000] [174] [INFO] Worker exiting (pid: 174)

Topic		Replies	Views
How to deploy airflow together with DBT? Archive	0	3035	December 24, 2020
What is the best practice for deploying Airflow together with dbt? In-Depth Discussions airflow , orchestration-and-deployment	1	5285	December 29, 2020
Airflow Integration Archive	1	4449	May 28, 2020
Airflow on Docker: How to update git version from 2.20 to atleast 2.25 when creating a DBT package Help airflow	1	876	January 30, 2023
what is the cause of airflow deployment error on docker image Help dbt-core	0	680	December 15, 2023

DBT on airflow docker container

Related topics