I am testing out connecting my dbt model with airflow.
This is my setting in airflow and i just use a simple bash operator to call for dbt run:
dag = DAG(
'dbt_id_links'
, start_date=datetime(year=2019, month=12, day=27, hour=0, minute=0)
, schedule_interval='@hourly'
, default_args=default_args
, max_active_runs=1
, catchup=False
, concurrency =1
)
I have an incremental model where I use a post-hook to vacuum the table to keep the sortkey working.
{"sql": "vacuum {{this.schema}}.{{this.name}}", "transaction": False}
Our db is in aws redshift.
Unfortunately, I keep on getting issues with the dag where it is failed because the vacuum job is hanging from the previous run. Does anyone has any insight into this?
Here is a sample of the error code
