DAG with two DbtTaskGroup not importing

Help
, ,
1

DAG with astronomer cosmos is not loading from Dagbag with two DbtTaskGroup

In logs:
WARNING - Killing DAGFileProcessorProcess

image
image1280×159 120 KB

With one DbtTaskGroup - dag loaded good
start_task >> dbtTaskGroup1 >> end_task

With two DbtTaskGroup’s - not loaded without any errors in logs of scheduler or dag log file
start_task >> DbtTaskGroup1 >> DbtTaskGroup2 >> end_task

And some magic - is working on local Docker on workstation, but not on server (with ~same config)

Airflow - “apache/airflow:2.7.3”
16 Gb RAM
Docker Container in Portainer with Airflow and dbt with Astronomer Cosmos
AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT = 600
Python 3.8.18
astronomer -cosmos 1.2.4

dbtTaskGroup1 and dbtTaskGroup2 generated by tags - 11 and 30 tasks

After inserting DAG with code containing two tasks:

  • i see new dag Code in web-gui in Airflow, but not Grid and Details
  • in 1-2 minutes i see warning “DAG seems to be missing from DagBag.” (red message in Airflow DAGs page)
  • and later DAG disappears from list
    start_task = EmptyOperator(task_id="pre_load")

    dbtTaskGroup1 = DbtTaskGroup(
        group_id='load_source1',
        project_config=DBT_PROJECT_CONFIG,
        profile_config=DBT_CONFIG,
        render_config=RenderConfig(
            load_method=LoadMode.DBT_LS,
            select=['tag:dwh_staging'] 
        )
    )

    dbtTaskGroup2  = DbtTaskGroup(
        group_id='load_source2',
        project_config=DBT_PROJECT_CONFIG,
        profile_config=DBT_CONFIG,
        render_config=RenderConfig(
            load_method=LoadMode.DBT_LS,
            select=["tag:report_432"]
        )
    )
    
    end_task = EmptyOperator(task_id="post_load")
2

Seems, fixed with:

dag_discovery_safe_mode = False
(was True)

dag_file_processor_timeout = 100
(was 50)

3

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.