Trigger dbt models after snowpipe
Hello,
I want to check the possibility of triggering dbt models after snowpipe load data into Snowflake database. I have a requirement where realtime xml data comes everytime and I’ll use snowpipe to ingest that data into snowflake. I want to know if there is a way to trigger dbt models that will load the variant field data to raw tables.
We are using airflow as orchestration to trigger the job periodically but today when 2500 files got accumulated in snowflake variant column table. dbt job parsing the xml got memory issues during execution as processing 2500 xml files in on run caused this issue.
Is there a way we can batch the number of files we process or handle the snowflake memory issue
Thanks in advance !