The problem I’m having
I want a model to run when all the data is available. The issue is we have many data sources and we can’t wait for everything to be ready to wait to run our models. We have large amounts of data and if data is available for model A but not model B we want to run model A right away. Assume that model B needs the data from model A but also other data that model A does not need.
The context of why I’m trying to do this
I want to create simple models with all the dependencies clearly expressed in dbt. I don’t want some of this logic in Airflow as this is too complicated.
Some example code or error messages
I can do this in Airflow, with Airflow using S3 sensors for the data feeds into model A and B. Airflow will then determine when to run model A and model B. The issue is this eliminates the automatic dependencies that dbt has (the refs) and splits the logic across Airflow and dbt, making it more difficult to understand the system.
Basically, all Airflow would do is run all the models with one command, but dbt would determine when the upstream data is ready. dbt would have an S3 sensor built in for this type of complex dependency management.