I’m wondering what the best way to manage jobs/pipelines is according to you. Furthermore I wonder how to prevent certain models to be ran multiple times because the are used by multiple jobs?
I might be thinking about this the wrong way, but our situation is that sources are build/loaded daily. These will have staging models that are used by more than one job/pipeline.
How would you prevent that each job will execute the model instead of just using the materialization directly? I could imagine some conditional logic based on freshness and then either use ref or source, but that seems a bit hacky.
how do you handle this?
Furthermore I wonder how you specify the jobs; do you select the end result and process everything upstream?