Hi,
we are currently exploring options to split up our dbt project to separate smaller projects that can be handled by the different domain teams. (okay, we are trying to build [our understanding of] a data mesh I guess)
To get started we already separated certain business domains and integrated them into our main dbt project as a package, but lets assume we simply have connected dbt projects where models from project A are sources for project B.
There are two features we would like to look deeper into but wanted to ask for community feedback first - maybe there are already ways to accomplish this.
-
run dbt compile on single project level and for all projects.
Use case 1 would be to check for individual domain teams their project status.
Use case 2 would be to see if new introduced changes would break downstream projects. -
Have sources with version
APIs in engineering system often evolve by simply adding a new endpoint version and consumer teams would be up to decide when to switch. By versioning sources you could accomplish the same (if you are temporarily fine with the additional storage and compute)
Are you aware of any work in this direction? Is this something more people would be interested in (and if not, please tell us why 
). Happy for any thought around this.
Stephan