We’re using multiple dbt projects for various similar data pipelines.
You don’t need to combine the code into a single project, rather, simply import one package as a reference into another.
Add namespacing for dbt resources · Issue #1269 · fishtown-analytics/dbt is blocking us from having a documenation meta project that imports all other projects as packages, because of the similarity a lot of the models share the same name and might also rely on the same models from a shared package.
We do have promising early results by recursively merging manifest.json, catalog.json, and run_results.json and then just serving that with one of the identical index.html files.
The graph is a bit messed up (doesn’t seem to distinguish packages), but all of the rest seems to work fine. The best thing is seeing which project models reference a specific model from the shared package.
Using jq to merge JSON files recursively in the command-line.
deepmerge=‘def deepmerge…’
mkdir merged
rsync -a */target/compiled merged/
for file in {manifest,catalog,run_results}.json; do
jq --slurp “$deepmerge” */target/$file > merged/$file
done
rsync */target/index.html merged/