Single Documentation page for all projects

Archive
1

Hi All,

How do we generate / push multiple projects Documentation to a single Documentation place. Did any worked on this approach similar to some thing build by gitlab - GitLab dbt Docs

2

Check out the documentation on dbt Docs: https://docs.getdbt.com/docs/documentation-website

3

Hi Drew,

The documentation shows about the Single Project, but if we need to merge multiple projects, I donā€™t see any details. Can you please guide.

4

As long as all the projects are imported into a master project (through packages), the documentation will automatically contain all models.

gitlab_snowflake is their main project, and the other 3 (snowflake_spend, logging, and snowplow) are actually packages.

You cannot combine the docs from multiple, independent projects at this time (well, not without working under the hood to merge relevant data from the manifest.json files anyway).

5

Hi Martin,

Thanks for taking time on reply, I am not still clear on the action, I have 2 Projects Project 1 & Project 2, which has their own DBT_Project.yml. are you suggesting to combine the 2 Projects under one DBT_Project.yml ? If you have any documentation or details can you please help me.

6

@sudhirnune dbt has a native package manager that allows you to import one project into another as a dependency. You donā€™t need to combine the code into a single project, rather, simply import one package as a reference into another. More info here.

7

Weā€™re using multiple dbt projects for various similar data pipelines.

You donā€™t need to combine the code into a single project, rather, simply import one package as a reference into another.

Add namespacing for dbt resources Ā· Issue #1269 Ā· fishtown-analytics/dbt is blocking us from having a documenation meta project that imports all other projects as packages, because of the similarity a lot of the models share the same name and might also rely on the same models from a shared package.

We do have promising early results by recursively merging manifest.json, catalog.json, and run_results.json and then just serving that with one of the identical index.html files.
The graph is a bit messed up (doesnā€™t seem to distinguish packages), but all of the rest seems to work fine. The best thing is seeing which project models reference a specific model from the shared package.

Using jq to merge JSON files recursively in the command-line.
deepmerge=ā€˜def deepmergeā€¦ā€™
mkdir merged
rsync -a */target/compiled merged/
for file in {manifest,catalog,run_results}.json; do
jq --slurp ā€œ$deepmergeā€ */target/$file > merged/$file
done
rsync */target/index.html merged/

2 Likes