The problem I’m having
I need multiple run_results.json and manifest.json files under different directories in dbt. So, is that possible?
The context of why I’m trying to do this
I have different parallel executions of dbt commands so I need these files to be separated from each other in order to avoid data loss/overlapping.
I think we cannot have multiple json artifacts files in dbt. I got the idea from following
08:01PM - 19 Oct 21 UTC
12:53PM - 22 Oct 21 UTC
### Is there an existing feature request for this?
- [X] I have searched the
… existing issues
### Describe the Feature
Allow to define a custom path for auto-generated documentation files.
Currently, there's no way to customize the destination path for the documentation files generated by `dbt docs generate`. They all end up in the `target` folder, or in [whichever folder is configured as such](https://docs.getdbt.com/reference/project-configs/target-path).
**Note**: the configuration parameters [`docs-paths`](https://docs.getdbt.com/reference/project-configs/docs-paths) and [`asset-paths`](https://docs.getdbt.com/reference/project-configs/asset-paths) don't provide a solution for this, as far as I can tell.
### Describe alternatives you've considered
The same way there are configuration parameters for other paths (e.g. `target-path`, `docs-paths`, `asset-paths`), we can add a new one for this purpose.
Because we want to make this change backward-compatible, by default it should take the same value as `target-path`.
Potential names for the new parameter:
- **Please propose a name**
### Who will this benefit?
I'm proposing this feature due to a real scenario in which specifying a separate folder for the auto-generated documentation files would simplify a process. We intend to expose the generated files through a web service, and it's easier to perform the upload process if the folder is self-contained (i.e. doesn't contain other unrelated files).
### Are you interested in contributing this feature?
Yes. I'm familiarized with dbt but not with its code base, so I'd like some tips to get me in the right direction (e.g. "you'll probably have to change this, this, and that").
### Anything else?
My guess is that we can do this by changing mostly the [`generate` task](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/task/generate.py#L219).
@adilrehman you could set a different target-path on each run, e.g. taking including the current timestamp in your command:
dbt --target-path 1680136368 run (that’s the current unix timestamp, you could do whatever you wanted)
Alternatively, if you’re using dbt Cloud, you can use the API to access each individual step’s artifacts
using the .
This works for me!
April 6, 2023, 2:32am
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.