Hi, I’m a developer working on a small data warehouse built using dbt / BigQuery. I’m planning to upgrade to dbt 1.5, as those features look really amazing. One of the most useful features for us is Model Access. I would like to set many of our models as private and I already created a group and assigned the models into it. I also added tests and snapshots to the same group. However, these models are also referenced by an exposure. When I attempt to add the exposure to the group (either through dbt_project.yml or in the exposure’s config), it either doesn’t do anything or it fails on compile.
Please, is it somehow possible to add the exposure into the group? Or can the exposure only reference public models? The 1.5 documentation doesn’t mention exposures at all, the only thing I was able to find was an idea which presumes exposures won’t be able to see private models in any way. Is that still the case?Thanks a lot in advance!
1 Like
I’m glad model access is exciting for you!
Can you expand on what you want to achieve by having the exposures in a group? Is it solely for categorisation (i.e. you’re using groups to create collections instead of putting them in a common folder)? Exposures aren’t ref-able, so the access restrictions don’t really apply to them.
Hi, thanks for the response. The reason is a bit complex, which is why I didn’t go into it in the issue description.
Our project evolved over time, so that now it consists of a few parts which are essentially independent: financial data, product data, commercial data and then the main data (we should have a better name for it, but it’s still called the main data for now). The process now is that financial data, product data and commercial data are built independently into their own datasets and then the main data build uses these datasets as source (rather than ref) for the build.
Originally we were building the whole data warehouse in one go, but over time we realized we needed to separate away a few logical parts. I do realize that these separate parts should correctly be their own separate projects, but such a big separation is not really possible right now. In order to keep these parts independent, we of course need to refrain from any ref between them.
While that sounds easy, every once in a while it happens that somebody forgets this and creates a ref that e.g. introduces dependency of financial data on product data (or worse yet: on the main data, leading to a cyclic dependency). Also, our team members sometimes struggle with the above principles and put in wrong refs. While I catch most of these in pull requests, we’ve already multiple times encountered issues coming from such broken refs.
Model access provides us an amazing solution to these issues, as we’re automatically testing every pull request, so such issues would be immediately obvious and the tests wouldn’t even compile.
Unfortunately we’re also using exposures and from your comment I understand that it’s not possible to use exposures when there are access restrictions in place? Since we’re only using the exposures for documentation, I’m considering to drop them and use the model access instead. Thanks!
Ohhhh I see - you’re trying to create an exposure, but the ref won’t resolve?
#something.yml
groups:
- name: finance
owner:
email: finance@jaffleshop.com
models:
- finance_public_model
access: public
config:
group: finance
- name: finance_private_model
access: private
config:
group: finance
exposures:
- name: some_exposure
depends_on:
- ref('finance_public_model')
- ref('finance_private_model') #doesn't work!
If so, this is probably an oversight and it’d be worth opening an issue for one of the Core maintainers to check out! I guess it’s possible that we think private models shouldn’t be being accessed from exposures, but IMO exposures should be descriptive of what is happening even if that’s not “perfect”