How do we build/load a package's model?

Help
1

Hi Folks

I’ve developed a package for accessing shared tables that are used across other dbt projects. The issue here is that although I can {{ ref() }} the package/table to get the data, what I really need to do is trigger the re-load of that ref’d table when the dependent model is run so that the data is up-to-date.

What I don’t want is to run the entire package’s set of models every time, only the specific model that is being referred to

For example, my package contains a dim_customer model and in my parent project (the one using the package) I want to have the customer table refreshed - not just referred to - when I specify {{ ref( ‘dim_customer’) }}

Thanks in advance for any insights

  • russell
2

Looks like the use of tags and/or selectors may be the answer here… still digging but thought I’d share the thought.

3

@rlmagidson
when u r referring to dim_customer in your current_model that means the current_model depends on dim_customer. If you place + at the front of the model selector then it will select all parents of the selected model

dbt run --select +my_model

4

Hi Surya - you are correct, but that won’t run the package’s dependencies, only the current project’s dependencies. The package’s model will be referenced but not rebuilt.

5

did u try running the command

6

Here’s what I’ve come up with as a solution (and it works)… remember that my requirement is to rebuild the necessary models (but not all) within the package along with the ones in the referencing project.

The dbt project that’s used as a package (the package-project) needs to have its models uniquely named so as not to introduce conflicts that will result in an error. There should also be sufficient tags to specify the models to be ru.

The referencing project uses a selectors.yml file that specifies the union of tags to process. Some of the tags will be from the referencing project, others from the package-project. For example, my selectors.yml looks something like this:
selectors:

  - name: timeentry-customer-run
    description: runs the tags for dim_customer and timeentry
    definition: 
      union: 
        - 'tag:dim_customer'
        - 'tag:timeentry'

A selectors.yml file can get very complex but my needs are simple for now and the file reflects that simplicity.

The dim_customer tag exists in the package-project while the timeentry tag is in the referencing project. Info on selectors use is at dbt yaml selectors as well as Stack Overflow.

Once the yml file and tags are set, dbt ls can be used to see what will be included, eg: dbt ls --selector timeentry-customer-run

Of course, setting up a Git project as a dbt package is another topic in itself.

Hope this helps someone and gives a starting point for your work.
- rlm