dbt & dag model structure

Help
,
1

Intro

Hi, new to this forum, senior data analyst here, asking some help regarding dbt & dag structures.

The problem

Early days of dbt at my workplace, we are trying to find better ways to make it more robust. Looking at options , ideas and best practises in utilisation of dbt & dag’s.

The context of why I’m trying to do this

We have this setting (let’s assume we have two domains: marketing and finance). Each project(domain) has its own dag, with each project has staging, intermediate and marts models.
Sometimes a cross-domain model (via {{source}} instead of {{ref}} ) is needed, but that is not good:

  1. running dbt build --models etc, no tests can be done out of domain
  2. cannot see all up/downstream models using lineage graph.

Final thoughts

The problem I guess here is with dags (more than dbt). For analogy with Jaffle Shop, they have both domains within one project, but what if there are more domains, (such as clients, experts, some_specific_to_business), than having ONE dag for all seem not right, no?

Thanks

2

Hi, the purpose is to test that you have unique values in that column? There is a built in test in dbt which is called unique. You need to create a .yml folder where you have those models, like schema.yml for instance, and write the code like this:

For more info: https://docs.getdbt.com/docs/build/data-tests

Note: @sergio.salvador originally posted this reply in Slack. It might not have transferred perfectly.