Dbt dev/prod data pipelines

In-Depth Discussions
, , ,
1

Hello,

First time adopter of dbt in the last couple months. I had a question around best practices in handling dbt runs for dev and prod. I’d like to:

  1. Have a daily job that runs dbt run to target a dev schema
  2. Run dbt test to make sure all the unit tests pass
  3. Run dbt run --target prod

A couple of questions:

  • Is this order of operations typical?
  • We’re using big query. Is it possible to have the the dev target in one GCP “dev” project and the prod target in a GCP “prod” project?
2

Hello there,

You can definitely target two different GCP projects.

Ex:

# profiles.yml
templates:
  bigquery: &bigquery
    # run the following command to authenticate:
    # $> gcloud auth application-default login --scopes=https://www.googleapis.com/auth/bigquery
    dbname: analytics
    type: bigquery
    method: oauth
    threads: 10
    timeout_seconds: 300
    priority: interactive
    retries: 1

default:
  outputs:
    dev:
      <<: *bigquery
      project: acme-dev
    prod:
      <<: *bigquery
      project: acme-prod
  target: dev
2 Likes
3

@pcreux I haven’t seen this <<: syntax before - does this let you take all of the contents of an anchor and then append extra keys? That’s very exciting because it has always annoyed me that I couldn’t make that work!

4

@joellabes Yes! You’re totally right!

1 Like
5

Watch this then :grinning: YAML Basics: Anchors and Aliases - YouTube

1 Like
6

Super interesting. How does that translate to dbt cloud? @pcreux

I currently want to run my production jobs to another gcp project called ‘production’ but when I insert production in the target dbt jobs simply don’t work … any clue on how to approach this?

7

@emiliobiz98 I’m not familiar with dbt cloud, unfortunately.

8

No worries, I appreciate the fast answer. Any chance @joellabes? :eyes:

9

You can use Extended Attributes to add things that are configurable in profiles.yml but which dbt Cloud doesn’t expose directly.

They’re not compatible with yaml anchors etc as it’s just a dumb textbox, but it should work for your use case