One CI/CD pipeline for two separate Snowflake instaces

Help
, ,
1

Hi community,

dbt is a new tool at our company and we are looking for a best possible way on how to integrate it. I really appreciate any time you spend on my topic.

The problem I’m having

My company is using two separate Snowflake instances and recently we decided to adopt dbt. We are using dbt core and we are now designing ci-cd pipeline to build our models, lint sql, regenerate docs, etc…

The context of why I’m trying to do this

Because we are using two Snowflake instances, I am looking for an approach where we won’t need two separate ci-cd pipelines, and instead have one and for example depending on the model’s source we could set the appropriate --target value.
I am not really sure whether this would work or not, can we put (for example) region parameter in sources.yaml?
Any other thoughts, suggestions, advices would be highly appreciated

Kindest regards,
Milos

2

Hi Milos,

What I understand is, you are trying to connect to different source(instance) for different environments(dev, test, etc). I hope my understanding is right here. Based on this I have given the solution below.

We are using dbt with bigquery in our project. We point the source to different Bigquery projects using the project config in source yaml file. The contents of the file is given below for your reference.

sources:

  • name: source_oursource
    project: |
    {%- if target.name == “local” -%} oursource-d-xyz1s
    {%- elif target.name == “dev” -%} oursource-d-ec24r
    {%- elif target.name == “test” -%} s oursource-t-1df4
    {%- elif target.name == “prod” -%} oursource-p-3ef4
    {%- else -%} invalid_database
    {%- endif -%}
    dataset: oursource_dataset
    tables:
    • name: source_table_name

The part where project is given is what makes it point to different bigquery projects based on “target” we set. I hope this helps!

3

Hi Bhuvaneswaran,

Thank you so much for taking the time to address my issue.

My company has two prod envs and I am more interested in the following:

If my model is supposed to run in (prod)source X, then what I want is, within my ci-cd pipeline, that the target should be CI_X, but if it’s supposed to run in source Y, then the target should be CI_Y, or if it’s supposed to run in both envs, then I want to run dbt build --select model_name(s) twice, once using CI_X as target and other one using CI_Y.

Please let me know if there is anything else I should add.