Integrating dbt Projects with Github in Snowflake Workspaces

[nagendrababu.l]

A Step-by-Step Guide:

Integrating dbt (data build tool) directly into Snowflake Workspaces allows data engineers to build, version-control, and execute data models without ever leaving the Snowflake interface. By leveraging native Git integration, you can ensure your transformations are modular, tested, and fully governed.

Follow this guide to set up your repository, secure your credentials, and initialize your first dbt project in Snowflake.

Phase 1: GitHub Configuration

Before connecting to Snowflake, you must prepare your repository and authentication credentials.

1.1 Repository Setup

1. Log in to your GitHub account.

2. Create a new Repository or select an existing one.

3. Important: Ensure the repository has at least one branch other than main (e.g., a dev or feature branch) to facilitate dbt workflows.

1.2 Generate Personal Access Token (PAT)

Snowflake uses a PAT for Basic Authentication to communicate with GitHub.

1. Navigate to GitHub Profile > Settings > Developer Settings > Personal Access Tokens > Tokens (Classic).

Press enter or click to view image in full size

2. Select Generate New Token (Classic).

Press enter or click to view image in full size

3. Select the necessary scopes (typically Repo access) and verify your identity via email if prompted.

4. Copy and save the token immediately; it will not be displayed again.

Phase 2: Snowflake Security & Integration

With your GitHub PAT ready, use the following SQL commands in Snowflake to establish a secure connection.

Press enter or click to view image in full size

2.1 Create a Secret

Secrets securely store your GitHub credentials so they are not exposed in plain text within your code.

SQL

CREATE OR REPLACE SECRET database.schema.dbt_git_secret
  TYPE = PASSWORD
  USERNAME = 'Your_GitHub_Username' 
  PASSWORD = 'ghp_Your_GitHub_PAT'      
  COMMENT = 'Secret for Git Integration';

2.2 Create API Integration

This object acts as a secure bridge, allowing Snowflake to communicate with the GitHub API.

SQL

CREATE OR REPLACE API INTEGRATION git_api_integration
  API_PROVIDER = git_https_api
  API_ALLOWED_PREFIXES = ('https://github.com/Your_Github_Username')
  ALLOWED_AUTHENTICATION_SECRETS = (database.schema.dbt_git_secret)
  ENABLED = TRUE;

Phase 3: Creating the Workspace

Now, connect your repository to the Snowflake UI to manage your project.

• In Snowflake, navigate to Projects > Workspaces.

Press enter or click to view image in full size

• Click the Create button and select From Git Repository.

Press enter or click to view image in full size

• Fill in the configuration details:

• Repository URL: Your GitHub repository URL.

• Workspace Name: A descriptive name for your workspace.

• API Integration: Select the git_api_integration created earlier.

• Under Authentication, select Personal Access Token.

• In the Credentials Secret section, browse to the Database and Schema where you stored your secret and select it.

• Click Create.

Press enter or click to view image in full size

Phase 4: Initializing the dbt Project

Once your workspace is active, you can initialize the dbt framework.

• Open your newly created workspace and click Add new > dbt Project.

• Configure your project settings:

• Project Name: Enter a unique name for your project.

• Role and Warehouse: Select the compute resources for running transformations.

• Database and Schema: Define where dbt will materialize your views and tables.

• Click Create.

Press enter or click to view image in full size

Your dbt project is now linked to your Git repository, enabling version-controlled data modeling directly within the Snowflake interface.

Phase 5: Handling Dependencies and Running Models

To successfully run your dbt models, you must ensure all external dependencies (packages) are properly installed. Since Snowflake Workspaces run in a secure environment, you must explicitly grant permission for dbt to reach external hubs like GitHub or the dbt Package Hub.

5.1 Configure External Access in Snowflake

Snowflake requires an External Access Integration supported by a Network Rule to download dbt packages. Run the following commands in your Snowflake worksheet:

SQL

-- Step A: Create a NETWORK RULE to allow outbound traffic 
CREATE OR REPLACE NETWORK RULE my_dbt_network_rule
  MODE = EGRESS
  TYPE = HOST_PORT
  -- Minimal URL allowlist required for dbt deps 
  VALUE_LIST = (
    'hub.getdbt.com',
    'codeload.github.com'
  );

-- Step B: Create an EXTERNAL ACCESS INTEGRATION using the rule 
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION my_dbt_ext_access
  ALLOWED_NETWORK_RULES = (my_dbt_network_rule)
  ENABLED = TRUE;

5.2 Define Your Packages

In your dbt project, locate or create a packages.yml file and add the required dependencies:

YAML

packages:
  - package: dbt-labs/dbt_utils
    version: 1.1.1
  - package: godatadriven/dbt_date
    version: 0.10.0
  - package: Snowflake-Labs/dbt_semantic_view
    version: 1.0.3

References:- Snowflake documentation

5.3 Install Dependencies (dbt deps)

Once the network rules and yaml files are ready, use the Snowflake Workspaces UI to install them:

• Open your dbt project in the Snowflake Workspace.

• Select the Operation dropdown and choose deps.

• In the External Access Integration field, select the integration you created (e.g., MY_DBT_EXT_ACCESS).

• (Optional )Add additional flags such as --upgrade .

• Execute the command to install the packages into your workspace project.

Press enter or click to view image in full size

• After a successful run, the package-lock.yml file is automatically added to your project, ensuring your team uses the exact same package versions.

Press enter or click to view image in full size

Pro Tip: You cannot update a deployed dbt project object directly with new dependencies; you must add a new version to the object to incorporate changes.