I am attempting to use the dbt cloud REST API to perform the following steps:
- Create a new environment
- Configure that environment to use a unique Google Service Account that has specific access to a BigQuery dataset
- Create a job in the environment that runs using the Google Service Account
Depsite several approaches, I cannot successfully configure the environment’s deployment credentials when using only API calls.
I can get an environment’s deployment credentials working as expected if I use the dbt cloud UI.
Approach
My current steps are:
- Create an Account Connection
https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Create%20Account%20Connection
I create an account connection and provide the service account details.
// ...
const requestBody = {
account_id: dbtAccountId,
name: datasetId,
adapter_version:"bigquery_v1",
config: {
deployment_env_auth_type:"service-account-json",
dataset:datasetId,
...serviceAccount
}
};
const url = `${dbtApiUrl}/api/v3/accounts/${dbtAccountId}/connections/`;
const response = await fetch(url, {
method: "POST",
headers: {
"Authorization": `Bearer ${authToken}`,
"Content-Type": "application/json",
},
body: JSON.stringify(requestBody),
});
// ...
- Create Credentials
https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Create%20Credentials
I create credentials that will be used as the environment’s deployment credentials.
// ...
const requestBody = {
account_id: dbtAccountId,
project_id: dbtProjectId,
type:"bigquery",
schema:datasetId,
threads:4,
state:1
};
const url = `${dbtApiUrl}/api/v3/accounts/${dbtAccountId}/projects/${dbtProjectId}/credentials/`;
const response = await fetch(url, {
method: "POST",
headers: {
"Authorization": `Bearer ${authToken}`,
"Content-Type": "application/json",
},
body: JSON.stringify(requestBody),
});
// ...
- Create Environment
https://docs.getdbt.com/dbt-cloud/api-v3#/operations/Create%20Environment
I then create the environment with the previously created connection and credentials.
// ...
const requestBody = {
account_id: dbtAccountId,
project_id: dbtProjectId,
connection_id: dbtConnectionId,
credentials_id: dbtCredentialsId,
name: datasetId,
type: "deployment",
use_custom_branch:false,
supports_docs:true,
deployment_type:"production",
state:1
};
const url = `${dbtApiUrl}/api/v3/accounts/${dbtAccountId}/projects/${dbtProjectId}/environments`;
const response = await fetch(url, {
method: "POST",
headers: {
"Authorization": `Bearer ${authToken}`,
"Content-Type": "application/json",
},
body: JSON.stringify(requestBody),
});
// ...
- Create a job in the new environment
https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job
// ...
const requestBody = {
account_id: dbtAccountId,
project_id: dbtProjectId,
environment_id: dbtEnvironmentId,
name: `Test Job`,
description: `A test job`,
execute_steps:[
`dbt run --models test`
],
execution: {
timeout_seconds:0
},
generate_docs:false,
job_type: "scheduled",
lifecycle_webhooks:false,
run_compare_changes:false,
run_generate_sources:false,
run_lint:false,
errors_on_lint_failure:false,
settings: {
threads:4,
target_name:""
},
state:1,
triggers: {
schedule:true
},
schedule:{
date:{
type:"every_day"
},
time:{
type:"every_hour",
interval:1
}
}
};
const url = `${dbtApiUrl}/api/v2/accounts/${dbtAccountId}/jobs/`;
const response = await fetch(url, {
method: "POST",
headers: {
"Authorization": `Bearer ${authToken}`,
"Content-Type": "application/json",
},
body: JSON.stringify(requestBody),
});
// ...
After making these calls, I see the new environment in dbt and the new job.
When I run the job from the dbt cloud UI, I receive the following error:
Failed to generate profile due to incorrect credentials. Please double check credentials and try again.
When inspecting the environment, the “Deployment credentials” appear correct:
Deployment credentials
| Authentication Method | Service Account JSON |
| Dataset | value provided to schema property when creating credentials |
Editing the environment then shows:
Deployment credentials
| Authentication Method | empty, with Service Account JSON available in the dropdown |
| Dataset | value provided to schema property when creating credentials |
When I choose to set the Authentication Method and then press “Test Connection”, the tests complete successfully.
However, when I then save the edited environment, I get the error:
An unknown error occurred, please try refreshing. If the error persists, please contact support.
Inspecting the Javascript console, I see the failing call is:
PATCH https://AAAA.us1.dbt.com/api/v3/accounts/BBBBB/projects/CCCCCC/credentials/DDDDDDDDDDDDDDD/
{
"id": DDDDDDDDDDDDDDD,
"credential_details": {
"fields": {
"auth_type": {
"metadata": {
"label": "Authentication Method"
"description": "The authentication method to use for this credential.",
"field_type": "select",
"encrypt": false,
"overrideable": false,
"is_searchable": false,
"options": [
{
"label": "Service Account JSON",
"value": "service-account-json"
},
{
"label": "Native OAuth",
"value": "outh-secrets"
},
{
"label": "Workload Identity Federation",
"value": "external-oauth-wif"
}
],
"validation": {
"required": true
},
},
"value": "service-account-json"
}
}
}
}
The response is:
{
"status": {
"code": 400,
"is_success":
false,
"user_message":"The request was invalid. Please double check the provided data and try again.",
"developer_message" :""
},
"data": "Additional properties are not allowed ('credential_details' was unexpected)",
"extra": {},
"error_code": null
}
The id property in the above request is confirmed to be the id for the credentials object that was created previously in step 2.
Alternate Approach
As an additional test, I have captured the API calls that are generated when I use the dbt Cloud UI to successfully create a new environment and configure the deployment credentials. In that flow I see the following call:
POST https://AAAA.us1.dbt.com/api/v3/accounts/BBBBB/projects/CCCCCC/credentials/
{
"id": null,
"account_id": BBBBB,
"project_id": CCCCCC,
"type": "adapter",
"state": 1,
"threads": 4,
"credential_details": {
"fields": {
"auth_type": {
"metadata": {
"label": "Authentication Method",
"description": "The authentication method to use for this credential.",
"field_type": "select",
"encrypt": false,
"overrideable": false,
"is_searchable": false,
"options": [
{
"label": "Service Account JSON",
"value": "service-account-json"
},
{
"label": "Native OAuth",
"value": "oauth-secrets"
},
{
"label": "Workload Identity Federation",
"value": "external-oauth-wif"
}
],
"validation": {
"required": true
}
},
"value": "service-account-json"
},
"workload_pool_provider_path": {
"metadata": {
"label": "Workload Pool Provider Path",
"description": "The fully specified resource name of the workload pool provider (ex: //iam.googleapis.com/projects/<project-id>/locations/global/workloadIdentityPools/<workload-identity-pool>/providers/<workload-identity-provider>)",
"field_type": "text",
"encrypt": false,
"depends_on": {
"auth_type": [
"external-oauth-wif"
]
},
"overrideable": false,
"validation": {
"required": true
}
},
"value": ""
},
"service_account_impersonation_url": {
"metadata": {
"label": "Service Account Impersonation URL",
"description": "The URL for the service account impersonation request",
"field_type": "text",
"encrypt": false,
"depends_on": {
"auth_type": [
"external-oauth-wif"
]
},
"overrideable": false,
"validation": {
"required": false
}
}
},
"schema": {
"metadata": {
"label": "Dataset",
"description": "In development, dbt will build your models into a dataset with this name. This dataset name should be unique to your personal development environment and should not be shared by other members of your team.",
"field_type": "text",
"encrypt": false,
"overrideable": false,
"validation": {
"required": true
}
},
"value": "nexus_com_directtest"
},
"target_name": {
"metadata": {
"label": "Target name",
"description": "",
"field_type": "text",
"encrypt": false,
"overrideable": false,
"validation": {
"required": true
}
},
"value": "default"
},
"threads": {
"metadata": {
"label": "Threads",
"description": "The number of threads to use for dbt operations.",
"field_type": "number",
"encrypt": false,
"overrideable": false,
"validation": {
"required": false
}
},
"value": null
}
}
},
"adapter_version": "bigquery_v1"
}
Here it is noted this is type: "adapter" and not type: "bigquery".
When I change my code to follow this appraoch using type adapter, I receive errors suggesting the JSON payload does not adhere to the expected format:
{
"status": {
"code": 400,
"is_success": false,
"user_message": "The request was invalid. Please double check the provided data and try again.",
"developer_message": "'metadata' is a required property\n\nFailed validating 'required' in schema['properties']['fields']['properties']['auth_type']:\n {'additionalProperties': False,\n 'description': 'SelectField(metadata: '\n 'schemas.fields.select_field.SelectFieldMetadata, '\n 'value: Optional[str] = None)',\n 'properties': {'metadata': {'$ref': '#/definitions/SelectFieldMetadata'},\n 'value': {'oneOf': [{'type': 'string'},\n {'type': 'null'}]}},\n 'required': ['metadata'],\n 'type': 'object'}\n\nOn instance['fields']['auth_type']:\n {'value': 'service-account-json'}"
},
"data": {
"fields": {
"auth_type": "'metadata' is a required property"
}
},
"extra": {},
"error_code": null
}
Interestingly, I receive this error even when I replay the payload captured from the browser that was submitted successfully from the dbt Cloud interface.
I find it unexpected that the POST body requires all the metadata, but whether or not I include the metadata, I receive the same error.
I have not been able to create a credentials object of type adapter.
Clearly I’m doing something wrong! Any help or tips would be greatly appreciated.
Thanks!