DBT Cloud webhook with AWS lambda

Help
1

The problem I’m having

I can’t seem to configure a webhook that triggers an AWS lambda

The context of why I’m trying to do this

I’m trying to trigger an AWS lambda after a dbt job ends (either successfully or failed). I was looking into creating a webhook to achieve this.

What I’ve already tried

I’ve tried enabling a Lambda endpoint but I don’t want to expose it to the public internet. the other option AWS offers is through an IAM role which would allow authentication from dbt cloud. This one doesn’t seem to work because on the webhook configuration there’s no mention to any role. Unfortunately neither work… is there any other option or workaround that you’ve found?

2

You must expose your endpoint over the public internet, because dbt Cloud is going to attempt to communicate with you over the public internet. There is no alternative to this.

dbt Cloud does not provide custom authentication options (ie, providing account credentials on AWS) but it does provide a way to validate requests https://docs.getdbt.com/docs/deploy/webhooks#validate-a-webhook

Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.

1 Like
3

Got it, that was what my hunch. Is there any setup that I need to do to allow dbt Cloud to talk to the lambda? Meaning having some security groups to be set up, etc?

4

Hi <@U05HW5JBVK3>,
Hope you are fine!
I was trying to validate a webhook from dbt cloud on our product but don’t understand what is the way dbt Cloud wants us to validate the requests. Concretely, what is the supposed value for MY_DBT_CLOUD_AUTH_TOKEN?
Thanks!

Note: @josep.franquet originally posted this reply in Slack. It might not have transferred perfectly.

5

When you create the webhook, dbt provides you with the token

Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.

6

it essentially provides you the same information twice: the first time when you create the webhook it provides you with the token, and then when it sends you an API call, it also signs its message with that same token and you can sign the message with that token too to check the authenticity. That’s what this code example from the docs is doing

app_secret = os.environ['MY_DBT_CLOUD_AUTH_TOKEN'].encode('utf-8')
signature = hmac.new(app_secret, request_body, hashlib.sha256).hexdigest()
return signature == auth_header```

<sub>Note: `@Mike Stanley` originally [posted this reply in Slack](https://getdbt.slack.com/archives/CBSQTAPLG/p1704815474757249?thread_ts=1699958893.570349&cid=CBSQTAPLG). It might not have transferred perfectly.</sub>
7

The value to use there is provided only once, when you create the webhook, and never again

Note: @Mike Stanley originally posted this reply in Slack. It might not have transferred perfectly.

8

Thanks <@U05HW5JBVK3>! It perfectly worked:wink:

Note: @josep.franquet originally posted this reply in Slack. It might not have transferred perfectly.