The problem I’m having
I am new to dbt and working with our data team. We have built our models and we are thinking about how to deploy them to our dev and production warehouse which is running on Snowflake.
We are currently using corn jobs with GitHub action to run the command and update our models.
The issue we are having is when we run a simple dbt build command it is taking around 11 minutes to complete.
I am looking for optimal ways of doing this. I don’t want the command to hang on GitHub action while the queries are being executed on Snowflake. We are being billed for this on Github.
What I am thinking about as a solution
Is there a way to run the command dbt build and not wait for the query to complete on Snowflake to finish the job? Then on the data warehouse use a sort of hook that will send a message let’s say in a slack channel if there was an error on the queries we are running.
Did someone face a similar problem in his job and what approach did they take to solve it?
I am trying to avoid using debt cloud to run the query because it does not offer the flexibility and customization we want.