The problem I’m having
In our project we have the following code to upload dbt artifacts as an audit method. It works just fine when all the models succeed. If a model failed it causes on-run-end to exit with the following error. I understood that on-run-end should complete always even if some models failed. Its worth saying the job is configured for 20 threads and there is no fast fail configured, also the command is a dbt run --select path:[...].
If this expected to happen, what are your suggestions to upload audit metadata?
project.yml:
on-run-end:
- "{% if env_var('DBT_ENVIRONMENT') == 'PROD' or env_var('DBT_ENVIRONMENT') == 'UAT' or env_var('DBT_ENVIRONMENT') == 'TEST' %}{{ dbt_artifacts.upload_results(results) }}{% endif %}"
Error:
16:19:29 1 of 3 START hook:[project_name].on-run-end.0 .................................... [RUN]
16:19:29 1 of 3 OK hook: [project_name].on-run-end.0 ....................................... [OK in 0.13s]
16:19:29 Uploading model executions
16:19:30
16:19:30
16:19:30 Exited because of keyboard interrupt
The context of why I’m trying to do this
I need to upload audit data to snowflake always, even when the models failed. Its specially important when models failed because we are pulling the data in a custom dashboard to monitor the pipeline.