Monitor test duration vs actual transformation

Help
, ,
1

The problem I’m having

I would like to be able to keep my production pipeline within a reasonable time. To do so i would like to monitor easily how long my tests take vs the actual transformation.

The context of why I’m trying to do this

I am using dbt cloud and my prod pipeline contain only the following command:

dbt build

Since the build does both run and test, when my pipeline duration is increasing i cannot see easily if this because of a slow test or a slow transformation

Do you have any idea how i can solve this problem ?

Should i split my pipeline in two steps:

dbt run
dbt test

In this case i am loosing the “run test directly after transformation, in DAG order” which i like

Thanks,

2

The run_results.json includes execution time of each node that ran (models, tests, etc.) so you can parse that to get the data you’re looking for.

There are some packages that make this much easier to track over time, though. Including:
• https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest/|dbt-artifacts
• https://hub.getdbt.com/elementary-data/elementary/latest/|elementary

Note: @Owen originally posted this reply in Slack. It might not have transferred perfectly.

3

Oh I should have included a link to more info about run_results.json: https://docs.getdbt.com/reference/artifacts/run-results-json

Note: @Owen originally posted this reply in Slack. It might not have transferred perfectly.

4

hey <@U0291GP2AQ0>,
do you happen to know what are the key differences between dbt-artifacts & elementary? I am currently looking into data observability tools/solutions for our dbt project and would welcome any advice on this topic.

Note: @erik.kitzing originally posted this reply in Slack. It might not have transferred perfectly.

5

I don’t know, so my advice is to read the documentation on both and then choose whichever seems best-suited to your use cases.

Note: @Owen originally posted this reply in Slack. It might not have transferred perfectly.