Using dbt to manage user defined functions

Show and Tell
22

Hey! Hope you’re doing well.

I just wanted to add something that I think might be helpful to this excellent original idea.

The thing is, the on-run-start hook can add some seconds to each model/test run to apply all the UDFs. In my case, the overall sum of those seconds was around 10-15% of the entire dbt pipeline runtime, and I considered it was too much time for something that is a static thing.

To tackle this, I decided to do the same idea described here, but instead of executing it with the hook, it’s executed in a step of the CI pipeline, in case some file that matches the glob pattern macros/udfs/**/*.sql is modified. You may need to make some adjustments to your infrastructure (your runners), but it worthed a while for my case.

Thanks!

1 Like
23

Hey, I’ve written an article on udfs as a custom dbt materialization, which allows you to incorporate them into the DAG and so solve most (all?) of the problems I’ve seen mentioned here elegantly.

Writeup here Materialized UDFs in a dbt World - Tempered Works

Example here pypi_vulnerabilities/models/published/udfs/matches_multi_spec.sql at main · brabster/pypi_vulnerabilities · GitHub

1 Like