implementing threads dynamically

Help
1

The problem I’m having

I understand the use of threads in dbt - to simultaneously run multiple models without violating dependencies. However, we heavily use custom materialisations (1 model materialises numerous tables via custom materialisations) and I’d like a dynamic way for dbt to run my 1 x custom materialisation but in multiple threads that I pass in at run time to achieve parallelism. Is this possible?

The context of why I’m trying to do this

I want to do this as my custom materialisation creates numerous tables 1 by 1 at the moment. I would get some good performance gains if I can run them concurrently.

What I’ve already tried

I’m currently achieving this by replicating my single model so I have 4 instances / models in dbt so that I can run them concurrently. But this means a lot of replication & manual labour.

Some example code or error messages

no errors generated… just looking for a systematic way to break 1 model into several programatically so I can get some concurrency gains.

2

Here’s an example of my current model run… you can see lots of SQL statements being issued and are all relatively small… this would benefit heavily from being executed concurrently but how do I achieve that via dbt other than creating lots of physical models?

image
image850×1065 22.2 KB

3

hey dbt admins / forum… any ideas?

4

@joellabes hey can you help?