I’m getting ready to set up a fresh DBT install on a redshift cluster and am looking for advice / tips on how people have configured their WLM settings. We have a few queues (one for fivetran, one for analysts) and I’m trying to figure out if we should carve out a separate queue for dbt and how much memory we would allocate to it
Any suggestions / advice / ideas would be much appreciated!
Yes, after some trial and error that’s where I landed. I have 3 queues that are mapped to user groups.
In general, I try to run pretty low concurrency, because Redshift recommends it. SQA helps make that work.
SQA: Enabled
Q1: Loaders: Fivetran, Stitch, etc. Concurrency: 2, Memory: 10%
Q2: Transformers: DBT, Looker PDTs, and admin users (ad hoc only, usually for DBT development). Concurrency: 2. Memory: 40%
Q3: All others: Mostly Looker (~50 WAU). Concurrency: 5. Memory: 50%. Timeout: 100s
Good luck! I found that tweaking WLM was more important than any table-level confit to get the whole deployment humming.
Another note: Looker also lets you configure queueing on their side… I let them maintain a few more than 5 connections so that Redshift can decide what gets executed (some will sneak through SQA). I found that if you let Looker open 50 connections it crushes the Leader node and provides a worse UX