# SET spark.sql.shuffle.partitions=auto

**URL:** <https://discourse.getdbt.com/t/set-spark-sql-shuffle-partitions-auto/10246>\
**Category:** Help\
**Tags:** databricks\
**Created:** [October 6, 2023, 11:12am UTC](https://discourse.getdbt.com/t/set-spark-sql-shuffle-partitions-auto/10246 "2023-10-06T11:12:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![martin\_larsson](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/martin_larsson/32/1971_2.png) [@martin\_larsson](https://discourse.getdbt.com/u/martin_larsson)\
**Post date:** [October 6, 2023, 11:12am UTC](https://discourse.getdbt.com/t/set-spark-sql-shuffle-partitions-auto/10246/1 "2023-10-06T11:12:57Z")

</div>

## The problem I’m having

Me and my team are running our dbt project on an all purpose cluster. When running some of our larger tables we get a performance hint in the Spark UI telling us to increase the `spark.sql.shuffle.partitions` setting to increase performance. I found [this stack overflow answer](https://stackoverflow.com/questions/67933530/how-to-pass-spark-configuration-parameters-to-dbt) on how to set this setting in dbt. Instead of putting this setting in the model config I added it to `dbt_project.yml` as you can see below. When I ran one of the big models again I still get the performance warning so it seems that the setting did not have the effect I was hoping for.

Is there anybody here that knows how to set this?

Im not sure but I think this setting is on the cluster level in Databricks. So it does not really make sense to set this for every model.

## Context

We are running Azure Databricks and we tried this on an all purpose cluster with 96 cores and 384 Gib memory.

## What I’ve already tried

## Some example code or error messages

```auto
models:
  +file_format: delta
  +pre-hook:
    sql: "SET spark.sql.shuffle.partitions=auto"
    transaction: false

```

---

<div class="post-metadata">

**Author:** ![charlbotha](https://sea2.discourse-cdn.com/flex020/user_avatar/discourse.getdbt.com/charlbotha/32/4844_2.png) [@charlbotha](https://discourse.getdbt.com/u/charlbotha)\
**Post date:** [November 2, 2024, 8:30am UTC](https://discourse.getdbt.com/t/set-spark-sql-shuffle-partitions-auto/10246/2 "2024-11-02T08:30:43Z")

</div>

Databricks / Spark saves the configured spark.sql.shuffle.partitions into the stream checkpoint.

I had to do a full refresh on the aggregation tables in question to get my new setting to take effect.

See e.g. [Solved: Changing shuffle.partitions with spark.conf in a s... - Databricks Community - 27389](https://community.databricks.com/t5/data-engineering/changing-shuffle-partitions-with-spark-conf-in-a-spark-stream/td-p/27389)