Is it possible to customize incremental_strategy?

Help
1

As per topic, is it possible to roll your own incremental_strategy? For example if the unique_key already exists, do nothing so that the original record is kept as is.

2

Yes, it’s possible to make your own strategy, check out Creating new materializations | dbt Developer Hub for an introduction.

Before you go down that path, there’s another option which I haven’t tried but feels like it has decent odds of working:

There are strategy-specific configs available when configuring incremental models, and one of them is merge_update_columns if you use the merge strategy on adapters which support it (including Snowflake, BigQuery, Apache Spark, and Databricks). I wonder whether passing an empty list to that would mean nothing would be updated if the key already exists?

{{
  config(
    materialized = 'incremental',
    unique_key = 'id',
    merge_update_columns = [],
    ...
  )
}}

select ...
1 Like
3

thanks for the reply @joellabes. I did arrive at the same thinking with merge_update_columns after reading other discussions here.

unfortunately we are using postgres, which does not support merge strategy.

I will have a read on Creating new materializations

1 Like
4

If you get it working, I’d love to read about it in #show-and-tell! Feel free to open another topic if you run into any implementation issues along the way

5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

7

@mwbong dbt now makes it easier to define custom incremental strategies in your project and you can also use those published within a package.

8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.