Since custom incremental strategies are not available for BigQuery adapter, I would like to override the default behavior of the insert_overwrite incremental strategy.
There are certain scenarios where I would like to execute a plain INSERT INTO statement, i.e. when adding new data for a cluster that wasn’t available in the source before. I don’t want to run a full refresh because it would be more expensive, so I need to go to BigQuery console and run the statement myself one by one on all the affected models. I would like to do this through dbt instead to make it quicker, so I have thought on editing the default macro to control the logic through a variable like this:
{% macro get_incremental_insert_overwrite_sql(arg_dict) %}
{#
We're overriding the default "insert_overwrite" strategy.
If a "insert_only" variable is received and we're on an
incrmeental run, we will perform run a INSERT INTO statement
without overwriting any partition
-#}
{{ log("Entered macro") }}
{% if var("insert_only") and is_incremental() %}
{%- set dest_cols_csv = get_quoted_csv(arg_dict["dest_columns"] | map(attribute="name")) -%}
INSERT INTO {{ arg_dict["target_relation"] }} ({{ dest_cols_csv }})
(
SELECT {{ dest_cols_csv }}
FROM {{ arg_dict["temp_relation"] }}
)
{% else %}
{{ return(adapter.dispatch('get_incremental_insert_overwrite_sql', 'dbt')(arg_dict)) }}
{% endif %}
{% endmacro %}
However, even I have purposedly overrided the get_incremental_insert_overwrite_sql macro, it looks like my models are always using the default macro instead of this new logic when I feed the `–vars ‘{insert_only: true}’ variable. I know this because no log is being output to the log.file
Is it possible to override this macro on BigQuery, or is it that I need to name it differently?