The problem I’m having
Using incremental models it doubles costs.
The context of why I’m trying to do this
We tried to implement incremental models to decrease processing costs on BigQuery
What I’ve already tried
We tried static and dynamic strategies and didn’t work either
Some example code or error messages
{{
config(
unique_key='trip_id',
materialized='incremental',
incremental_strategy='insert_overwrite',
partition_by={
"field": "provider_trip_ended_at",
"data_type": "timestamp",
"granularity": "day"
},
cluster_by = ["city_name","type_name","status"]
)
}}
Second Run with Incrementals
PS: The query cost doesn’t decrease even if we use this where clause
{% if is_incremental() %}
where date( created_at) > date_add(current_date(), interval -1 day)
{% endif %}