I have a doubt regarding incremental model the config I have for this model is below:
I have changed the logic for one of the existing column(and it is not unique key) to different field name from what it was before so just wondering if I need to do the full refresh for syncing those changes as I can’t see the updated value for that particular column.
The incremental materialization has a setting called
From the docs:
" The possible values for
ignore: Default behavior (see below).
fail: Triggers an error message when the source and target schemas diverge
append_new_columns: Append new columns to the existing table. Note that this setting does not remove columns from the existing table that are not present in the new data.
sync_all_columns: Adds any new columns to the existing table, and removes any columns that are now missing. Note that this is inclusive of data type changes. On BigQuery, changing column types requires a full table scan; be mindful of the trade-offs when implementing.“”
So if you want to see your new column you can use
But notice this will not backfill the data for your new column. If you want to populate your new column with historical data you need to run the model in
Yeah I know about this feature but currently avoiding to use that so just trying to understand if i have changed the logic of existing field do i need to do the full refresh ?
If you want to apply this logic to historical data then yes,
if you are ok with just recent data having this logic and historical data a different logic then no