Changing the logic of column in incremental model

Help
,
1

Hello,

I have a doubt regarding incremental model the config I have for this model is below:
image

I have changed the logic for one of the existing column(and it is not unique key) to different field name from what it was before so just wondering if I need to do the full refresh for syncing those changes as I can’t see the updated value for that particular column.
Please guide.

Thanks,
Arti

2

Hi @arti.aga16

The incremental materialization has a setting called on_schema_change.

From the docs:
" The possible values for on_schema_change are:

  • ignore: Default behavior (see below).
  • fail: Triggers an error message when the source and target schemas diverge
  • append_new_columns: Append new columns to the existing table. Note that this setting does not remove columns from the existing table that are not present in the new data.
  • sync_all_columns: Adds any new columns to the existing table, and removes any columns that are now missing. Note that this is inclusive of data type changes. On BigQuery, changing column types requires a full table scan; be mindful of the trade-offs when implementing.“”

So if you want to see your new column you can use append_new_columns or sync_all_columns.

But notice this will not backfill the data for your new column. If you want to populate your new column with historical data you need to run the model in full-refresh mode

3

Hi brunoszdl,
Yeah I know about this feature but currently avoiding to use that so just trying to understand if i have changed the logic of existing field do i need to do the full refresh ?

Thanks,
Arti

4

If you want to apply this logic to historical data then yes,

if you are ok with just recent data having this logic and historical data a different logic then no

1 Like
5

cool thanks so much :slight_smile:

1 Like