I have the following snapshot:
{% snapshot meter_readings_snapshot %}
{{
config(
target_schema=get_snapshot_schema(),
unique_key='id',
strategy='timestamp_with_nulls',
updated_at='updated_at',
invalidate_hard_deletes=True,
)
}}
select id,
meter_id,
estar_meter_reading_id,
from_date,
to_date,
cost,
usage,
exported,
demand,
demand_charge,
is_custom,
created_at,
is_estimate,
updated_at,
_sdc_extracted_at,
_sdc_batched_at,
_sdc_deleted_at
from {{ source('CORE', 'METER_READINGS') }}
{% endsnapshot %}
We added two new columns to this METER_READINGS table in Production (updated_by and created_by), but now I would like to add those columns to the snapshot. I expected that the next time dbt ran it would pull in those new columns, pull in the current values for the columns, and then all the historical data where dbt_valid_to is not null would have null values for the new fields. That’s not what happened. Instead I got two new columns that were almost entirely null, and then I’m slowly getting values, presumably as people make changes, again presumably because of the update strategy, I imagine?
So… my question is, how do I need to go about this so that I preserve all the historical data we’ve been collecting for years, while also bringing in the current values of the two new columns for all the current rows? Thanks all.