I’m running a snapshot on a table, but on specific fields like follows:
{% snapshot prestaging_pipeline_check %}
{{
config(
target_schema='snapshots',
strategy='check',
unique_key='col_V',
check_cols=['col_Y', 'col_Z'],
)
}}
select * from {{ref("table_X")}}
{% endsnapshot %}
However, the snapshot seems to fail on a completely different field in the table that has changed it’s data type.
08:58:52
08:58:52 Database Error in snapshot prestaging_pipeline_check (snapshots/prestaging_pipeline_snapshot.sql)
08:58:52 Value has type STRING which cannot be inserted into column col_A, which has type ARRAY<STRING> at [16:93]
08:58:52 compiled SQL at target/run/datapipelines/snapshots/prestaging_pipeline_snapshot.sql
08:58:52
Looking at the logs, snapshot does try to access the entire table, but shouldn’t it be using the specific columns mentioned in check_cols?
Any suggestions on how to fix this without having to drop the original table?
Information:
Running on dbtCloud