Use Case: I have a table in Snowflake which is synced through Fivetran. Currently, the team which manages the SOURCE data has decided to delete the entire data from the source side.
I want to retain the entire data on destination side(Snowflake) using another temp table(historical table) and I want to keep adding the incremental records from the source table to this temp table and eventually use this temp table in my pipelines. How can I achieve this using INCREMENTAL materialized option? Any other method can also be recommended.
So the Source is getting deleted. so … it’s gone. But you have a copy of the information in a history table. Where do the new records of data come from? I’ve gotten lost in the use case.
some general thoughts -
if you just want to incrementally append new records from table_a on to table_b, that’s pretty standard incremental model functionality. some things to thing about - do you need to handle updates? does your data have a unique key?
if you want to watch table_a and monitor data changes to use later you might want to read up on snapshots which basically create a type2 SCD.