@krudflinger : I would love to learn more about how you tackled this. We are building up our models now, and we are struggling with some of the key realities that we think snapshots miss:
-
What happens when there is an ETL failure (@claire : would like to see some examples of this from your ‘resilient’ section)
-
What happens when we learn that a field was incorrect, and we need to ‘correct’ history ?
-
How do you tackle this idea of pre-snapshot history? (@claire : would love to see some ideas/examples on this – i really like the idea of quality column)
-
What happens when you want (or need) to rebuild an environment?
We are exploring the idea of leveraging incrementals, and custom incrementals, to create ‘archives’ of the source data so we can play it back, but I’m not seeing a way to set a loop for operations (while not_finished do dbt run).
We’re just a month into this, so still eyes wide open, so any guidance from your experience would be great!
Thanks,
Ron