How to combine static historical data with live data?

I would start simple, and just union the two datasets together and materialize that union model as a view. For downstream models, you can either materialize them as views or rebuild the downstream tables every hour, so they contain fresh data. This will probably scale better than you think, and may or may not be wasteful, resource-wise, depending on your warehouse.

A second option would be to pursue a strategy called Lambda Views. There’s an excellent discussion of that in this post: How to create near real-time models with just dbt + SQL

1 Like