I am working on a project that has multiple incremental models in the data mart. These tables will feed an application that consumes data from all the data mart tables.
I would like to refresh the data of all the data mart models at once instead of one by one. This is to avoid showing inconsistent data in the application while the model is executed.
I need all the models to be completed at the same time. I have an accounts model and purchases model materialized as tables and a dashboard which queries both these tables. I would like to complete the models at the same time, so the user will not see any discrepancies in the dash board data.
The accounts model will take less time than purchases model, so there will be a brief period where the data is not synced, like the user might see 10 purchases when queried from the accounts table but sees only 8 purchases when the details are pulled from the purchases table.
I am trying to avoid that brief discrepancy. Do you know if we can accomplish this?
You can’t force them to complete at the same time. Without knowing the relationships between the tables, the only suggestion I have is to have some sort of an inner join between them in the dashboard to prevent showing rows that are in one but not the other.