The problem I’m having
I have a source table orders. The table has three timestamp columns timestamp1, timestamp2 and timestamp3. I define the source table as shown below.
version: 2
sources:
- name: source1
database: source_database
schema: source_schema
tables:
- name: orders
freshness:
warn_after:
count: 3
period: day
loaded_at_field: timestamp1
When I run dbt source freshness --select source:source1.orders, dbt will check the data freshness in the table orders.
Now, I want to also check the source data freshness based on the other two timestamp columns timestamp2 and timestamp3. How can I do that?
The context of why I’m trying to do this
In my business, I need to check the source data freshness in the orders table by the three timestamp columns respectively. Means I need dbt to run three different SQL queries.
What I’ve already tried
(1) I tried to add multiple freshness under the same table block, does not work. I got error Map keys must be unique.
(2) I tried to add multiple loaded_at_field under the same table block, does not work. Same error.
(3) I tried to add multiple columns to the loaded_at_field property and split those columns by ,, does not work.
(4) I tried to add multiple sources with different names, like source1, source2 and source3. Then put freshness under the same table under those different sources. This solution works. But I don’t want to create multiple sources just because of this.
I expect a solution that I can configure the same source and the same table with multiple loaded_at_field columns, and dbt could do multiple data freshness checks with multiple individual SQL queries.
Does anybody know if this is supported or not by dbt?
Thank you.
Jeep