The problem I’m having
I have a source table orders
. The table has three timestamp columns timestamp1
, timestamp2
and timestamp3
. I define the source table as shown below.
version: 2
sources:
- name: source1
database: source_database
schema: source_schema
tables:
- name: orders
freshness:
warn_after:
count: 3
period: day
loaded_at_field: timestamp1
When I run dbt source freshness --select source:source1.orders
, dbt will check the data freshness in the table orders
.
Now, I want to also check the source data freshness based on the other two timestamp columns timestamp2
and timestamp3
. How can I do that?
The context of why I’m trying to do this
In my business, I need to check the source data freshness in the orders
table by the three timestamp columns respectively. Means I need dbt to run three different SQL queries.
What I’ve already tried
(1) I tried to add multiple freshness
under the same table block, does not work. I got error Map keys must be unique
.
(2) I tried to add multiple loaded_at_field
under the same table block, does not work. Same error.
(3) I tried to add multiple columns to the loaded_at_field
property and split those columns by ,
, does not work.
(4) I tried to add multiple sources
with different names, like source1
, source2
and source3
. Then put freshness
under the same table under those different sources. This solution works. But I don’t want to create multiple sources just because of this.
I expect a solution that I can configure the same source and the same table with multiple loaded_at_field
columns, and dbt could do multiple data freshness checks with multiple individual SQL queries.
Does anybody know if this is supported or not by dbt?
Thank you.
Jeep