I have a source table
orders. The table has three timestamp columns
timestamp3. I define the source table as shown below.
version: 2 sources: - name: source1 database: source_database schema: source_schema tables: - name: orders freshness: warn_after: count: 3 period: day loaded_at_field: timestamp1
When I run
dbt source freshness --select source:source1.orders, dbt will check the data freshness in the table
Now, I want to also check the source data freshness based on the other two timestamp columns
timestamp3. How can I do that?
In my business, I need to check the source data freshness in the
orders table by the three timestamp columns respectively. Means I need dbt to run three different SQL queries.
(1) I tried to add multiple
freshness under the same table block, does not work. I got error
Map keys must be unique.
(2) I tried to add multiple
loaded_at_field under the same table block, does not work. Same error.
(3) I tried to add multiple columns to the
loaded_at_field property and split those columns by
,, does not work.
(4) I tried to add multiple
sources with different names, like
source3. Then put
freshness under the same table under those different sources. This solution works. But I don’t want to create multiple sources just because of this.
I expect a solution that I can configure the same source and the same table with multiple
loaded_at_field columns, and dbt could do multiple data freshness checks with multiple individual SQL queries.
Does anybody know if this is supported or not by dbt?