The problem I’m having
My Elementary tests look at the last 24 hours of data to decide if something is wrong, but I want them to only look at the last 1 hour. Right now, even though my tests run every hour, they still compare against a full day’s worth of history.
Note that since I have time_bucket = 1 hour, and days_back = 1 day, every test will be splitted to 24 sub-tests on hourly chunk for the last day, and eventually any failure within these chunks will cause the entire test to fail.
The context of why I’m trying to do this
Our data’s freshness SLA is 1 hour, if the test looks like past day it will for sure capture last hour’s error but this will generate a lot of false positive alerts and cause alert fatigue
What I’ve already tried
I tried setting days_back: 0.5 or even days_back: 0.04 (which should be 1 hour), but Elementary still uses a full day.
I also tried using training_period and detection_period params like the documentation said: detection_period - Elementary, but I get an error saying : test compilation error Missing mandatory configuration: [‘backfill_days’]
Some example code or error messages
name: table_A_freshness_anomalies_error
seasonality: hour_of_day
anomaly_sensitivity: 2
time_bucket:
period: hour
count: 1
days_back: 1
backfill_days: 1
