Prometheus / Advanced: scale, persistence, and safe validation / Lesson 14

Staleness, absence, and time comparisons

Interpret failed scrapes, absent series, stale markers, lookback, and shifted evaluation without converting missing evidence into a successful zero.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate zero, failed collection, and absent evidence.
Compare absence functions with failure selectors.
Review offset, timestamps, and historical query assumptions.

Zero and absence are different measurements

A counter rate of zero can describe observed inactivity, up equal to zero describes a failed scrape, and no selected series describes absence under query semantics. Those states cannot be merged without a population and coverage policy. An intentionally retired target differs from an unexpectedly missing one. Staleness handling influences when instant selectors stop returning old series; lookback and explicit timestamps add further context. Read the pinned semantics rather than assuming every missing point means exactly one fixed delay. A continuous-looking graph can reuse recent samples while collection has already become unreliable.

Ask failure and absence questions separately

The first unexecuted expression selects explicit failed scrapes. The second asks whether the selected job has no samples in the supplied range under absent_over_time semantics. A present up series with value zero is still evidence of samples and therefore differs from absence. These are teaching expressions, not a complete expected-target alert design: absence at the job level may not identify one missing instance among healthy peers. Use an authorized expected inventory when per-target completeness matters. Do not invent target identities from an empty result or assume that inferred output labels reproduce an inventory.

promql Illustrative failed-versus-absent queries

up{job="warehouse-lab"} == 0
absent_over_time(up{job="warehouse-lab"}[10m])

The ten-minute range is illustrative, not a paging recommendation. Neither query runs here, and the second is not a per-instance inventory comparison.

Keep known zero and unknown distinct

The offline classifier uses declared sample presence and freshness before reading a value. A fresh measured zero remains known zero, while missing or old evidence remains unknown. It models an application’s interpretation policy, not Prometheus stale markers or their storage representation. Real range functions can use earlier samples according to their own rules even when an instant selector no longer returns a series. Verify each chosen function rather than applying one blanket absence policy. Explicitly state how dashboards and alerts represent unknown evidence so a reassuring color does not conceal a blind source.

python Offline evidence-state model

def interpret(present, fresh, value):
    if not present or not fresh:
        return "unknown-evidence"
    return f"known-value={value}"
print(interpret(True, True, 0))
print(interpret(False, False, None))
print(interpret(True, False, 0))

Expected output

known-value=0
unknown-evidence
unknown-evidence

Presence and freshness are fixed inputs. No Prometheus lookback, stale marker, exporter timestamp, or query engine is implemented.

Historical comparison needs aligned definitions

offset shifts the time context of a selector, and the @ modifier can fix evaluation to a specified timestamp under documented rules. These tools can support comparisons, but they do not guarantee equivalent traffic mix, label schema, deployment identity, or data completeness. A renamed job can make yesterday’s selector empty while today’s succeeds. Explicitly timestamped exporter data can have different age behavior from ordinary scrape-timestamped samples. Record timezone, evaluation time, window, and contract revision. Compare like populations before interpreting a historical difference as a service improvement or regression.

Test absence transitions, not only healthy values

Use synthetic cases for no traffic, failed collection, target removal, delayed observations, restart, and label migration. Decide which system owns expected activity and how long evidence can remain useful. A fallback zero may be justified for one explicitly known population but harmful for another; keep the justification and matching labels reviewable. This lesson executes only the local classifier. It performs no historical query, timestamp injection, target change, data deletion, or monitoring operation, and does not claim to validate actual staleness timing.

Common errors and fixes

No samples is presented as no errors

Why it happens: Unknown coverage was converted to measured zero.
How to fix it: Preserve absence and freshness as distinct states.
Prevent it next time: Test missing-source and no-traffic cases.

A job-level absence check misses one missing instance

Why it happens: The query lacked an expected per-target population.
How to fix it: Compare authorized inventory with observed identities.
Prevent it next time: Review scope and target retirement rules.

Historical improvement follows a label rename

Why it happens: The shifted selector no longer matches comparable data.
How to fix it: Align old and new contracts or explain the migration.
Prevent it next time: Version query definitions and identity changes.

Check your understanding

1. Why does absent_over_time(up[10m]) not mean that all targets were healthy?

Show worked solution

It asks about sample absence for the selected population. Failed scrapes can still produce up samples with value zero, and one healthy peer can keep a broad job selection nonempty. Interpret the exact selector and expected target inventory; absence and failed collection answer different questions.

2. Can offset 1d guarantee a fair day-over-day comparison?