Data Engineering / Advanced: distributed correctness and reliability / Lesson 17

Freshness, observability and cost

Measure consumer-visible data health and resource demand with clear populations and clocks. Distinguish timely task completion from complete current data and explain the cost of repair.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Define freshness, completeness and lag evidence separately.
Evaluate cost alongside correctness and service objectives.

Define what the consumer must observe

A data SLI should identify dataset, accepted version, population, time boundary and expected behavior. Publication age, source-event age, processing lag and completeness are different metrics. A recently finished job may publish yesterday’s source snapshot, while the largest event timestamp can hide a missing older partition. Keep failed and unmeasured states visible rather than converting them to zero. Alert on the consumer-impacting condition with owner and lineage context. An orchestration dashboard is useful but cannot replace evidence that the promised records and corrections reached the accepted output.

A recent publication can still have a progress gap

The offline model compares the latest expected source position with the accepted output position and separately records publication age. A one-minute publication age does not remove the twenty-event progress gap. Positions must refer to the same declared history; subtracting unrelated offsets across partitions is not meaningful. Real pipelines need per-source/partition semantics, clock accuracy and completeness checks rather than one universal lag number. The example uses invented integer positions and performs no monitoring query, network request or production clock measurement.

python Offline publication-age/progress distinction

latest_expected = 100
accepted_position = 80
publication_age_minutes = 1
assert publication_age_minutes <= 5
assert latest_expected - accepted_position == 20
assert accepted_position != latest_expected

The recently published output still trails by twenty positions in this synthetic history. The units are not a real service SLO.

Correlate data symptoms with execution evidence

When freshness degrades, inspect source availability, input volume, rejects, queueing, task attempts, skew, file layout, sink latency and checkpoint state at matching times. Logs should reference identities and scoped reasons without leaking private records. A retry storm can increase cost while delaying recovery, and a low row count can mean legitimate quiet input or a missing partition. Keep those hypotheses distinguishable. A single average runtime can hide a few very slow partitions, so distributions and oldest outstanding work often matter more than one dashboard total.

Expose the assumptions behind a cost comparison

The toy cost function assigns invented units to files opened and data bytes processed. Two layouts with equal bytes but different file counts receive different modeled overhead. This is not cloud pricing, a benchmark or a universal engine cost formula. Real cost also depends on compute duration, requests, network movement, storage, retention and maintenance under the selected service. Preserve correctness and freshness while comparing alternatives; dropping required data can trivially make a wrong pipeline cheaper. Report the workload and assumptions that justify a change rather than a percentage without its denominator.

python Offline hypothetical file-overhead model

def modeled_cost(file_count, byte_units):
    return file_count * 3 + byte_units

assert modeled_cost(10, 100) == 130
assert modeled_cost(2, 100) == 106
assert modeled_cost(2, 100) < modeled_cost(10, 100)

The computed values are 130 and 106 invented units. No cloud rates, bill or real query execution are consulted.

Make the next action accountable and bounded

A useful alert states which dataset/version is affected, the consumer objective violated, the observed evidence, owner and approved next step. Avoid making every anomaly trigger a full historical replay. Scope repairs from lineage and protect live capacity with admission and bounded concurrency. Re-evaluate metrics when schemas, partitions or consumer contracts change. Optimize cost only after result equivalence and recovery behavior are verified. Evidence should say what was measured, what is modeled and which uncertainty remains; a green budget graph does not certify complete, current or privacy-safe data.

Common errors and fixes

Recent job completion is called fresh source data

Why it happens: Publication and source-progress clocks were conflated.
How to fix it: Measure accepted source coverage and publication age separately.
Prevent it next time: Test stale input with a recent output timestamp.

A mean runtime hides a stranded partition

Why it happens: Distribution and oldest work were not observed.
How to fix it: Inspect partition-level progress and consumer impact.
Prevent it next time: Monitor tails and completeness, not only averages.

A cheaper pipeline omits required history

Why it happens: Cost optimization changed the accepted population.
How to fix it: Restore result equivalence before evaluating savings.
Prevent it next time: Make correctness a cost-change acceptance gate.

Check your understanding

1. A dataset was published one minute ago but its accepted source boundary is an hour old. Which freshness claim is valid?

Show worked solution

The publication is recent, but source freshness is not demonstrated by that timestamp. Report both boundaries and determine whether the promised source population is complete through the required point. A consumer objective must specify which clock matters instead of accepting any recent-looking time.

2. Why should a cost model label file and byte assumptions explicitly?