Data Engineering / Intermediate: change, time and coordination / Lesson 12

Data quality, rejects and lineage

Turn quality rules into scoped evidence and retain enough lineage to explain a published result. Distinguish absence of detected failures from proof of complete correctness.

About 4 min readContent reviewed 2026-10-26

On this page

Lesson goals

Define completeness, uniqueness and consistency checks at the correct grain.
Record input/transform/output lineage and reject handling.

A quality assertion needs a population and meaning

Completeness asks whether the promised population arrived; validity asks whether values satisfy a rule; uniqueness concerns identity; consistency checks relationships and meanings. Freshness adds time, but a recent publication can contain old or incomplete events. Define which checks block publication and which create an explicitly labeled warning or partial result. A universal quality score can hide a critical failed invariant behind many easy passes. Keep applicable, failed, passed and unmeasured checks distinguishable, with owner and evidence freshness rather than only a green percentage.

Account for accepted and rejected input explicitly

The offline classifier separates three invented quantity records into accepted and rejected sets. The accounting invariant includes both outcomes so filtering cannot silently erase a row from the run record. It validates only the declared integer/nonnegative rule, not every field or the completeness of upstream delivery. Keep the rejected identity and scoped reason for repair, with privacy-conscious retention. If the dataset permits partial publication, that must be visible to consumers; a balanced accepted-plus-rejected count does not automatically mean the accepted subset is the promised complete dataset.

python Offline input-accounting invariant

rows = [{'id': 'a', 'qty': 2}, {'id': 'b', 'qty': -1}, {'id': 'c', 'qty': 0}]
accepted = [r for r in rows if type(r['qty']) is int and r['qty'] >= 0]
rejected = [r for r in rows if not (type(r['qty']) is int and r['qty'] >= 0)]
assert [r['id'] for r in accepted] == ['a', 'c']
assert [r['id'] for r in rejected] == ['b']
assert len(accepted) + len(rejected) == len(rows)

Two accepted and one rejected record account for all three supplied records. This is not proof that no source file was missing.

Lineage connects a result to its production context

A lineage record should identify source versions, transformation code/configuration, logical interval, output version and relevant checks. W3C PROV provides a formal vocabulary for entities, activities and agents; the exercise record is simpler application metadata, not a complete PROV serialization. Lineage can explain where a result came from, but trustworthy claims also need authenticated collection and access policy. A hand-edited label is not cryptographic provenance. Keep enough detail to reproduce or investigate the result without dumping credentials or unnecessary private payloads into a shared catalog.

Write a bounded synthetic lineage record

The JSON binds one output version to declared input and transform identities and a local acceptance-check identifier. It does not assert that a real run passed or load any orchestration system. In production, use stable identities that remain resolvable under retention policy, and record the actual evidence state. A source path that later points to different bytes cannot reliably explain an old result. Capture failed attempts as well as accepted publications so an incident can distinguish a transformation defect, missing input, stale dependency or mistaken manual intervention.

json Application-defined lineage metadata

{
  "outputVersion": "lab-shipments-clean-v2",
  "inputs": ["lab-source-snapshot-v1"],
  "activity": "normalize-quantities-v2",
  "interval": "2026-10-01",
  "checks": ["LOCAL-INPUT-ACCOUNTING"],
  "evidenceState": "synthetic-example"
}

The record identifies intended relationships only. It is not a deployed lineage service, PROV conformance claim or real execution receipt.

Use quality evidence to drive repair and change

Test accepted, rejected, missing, duplicate and contradictory cases with known expected outcomes. Compare data-level acceptance across source intervals and schema changes rather than allowing a passing historical test to certify every future run. Alerting should identify affected datasets, owners and consumer impact, not merely a generic row count. Lineage and quality together help scope a backfill or rollback, but do not themselves repair state. Retain an approved rerun plan and verify downstream acceptance after correction, including whether consumers replaced or re-added an updated result.

Common errors and fixes

A quality score hides one critical failure

Why it happens: Unrelated passes were averaged over a blocking invariant.
How to fix it: Report critical acceptance checks individually with ownership.
Prevent it next time: Separate unmeasured and failed states from passed checks.

A lineage path now contains different input

Why it happens: Mutable location was treated as stable input identity.
How to fix it: Use retained version/content references and preserve the accepted evidence.
Prevent it next time: Test reproducibility across source replacement.

Rejected records are counted but cannot be repaired

Why it happens: Identity, reason or owner was omitted from the reject protocol.
How to fix it: Keep scoped repair evidence under an approved retention policy.
Prevent it next time: Exercise the complete reject-to-correction workflow.

Check your understanding

1. Accepted plus rejected count equals input count. Does that prove the promised source population is complete?

Show worked solution

No. It accounts for supplied records but cannot detect an entire file or source interval that never arrived unless another expected-population check exists. Validate both internal accounting and external completeness against an identified source manifest or contract. Distinct assertions support distinct claims.

2. Why retain failed-attempt lineage when only accepted outputs are consumed?