Apache Spark / Advanced: streaming and recoverable operation / Lesson 16

Quality tests and operational evidence

Combine exact synthetic fixtures, source reconciliation and scoped runtime observations before accepting a pipeline result.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Test domain invariants and adversarial data rather than only happy paths.
Separate local model checks, Spark assertions and operational metrics.

A passing job is not a passing data product

A pipeline can finish while omitting malformed rows, duplicating a reference join or publishing incomplete daily totals. Define acceptance invariants before execution: source equals accepted plus rejected plus intentionally excluded, required identities are valid, revisions follow the winner rule and energy totals use the intended unit. A fixed count alone cannot prove correct identities. Keep fixtures with duplicate versions, missing sites, invalid numeric strings and exact time boundaries. These are meaningful tests because they challenge the business contract rather than merely repeat the implementation’s syntax.

Validate a tiny exact model without Spark

The complete Python program below uses Decimal to parse two known energy values and reject one invalid string. It prints accepted=2, rejected=1 and total=3.75. It is a standard-library model only, not Spark’s parser, Decimal engine or distributed rejection behavior. The source domain here requires valid decimal text; a broader production contract would also define missingness, negative values and bounds. Running this safe program proves the stated fixture arithmetic and classification scope. It does not justify claiming that a configured pipeline or external file was tested.

python Illustrative technical example; logical results are reasoned, not engine-tested.

from decimal import Decimal, InvalidOperation
raw = ['1.25', '2.50', 'offline']
accepted = []
rejected = []
for value in raw:
    try:
        accepted.append(Decimal(value))
    except InvalidOperation:
        rejected.append(value)
assert len(raw) == len(accepted) + len(rejected)
assert sum(accepted) == Decimal('3.75')
print(f'accepted={len(accepted)}, rejected={len(rejected)}, total={sum(accepted)}')

Expected output

accepted=2, rejected=1, total=3.75

Executed only as an isolated standard-library program during authoring. No PySpark imports, files or services are involved.

DataFrame equality needs an explicit comparison policy

Spark’s testing helpers can compare expected and actual DataFrames in an independently authorized engine environment. Specify whether row order matters and choose numeric tolerances deliberately. An unordered relation should not fail solely because partitions returned rows differently; a report with a declared order should test that order. Schema and nullability expectations also need review. The fragment names prepared actual and expected relations without executing the engine. A local list comparison is not a substitute for testing Spark type conversion, ANSI behavior, query planning or distributed joins when those integrations matter.

python Illustrative technical example; logical results are reasoned, not engine-tested.

from pyspark.testing.utils import assertDataFrameEqual
assertDataFrameEqual(actual, expected, checkRowOrder=False)

Platform test fragment only. actual and expected require a separately prepared Spark lab; no engine assertion is executed here.

Metrics explain execution within their collection scope

Use approved application, job, stage and task identifiers to connect a data-quality defect with execution evidence. Source counts, rejected counts and publication versions belong beside task duration, shuffle bytes and spill. Event logs can support later history review when enabled and retained under the deployment policy; they can also contain sensitive information and need access controls. A green dashboard or successful application state does not prove energy completeness. Label estimates, measured values and synthetic expectations distinctly so a future maintainer can understand what the evidence actually establishes.

Publish only a reconciled version with known limits

An acceptance record should name the source snapshot, schema and transformation version, accepted and rejected population, key checks, energy reconciliation and unresolved exclusions. Use representative failure cases and restart tests in a separate authorized environment when operational readiness requires them. Repeating every expensive test after no relevant change adds little evidence; select checks according to the changed contract and unresolved risks. This course performs the stated small standard-library checks only. It runs no Spark unit suite, event-log collector, database operation or load test, and preserves all engine fragments as non-runnable.

Common errors and fixes

A successful job certifies source completeness

Why it happens: Execution status replaced row and identity reconciliation.
How to fix it: Check source, accepted, rejected and excluded populations explicitly.
Prevent it next time: Publish a versioned acceptance manifest.

An unordered equality test depends on incidental row order

Why it happens: Physical return order was mistaken for the relation’s contract.
How to fix it: Use the intended ordering policy and explicit numeric tolerance.
Prevent it next time: Test ordered reports and unordered relations differently.

Synthetic model results are labelled engine-tested

Why it happens: Local validation scope was overstated.
How to fix it: State exactly which program and assertions ran.
Prevent it next time: Keep model, engine and operational evidence separate.

Check your understanding

1. What does the Decimal fixture prove, and what does it not prove?

Show worked solution

It proves two accepted values, one invalid string, count reconciliation and the exact local total3.75. It does not prove Spark parsing, executor behavior or production completeness. Keep those limits with the result and use separately authorized integration checks when required.

2. Why are duplicate-key fixtures needed even when totals match?