Data Analysis / Advanced: Defensible and Reproducible Decisions / Lesson 16

Reproducible Analytical Pipelines

Make inputs, transformations, assumptions, and outputs explicit enough that another run can explain both agreement and change.

About 5 min readContent reviewed 2026-10-25

On this page

Lesson goals

Separate raw data, validated records, and derived artifacts.
Record code, data, environment, and random-stream context.
Use invariants and pure transformations for auditable reruns.

A pipeline is a sequence of contracts

Define stages for acquisition, schema validation, cleaning, joining, metric computation, and presentation. Each stage should state its input/output grain and checks. Keep raw records separate from derived values so rerunning a notebook cell cannot silently apply a conversion twice or overwrite the only evidence of an earlier state.

Prefer functions with explicit inputs and returned results over hidden mutable global tables. Exploratory notebooks can be useful, but an out-of-order execution history is not a reproducible workflow. Restart and run in declared order, or extract a bounded script whose dependencies and configuration are visible.

Content identity differs from a filename

The first program canonically serializes two equivalent synthetic mappings and one changed mapping before hashing. Equal content under that serialization yields equal digests despite insertion order, while changing a value changes the digest. Canonicalization choices—field ordering, encoding, number/date representation—are part of the identity contract. A checksum does not prove truth, completeness, or permission to use data.

For real workflows, record raw-source identity and derived-output identity alongside code revision and parameters. Do not embed secrets in metadata or use a content hash as an anonymization guarantee. Hashes of small predictable sensitive values can still reveal information through guessing.

python Canonical synthetic artifact identity

import hashlib
import json

def digest(record):
    payload = json.dumps(record, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
    return hashlib.sha256(payload).hexdigest()

first = {"metric": "orders", "count": 3}
reordered = {"count": 3, "metric": "orders"}
changed = {"metric": "orders", "count": 4}
print("same content", digest(first) == digest(reordered))
print("changed content", digest(first) == digest(changed))

Expected output

same content True
changed content False

Reordering fields preserves the canonical content; changing count does not.

Return a reconciled result without altering input

The second program is a pure bounded transformation of three synthetic count tokens. It returns input, accepted, rejected, and total fields and asserts the reconciliation independently of formatting. It also verifies that the original sequence has not been mutated. These properties make later pipeline failures easier to locate than a large notebook full of in-place changes.

Keep tests based on meaningful properties and known cases. Do not merely duplicate the implementation’s formula in every expectation. Include a malformed row, a boundary value, an empty input, and conflicting keys where applicable. A successful rerun is evidence about the declared data and environment, not proof of every possible input.

python A pure reconciled metric stage

import json

def analyze(tokens):
    accepted = [int(token) for token in tokens if token.isascii() and token.isdecimal() and 0 <= int(token) <= 100]
    return {"input": len(tokens), "accepted": len(accepted), "rejected": len(tokens) - len(accepted), "total": sum(accepted)}

raw = ["0", "12", "bad"]
before = list(raw)
result = analyze(raw)
assert result["accepted"] + result["rejected"] == result["input"]
assert result["total"] == 12 and raw == before
print(json.dumps(result, sort_keys=True))

Expected output

{"accepted": 2, "input": 3, "rejected": 1, "total": 12}

Two accepted values sum to12, one is rejected, and the raw list remains unchanged.

A seed alone is not an environment record

Record Python and package versions, the SQL engine/dialect, locale, time-zone rules, configuration, and data extraction time when they affect the result. NumPy stream compatibility has strict conditions involving generator, calls, build, and environment. Use local generators rather than shared global state so unrelated code does not silently consume your stream.

Pin computational choices such as ddof, quantile method, grouping options, and sort order. Display formatting can change without a substantive metric change, so compare typed intermediate results and meaningful tolerances where appropriate. Preserve a human-readable method note rather than expecting a dependency file to explain analytical intent.

Publish evidence with a reviewable run record

A reproducible deliverable can include the question, source identities, code/version, parameters, checks, output tables, figure specifications, and limitations. Separate generated results from manually edited presentation so an updated run cannot leave stale numbers beside new charts. Have a reviewer trace one important figure back to its denominator and source records.

Run only on synthetic or authorized local data, avoid hidden network dependencies, and keep outputs within a controlled destination. A successful local analysis does not authorize publishing sensitive records or writing to production systems. The course’s example artifacts are temporary; its durable product is the transparent analytical method.

Common errors and fixes

A rerun changes after an unrelated random call.

Why it happens: The analysis shares a global random stream.
How to fix it: Use a local recorded generator and call sequence.
Prevent it next time: Record environment and generator context.

A notebook contains new charts with old totals.

Why it happens: Manual edits and execution order are mixed.
How to fix it: Restart/run the declared pipeline and generate linked outputs together.
Prevent it next time: Review artifact consistency.

A checksum is treated as evidence that data is correct.

Why it happens: Content identity is confused with validity.
How to fix it: Validate schema, meaning, and provenance separately.
Prevent it next time: Document what each check can and cannot establish.

Check your understanding

1. Name the minimum run record needed to reproduce an overall completion rate after cleaning and joining.

Show worked solution

Record the metric’s numerator/denominator and eligible unit, raw source versions, cleaning and join rules, code revision, parameters and date window, software/dialect versions, and reconciliation checks. Include unmatched and rejected counts. A final percentage and a script filename do not reveal which population was analyzed or whether a join multiplied events.

2. Why is rerunning a cleaning cell twice potentially different from rerunning a complete pure pipeline?