MLOps / Prediction and evidence foundations / Lesson 4

Experiment records and lineage

Connect artifacts, executions, metrics, and selection decisions so a result can be investigated later.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate a run from the artifacts it consumes and produces.
Record metric definitions and comparable cohorts.
Use lineage to answer a concrete diagnostic question.

A run is an execution rather than a model name

One execution can consume several artifacts and produce several others: transformed data, a fitted model, an evaluation report, and logs. Repeating the same code with another input snapshot creates another execution even if the display name stays unchanged. An artifact has its own identity and interpretation, while an experiment groups related questions or runs. Record code and configuration identity, inputs, outputs, start and end evidence, status, and important environment assumptions. A timestamp or human-friendly name alone rarely identifies enough state to explain why two predictions differ.

Lineage records dependencies with meaning

The first fixture defines artifacts and execution edges, checks that referenced identities exist, and finds the producer of a model and report. A real lineage system can store richer contexts and properties, but the important question remains concrete: which data, split, transformation, and execution produced the artifact under review? Distinguish a copied file from a newly computed model and preserve the relationship. Existence of a parent id does not prove the parent content is immutable or correct. Keep content identity and contract versions where needed, and avoid recording secrets or unnecessary private examples as metadata.

python Deterministic offline standard-library fixture

artifacts = {"data-v1":{"kind":"dataset"}, "split-v1":{"kind":"split"},
             "model-v1":{"kind":"model"}, "evaluation-v1":{"kind":"report"}}
executions = [{"id":"fit-1", "inputs":["data-v1","split-v1"], "outputs":["model-v1"]},
              {"id":"assess-1", "inputs":["model-v1","split-v1"], "outputs":["evaluation-v1"]}]
for execution in executions:
    assert all(key in artifacts for key in execution["inputs"] + execution["outputs"])
producer = {output:run["id"] for run in executions for output in run["outputs"]}
print("model producer", producer["model-v1"])
print("report producer", producer["evaluation-v1"])
print("model inputs", executions[0]["inputs"])

Expected output

model producer fit-1
report producer assess-1
model inputs ['data-v1', 'split-v1']

Artifact identity and execution identity are separate; these in-memory edges support a small provenance query without a metadata backend.

A metric value needs a measurement contract

Record the metric name, direction, units, eligible cohort, denominator, slices, thresholds, label-maturity policy, and relevant uncertainty. A value called score can mean training fit, validation selection, a delayed operational outcome, or a final untouched assessment. Those are different evidence roles. The second fixture compares only records sharing its declared metric and cohort; a lower loss from another cohort is not automatically superior. Different weighting, preprocessing, class mappings, or missing-label rules can also invalidate a comparison even when metric names match. Preserve the evaluation implementation identity alongside the reported number.

python Deterministic offline standard-library fixture

runs = [{"id":"a", "metric":"mae", "cohort":"validation-v1", "value":2.0},
        {"id":"b", "metric":"mae", "cohort":"validation-v2", "value":1.5},
        {"id":"c", "metric":"mae", "cohort":"validation-v1", "value":1.8}]
comparable = [run for run in runs if run["metric"] == "mae" and run["cohort"] == "validation-v1"]
best = min(comparable, key=lambda run:run["value"])
print("comparable ids", [run["id"] for run in comparable])
print("selected", best["id"], best["value"])

Expected output

comparable ids ['a', 'c']
selected c 1.8

The smaller value from another cohort is not silently treated as a better result under the declared comparison contract.

Selection history explains which evidence was consumed

Track which candidates were tried, why one was selected, and what later changes were prompted by inspected results. Automatic tracking can collect many fields while still missing a manual exclusion, a revised label rule, or the fact that a final test guided tuning. Treat logs as useful records whose completeness must be reviewed, not an automatic guarantee of scientific validity. Keep failed and rejected runs when they explain the decision. A result copied into a report should remain linked to its original cohort and artifact rather than becoming a detached favorable number.

Query lineage to test a hypothesis

If an active model behaves differently, compare its full dependency path with the earlier artifact: input snapshot, schema, fitted transformation, training configuration, evaluation procedure, and packaging. Locate the earliest differing dependency before rerunning expensive work. Missing or contradictory edges should block claims that the artifact is fully explained. These standard-library fixtures use only tiny dictionaries and lists; no tracking service, registry, metadata database, model fit, or external experiment is created. They verify graph references and comparison arithmetic, while operational storage, access, and retention require a separately implemented system.

Common errors and fixes

Two different runs are merged under one display name

Why it happens: Human-readable naming was used as execution identity.
How to fix it: Assign stable execution and artifact ids and preserve their edges.
Prevent it next time: Test repeated names with different inputs in the lineage fixture.

The smallest displayed loss is selected across incompatible cohorts

Why it happens: Metric context and evaluation boundaries were omitted.
How to fix it: Filter by a complete comparable measurement contract.
Prevent it next time: Store cohort, weighting, units, and evaluator identity with each value.

Automatic logs are treated as complete experimental provenance

Why it happens: Manual choices and external transformations were not captured.
How to fix it: Review the dependency and selection path explicitly.
Prevent it next time: Record exclusions, revised rules, failures, and consumed assessment data.

Check your understanding

1. Why should an evaluation report have a different identity from its model?

Show worked solution

It is produced by a separate procedure using a model plus a particular dataset, target policy, metric implementation, and configuration. The same model can have several reports with different legitimate questions. Keeping separate identities preserves those dependencies instead of suggesting that one score is an intrinsic property of the weight file.

2. Run b has lower loss but another cohort. What should happen before declaring it better?