MLOps / Pipelines and controlled delivery / Lesson 8

Evaluation slices and acceptance gates

Turn a candidate comparison into explicit evidence without letting one favorable average erase important failures.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Compare candidates and baselines under the same evaluation contract.
Report slice values, counts, uncertainty, and coverage.
Define missing or insufficient evidence as a distinct gate outcome.

An acceptance rule begins with the decision

Choose primary metrics, important slices, resource limits, compatibility conditions, and the permitted evaluation cohort before reviewing final outcomes. A loss measures one prediction property and may not capture the action’s cost, subgroup impact, or fallback behavior. Compare the complete candidate and baseline on the same eligible observations and target definition. A model that cannot support the required input population needs an explicit coverage statement. Keep selection evidence separate from the final assessment, and record the comparison budget so repeated tuning on favorable results is visible rather than disguised as one independent test.

Average improvement can conceal a critical regression

The first fixture shows lower overall absolute error for a candidate while its critical slice worsens. The slice has only one observation, so its number is both operationally relevant under the fixture and statistically weak evidence. These are separate points: ignoring the slice is not justified by the average, and treating its single error as a precise population estimate is also unjustified. Report counts, definition, error distribution where useful, and uncertainty appropriate to the sampling and dependence. A slice chosen only after searching for an attractive result consumes selection evidence and needs that context.

python Deterministic offline standard-library fixture

baseline = {"common":[2.0,2.0,2.0,2.0], "critical":[4.0]}
candidate = {"common":[1.0,1.0,1.0,1.0], "critical":[5.0]}
def overall(groups):
    values = [value for group in groups.values() for value in group]
    return sum(values)/len(values)
print(f"baseline overall={overall(baseline):.2f}")
print(f"candidate overall={overall(candidate):.2f}")
print("critical slice worsens", sum(candidate["critical"])/len(candidate["critical"]) > sum(baseline["critical"])/len(baseline["critical"]))

Expected output

baseline overall=2.40
candidate overall=1.80
critical slice worsens True

The invented candidate improves average absolute error while worsening a one-observation critical slice; the tiny slice also supplies weak evidence.

Thresholds express a task-specific policy

The second fixture checks an overall bound, a critical-slice bound, a minimum evidence count, and input compatibility. The fixed thresholds teach gate logic and are not recommended values for other applications. A minimum count can prevent a trivially thin report from passing, but does not itself guarantee adequate power, representativeness, or confidence. Decide how missing metrics, nonfinite values, delayed targets, and unsupported slices affect the outcome. An absent value should not be silently replaced by zero or omitted from all checks. Preserve pass, fail, and insufficient-evidence meanings explicitly in a real gate.

python Deterministic offline standard-library fixture

evidence = {"overall_mae":1.8, "critical_mae":5.0, "critical_count":1, "compatible":True}
checks = {"overall bound":evidence["overall_mae"] <= 2.0,
          "critical bound":evidence["critical_mae"] <= 4.0,
          "critical minimum count":evidence["critical_count"] >= 3,
          "input compatibility":evidence["compatible"]}
failed = [name for name,passed in checks.items() if not passed]
print("failed", failed)
print("passes declared gate", not failed)

Expected output

failed ['critical bound', 'critical minimum count']
passes declared gate False

These illustrative thresholds are fixed by the fixture contract, not universal recommendations or statistical confidence guarantees.

Evaluate the procedure that will make decisions

Include fitted preprocessing, calibration, thresholds, abstention, batching behavior, and fallback if they affect the delivered result. Probability accuracy, calibration, and decision usefulness are different properties; a lower log loss does not automatically settle every requirement. Offline evaluation may miss interactions caused by users responding to predictions, but an online experiment also needs its own valid design. State the scope of each evidence source. A gate can block an unsupported candidate without declaring that its underlying architecture is universally bad. Rejection can reflect insufficient evidence, an incompatible package, or a specific unfulfilled requirement.

Preserve the gate report and its inputs

Link the result to candidate and baseline identities, cohort manifest, evaluator version, metric definitions, slice rules, threshold policy, and limitations. Test the evaluator itself with independently known small cases, including a missing slice and a candidate that improves the average but fails one required condition. Review material changes to gate policy rather than lowering thresholds after a disappointing score. These programs calculate fixed synthetic losses and Boolean checks only. They do not train models, run a model-analysis framework, estimate real population uncertainty, or authorize delivery to any live system.

Common errors and fixes

A lower overall loss overrides a required failing slice

Why it happens: The acceptance contract was reduced to one favorable aggregate.
How to fix it: Apply all declared required checks and report their evidence.
Prevent it next time: Keep an average-improves/slice-worsens fixture.

A missing metric is treated as a passing zero

Why it happens: Absent evidence was confused with a measured value.
How to fix it: Represent missing or insufficient evidence explicitly.
Prevent it next time: Test empty, delayed, nonfinite, and unsupported metric cases.

A minimum sample count is described as a confidence guarantee

Why it happens: A policy guard was mistaken for statistical adequacy.
How to fix it: Assess uncertainty and sampling assumptions separately.
Prevent it next time: Label count rules as one evidence requirement with limits.

Check your understanding

1. Why can the candidate’s overall error of 1.80 fail the fixture gate?

Show worked solution

The gate requires more than an overall value at most two. The critical error is five against a bound of four, and its count is one against the illustrative minimum of three. The result fails those declared conditions even though the overall mean improves; the numbers are synthetic policy examples.

2. What should a report say when delayed labels leave a critical slice unmeasured?