Generative AI / Intermediate: retrieval and controlled workflows / Lesson 9

Evaluation fixtures and error analysis

Define representative tasks, labels, metrics, and regression decisions without turning a selected example or one average into a universal quality claim.

About 5 min readContent reviewed 2026-10-26

On this page

Lesson goals

Explain the assumptions and trust boundaries of evaluation fixtures and error analysis.
Interpret both examples and distinguish offline fixture results from model or production evidence.
Diagnose the listed failures and justify the worked decisions while preserving task, data, permission, and uncertainty contracts.

Define what success means before comparing systems

An evaluation needs a task, input distribution, allowed evidence, accepted output, failure policy, and scoring rule. Different properties require different measures: retrieval relevance, factual support, format validity, authorization, abstention, latency, and cost should not be collapsed without justification. A system can improve one and harm another. The first offline example computes retrieval precision and recall for hand-authored relevant IDs. It verifies a metric calculation, not a model benchmark. Preserve the denominator and scenario so a percentage cannot hide excluded failures or a changed test population.

python Offline retrieval precision and recall

relevant = {"doc-a", "doc-b"}
ranked = ["doc-a", "doc-x", "doc-b"]
for k in [2, 3]:
    selected = ranked[:k]
    hits = len(set(selected) & relevant)
    precision = hits / len(selected)
    recall = hits / len(relevant)
    print(k, f"{precision:.2f}", f"{recall:.2f}")

Expected output

2 0.50 0.50
3 0.67 1.00

At two results, precision and recall are 0.50; at three, recall becomes 1.00 while precision is 0.67. These are exact fixture metrics, not observed model quality.

Build representative and protected fixtures

Include common requests, difficult boundaries, multilingual cases, rare terms, conflicting versions, missing evidence, and prohibited actions where relevant. Keep development examples separate from held-out evaluation, grouping related source documents or users when leakage is possible. Repeatedly tuning against a visible test set can overfit it even without weight training. Version the corpus, labels, prompts, model, retrieval configuration, and validators. The second synthetic protocol record states those identities and a release gate. It is not an evaluation framework configuration and does not claim any actual system passed.

yaml Synthetic evaluation protocol

evaluation_revision: lab-eval-4
held_out_groups: separated-from-prompt-development
corpus_revision: lab-corpus-7
required_checks: [support, format, access, abstention]
release_gate: no-required-check-regression-without-review
measurement_status: no-real-model-run-in-this-record

The record names reproducibility and review requirements. It does not execute a benchmark, supply gold labels, or certify that a candidate passed.

Use labels and rubrics that can be audited

A reference answer can be incomplete, stale, or wrong, so keep source evidence and a review process for ambiguous cases. Human raters need a concrete rubric and examples of supported, unsupported, incomplete, and correctly abstained responses. Record disagreement rather than manufacturing certainty through an unexplained majority. Automated graders can help scale checks but can share model biases, reward style, or be influenced by untrusted text. Validate them against independent labels for the intended task. A judge score is another measurement with error, not a replacement for application authorization or proof that every factual claim is true.

Analyze slices and failures before one winner

Report important task and user slices with adequate sample context. An aggregate improvement can hide a severe regression in a low-volume language, tenant rule, or unanswerable case. Compare candidates on the same frozen cases and inspect paired errors, variability, and measurement uncertainty. A tiny fixture or one generated answer cannot justify a broad superiority claim. Preserve failure artifacts safely, with sensitive input redacted and access controlled. Separate stage failures so a missing document is not scored as only a generation problem and an invalid tool proposal is not treated as an authorized action merely because its wording was useful.

Turn scores into an accountable release decision

Set acceptance and stop criteria before observing results, with mandatory safety or correctness gates that cannot be averaged away. Name the reviewer, known limitations, rollback or fallback behavior, and follow-up monitoring. Real model and performance evaluation must be authorized and use the chosen runtime; none occurs in this lesson. Both examples are offline arithmetic or non-runnable editorial metadata. Their purpose is repeatable evaluation design and honest reporting. A trustworthy decision can explain what was measured, what changed, and what remains unknown, rather than presenting a leaderboard number without the task, evidence, or failure costs that give it meaning.

Common errors and fixes

One selected answer is called a benchmark

Why it happens: Anecdote replaced a predefined representative test.
How to fix it: Use frozen cases and explicit scoring with limitations.
Prevent it next time: Record selection and development history.

An average hides a mandatory failure

Why it happens: Critical properties were combined into one score.
How to fix it: Keep nonnegotiable gates and slice-level evidence.
Prevent it next time: Define acceptance before seeing results.

The evaluator is treated as infallible

Why it happens: Judge and label error were ignored.
How to fix it: Validate scoring against independent reviewed evidence.
Prevent it next time: Track disagreement and evaluator version.

Check your understanding

1. Adding one irrelevant retrieved item raises recall but lowers precision. Is the system necessarily better?

Show worked solution

No. The application may need the extra relevant passage, but distractors, context budget, latency, and support quality also matter. Compare the end-to-end task and access constraints on the same cases instead of optimizing one retrieval metric in isolation.

2. A candidate improves overall accuracy but leaks one protected source. How should a release gate behave?