API Testing / Advanced: versioned evidence and review / Lesson 15

Build failure evidence that preserves context

Connect assertions, synthetic event records, and boundary hypotheses without exposing private payloads.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish a symptom from a supported causal explanation.
Design minimal structured diagnostics and compare attempts with effects.

Separate observation from explanation

A failed assertion is an observation, not automatically a diagnosis. “Expected one entry, observed two” supports a duplicate-effect symptom; it does not yet identify a retry bug, transaction error, or repeated fixture setup. List plausible boundaries and seek the smallest evidence that distinguishes them. Record the case name, fixture version, operation, and redacted difference before rerunning anything. An unexplained rerun that passes can erase the only useful trace. Keep uncertainty visible and avoid assigning a root cause merely because a familiar error message appears.

Prefer allowlisted diagnostic records

A structured diagnostic can carry operation, case, outcome category, and a synthetic correlation marker without copying the entire request. An allowlist is often easier to review than attempting to redact every dangerous key after the fact. The first example selects safe fields from a local observation that also contains a classroom-only private note. It does not configure logging handlers or write a log file. The exact fields belong to this exercise; real systems require reviewed retention, access, and disclosure rules in addition to careful field selection.

python Check a minimal synthetic diagnostic projection

observation = {"case":"duplicate-add", "operation":"add-book", "outcome":"conflict", "correlation":"lab-17", "privateNote":"classroom-only"}
allowed = ("case", "operation", "outcome", "correlation")
diagnostic = {key: observation[key] for key in allowed}
assert "privateNote" not in diagnostic
assert diagnostic["case"] == "duplicate-add"
assert len(diagnostic) == 4
print("diagnostic allowlist checked")

Expected output

diagnostic allowlist checked

The complete program checks the selected field projection only. It does not prove that every logging path is safe. Correlation values must also be reviewed for privacy and trust rather than accepted blindly from arbitrary callers.

Count attempts and committed effects separately

Two request attempts may legitimately map to one committed action under a replay policy. Conversely, one visible request can produce multiple effects through a faulty consumer. Use distinct event categories and stable synthetic business identifiers to reconstruct the relationship. The second example groups local records and confirms that two attempts correspond to one commit. It demonstrates an evidence query, not an event-delivery guarantee. Missing logs can reflect incomplete instrumentation rather than missing work, so corroborate important effects with the protected state at the proper boundary.

python Interpret a synthetic event sequence

events = [{"kind":"attempt","action":"a1"}, {"kind":"attempt","action":"a1"}, {"kind":"commit","action":"a1"}]
attempts = sum(e["kind"] == "attempt" for e in events if e["action"] == "a1")
commits = sum(e["kind"] == "commit" for e in events if e["action"] == "a1")
assert (attempts, commits) == (2, 1)
print("two attempts and one commit observed")

Expected output

two attempts and one commit observed

The complete program computes counts from three explicit local records. It neither receives real telemetry nor proves events are complete or trustworthy. The commit category must be tied to a documented durable boundary in any real diagnosis.

Use temporal evidence with appropriate clocks

Wall-clock timestamps help correlate records, but clock adjustments and skew can make them unsuitable for measuring durations across components. Use an appropriate monotonic source for local elapsed time and document units. An asynchronous failure may involve queue delay, processing duration, or observation delay; one total duration cannot isolate all three. Do not subtract timestamps from unsynchronized machines and call the result exact latency. The course performs no timing experiment. Instead, it teaches which measurements an approved diagnostic plan would need before making a causal or performance claim.

Verify the causal hypothesis after a narrow repair

Create a minimal synthetic reproduction that fails for the suspected defect, apply the narrow correction, and rerun the relevant case. Include an adjacent case to ensure the correction did not simply bypass validation or suppress every event. Preserve the before/after evidence and state which hypothesis it supports. If the reproduction does not fail before the change, it may not explain the incident. Do not replace the expected output with the latest observation merely to obtain a green result. A useful diagnostic conclusion remains tied to its actual observations and limitations.

Common errors and fixes

A passing rerun closes an unexplained failure

Why it happens: The original observation was not preserved.
How to fix it: Capture the mismatch and reproduce the decisive boundary.
Prevent it next time: Keep failure artifacts and distinguish flaky symptoms from causes.

Request bodies are copied into every failure report

Why it happens: Convenient context bypasses privacy review.
How to fix it: Use a reviewed allowlist and synthetic reproductions.
Prevent it next time: Test diagnostic projections and restrict retained evidence.

Attempt count is reported as committed work

Why it happens: Event categories do not distinguish transport and business effects.
How to fix it: Correlate attempts with durable effects using the documented boundary.
Prevent it next time: Keep separate counters and verify state independently.

Check your understanding

1. Two attempts and one commit appear in logs. Does that prove replay is correct?

Show worked solution

It is consistent with replay, but log completeness and commit meaning must be established. Compare protected state and relevant side effects, then confirm caller scope and payload binding. Three local event records alone cannot establish the behavior of a deployed replay store.

2. A report says the database caused a duplicate because the API returned two ids. Improve it.