API Testing / Advanced: versioned evidence and review / Lesson 18

Review a versioned API evidence package

Integrate contract, negative cases, compatibility, and honest limitations into one reviewable decision.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Trace an important promise through cases, observations, and conclusions.
Identify incomplete evidence without inventing service-level guarantees.

Choose a coherent change scenario

The fictional catalog is adding an optional reading note to an existing add-book operation. Supported callers may continue sending only bookId, explicit null clears the note where permitted, and text is bounded under a documented unit. Ownership still comes from trusted subject context. Duplicate additions must not create another entry or notification. This change connects representation, business state, permission, and compatibility without requiring a real service. Begin by writing those promises and their assumptions as a versioned contract before selecting tools or producing a release recommendation.

Trace each promise to a case and observation

A traceability table should identify the promise, case, observation, and boundary that supplied evidence. The first program checks that each required promise has an assigned case and that the evidence state is not silently absent. It deliberately allows pending evidence as a recorded status; that is different from a pass. The program verifies bookkeeping only. It does not execute the named cases or establish their quality. A complete package must contain actual results and enough context for a reviewer to assess whether each result supports the stated promise.

python A synthetic promise-to-evidence inventory

required = {"old-request-accepted", "owner-from-context", "duplicate-no-effect"}
evidence = {"old-request-accepted":{"case":"omit-note","status":"local-pass"}, "owner-from-context":{"case":"trusted-owner","status":"local-pass"}, "duplicate-no-effect":{"case":"duplicate-state","status":"pending-integration"}}
assert required <= evidence.keys()
assert all(evidence[key]["case"] and evidence[key]["status"] for key in required)
assert evidence["duplicate-no-effect"]["status"] != "local-pass"
print("coverage inventory complete; integration remains pending")

Expected output

coverage inventory complete; integration remains pending

The complete program validates inventory presence and preserves a pending integration status. The statuses are classroom data, not results from a service. The output explicitly prevents completeness of the table from being confused with completeness of verification.

Keep conclusions no stronger than evidence

A release review can accept a local policy result while withholding a claim about persistence or runtime integration. The second example requires every named critical boundary to be verified before its fictional readiness function returns ready. It does not automate a real release policy and cannot decide business risk on behalf of an owner. Missing evidence, known failure, and an explicit accepted exception are different states and should not all become a false boolean with no explanation. Preserve the reason and accountable decision alongside any exception.

python A local evidence-completeness decision

critical = {"contract", "permission", "persistence"}
def readiness(states):
    missing = sorted(key for key in critical if states.get(key) != "verified")
    return {"ready": not missing, "needsEvidence": missing}

result = readiness({"contract":"verified", "permission":"verified", "persistence":"not-executed"})
assert result == {"ready": False, "needsEvidence": ["persistence"]}
print("readiness blocked by unverified persistence")

Expected output

readiness blocked by unverified persistence

The complete program checks one fictional decision rule. It performs no deployment, merge, publication, or service verification. The critical boundary list must be tailored and approved for any actual change rather than copied as a universal release checklist.

Read failures as contract questions

Suppose an old caller omitting note is rejected. First determine whether the new implementation wrongly requires the field or whether the approved compatibility promise changed through a reviewed migration. Do not update the expected failure automatically. If a duplicate request sends a notification, identify the side-effect boundary and verify a narrow repair. Include adjacent positive and negative cases so a fix does not simply disable creation. Review schema, semantic units, permission, and state effects together while retaining focused cases that explain each independent property.

Write a package another reviewer can use

The final package should contain the contract revision, supported consumer assumptions, fixture definitions, executed case inventory, actual outputs, unresolved findings, and boundary limitations. State exactly which local programs ran and which schema, HTTP, middleware, storage, concurrency, or hosted CI checks did not. Official references explain semantics; they do not replace implementation evidence. A concise conclusion can say the local policy examples passed while integration remains pending. That honest result is more actionable than an unsupported assurance that the entire API is secure, compatible, fast, and ready.

Common errors and fixes

A complete checklist is treated as completed testing

Why it happens: Assigned cases are mistaken for executed evidence.
How to fix it: Store actual result and boundary status separately from case presence.
Prevent it next time: Require traceability to concrete outputs and unresolved findings.

An old caller is silently removed from support

Why it happens: A failing compatibility case is updated without a product decision.
How to fix it: Review the support promise and migration before changing expectations.
Prevent it next time: Version consumer assumptions and retain relevant old-request cases.

Local success becomes a universal readiness claim

Why it happens: Unexecuted integration boundaries disappear from the report.
How to fix it: List verified and pending boundaries with their actual evidence.
Prevent it next time: Keep conclusion strength aligned with observation scope.

Check your understanding

1. The inventory covers every promise but persistence is not executed. Can the package claim full verification?

Show worked solution

No. Coverage planning is complete, while execution evidence remains incomplete. Report the local results and the pending persistence question explicitly. The accountable release owner may evaluate risk using approved policy, but the test report must not relabel missing evidence as a pass.

2. Write the conclusion for this course’s executable examples.

Show worked solution

The complete standard-library programs passed their stated synthetic local assertions. They do not establish actual HTTP parsing, authentication integration, schema-engine behavior, storage atomicity, worker delivery, service performance, or hosted CI execution. Those boundaries require separate authorized evidence before any corresponding deployment claim.

Key takeaways

Trace each promise to actual evidence and its boundary.
A complete plan and a complete execution are different achievements.
Conclusions must preserve pending and unexecuted work.