API Testing / Advanced: versioned evidence and review / Lesson 17

Make CI selection and results auditable

Plan deterministic checks, explicit selection rules, and evidence artifacts without running a hosted pipeline.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish a passing selected suite from complete intended coverage.
Bind artifacts to source, fixtures, and the actual executed cases.

Treat selection as part of the test logic

A pipeline can be green because the relevant tests were never selected. Path filters, tags, naming conventions, and conditional jobs are themselves logic that can be wrong. Define a small always-required contract set and a reviewed mapping from changed areas to additional checks. Cross-cutting changes to authentication, serialization, or shared schemas may affect many operations even when only one file changes. Record the selected case identifiers and skipped reasons so a reviewer can distinguish a deliberate boundary from an accidental omission. “No failures” and “all intended checks executed” are separate claims.

Review a deterministic selection model

The first example maps a schema change to both representation and consumer checks while retaining a core refusal case. It checks selection only; none of the named tests is executed by this program. The mapping is fictional and finite, so it cannot establish that a repository’s real impact analysis is complete. Use such a model to expose assumptions for review and include a case for unknown change categories. A conservative fallback can be appropriate when impact is uncertain, but the actual policy should consider meaningful coverage and available CI resources.

python A local change-to-check selection model

core = {"unknown-book-refusal"}
impact = {"schema": {"representation-shape", "consumer-title"}, "policy": {"owner-read", "cross-tenant-denial"}}
def select(changes):
    selected = set(core)
    for change in changes:
        selected |= impact.get(change, set().union(*impact.values()))
    return selected

assert select(["schema"]) == {"unknown-book-refusal", "representation-shape", "consumer-title"}
assert "cross-tenant-denial" in select(["unclassified"])
print("selection policy checked")

Expected output

selection policy checked

The complete program checks the declared set-selection rules and unknown-category fallback. It does not discover changed files, run a test, create CI jobs, or evaluate the correctness of the named contract checks.

Bind evidence to immutable inputs

A useful result artifact identifies source revision, test inventory, fixture revision, runtime, and boundary assumptions. A content digest can help detect accidental artifact changes, but a digest alone does not establish authenticity or approval. The second program fingerprints a tiny synthetic manifest with deterministic JSON serialization. It demonstrates repeatability and change detection only. A real evidence chain needs trusted build provenance and access policy in addition to hashes. Avoid embedding secrets or private payloads in artifacts merely because the artifacts have restricted access or a short retention period.

python Fingerprint a synthetic evidence manifest

import hashlib, json
def digest(value):
    raw = json.dumps(value, sort_keys=True, separators=(",", ":")).encode("utf-8")
    return hashlib.sha256(raw).hexdigest()

manifest = {"fixture":"catalog-lab-1", "cases":["owner-read", "unknown-book-refusal"]}
assert digest(manifest) == digest(dict(reversed(list(manifest.items()))))
changed = {**manifest, "fixture":"catalog-lab-2"}
assert digest(manifest) != digest(changed)
print("manifest change detection checked")

Expected output

manifest change detection checked

The complete program performs local SHA-256 calculations on classroom data. It does not sign an artifact, attest a build, or verify repository provenance. Canonicalization here is sufficient for these simple objects, not a universal cross-language canonical JSON standard.

Keep retries from hiding weak evidence

Automatic test reruns can help collect diagnostic information, but converting any eventual pass into an unconditional green result hides instability. Preserve first-failure artifacts and distinguish infrastructure interruption, fixture leakage, and a product race. Quarantining a flaky case needs an owner, reason, expiry review, and replacement coverage for important promises. Do not solve a failing access-control test by excluding the entire security tag. A reliable pipeline reports both the test outcome and whether the evidence is complete enough for the intended decision.

Separate an authored plan from a pipeline run

Hosted CI behavior depends on workflow syntax, runner environment, permissions, dependency provenance, and event context. Those are deployment-specific assumptions and are not exercised in this lesson. An approved implementation should use minimal required permissions and isolated test resources, preserve actual exit results, and publish only reviewed nonsecret evidence. Keep commands and workflow fragments non-runnable until the complete environment and authorization are defined. The local selection and hashing checks provide narrow reproducible evidence; they do not mean a hosted workflow ran or that a release is approved.

Common errors and fixes

A green run contains zero relevant tests

Why it happens: Selection filters omit the changed contract.
How to fix it: Compare intended and executed inventories and correct the mapping.
Prevent it next time: Test selection rules and keep an always-required core.

An eventual pass erases the first failure

Why it happens: Rerun policy hides instability and evidence.
How to fix it: Preserve the first mismatch and classify the cause.
Prevent it next time: Track quarantines and require reviewed replacement coverage.

A digest is described as proof of trusted execution

Why it happens: Integrity comparison is confused with provenance.
How to fix it: State what was hashed and obtain trusted execution evidence separately.
Prevent it next time: Keep authenticity, authorization, and integrity claims distinct.

Check your understanding

1. The schema tests passed but consumer checks were skipped. What should the release evidence say?

Show worked solution

State that schema checks passed and consumer compatibility remains unverified for this run. Inspect why selection skipped those cases and run the authorized missing checks before claiming that compatibility promise. A green status cannot substitute for a complete intended inventory.

2. Two manifests have the same SHA-256 value. Does that prove the tests ran?