DevOps QA / Basic: delivery evidence and suite design / Lesson 1

Quality evidence in a delivery pipeline

Connect a pipeline result to a specific claim about a booking application, its artifact, and the risks that remain.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish execution status from evidence of product behavior.
Trace checks, artifacts, and decisions to one source revision.

Start with a claim and its limits

A green pipeline means configured jobs satisfied their configured success rules in one execution context. It does not establish that every requirement was tested, the environment represented users, or the released artifact matched the tested one. For a booking application, a useful claim is that a particular artifact rejects reservations above available capacity under a documented fixture. Name the behavior, evidence source, assumptions, and untested conditions. This turns a vague quality badge into a reviewable argument and helps teams decide which additional evidence matters before promotion.

Separate feedback from authorization

Continuous integration gives rapid feedback when changes are combined. Delivery readiness adds decisions about packaging, compatibility, security, operational risk, and recovery. A successful test job can supply evidence to a gate, while a gate applies policy to that evidence. Keep the distinction explicit: a report generator should not silently decide whether a high-impact change is safe. Different changes need different review depth. A documentation correction and a capacity-allocation algorithm can share a pipeline framework while requiring different checks and human judgment.

json An application-specific evidence record

{
  "claim": "capacity-boundary",
  "suite": "reservation-unit",
  "sourceRevision": "lab-revision-17",
  "artifactDigest": "lab-artifact-17",
  "result": "passed",
  "environment": "synthetic-single-process"
}

This is a teaching schema, not a CI vendor format or a real cryptographic digest. It records which claim was checked and where. A production implementation must use verified immutable artifact identifiers and validate required fields before accepting the record.

Keep revision and artifact identity connected

A source revision alone may not identify the deployed bytes because dependencies, build inputs, and packaging configuration also affect output. Build once where practical, identify the resulting artifact, and attach test evidence to that identity. Rebuilding later for release can invalidate an earlier claim unless reproducibility is established. Reports need enough metadata to reveal this mismatch. Keep synthetic environment identity, suite version, and execution time alongside results, while avoiding secret values and private user data. Traceability supports diagnosis as well as compliance.

Preserve the test process result

The runner communicates success through an exit status, while reports communicate detail. Accidentally replacing that status with a later successful shell command makes a failing suite appear green. Avoid blanket error suppression and understand how the selected shell handles pipelines. The example below saves a test command’s result, writes a small diagnostic marker, and returns the original status. It assumes the application already has a configured local test script. Artifact publication should run through the vendor’s failure-aware mechanisms without changing the test outcome.

bash Preserve status in a POSIX-style shell step

status=0
npm test -- --runInBand || status=$?
printf 'test_exit_status=%s\n' "$status" > test-status.txt
exit "$status"

The existing npm test script must support --runInBand, as Jest does; other runners need their own flags. The file is diagnostic evidence, not a substitute for a structured report. This excerpt is not a standalone pipeline and should be adapted to the configured runner.

Record what the decision actually means

A gate should distinguish pass, fail, missing evidence, and infrastructure failure. Missing results are not passing tests, and a runner outage is not automatically a product defect. State who owns triage and how an exception expires. Keep the decision proportional to risk while preserving an audit trail. At review time, ask which claim the evidence supports, whether it belongs to the candidate artifact, and whether unresolved risks are acceptable for the intended exposure. Quality engineering in delivery is the discipline of making these connections dependable.

Common errors and fixes

A failing suite produces a green job

Why it happens: A later shell command replaced its exit status.
How to fix it: Return the original status and publish reports separately.
Prevent it next time: Test the pipeline with an intentionally failing synthetic check.

Evidence refers to a different artifact

Why it happens: Release rebuilt the application after testing.
How to fix it: Promote the tested immutable artifact or revalidate the new one.
Prevent it next time: Carry artifact identity through every report and gate.

Absent reports are treated as success

Why it happens: The gate counts only reported failures.
How to fix it: Require expected suites and valid evidence metadata.
Prevent it next time: Model missing and infrastructure states explicitly.

Check your understanding

1. A pipeline passes 400 tests but skips the only capacity test. What claim can the team make?

Show worked solution

The executed cases passed in that context; capacity correctness remains unsupported. Identify why the required case was skipped and restore evidence before applying a capacity-risk gate. Test count alone says little about the risk the change introduced.

2. A test report has a revision but no artifact id. Design the minimum improvement.