DevOps QA / Basic: delivery evidence and suite design / Lesson 6

Pipeline graphs, reports, and artifact gates

Make a pipeline preserve useful failure evidence and enforce the checks actually required for a candidate.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish job dependencies, caches, artifacts, and report ingestion.
Handle missing, failed, skipped, and cancelled work explicitly.

Model dependencies rather than decorative stages

A pipeline is a graph of work and evidence dependencies. A component test may need a build artifact while a static check can start immediately. A reporting job may need results from several shards, including failed ones. Distinguish ordering from artifact transfer; depending on a job does not always provide its files automatically. Cancellation, skipped jobs, and conditional execution complicate the graph. Write down which outputs the final gate expects and how each state is classified before writing vendor YAML. This prevents a visually tidy pipeline from hiding a missing evidence path.

Publish diagnostics without changing the verdict

JUnit XML and browser reports are evidence formats, not universal definitions of job success. A CI provider can ingest a test report while still relying on the script exit status. GitLab’s artifacts:reports:junit declares report files; artifacts:when controls ordinary artifact collection policy. Verify behavior on the actual GitLab version and runner. The example names a pre-existing test command that must generate the declared file. It is intentionally a job fragment, so it cannot be pasted into an unrelated repository and assumed to work.

yaml GitLab CI job fragment reviewed against current documentation

component_tests:
  stage: test
  script:
    - python -m pytest tests/component --junitxml=reports/component.xml
  artifacts:
    when: always
    paths:
      - reports/
    reports:
      junit: reports/component.xml
    expire_in: 7 days

Requires a configured runner environment containing the pinned application and pytest dependencies, an existing test stage, and synthetic component fixtures. Seven days is a teaching retention choice, not a universal policy. Report upload does not suppress a nonzero pytest result.

Give caches and artifacts different trust

A cache accelerates reusable work and may be absent, stale, or restored under a fallback key. An artifact is an identified output retained for transfer, inspection, or promotion. Do not treat a cache hit as proof that dependencies or generated code match the current revision. Key caches by relevant lockfiles and runtime dimensions, and keep untrusted writes away from privileged jobs. Build outputs used for promotion need stronger identity and integrity checks than an opportunistic dependency cache. Keep sensitive test traces under deliberate retention and access rules.

Require complete evidence explicitly

The final gate should compare expected suite names with received results and verify identity before evaluating outcomes. A missing shard, malformed report, or cancelled job must not vanish from the denominator. The example is a small pure policy function; it deliberately returns hold for incomplete evidence. Real ingestion also validates authenticity, freshness, duplicate records, and artifact identity. Keep this logic in reviewed trusted code rather than accepting arbitrary policy generated by an untrusted branch with access to release privileges.

python A conservative teaching gate

def gate(expected, results):
    if set(results) != set(expected):
        return "hold: incomplete evidence"
    if any(value != "passed" for value in results.values()):
        return "hold: review results"
    return "eligible: evidence complete"

Inputs are already-validated synthetic suite results. eligible means this evidence policy is satisfied, not that the product is defect-free or deployment is authorized. Add artifact matching and provenance validation before using any comparable logic in a real pipeline.

Design a report for the next maintainer

A useful failure bundle contains the failing case, first relevant error, redacted logs, environment manifest, artifact identity, seed, and a bounded trace when needed. Huge unstructured archives increase triage time and may leak data. Distinguish a report publication failure from the original test failure so the root cause is not overwritten. Validate report parsers with a known failing synthetic suite and a deliberately missing file. The gate is only as dependable as the evidence path that feeds it, so report handling deserves its own small verification scenarios.

Common errors and fixes

A report is uploaded but the job is green incorrectly

Why it happens: The runner result was ignored or suppressed.
How to fix it: Preserve the test exit status independently of publication.
Prevent it next time: Verify a known failing synthetic case end to end.

One shard disappears from the final decision

Why it happens: Aggregation evaluates only files that arrived.
How to fix it: Compare against the expected shard manifest.
Prevent it next time: Treat missing and cancelled evidence as explicit states.

Cached output from another revision is promoted

Why it happens: A performance cache was trusted as a release artifact.
How to fix it: Use an immutable verified build output for promotion.
Prevent it next time: Separate cache policy from artifact identity policy.

Check your understanding

1. All three received shard reports pass, but the plan expected four. What is the gate outcome?

Show worked solution

Hold for incomplete evidence. Determine whether the fourth shard was skipped, cancelled, failed before reporting, or never scheduled. Do not report a 100% passing suite by changing the denominator to the three convenient reports.

2. A trace archive contains session tokens. What changes belong in the reporting design?