Jenkins / Evidence, reuse, and authority / Lesson 7
Test reports and build-result policy
Keep test execution, JUnit publication, unstable results, and release gates distinct.
On this page
Lesson goals
- Explain why report publication is different from running tests.
- Define required-result handling for failed, unstable, absent, and aborted evidence.
A publisher does not execute assertions
A test command produces observations and often a machine-readable report. Jenkins can publish supported JUnit XML through the JUnit plugin, but publication is not the test itself. Parcel’s unit suite checks label normalization, while the publisher records cases and failures for navigation and history. If a test command never ran, an old report in the workspace can be misleading. Bind reports to the run and source, ensure the intended producer created them, and distinguish missing report from zero failed tests.
Success and unstable carry different promises
JUnit publication can mark a build unstable when test failures are reported under its documented defaults and options. A command may also fail independently, and an interrupted run can be aborted. Decide which outcomes block downstream release. An organization can intentionally retain a completed but unstable validation run while refusing promotion; it should not silently equate unstable with successful required evidence. Declarative options and stage conditions need review against that policy. Colors and familiar status labels are presentation, while the release contract must state acceptable observations explicitly.
Make report paths and absence meaningful
The following fragment publishes a declared report pattern and does not allow empty results. It assumes the JUnit plugin and matching newly produced files in an allocated workspace. It is not executed and has no fabricated Jenkins result. Choose the report pattern narrowly enough to avoid unrelated or stale suites. If reports must survive a failed command, place publication in an appropriate reviewed post or result-handling path while preserving command failure. Setting allowEmptyResults merely to silence a path error can erase required evidence.
junit testResults: "reports/unit-*.xml", allowEmptyResults: falseThis invokes a plugin step only in a real authorized Pipeline. It does not run tests, create reports, or prove the XML schema and semantics of an arbitrary producer. No plugin installation occurs here.
Inspect a small explicit report locally
The complete local program below parses a synthetic XML suite and counts failure elements. It compares the declared test count with observed testcase elements. This is a narrow structural exercise, not validation of every JUnit dialect or Jenkins plugin behavior. Producers differ in time units, skipped cases, nested suites, and error representation. Use the publisher’s documented format expectations and verify representative real reports in a separately authorized lab when needed. A well-formed XML document can still describe the wrong run or incomplete coverage.
from xml.etree import ElementTree as ET
xml = '<testsuite tests="2"><testcase name="trim"/><testcase name="uppercase"><failure message="expected uppercase"/></testcase></testsuite>'
suite = ET.fromstring(xml)
cases = suite.findall("testcase")
print(len(cases) == int(suite.attrib["tests"]))
print(sum(c.find("failure") is not None for c in cases))True
1The program reads only the embedded XML string. One failure is intentionally present; no real test suite or Jenkins publisher is run.
Use fresh complete evidence for the decision
A release gate should know the required suites, their producer revision, acceptable result states, and report freshness. A successful upload is evidence that publishing worked, not that the tests passed. Avoid retries until green without retaining earlier attempts, and assign instability an owner rather than hiding it in optional reporting. Keep application failure and report-channel failure separately actionable. After correcting a report path, verify that the intended cases appear and the known failing synthetic case remains capable of blocking the required contract.
Common errors and fixes
Publishing JUnit XML is described as running the unit tests.
- Why it happens
- The report consumer was confused with the assertion producer.
- How to fix it
- Run the declared suite and publish its newly produced report separately.
- Prevent it next time
- Document command, report path, and result policy together.
allowEmptyResults hides a missing required report.
- Why it happens
- A diagnostic path failure was converted into acceptable absence.
- How to fix it
- Correct the producer or path and preserve missing-evidence rejection.
- Prevent it next time
- Exercise the no-report case in a controlled validation plan.
An unstable required suite is promoted as if it passed.
- Why it happens
- The release policy relies on a vague green-or-not-failed interpretation.
- How to fix it
- Require explicitly acceptable outcomes for all mandatory suites.
- Prevent it next time
- Review success, unstable, failed, skipped, and aborted cases.
Check your understanding
1. A stale report has zero failures while the current test command never started. Is the candidate tested?
Show worked solution
No. The report is not evidence for the current execution. Establish producer identity and run freshness, then obtain the required suite result. A valid XML file and successful publication do not connect it to the intended source.
2. Why retain an unstable run instead of deleting it after a successful rerun?
Show worked solution
It contains useful failure and instability evidence. Preserve attempt history, investigate the cause, and apply the declared retry policy. A later pass can be relevant without erasing the fact that an earlier identical-input attempt failed.
Key takeaways
- Test execution, publication, and release eligibility are separate.
- Missing or unstable required evidence needs an explicit policy.