Software QA / Advanced: strategy, resilience, and leadership / Lesson 17

Coverage, reliability metrics, and release evidence

Read quality measurements without confusing activity with assurance. Build a release evidence summary that connects user outcomes, coverage, known defects, and reliability objectives.

About 4 min readContent reviewed 2026-10-23

On this page

Lesson goals

Interpret coverage and defect metrics with denominators.
Calculate a simple success indicator and error budget.
Write a bounded evidence-based release recommendation.

Every metric needs a question

Test count measures activity, execution pass rate measures outcomes for the executed set, and coverage measures a chosen relation to code or requirements. None directly measures total product quality. Ask which population, time window, exclusions, and collection method define each number. A rising defect count might mean worsening software, better investigation, or a broader scope. Use trends with context instead of ranking people by defects found. Choose a few measurements that inform decisions, such as unresolved high-impact risks, escaped defect themes, and investigation time lost to unreliable tests.

Worked case: the changing denominator

A team planned 120 cases, executed 100, passed 98, failed two, and did not execute 20. The executed pass rate is 98%, while execution completion is about 83.3%; neither says that 98% of the product is correct. If the 20 omitted cases include all refund-retry checks, the release may retain a serious gap despite the attractive pass rate. Add requirement and risk mapping, and explain blocked cases separately. Code coverage can still be high when tests execute lines without checking meaningful outcomes.

text Three denominators, three different claims

Quantity | Calculation | Meaning
Executed pass rate | 98 / 100 = 98% | outcomes of executed cases
Execution completion | 100 / 120 ≈ 83.3% | planned cases actually executed
Unexecuted | 20 / 120 ≈ 16.7% | no execution evidence
Product correctness | not derived | requires risk and oracle analysis

Show the missing refund evidence alongside these figures; a percentage alone cannot rank its harm.

Measure the user outcome you promised

A service-level indicator is a defined measurement; an objective sets a target for it over a window. For enrollment, “valid attempts completed correctly” may be more useful than “homepage returned 200.” Define eligibility for the denominator, what counts as good, handling of timeouts, and data collection limits. A request-based objective differs from a time-based availability objective; do not convert between them casually. Reliability measurement complements pre-release testing with observations from real operation, but historical achievement does not guarantee the next release or protect every user subgroup equally.

Worked case: request-based error budget

For an illustrative objective of 99.5% good eligible attempts in a fixed window, 20,000 attempts allow 100 bad attempts at the target boundary. If 60 are bad, 40 remain under that simple request-count budget. Specify that the numbers are synthetic and that future traffic changes the eventual denominator. The budget is a decision aid for balancing reliability work and change, not permission to harm users or a prediction of failure probability. Inspect whether failures concentrate in one language or payment method even if the overall objective is met.

python Synthetic request-count budget at a fixed denominator

eligible = 20_000
good = 19_940
allowed_bad = eligible * 5 // 1000
observed_bad = eligible - good
assert allowed_bad == 100
assert observed_bad == 60
assert allowed_bad - observed_bad == 40

The 0.5% allowance is specific to this example. It is not a recommended target for every service.

Write evidence that supports a decision

A release summary identifies build and configuration, changed behavior, important risks, executed evidence, unresolved defects, blocked checks, and limitations. State a recommendation with conditions: proceed with a disabled export feature, delay for missing authorization evidence, or release after a specified fix is confirmed. Identify who owns the decision and follow-up observations. Link representative cases and defect records rather than overwhelming readers with raw logs. Keep “not tested,” “failed,” and “passed” distinct, and avoid a certification-style stamp. The purpose is to make uncertainty and consequences understandable to accountable decision makers.

Common errors and fixes

Pass rate is presented as product correctness

Why it happens: The executed-case denominator is hidden.
How to fix it: Report planned, executed, failed, blocked, and risk mapping.
Prevent it next time: Require a definition beside every percentage.

An SLO excludes timeouts without explanation

Why it happens: The denominator silently removes harmed users.
How to fix it: Define eligible attempts and classify timeouts consistently.
Prevent it next time: Review indicator definitions with user representatives.

A release report hides untested critical paths

Why it happens: Status is optimized for reassurance rather than decisions.
How to fix it: Expose gaps, consequences, and concrete options.
Prevent it next time: Use an evidence template that requires limitations.

Check your understanding

1. Ninety tests pass and ten are blocked. Is the pass rate 100%?

Show worked solution

It is 100% of executed tests if all 90 executed cases passed, but execution completion is 90% of the planned 100. Report the ten blocked cases and their risks. Calling the release 100% passed without the denominator hides absent evidence.

2. At a fixed 10,000 attempts and 99% objective, 80 are bad. Calculate the simple remaining budget.