Software QA / Intermediate: integrated product evidence / Lesson 12
Regression selection and unreliable tests
Choose regression checks from change impact and risk, diagnose intermittent failures, and maintain a suite that people can trust. Use a pricing change to connect confirmation, regression, and evidence gaps.
On this page
Lesson goals
- Select checks using dependencies and user risks.
- Investigate flaky behavior with preserved evidence.
- Make quarantine and maintenance decisions explicit.
A fix can disturb neighboring behavior
Confirmation testing reruns the original failure to show the intended fix works. Regression testing investigates unintended effects elsewhere. If the discount calculation changes, inspect shared pricing helpers, invoices, refunds, API responses, and localized displays, not only the modified line. Map dependencies and user journeys, then prioritize checks by impact, likelihood, and cost of missing a failure. A selected regression set is a reasoned sample; record excluded areas and why. “All tests passed” is incomplete unless the reader knows which tests, build, and environment produced that statement.
Worked case: a rounding change
Assume a reviewed rule rounds a 10% discount on 105 cents to a discount of 11 cents, leaving 94 cents. The original bug rounded inconsistently between the page and invoice. Confirmation compares both with 94. Regression includes zero discount, full discount if allowed, refund of the charged amount, and another currency precision supported by the application. Expected values come from the agreed rule, not the old implementation. Note that currency behavior is product-specific; the example teaches impact selection rather than prescribing a universal financial rounding policy.
Check | Purpose | Expected
105 cents, 10% | confirm original fix | page=invoice=94
105 cents, 0% | nearby regression | 105
Refund charged purchase | downstream regression | refund equals persisted charge
Localized view | presentation consistency | same amount; agreed formattingThe set states why each case exists; additional supported rules may need more cases.
Intermittence is evidence to investigate
A flaky test passes and fails without a sufficiently explained change in the tested behavior. Common causes include shared state, uncontrolled time, asynchronous races, environment differences, and overly strict comparisons. Product intermittence can look the same, so do not assume the test alone is defective. Preserve the first failure’s trace, data identifiers, seed, timing, and environment. Compare a failing run with a passing one and change one suspected factor at a time. Repeating until green conceals frequency and can teach the team to ignore a genuine regression.
Worked case: an ordering dependency
A test expecting three courses passes alone but fails after a test that creates a fourth course. Run the pair in both orders against a clean isolated store. If failure follows the creator, inspect fixture cleanup and whether the reader’s scope is too broad. The fix may be independent namespaces and precise queries rather than deleting a legitimate assertion. Verify each case alone and the pair after repair. This demonstrates a specific dependency; it does not justify rerunning the entire suite repeatedly once the cause and targeted verification are settled.
Maintain confidence while repairing the signal
If an unstable check must be quarantined, give it an owner, issue, expiry or review date, and an alternative for the risk it covered. Keep its failures visible outside the blocking set; quarantine is a temporary management choice, not a fix. Track time spent diagnosing and the frequency of instability to prioritize repair. Remove obsolete tests when requirements disappear, but retain the historical reason where useful. A smaller reliable suite plus explicit gaps is more interpretable than an impressive count of checks nobody trusts enough to investigate.
Common errors and fixes
A fix is checked only with its original input
- Why it happens
- Change impact beyond the reported defect is ignored.
- How to fix it
- Add selected checks for shared dependencies and neighboring rules.
- Prevent it next time
- Review an impact map for behavior changes.
Retries convert failure into a silent pass
- Why it happens
- First-failure evidence and frequency are discarded.
- How to fix it
- Retain the failure and classify rerun outcomes explicitly.
- Prevent it next time
- Investigate instability with an owner and time limit.
Quarantine becomes permanent deletion of risk
- Why it happens
- No owner, deadline, or substitute evidence exists.
- How to fix it
- Restore visibility and an explicit repair plan.
- Prevent it next time
- Review quarantined risks alongside release evidence.
Check your understanding
1. A date-format fix touches a shared formatter. Choose three regression areas.
Show worked solution
Check a normal schedule date, an expiry or cancellation boundary with an explicit zone, and localized invoice/report presentation if they share the formatter. Derive each expected value from its requirement. Include the original reproducer for confirmation and record untested consumers of the shared helper.
2. A flaky test is retried five times and eventually passes. What should the report say?
Show worked solution
Report the initial failure and rerun outcomes, not an unqualified pass. Preserve evidence and state whether the risk is understood, assigned, or temporarily covered another way. A final successful attempt does not establish that earlier failures were harmless.
Key takeaways
- Select regression from risk and dependencies, and explain exclusions.
- Repair unreliable signals instead of erasing their observations.