DevOps QA / Intermediate: trustworthy validation systems / Lesson 8

Flaky-test diagnosis and bounded quarantine

Separate intermittent test behavior from deterministic defects and infrastructure outages while preserving the original failure evidence.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Classify failures using controlled comparisons.
Use retries and quarantine without laundering risk.

Define flakiness carefully

A flaky test produces inconsistent outcomes under inputs believed equivalent. The uncertainty may come from timing, shared state, environment drift, an unstable assertion, or genuine product nondeterminism. A deterministic defect that happens only on one supported platform is still a defect, not automatically flaky. An infrastructure outage is another category. Start with artifact identity, case inputs, worker placement, environment manifest, and first failure evidence. If those differ, determine whether the difference explains behavior before treating repeated execution as a statistical mystery.

Keep every attempt visible

Retries can collect diagnostic evidence and reduce disruption while a known intermittent condition is investigated. They also change the meaning of a passing run and consume capacity. Playwright distinguishes tests that pass initially, pass after retry, and continue failing. Its worker restart behavior matters because a retry may run with a cleaner process, hiding state leakage. Keep the initial trace and failure category instead of retaining only the successful attempt. Decide explicitly whether a recovered failure blocks, warns, or uses a temporary exception under the risk policy.

typescript A bounded Playwright retry policy

import { defineConfig } from "@playwright/test";

export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  use: {
    trace: "retain-on-failure",
    screenshot: "only-on-failure"
  }
});

Requires a pinned compatible Playwright Test project. One retry is a teaching choice, not a recommended universal count. retained traces can contain sensitive browser data, so use synthetic accounts and controlled artifact access. Reporting must preserve retry classification rather than converting it into an ordinary first-pass result.

Change one condition at a time

Repeat the same case on the same artifact with recorded inputs, then compare isolated execution, parallel execution, fresh data, and a controlled runner. A failure that appears only beside one neighboring case suggests state coupling; one that follows a particular dependency version suggests environment or compatibility drift. Do not repeatedly rerun an entire production-like load without a hypothesis. Use bounded local or test-environment reproduction. Preserve timelines and operation ids so a timeout can be distinguished from an assertion against stale data or an application request that actually failed.

Quarantine with a risk and expiry

Quarantine removes or changes enforcement of a known unstable check; it does not repair the test or restore missing confidence. Record the affected behavior, owner, issue, first evidence, expiry, and compensating check. Keep quarantined cases running in a visible diagnostic lane when safe. High-impact behavior may need a manual or alternative automated control before promotion. The example is a policy record interpreted by a trusted system, not a built-in CI keyword. Expired quarantine should trigger review instead of silently extending itself forever.

json A synthetic quarantine record

{
  "testId": "booking-email-status",
  "owner": "booking-maintainers",
  "issue": "LAB-41",
  "expiresOn": "2030-02-01",
  "risk": "confirmation feedback may be missed",
  "compensatingEvidence": "api-confirmation-state",
  "diagnosticLane": true
}

LAB-41 and the date are lab examples, not existing tickets or an active exception. A real policy validates ownership, expiration, and compensating evidence. Quarantine should remain searchable in reports and should not delete historical failures.

Measure instability without blaming the messenger

Track first-attempt failure rate, recovered failures, affected case count, time to diagnosis, and quarantine age. Report infrastructure and product nondeterminism separately when evidence supports the distinction. A high flake percentage can arise from a small number of heavily repeated cases, so keep denominators visible. Fix the cause with explicit waiting, ownership, dependency control, or product synchronization as appropriate. Verify the repaired case under the conditions that exposed it and nearby boundary conditions. A short streak of passes increases confidence but cannot prove intermittent failure is impossible.

Common errors and fixes

Retry success erases the first failure

Why it happens: Reports store only the final status.
How to fix it: Retain attempts and mark recovered failures explicitly.
Prevent it next time: Review first-pass and eventual-pass metrics separately.

A platform-specific defect is labeled flaky

Why it happens: Different environments are treated as equivalent.
How to fix it: Reproduce on the affected supported platform and classify the defect.
Prevent it next time: Attach environment dimensions to failure records.

Quarantine becomes permanent silence

Why it happens: No owner, expiry, or compensating evidence exists.
How to fix it: Create a bounded risk exception and diagnostic lane.
Prevent it next time: Review quarantine age and expired records automatically.

Check your understanding

1. A test fails consistently on one browser and passes on another. Is retrying enough?

Show worked solution

No. Browser identity is an explanatory input. Reproduce the failing supported configuration and inspect compatibility or assertion behavior. Keep the result as a deterministic configuration-specific failure until evidence shows otherwise; retries do not remove the support obligation.

2. A flaky email check is quarantined. What evidence is needed before a risky booking release proceeds?