DevOps QA / Editorial solution guide
Distinguish a flaky pipeline from a deterministic failure
Preserve the first failure and compare controlled inputs before using retries, quarantine, or a product-defect classification.
Read-only knowledge base ยท No public comments
Recognize the symptoms
A job fails once and passes after rerun, a browser case times out only on some workers, or one operating-system job fails every time while others pass. The final green status hides the original evidence. A retry can change the worker process, fixture state, dependency availability, or source revision, so the two outcomes may not represent equivalent experiments.
Understand the likely causes
- Possible causes include a deterministic supported-platform defect, shared test data, uncontrolled waiting, environment drift, a runner outage, and genuine application nondeterminism. Flaky is a provisional diagnosis about inconsistent outcomes under comparable conditions. It should not become a convenient label for any failure that disappears after another attempt.
Preserve identity and the first useful evidence
Record candidate artifact, source revision, case id, attempt, runner image, browser and dependency versions, environment manifest, seed, and shard placement. Keep the first relevant error plus bounded redacted logs or trace. Separate setup, action, and assertion phases. Confirm the rerun actually used the same candidate; a new commit is a different comparison. Avoid increasing timeouts or deleting caches before collecting this evidence, because those changes can erase the condition needed to understand the failure.
Compare one variable at a time
Repeat the same case with the same owned data on a controlled local test environment, then compare isolated and parallel execution. If the failure follows one supported browser consistently, investigate a configuration-specific defect. If it follows a neighboring case or shared account, investigate isolation. If setup cannot reach a local dependency, classify the infrastructure boundary before business assertions. The helper below is a triage prompt over pre-collected synthetic facts; it deliberately avoids declaring flakiness from one successful retry.
def triage_prompt(*, same_candidate, setup_complete, consistent_on_supported_config, parallel_only):
if not same_candidate:
return "collect comparable evidence"
if not setup_complete:
return "investigate setup boundary"
if consistent_on_supported_config:
return "investigate deterministic configuration defect"
if parallel_only:
return "investigate isolation and capacity"
return "investigate timing, inputs, and product nondeterminism"This function performs no reruns and makes no statistical claim. Inputs must come from documented observations, not guesses. Its output suggests the next investigation boundary and must not replace a diagnosis supported by traces and controlled reproduction.
Repair the cause and bound temporary workarounds
Replace fixed sleeps with relevant readiness or waiting assertions, allocate owned fixtures, pin compatible dependencies, or repair application synchronization according to the evidence. A bounded retry can collect diagnostics while investigation continues, but retain attempt classification. If quarantine is necessary, name the lost behavior claim, owner, expiry, and compensating evidence. Do not permanently suppress a high-impact case or classify a runner outage as a product bug simply to close the report. Keep unresolved uncertainty visible to the delivery gate.
Verify the fix
Re-run the original controlled reproduction and the condition that distinguished it, such as parallel neighbors or the affected browser. Include a deliberate nearby boundary case and verify expected reports retain every attempt. A streak of passes increases confidence but cannot prove an intermittent condition impossible. Confirm the fix addresses the identified cause, and remove a temporary quarantine only after its stated verification criteria are met.
Keep it from recurring
- Keep first-pass and eventual-pass metrics separate. Review quarantine age, shared-resource ownership, environment changes, and recurring timeout locations. Add diagnostic context at the relevant boundary rather than uploading unrestricted environments or private browser data. A future investigator should be able to reconstruct the failed experiment without requesting a real secret or contacting a production service.