DevOps QA / Editorial solution guide

Repair parallel test-data collisions

Trace shared mutable resources across pipeline runs, shards, workers, and retries, then allocate and clean only owned fixtures.

About 3 min readContent reviewed

Read-only knowledge base · No public comments

Recognize the symptoms

Tests pass serially but fail together with duplicate-key errors, missing reservations, unexpected quotas, or another test’s profile values. Failures may move between cases when shard count changes. Cleanup from one attempt can remove records still in use by another, making reruns appear to repair the issue while actually changing the shared state.

Understand the likely causes

  • Separate browser contexts do not isolate backend accounts, queues, mailboxes, object-store prefixes, caches, or feature flags. Worker indexes repeat across machines, and a transaction may not include writes from asynchronous jobs. Truncated or reused fixture names and broad cleanup selectors are common collision sources.

Inventory writes and correlate ownership

List every mutable resource a case touches, including setup and teardown. Attach pipeline run, shard, worker, case, and attempt metadata to diagnostic records. Reproduce with two controlled workers using synthetic local fixtures and capture which resource ids each creates, updates, and deletes. Serial success is useful evidence of coupling but not proof of its exact cause. Also inspect shared connection and quota pressure, because resource exhaustion can resemble a data collision without one worker actually overwriting another.

Allocate a namespace from complete inputs

Give each mutable fixture an ownership scope that remains unique across machines and concurrent runs. Do not use worker zero or a fixed test email globally. The example hashes an unambiguous serialized input tuple to produce a short local name. Hash shortening reduces accidental collision probability but is not an authorization mechanism or an absolute uniqueness proof. Persist the original ownership metadata and use storage uniqueness constraints. A retry should not silently reuse half-mutated resources from its failed predecessor unless that reuse is an explicit recovery scenario.

python A synthetic fixture-name helper
import hashlib
import json

def fixture_name(run_id, shard, worker, case_id, attempt):
    parts = [str(x) for x in (run_id, shard, worker, case_id, attempt)]
    encoded = json.dumps(parts, ensure_ascii=True, separators=(",", ":"))
    return "qa_" + hashlib.sha256(encoded.encode("utf-8")).hexdigest()[:20]

The function creates a name only; it provisions no account or database. Validate platform inputs and name-length rules before use. Keep the original tuple with the resource so cleanup can verify ownership instead of trusting a prefix alone.

Repair cleanup and external shared state

Delete exact resource ids owned by the current case, tolerate already-removed resources, and wait for bounded asynchronous work before teardown when required. Never issue a broad delete-all command against a shared test service. If jobs outlive the case transaction, use a dedicated schema or namespace that includes those writes. Partition mail and object storage by the same ownership contract. Add an expiry mechanism for abandoned resources after runner loss, with narrow permissions and diagnostic retention before removal. Document genuinely exclusive resources and schedule only those cases serially.

Verify the fix

Run the smallest two-worker reproduction repeatedly with separate owned resources, then restore the intended shard and worker counts in an isolated test environment. Verify every case sees only its own data, cleanup leaves other cases intact, expected reports are complete, and connection demand remains within the planned limit. Include a failed attempt followed by retry and an interrupted-run cleanup scenario. Serializing the whole suite is not evidence that isolation was repaired.

Keep it from recurring

  • Centralize fixture allocation and ownership checks, record namespace dimensions in reports, and review new shared dependencies before enabling parallelism. Add tests for cleanup scope and naming collisions. Treat browser, process, database, and external-service isolation as separate contracts rather than assuming one mechanism covers them all.

Official references