DevOps QA / Intermediate: trustworthy validation systems / Lesson 7

Parallel suites, shards, and data isolation

Reduce feedback time without introducing shared-state races or losing evidence from individual shards.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish runner workers from cross-machine shards.
Design namespace ownership and complete aggregation.

Count concurrency across every layer

A CI matrix can create several jobs, each job can run multiple test workers, and each worker can open several service connections. Total demand is the product of these choices, not the number shown in one runner setting. Sharding divides the suite across independent jobs, while workers parallelize within a shard. Increasing both can overload a shared database and make otherwise valid tests fail. Measure queue time, execution time, memory, connection count, and downstream limits before deciding that more concurrency will shorten useful feedback.

Allocate ownership before starting workers

A robust fixture identity includes the pipeline run, shard, worker, and case or resource scope. A retry may need a new attempt identity while preserving enough context to diagnose the original failure. Normalize names to the database or service constraints and detect truncation collisions. Cleanup must verify ownership and remove only the allocated resources. Do not use a worker number alone because two machines can both have worker zero. The example builds a deterministic synthetic namespace from explicit inputs; production helpers must validate the values supplied by the CI platform.

python An explicit namespace input contract

import hashlib
import json

def namespace(run_id, shard, worker, case):
    parts = [str(x) for x in (run_id, shard, worker, case)]
    raw = json.dumps(parts, ensure_ascii=True, separators=(",", ":"))
    suffix = hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16]
    return "qa_" + suffix

assert namespace("run-41", 1, 0, "reserve") != namespace("run-41", 2, 0, "reserve")

The hash shortens names and reduces accidental collisions; it is not authorization or a mathematical uniqueness guarantee. Store the original ownership metadata with created resources and enforce uniqueness constraints. Do not use this identifier as a credential.

Make the shard plan visible

A GitHub Actions matrix can distribute a Playwright suite across jobs. fail-fast controls cancellation after a matrix failure; disabling it can preserve diagnostics from remaining shards, but costs more work. max-parallel limits concurrent matrix jobs rather than workers inside each one. The fragment below assumes trusted checkout, dependency installation, and local application setup already exist in the job. Pin those actions and dependencies separately according to policy. Every shard must publish a uniquely named report and the aggregator must know the expected shard count.

yaml GitHub Actions job fragment with Playwright sharding

strategy:
  fail-fast: false
  max-parallel: 2
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - name: Run local browser shard
    run: npx playwright test --shard=${{ matrix.shard }}/4 --workers=2 --reporter=blob

Reviewed for the documented matrix and Playwright CLI syntax; exact project versions must be pinned. Four jobs with two workers each can create eight active test workers if the concurrency cap is later raised. The blob reports require separate artifact collection and merge steps.

Match isolation to shared resources

Browser contexts isolate browser state, separate processes isolate memory, and database schemas or owned rows isolate different portions of server state. None automatically isolates external quotas, mailboxes, queues, object-store prefixes, or feature flags. Inventory mutable resources and define a strategy for each. Serial execution can help diagnose a collision but is not a general repair. Some cases genuinely require serial ownership of an exclusive resource; label and schedule them deliberately instead of silently reducing the whole suite to one worker whenever failures appear.

Verify partitioning and completeness

A shard plan must execute each intended case under the intended projects without unplanned duplication or omission. Different sharding modes can balance files or individual tests depending on runner configuration, so measure actual distribution. Merge reports using the tool’s supported process and preserve shard identity for diagnosis. A rerun should not overwrite the first attempt’s evidence. Compare expected and received partitions, classify cancellations, and ensure a fast-failing shard cannot make the aggregate appear successful. Optimize the longest path only after isolation and evidence completeness are trustworthy.

Common errors and fixes

Parallel tests overwrite one account

Why it happens: Worker identity is not reflected in backend fixtures.
How to fix it: Allocate owned resources across run, shard, and case scope.
Prevent it next time: Run controlled collision tests before raising concurrency.

The database times out after adding shards

Why it happens: Nested worker counts exceed connection capacity.
How to fix it: Bound total concurrency and inspect pool pressure.
Prevent it next time: Capacity-plan all shared dependencies.

Merged reports omit a failed shard

Why it happens: Only successful jobs upload evidence.
How to fix it: Collect failure reports and require the complete shard manifest.
Prevent it next time: Test failure and cancellation paths in aggregation.

Check your understanding

1. Three shards each use four workers. The service supports eight test connections. What should be examined before enabling all shards together?

Show worked solution

At least twelve workers may compete, and each can need multiple connections. Measure actual per-worker demand, cap matrix or worker concurrency, and reserve capacity for setup and cleanup. Do not equate eight connections with eight safe workers without checking pooling behavior.

2. A retry passes after the first attempt deleted another test’s data. How should the run be classified?