Tests pass locally but fail in CI: compare environments and isolate data

Understand the likely causes

• Common causes are uncommitted local fixtures, different dependency or database versions, machine-default time zones, case-sensitive paths, shared mutable data, and order dependence. A slower runner can reveal an application race hidden locally. Environment differences are hypotheses, not permission to skip a failing assertion.

1. Compare relevant facts without dumping secrets

Confirm the same revision and test selection, then compare runtime, locked dependencies, database engine/version, locale, time zone, and configuration values relevant to the case. Record variable names and safe settings, never tokens or an entire environment dump. Note whether tests share a database and whether parallel execution changes resource naming. Reproduce the assertion with synthetic data in an isolated context before editing application logic. A different build makes the comparison inconclusive.

2. Replace inherited data with owned fixtures

A test expecting three courses should create or scope exactly those three courses. It should not count every row in a shared store. Give each case or worker a deterministic namespace and reset the state it owns. Fresh mutable objects must be returned for each fixture request. The example checks this principle in memory; a real database test needs equivalent isolation plus reliable cleanup, including cleanup after setup fails partway through.

def fresh_courses():
    return ["qa-A", "qa-B", "qa-C"]

first = fresh_courses()
second = fresh_courses()
first.append("qa-D")
assert len(first) == 4
assert second == ["qa-A", "qa-B", "qa-C"]

3. Make time and engine assumptions explicit

For expiry, inject a clock and name the required zone instead of depending on the runner’s default date. Compare instants in a common representation and test display conversion separately. If a local in-memory database passes while PostgreSQL fails, inspect constraints, transaction handling, and collation using an isolated target-engine case. Do not remove a constraint merely to make environments agree; determine which behavior matches the durable product rule.

4. Distinguish isolation defects from product races

Run the smallest failing case alone, then with the suspected preceding case in both orders. If only the pair fails, inspect global state and cleanup. If overlap triggers failure, coordinate the contested operations in a controlled test and assert the business invariant. Disabling parallelism can narrow diagnosis but is not automatically the final repair. Increasing timeout without evidence can hide both a shared-state leak and a real user-visible ordering defect.

Keep it from recurring

• Document runtime and data assumptions, lock dependencies, design resettable fixtures, and avoid machine-default clock or locale behavior. Use target-engine checks for engine-specific risks and make parallel resource ownership explicit. Keep test secrets out of diagnostic output.