Works locally, fails in CI: recover reproducible evidence

Understand the likely causes

• Local caches, untracked files, toolchain drift, architecture, working directory, timezone, and environment precedence often remain undeclared. Parallel tests may share data or ports. A contribution workflow may intentionally lack privileged credentials. Distinguish a missing documented input from a job that should never receive that privilege.

Freeze the compared identities

Record the exact source revision, lockfile identity, tool versions, architecture, test command, working directory, and first failing check from both environments. Compare safe allowlisted fields rather than dumping the environment. Confirm that the local run actually tests the committed snapshot, not untracked repairs. Preserve the original failure report before rerunning. The first synthetic comparison shows equal source but a different toolchain and filename behavior; it raises hypotheses rather than declaring a cause. No real runner or developer environment is inspected by the example.

field                local              CI
source_revision      lab-commit-17      lab-commit-17
lockfile             lab-lock-8         lab-lock-8
python               3.11               3.13
path_case_sensitive  false              true
tracked_config_name  Config.json        Config.json

Classify before changing the pipeline

Separate checkout or path failure, dependency resolution, compilation, test assertion, and publication permissions. A failing assertion deserves its input and expected behavior; an unauthorized publication step deserves a trust-boundary review. Do not make an untrusted pull request succeed by handing it a deployment secret. Compare the documented supported runtime rather than blindly installing the newest version. For filename problems, inspect the exact committed case and relative path. For tests, identify shared fixtures, time assumptions, and order dependencies before increasing retries or runner capacity.

Reproduce with declared inputs

Prepare an isolated clean reproduction using the reviewed source and declared dependencies, with synthetic data and external effects disabled. Change one identified difference and preserve the result. The second example compares public synthetic metadata in memory and prints only differing keys. It does not install tools or create a CI job. If a cache is implicated, compare an authorized clean run with a cached run and repair the key or missing dependency declaration. Keep the eventual fix in source and workflow review, not as a permanent manual adjustment on one runner.

local = {"source": "lab-17", "tool": "3.11", "case_sensitive": False}
ci = {"source": "lab-17", "tool": "3.13", "case_sensitive": True}
print(",".join(sorted(key for key in local if local[key] != ci[key])))

case_sensitive,tool

Repair the contract and keep the failure visible

Pin or document the required toolchain, include missing tracked inputs, correct path case, isolate test resources, or split privileged publication from contribution checks as evidence indicates. Preserve nonzero failure status for required gates. If a temporary quarantine is necessary, assign an owner, expiry, and replacement evidence. A broad skip or unconditional success can turn a visible defect into an unverified release. Explain which input assumption changed and why the repair addresses it, so the next environment does not need to rediscover the same hidden dependency.

Keep it from recurring

• Keep build inputs, runner assumptions, cache keys, and job permissions reviewable. Prefer isolated fixtures and explicit paths, and retain first-failure evidence. Treat recurring environment-only failures as defects in the delivery contract rather than a reason to normalize repeated reruns.