CI/CD / Integrity, coordination, and improvement / Lesson 14
Runner isolation and pipeline reliability
Treat the execution environment as a controlled dependency with bounded failure behavior.
On this page
Lesson goals
- Distinguish job isolation from convenient workspace cleanup.
- Classify infrastructure failures before selecting retries.
The runner is part of the trust base
A runner executes repository-controlled commands and may hold network access, local files, sockets, or machine identity. A reused worker can retain state between jobs even when a workspace directory is cleaned. Processes, tool caches, mounted volumes, or host credentials may remain. Parcel’s untrusted validation should not share an execution boundary with privileged release work unless the platform provides an appropriate isolation model. A container executor is one mechanism, not a universal guarantee against every host or tenant threat.
Know what disappears after a job
Ephemeral execution can reduce persistent cross-job state when a fresh environment is actually provisioned and destroyed for each trust boundary. It does not fix an overpowered identity inside the job or unsafe network access while it runs. Document lifecycle, base-image maintenance, supported tools, and evidence export before destruction. If a failure removes the worker, retained reports and invocation metadata must still explain what happened. Keep those exports bounded and scrub secrets rather than archiving the entire machine for convenience.
job inputs -> isolated execution -> declared outputs
retained: test reports, candidate identity, redacted diagnostics
discarded: temporary workspace and unneeded local stateThis is a conceptual boundary, not proof that any executor enforces isolation. Host processes, identity, mounts, and network reachability require their own platform-specific review. No runner is provisioned or destroyed.
Retry only a failure that can be retried
An assertion failure, missing source file, exhausted disk, and interrupted artifact download are different failure classes. Automatic retries can help a transient acquisition error if the step is safe to repeat. They can hide deterministic defects or duplicate side effects elsewhere. Record the first failure and retry reason. Bound attempts and total elapsed time, then surface exhaustion clearly. Avoid retrying an entire promotion merely because one report upload failed after the target already accepted the release.
Calculate an attempt budget
The complete local model computes a worst-case retry budget from declared attempt duration and backoff delays. It excludes queueing and platform cancellation latency, so it is an estimate under stated assumptions rather than an operational timeout guarantee. Budgets prevent a short job from quietly turning into an hour-long incident. A production timeout must also account for cleanup and evidence publication, and its cancellation behavior must be understood before applying it to work with external side effects.
attempt_seconds = 20
backoff_seconds = [2, 4]
attempts = len(backoff_seconds) + 1
budget = attempts * attempt_seconds + sum(backoff_seconds)
print(attempts)
print(budget)3
66Three attempts at at most twenty seconds plus two waits total sixty-six seconds in this synthetic model. It does not sleep, start jobs, or test a runner.
Measure the delivery platform separately
Queue latency, worker startup, execution, artifact transfer, and service availability are separate sources of feedback delay. If jobs spend nine minutes queued and one minute running, rewriting a ten-second test offers little benefit. Track failure class and resource saturation without exposing user code or secrets unnecessarily. Platform maintenance should have an owner and rollback path just like application changes. A reliable pipeline makes infrastructure uncertainty visible instead of forcing developers to rerun unexplained failures until a green result appears.
Common errors and fixes
Deleting a workspace is treated as complete isolation.
- Why it happens
- Persistent processes, mounts, and host identity were ignored.
- How to fix it
- Review the executor’s full lifecycle and trust boundary.
- Prevent it next time
- Separate privileged and untrusted workloads with an appropriate platform model.
Every failure gets five automatic retries.
- Why it happens
- Transient infrastructure faults and deterministic defects share one policy.
- How to fix it
- Classify failures and retry only safe transient operations within a budget.
- Prevent it next time
- Retain attempt history and define exhaustion behavior.
A report-upload failure repeats an already accepted release.
- Why it happens
- The retry unit includes an external side effect with ambiguous completion.
- How to fix it
- Observe the target state and retry only the incomplete safe step.
- Prevent it next time
- Separate adoption, verification, and report publication states.
Check your understanding
1. A job runs in a fresh container but mounts the host’s privileged socket. Is its authority minimal?
Show worked solution
No. The socket can expose powerful host operations despite a fresh filesystem. Isolation must include mounts, identity, network, and daemon access. Remove unnecessary authority through the approved runner design rather than relying on container freshness alone.
2. Queue time is twelve minutes and execution is two. Which measurement should guide the first investigation?
Show worked solution
Investigate runner availability, scheduling constraints, concurrency limits, and workload arrival patterns. Execution optimization may help later, but the dominant delay is outside the script. Compare equivalent workloads and avoid granting extra privileges merely to reach another runner.
Key takeaways
- Runner lifecycle, identity, and isolation are release inputs.
- Retry policies need failure classes, repeatability, and bounded budgets.