Linux / Advanced: boundaries, security, and recovery / Lesson 18
Build reliable services and diagnose incidents
Connect readiness, restart policy, sandboxing, evidence preservation, and controlled recovery into a practical service lifecycle.
On this page
Lesson goals
- Choose a service type that matches the process model.
- Distinguish readiness, liveness, retries, and rate limits.
- Investigate incidents through a testable causal timeline.
Startup is not the same as readiness
A process can exist before it has opened its socket or loaded required state. Type=exec confirms that the service executable was successfully invoked, but not that application initialization finished. Type=notify allows a supporting application to report readiness explicitly. Type=oneshot fits finite work, while legacy forking behavior requires an appropriate model and reliable main-process identification. ExecStart is not automatically an interactive shell command: shell operators need an explicitly invoked shell if they are truly required. Prefer a direct executable and explicit arguments so the manager can identify and supervise the intended workload.
Retries require a bounded policy
Restart=on-failure can recover from some unexpected exits, but it does not repair a bad configuration or unavailable credential. RestartSec= spaces attempts, while start-rate limits can stop repeated activation. A deliberate stop is different from an application crash. Define acceptable retry behavior and ensure startup is idempotent so repeated execution does not duplicate durable work. If the service consumes jobs, coordinate acknowledgement and checkpoint semantics with the application. Restart supervision improves availability only when the application’s state transitions remain correct across interruption and re-entry.
[Unit]
Description=Synthetic report worker for review
StartLimitIntervalSec=60
StartLimitBurst=3
[Service]
Type=exec
ExecStart=/usr/local/libexec/example-worker --foreground
User=example-worker
Restart=on-failure
RestartSec=5
NoNewPrivileges=yes
PrivateTmp=yes
The executable and account are fictional and must not be installed by copying this fragment. The unit illustrates bounded retries and two sandbox settings; it is not a complete application deployment.
Reduce access without breaking required state
Service sandboxing can separate temporary directories, prevent new privilege gains, restrict writable paths, and reduce capabilities. Each setting changes the execution contract and must be tested with the actual application. PrivateTmp does not mean every application file is private, and NoNewPrivileges does not remove privileges already held. Read-only filesystem protection needs explicit writable locations for legitimate state. User and group identity still matter. Use the installed systemd manual because options and their implementation depend on version, kernel support, and whether the service runs under a user or system manager.
Preserve the first failure and build a timeline
Start an incident record with impact, first observed time, affected scope, and recent changes. Capture unit result, invocation evidence, journal entries, relevant cgroup events, and dependency state before restarting. Correlate timestamps carefully because clocks and time zones can differ across machines. Separate observations from hypotheses: permission denied is evidence, while a suspected deployment mistake is a hypothesis. Choose one low-risk discriminating check, update the hypothesis, and avoid a sequence of unrecorded changes. Repeated restarts can destroy the original evidence and create an additional rate-limit symptom.
systemctl show example-worker.service -p Result -p ExecMainCode -p ExecMainStatus -p NRestarts
journalctl -u example-worker.service -b -n 80 --no-pager
systemctl cat example-worker.serviceThese read-only commands inspect a prepared fictional lab unit. Compare the first failure with later retry messages. No restart, reset-failed, or configuration mutation is included.
Recovery and prevention need different evidence
Recovery proves that useful service returned; root-cause analysis explains the mechanism that produced the incident and why existing controls missed it. A rollback can restore availability without fully explaining the defect, so record that uncertainty explicitly. Verify an application request, data consistency, and expected dependency behavior after an authorized recovery. Then reproduce the causal mechanism in a disposable environment and add the smallest meaningful protection, such as input validation or configuration checking. Document a concise timeline, contributing conditions, tested fix, and an operational check that would detect recurrence.
Common errors and fixes
Active is treated as ready
- Why it happens
- Process startup was confused with application initialization.
- How to fix it
- Use a supported readiness model and functional check.
- Prevent it next time
- Define readiness before dependency ordering.
A restart loop becomes the repair
- Why it happens
- Retries concealed a persistent configuration or state failure.
- How to fix it
- Capture the first cause and bound recovery attempts.
- Prevent it next time
- Pair restart policy with failure classification.
Sandbox settings are copied without testing
- Why it happens
- Required application paths and privileges were unknown.
- How to fix it
- Test the access contract on a matching disposable image.
- Prevent it next time
- Review each restriction against required behavior.
Check your understanding
1. A unit reaches the start limit after several identical missing-file errors. What should the next diagnosis address?
Show worked solution
Treat the start-limit message as a consequence of retries. Investigate the first missing-file error, effective working directory, executable arguments, mount availability, and deployment contents. Only after fixing the cause should the authorized lab recovery plan consider clearing the limit and starting again.
2. A rollback restores requests. Is the incident analysis complete?
Show worked solution
No. Recovery is established, but the failed change’s causal mechanism and prevention remain open. Preserve the before/after versions, correlate evidence, reproduce the failure safely, and verify a targeted fix. State any remaining uncertainty instead of presenting correlation as proof.
Key takeaways
- Choose a process model and test application readiness.
- Preserve evidence, bound retries, and verify recovery functionally.