Linux / Editorial solution guide

Diagnose a service that fails to start

Trace a fictional systemd startup failure from effective configuration to the first causal error, then verify controlled recovery.

About 3 min readContent reviewed

Read-only knowledge base · No public comments

Recognize the symptoms

example-worker.service is failed or repeatedly retries, and a later message may say the start request was repeated too quickly. Users cannot complete the expected report request. The unit may be enabled despite its failure; activation policy does not certify runtime health.

Understand the likely causes

  • Potential causes include a missing executable, invalid arguments, wrong working directory, unavailable user, denied data access, bad application configuration, an occupied port, missing dependency, or resource termination. Rate limiting often follows an earlier cause. A healthy finite oneshot can also be inactive, so confirm the intended lifecycle before treating a state label as failure.

1. Capture state before retrying

Record incident time, user impact, unit name, and recent deployment or configuration changes. Capture the current result and bounded logs before any restart, because later attempts can bury the first error. Stay in the correct manager context: system and user services are separate. The commands below inspect only a prepared disposable image containing the fictional unit; they do not alter the host service manager.

bash Read state and recent unit evidence 
systemctl show example-worker.service -p ActiveState -p SubState -p Result -p ExecMainStatus
journalctl -u example-worker.service -b -n 80 --no-pager

Preserve the earliest causal error in the interval, not merely the final retry-limit line. Output and exit statuses depend on the lab scenario.

2. Inspect the effective execution contract

Use systemctl cat to read the vendor fragment and drop-ins, and show for relevant effective properties. Check ExecStart, User, WorkingDirectory, environment sources, and required paths. ExecStart is not an interactive shell line, so pipes and redirection do not acquire shell behavior automatically. Compare the intended executable with installed package or deployment evidence. If configuration files changed on disk, distinguish manager reload from application reload; neither is performed automatically by reading the files.

bash Compare source and effective properties 
systemctl cat example-worker.service
systemctl show example-worker.service -p ExecStart -p User -p WorkingDirectory -p ControlGroup

Inspect only the necessary properties; environment or command output can contain sensitive data. This example performs no daemon-reload or restart.

3. Test one cause in the matching context

Turn the first error into a narrow hypothesis. For a missing file, inspect the exact path, mount, deployment contents, and working-directory assumption. For permission denial, use the service identity, parent traversal, ACL, and sandbox policy. For an occupied socket, inspect the intended address and port before changing it. A command that works in your login shell may fail under the service’s different environment and restrictions. Use the application’s documented configuration validation on a copy when available.

4. Correct the cause through the authorized process

Prepare the smallest reviewed lab change and preserve the previous configuration. Distinguish the causal defect from the start-limit consequence; clearing a rate limit cannot create a missing file or repair an invalid configuration. An authorized operator can apply the application’s documented reload or restart procedure after validation. Do not weaken sandboxing or run the workload as root merely to hide an access mistake. Record the exact change and retain a rollback path before attempting recovery.

Verify the fix

Confirm the expected unit lifecycle and a successful application-level report operation. Review fresh logs for the same causal error, verify required dependencies, and observe another normal work cycle for repeated failure. For oneshot work, successful completion and correct output may be the acceptance condition rather than active status. Keep evidence linking recovery to the tested change.

Keep it from recurring

  • Validate unit and application configuration during deployment, test under the real service identity, define readiness and bounded retries, and keep a known recovery route. Preserve persistent logs with suitable retention. Write a short incident record separating observed cause, contributing conditions, successful recovery, and any remaining uncertainty.

Official references