Linux / Intermediate: system administration reasoning / Lesson 10
Read service state and structured logs
Understand systemd units, runtime state, activation, and time-scoped journal evidence on distributions that use systemd.
On this page
Lesson goals
- Distinguish active, enabled, failed, and installed.
- Read unit definitions and follow dependencies.
- Query journal evidence without assuming persistence.
A unit expresses managed behavior
systemd manages units such as services, sockets, timers, and mounts. A service describes supervised processes, while a socket or timer can activate work when needed. Installed means the definition exists; active describes runtime state; enabled describes installation links used by activation targets. These are independent dimensions. A disabled service may still be active through manual or dependency activation, and an enabled service may have failed. Always identify the actual unit name rather than infer it from a package name or the application’s marketing name.
Read the effective unit before changing it
systemctl status provides a compact human overview, while show exposes properties and cat displays the unit fragment plus drop-ins. A drop-in can override packaged settings without editing the vendor file. Unit search paths and manager context matter: a user manager and the system manager are separate. Check which context owns the workload. Reading a unit file alone does not prove that the manager has loaded the latest on-disk definition. A daemon-reload refreshes manager definitions; it is different from reloading an application’s configuration and is not performed in this course.
systemctl cat example-report.service
systemctl show example-report.service -p ActiveState -p SubState -p ResultThese are read-only study commands for an image containing example-report.service. The properties distinguish the current state from the result of the last run; outputs depend on that lab.
Ordering and requirement are different
After= establishes ordering when both units participate in a transaction; it does not by itself start the other unit. Wants= pulls in a dependency weakly, while Requires= establishes a stronger requirement with documented failure relationships. Neither declaration proves an external network endpoint is ready to answer. Services must still cope with unavailable dependencies. Read the unit’s type: oneshot work can complete successfully and become inactive, whereas a long-running service needs an appropriate process and readiness model. Do not interpret every inactive status as a fault.
Scope log queries by unit, boot, and time
journalctl can filter structured entries by unit, boot, priority, and time range. Start with the affected unit around the reported incident, then widen only when needed. -b selects a boot and --list-boots shows available recorded boots. Older boots may be absent because the journal is volatile or retention removed them. Logs are evidence of what was recorded, not proof that unrecorded work succeeded. Standard output and error of a managed service commonly enter the journal, but applications may also use separate files or remote logging.
journalctl -u example-report.service -b --since '2026-10-23 10:00:00' --until '2026-10-23 10:10:00' --no-pager
journalctl --list-boots --no-pagerUse the lab’s actual incident time and available privileges. These commands read accessible entries only and do not restart, clear, or vacuum anything.
State and useful behavior both matter
A service process can be active while unable to serve users. Verify a small application-level behavior in addition to manager state, and distinguish startup success from ongoing health. Conversely, a scheduled oneshot job may be healthy after it exits. Preserve first-failure logs before considering a retry, because repeated restarts can bury the original symptom and trigger rate limits. Record unit name, boot identifier, timestamps, and the precise result. Avoid publishing secrets that may appear in command arguments or application diagnostics when sharing evidence.
Common errors and fixes
Enabled is mistaken for running
- Why it happens
- Activation links were confused with current runtime state.
- How to fix it
- Inspect ActiveState and the unit’s documented lifecycle.
- Prevent it next time
- Report enabled and active independently.
After= is expected to start a dependency
- Why it happens
- Ordering was confused with requirement relationships.
- How to fix it
- Review Wants= or Requires= and application readiness.
- Prevent it next time
- Design dependency and ordering policy separately.
Previous boot logs are missing
- Why it happens
- Retention or volatile storage did not preserve them.
- How to fix it
- Check --list-boots and the documented logging configuration.
- Prevent it next time
- Plan persistence and retention before incidents.
Check your understanding
1. An enabled oneshot service is inactive with Result=success. Is it necessarily broken?
Show worked solution
No. A oneshot may finish its work successfully and return to inactive unless its configuration retains an active state. Check its timer or other trigger, expected output, and last invocation evidence. The appropriate health test is completion of the intended task, not continuous process presence.
2. Choose the first evidence for a failure reported at 10:04 in a lab service.
Show worked solution
Capture status and the effective unit, then query that unit’s journal across a short interval surrounding 10:04 in the correct boot. Identify the first failure and associated result before widening to dependencies. Preserve the evidence before any authorized recovery action.
Key takeaways
- Activation policy, runtime state, and application health differ.
- Read effective units and narrowly scoped journal evidence.