Linux / Advanced: boundaries, security, and recovery / Lesson 13
Trace boot and plan controlled recovery
Locate failures across firmware, bootloader, kernel, initramfs, mounts, and system targets using a disposable-machine recovery model.
On this page
Lesson goals
- Explain the stages that lead to a usable system.
- Use boot-specific evidence to narrow failure location.
- Plan reversible recovery with data protection.
Follow the handoffs
Firmware initializes enough hardware to select a boot path. A bootloader commonly loads the kernel and an initramfs, although some firmware can load a kernel directly. The kernel initializes devices and may run an early userspace environment to locate and unlock the real root filesystem. Control then passes to the installed system manager, which starts required mounts and services. A failure before the real root is available has different evidence and recovery tools from a failed application after multi-user startup. Identify the last successful handoff before selecting a repair.
Early userspace must find the intended root
The initramfs needs the drivers and tools required for that machine’s root storage stack. A missing storage driver, wrong root identifier, unavailable encrypted volume, or broken logical-volume assembly can stop progress before normal services exist. Record the exact early error and compare it with the intended storage design. Do not immediately run a filesystem repair because failure to find a device is not proof of filesystem corruption. Rebuilding early-boot images and changing bootloader configuration are distribution-specific administrative actions that belong in a reviewed disposable-machine exercise.
Associate evidence with the correct boot
On a systemd lab, journalctl --list-boots enumerates retained boot records and journalctl -b -1 reads the previous recorded boot when available. Kernel messages and unit failures help locate the boundary between device discovery and userspace activation. An absent previous boot can reflect volatile logging or retention rather than a clean shutdown. /proc/cmdline exposes the current kernel command line, which may reveal the selected root and recovery parameters. Treat identifiers and log timestamps as evidence, and redact secret-bearing parameters before sharing a report.
journalctl --list-boots --no-pager
journalctl -b -1 -p warning --no-pager
cat /proc/cmdlineThese read-only examples require a systemd lab with retained journal access. No output is fabricated; previous-boot availability and command-line contents vary.
Recovery environments have different assumptions
A rescue or emergency environment may offer fewer services, a read-only root, no network, and a different authentication path. A live image is a separate operating system whose /etc is not automatically the installed system’s /etc. Before inspecting another root, identify the correct device and mount state. A chroot changes pathname resolution but is not a virtual machine and does not provide container-level isolation. Never assume its process, device, and network views are independent. Keep the original disk protected and use a snapshot or copy when practicing uncertain recovery steps.
systemctl --failed --no-pager
findmnt --target /In the running disposable image, these commands inspect failed units and the root mount. They neither enter rescue mode nor remount or repair storage.
A recovery plan includes proof and rollback
Write a hypothesis tied to a specific failed stage, preserve the relevant configuration, and choose the smallest change that tests it. A previously installed kernel can be a diagnostic comparison when boot media and policy support that choice; it is not proof that a new kernel is defective. After recovery, verify mounts, required services, network access, and a small application task. Review the original change that caused the incident and ensure future reboot behavior is tested. A machine that works only until the next reboot has not completed recovery.
Common errors and fixes
Repairing a filesystem before identifying the device
- Why it happens
- Missing-device and corruption symptoms were conflated.
- How to fix it
- Map the boot stage and storage stack before selecting tools.
- Prevent it next time
- Require a device identity and backup plan for recovery.
Editing the live image instead of the installed system
- Why it happens
- The recovery root was mistaken for the target root.
- How to fix it
- Verify mount points and the exact target pathname.
- Prevent it next time
- Label recovery and target environments explicitly.
Declaring success before a controlled reboot test
- Why it happens
- Only the current session was tested.
- How to fix it
- Verify persistent configuration and reboot in the disposable plan.
- Prevent it next time
- Include next-boot behavior in acceptance criteria.
Check your understanding
1. The bootloader appears, but early userspace cannot find the root UUID. Where should investigation start?
Show worked solution
Start at root discovery: compare the kernel command line, available devices, storage drivers, encryption, and intended identifiers. Normal application service settings are downstream of this failure. Preserve the disk and avoid guessing a formatting or repair target.
2. What evidence distinguishes a failed application from a failed boot?
Show worked solution
Show that the real root mounted, the system manager reached its intended target, and other required services became usable, then isolate the application’s unit result and logs. A single failed unit does not prove the whole machine failed to boot.
Key takeaways
- Locate the failed handoff before selecting a recovery action.
- Protect source data and test persistent recovery.