Linux performance / Editorial solution guide

Separate high CPU, high load, and guest steal

Diagnose a slow fictional report service by separating execution demand, blocked work, scoped budgets, and virtual-machine scheduling.

About 3 min readContent reviewed

Read-only knowledge base · No public comments

Recognize the symptoms

Reports miss their deadline while a dashboard shows high CPU or load. Different tools disagree, and the guest may also report steal time. The service can be constrained even when the host has idle capacity. Preserve the exact incident interval and report class before interpreting any percentage.

Understand the likely causes

  • Possible mechanisms include more CPU work per request, one serialized hot thread, runnable competition, CPU quota throttling, affinity restrictions, blocked uninterruptible tasks, or hypervisor scheduling delay. Load average mixes demand states and is not a CPU percentage. Steal is guest accounting, while iowait is not a disk-latency measurement.

1. Establish scope and percentage denominator

Record whether the display is per CPU, per process, per thread, or host-wide, and whether process percentages can exceed one CPU. Identify the application’s actual cgroup, allowed CPUs, kernel, virtualization platform, and recent deployment. Compare the same request mix with a healthy window. A copied screenshot without units and timing cannot distinguish a new algorithmic cost from a reduced resource budget. Collect only relevant process metadata, because command arguments can contain sensitive values.

bash Read demand and a fictional process’s thread states 
uptime
vmstat 1 5
ps -L -p 4242 -o pid,tid,psr,stat,pcpu,comm

PID 4242 is a prepared lab identifier. Use vmstat interval rows after its initial summary and preserve the installed tool’s definitions. No host workload is targeted or generated.

2. Classify execution versus waiting

If user or kernel execution rises with useful work, inspect which threads and request stages consume it. If load grows with many D-state tasks and little execution, investigate the blocking path and related I/O evidence. If one thread is busy while other permitted CPUs are idle, inspect serialization, locks, or event-loop work before adding cores. A state label narrows the search but does not identify a device or source line; correlate it with the same interval and a bounded profile only when justified.

3. Check scoped budgets and virtualization

Read the effective CPU quota and throttling deltas, including ancestor policy, before interpreting spare host CPU as available to the service. Compare allowed CPU placement separately from time budget. If guest steal increases, seek authorized platform evidence for that interval rather than blaming the application automatically. A different VM size or noisy-neighbor theory remains a hypothesis until corroborated. Do not relax host quotas, scheduling, or security controls during diagnosis.

bash Inspect a synthetic service’s CPU-policy evidence 
systemctl show example-report.service -p ControlGroup -p CPUQuotaPerSecUSec -p AllowedCPUs
cat /sys/fs/cgroup/lab-report/cpu.stat

Both names refer to an existing disposable lab fixture. Capture repeated counters and effective ancestors; these commands do not change properties, move tasks, or restart services.

4. Prepare a response to the proven mechanism

Choose the smallest mechanism-specific candidate: reduce repeated computation, shorten a critical section, bound admitted concurrency, or propose justified scoped capacity. Preserve the original contract and compare equivalent lab work. A quota increase must include sibling impact; a code change must preserve output and error behavior. Write a stop condition and rollback. If evidence remains ambiguous, collect a discriminating observation rather than changing several unrelated parameters. The host kernel, scheduler, and services remain untouched throughout this guide.

Verify the fix

Repeat the matched report population and compare end-to-end tails, successful throughput, errors, thread states, and the specific event predicted to change. Confirm the improvement persists beyond a short burst and does not merely shift waiting downstream. A lower load average alone does not prove that user deadlines recovered.

Keep it from recurring

  • Keep per-thread and cgroup evidence alongside host charts, record virtualization context, and alert on service objectives plus relevant throttling or pressure. Version CPU-budget decisions and workload changes. Retain enough diagnostic headroom to inspect an incident without starting an uncontrolled profiler or load test.

Official references