Linux performance / Basic: measurement and resource signals / Lesson 3
Separate CPU utilization, load, and steal
Interpret busy processors, runnable demand, blocked work, and virtual-machine scheduling without equating their percentages.
On this page
Lesson goals
- Read CPU time categories with their measurement boundaries.
- Distinguish load average from CPU utilization.
- Recognize per-thread, quota, and hypervisor constraints.
CPU time describes execution categories
A CPU time view divides elapsed processor time into categories such as user execution, kernel execution, idle time, and platform-dependent virtualization accounting. A process percentage may be normalized to one CPU or the whole machine depending on the tool. A multithreaded process can therefore show more than 100 percent in one display without violating physics. Define the denominator before comparing charts. Short-lived work may disappear between snapshots, so a single top or ps screen cannot explain every request completed during a longer interval.
Load average is demand, not a utilization percentage
Linux load averages include runnable tasks and tasks in uninterruptible sleep, commonly associated with waiting in parts of the I/O path. They summarize smoothed demand over different horizons rather than a percentage of CPU time. A high load can accompany idle CPU time when many tasks are blocked. Compare load with available CPUs only as a starting clue, then inspect task states and the effective CPU allocation. A container’s reported host load may not reflect its own quota or cpuset, so a host-wide ratio can be misleading.
uptime
vmstat 1 5Study these diagnostics only in a prepared lab. The first vmstat report is commonly a since-boot summary; subsequent interval rows are the comparison of interest. Read the installed manual for field definitions.
Find the actual parallelism boundary
One saturated thread can limit an application while other CPUs remain idle. Serialization, a global lock, a single event loop, affinity, or a CPU quota can prevent extra cores from helping. Inspect threads and their states rather than only process totals. CPU affinity indicates where a thread may run, not how much time it will receive. A quota imposes a time budget across its period, while a relative weight matters under contention. Distinguish these mechanisms before deciding that the machine needs more cores or a different scheduler.
ps -L -p 4242 -o pid,tid,psr,stat,pcpu,comm
cat /proc/4242/statusPID 4242 is a synthetic lab identifier, not a target on this host. ps percentages may cover a different averaging interval from an interactive monitor; compare their definitions before drawing conclusions.
Virtualization and I/O wait need context
Steal time in a guest can indicate that a virtual CPU could not run because the hypervisor scheduled other work. It is not the guest application’s own kernel execution. Availability and meaning depend on the virtualization platform. Likewise, iowait is a CPU accounting category, not a direct disk latency measurement or a sum of all thread waits. Multiple CPUs and asynchronous I/O make simplistic interpretations especially fragile. Correlate guest timing with platform evidence, per-thread states, and device measurements before blaming either application code or a noisy neighbor.
Choose the next discriminating observation
If user CPU rises with a request mix change, investigate application work. If runnable queues grow with quota throttling, investigate the budget and arrival pattern. If uninterruptible tasks and device latency rise together, follow the I/O path. If steal increases across unrelated guest workloads, seek authorized hypervisor evidence. These are hypotheses, not automatic verdicts. Record the interval and compare a healthy window. Do not adjust nice levels, affinity, or host scheduling from this lesson; an unsupported change can move contention to another service without improving the user-visible goal.
Common errors and fixes
Treating load 8 as eight busy CPUs
- Why it happens
- Runnable and uninterruptible demand were mistaken for utilization.
- How to fix it
- Inspect interval CPU time, states, and effective allocation.
- Prevent it next time
- Label load and CPU percentage as different signals.
Calling every idle-looking guest healthy
- Why it happens
- Quota, affinity, or steal constrained useful execution.
- How to fix it
- Inspect the workload boundary and virtualization evidence.
- Prevent it next time
- Record effective capacity, not only host core count.
Using iowait as disk response time
- Why it happens
- CPU accounting was confused with per-operation latency.
- How to fix it
- Correlate with device and application I/O observations.
- Prevent it next time
- Use metrics from the layer named in the hypothesis.
Check your understanding
1. Load is high, CPUs are mostly idle, and many tasks are in D state. What should you investigate?
Show worked solution
Investigate the blocking path, especially related storage or kernel waits, using interval device and task evidence. D state alone does not name a disk or prove saturation, so preserve the timing and identify the actual waited resource before tuning.
2. A four-vCPU guest has one thread near one full CPU while three CPUs are idle. Will adding CPUs necessarily help?
Show worked solution
No. First determine whether the request path is serialized, affinity constrained, or quota limited. Additional CPUs help only if useful parallel work can use them without another bottleneck; a profile and thread-level evidence distinguish those cases.
Key takeaways
- Busy time, queued demand, and virtual CPU availability differ.
- Inspect threads and effective allocation before recommending capacity.