Linux performance / Intermediate: mechanism and scoped diagnosis / Lesson 9

Diagnose memory pressure and OOM boundaries

Use PSI, reclaim activity, cgroup events, and application evidence to distinguish expensive pressure from harmless occupancy.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Interpret PSI some, full, averages, and cumulative totals.
Separate memory.high pressure from memory.max enforcement.
Investigate OOM scope and the demand that triggered it.

Occupancy is not time lost

Memory usage states how much is accounted, while pressure asks how work stalls because resources are scarce. A large stable cache can be healthy; repeated reclaim and refaults can harm latency even before an OOM event. Compare application deadlines with reclaim and pressure during the same interval. A system that finishes work within its contract may tolerate some background reclaim, whereas a latency-sensitive service may not. Avoid an automatic rule that every nonzero pressure reading requires more memory; the relevant decision depends on duration, scope, and user impact.

Read stall proportions with the right scope

PSI exports cpu, memory, and io pressure on supporting kernels. some tracks intervals where at least some tasks are stalled; full tracks intervals where all non-idle tasks in the relevant scope stall together. Averages use recent windows, while total is cumulative stall time in microseconds and supports delta calculations. System-level CPU full has special documented semantics and should not be interpreted like memory full. Per-cgroup files provide a narrower workload boundary when supported. Do not sum percentages across resources or scopes as if every stall were independent.

bash Read pressure and reclaim-related counters

cat /proc/pressure/memory
cat /proc/pressure/io
cat /proc/vmstat

Use prepared-lab interval snapshots. Read the kernel’s field definitions before selecting vmstat counters, and preserve totals so short spikes are not hidden by rounded averages.

Pressure and hard limits have different mechanisms

In cgroup v2, memory.high introduces reclaim and throttling pressure rather than acting as an absolute usage ceiling. memory.max is a hard boundary and can trigger cgroup OOM handling when reclaim cannot satisfy demand. Ancestor limits can constrain a child whose local file appears unlimited. memory.events and memory.events.local answer related but different hierarchical questions; use their documented semantics. A memory controller event does not by itself identify the allocation site or a leak. Preserve the application phase and effective hierarchy with every event delta.

bash Study a fictional cgroup’s pressure boundary

cat /sys/fs/cgroup/lab-report/memory.current
cat /sys/fs/cgroup/lab-report/memory.events
cat /sys/fs/cgroup/lab-report/memory.pressure

lab-report is a synthetic existing lab cgroup path. These reads do not create a cgroup or alter limits. Correlate event deltas with the same request interval and inspect ancestor policy separately.

An OOM result describes a failure boundary

An allocation failure, a kernel OOM kill, a cgroup OOM event, and a service manager’s memory-pressure action are not interchangeable reports. Establish which mechanism acted and which process lifetime was affected. Kernel logs, cgroup counters, unit results, and application logs can corroborate one another but may have different retention. A killed process is not necessarily the original source of all demand. Inspect concurrency, retained data, cache policy, and other group members before assigning blame. Preserve evidence before restarting, because process-level memory mappings vanish when the process exits.

Correct demand before weakening protection

A targeted response can reduce admitted concurrency, bound cache or batch size, fix retained objects, or provide justified capacity. Raising a hard limit without host headroom may move the failure from one service to the whole machine. Changing reclaim policy can shift latency rather than remove the underlying working-set demand. Plan a reversible lab test with equivalent inputs, measure completed work and stalls, and retain diagnostic reserve. OOM disappearance alone is insufficient if the replacement state spends most of its time thrashing or silently drops required work.

Common errors and fixes

Treating memory.high as memory.max

Why it happens: Pressure control was confused with hard enforcement.
How to fix it: Inspect both files and their event deltas.
Prevent it next time: Document each limit’s mechanism and ancestry.

Blaming the killed process alone

Why it happens: The victim was assumed to explain all group demand.
How to fix it: Inspect group members, growth, and the triggering phase.
Prevent it next time: Capture pre-failure scope-specific evidence.

Raising limits until OOM disappears

Why it happens: Host capacity and stall impact were not evaluated.
How to fix it: Compare demand reduction and justified capacity safely.
Prevent it next time: Require headroom and latency acceptance checks.

Check your understanding

1. Memory usage is below the local maximum but the service stalls. What else matters?

Show worked solution

Inspect memory.high, ancestor limits, reclaim/refault activity, PSI, and other members of the group. A hard maximum is not the only boundary that can impose delay, and aggregate usage can hide the application’s actual working-set behavior.

2. An OOM counter increments after a larger batch size. What experiment would test the mechanism?