Linux performance / Editorial solution guide

Distinguish memory pressure, OOM, and useful cache

Explain a fictional service’s memory alarm using availability, reclaim, PSI, scope-specific events, and actual request impact.

About 3 min readContent reviewed

Read-only knowledge base · No public comments

Recognize the symptoms

A service becomes slow or exits while a dashboard reports high used memory. Free RAM is small, swap is nonzero, or an OOM-related message appears. These observations can describe different mechanisms, so preserve the first failure, request interval, and service identity before restarting or changing limits.

Understand the likely causes

  • Useful file cache, growing anonymous allocations, high admitted concurrency, repeated reclaim/refaults, a cgroup boundary, or whole-host scarcity can all raise different memory signals. Used swap may be historical rather than active thrashing. A kernel kill, allocation error, and manager-driven memory action require separate evidence.

1. Identify which memory boundary matters

Record the process lifetime, service cgroup, ancestor limits, container context, and host availability. A process’s virtual size is not its resident footprint, and summing RSS can double count shared pages. A container can hit a hard limit while the host retains RAM. Preserve application version, batch size, concurrency, and data phase so growth is interpreted against actual work. Do not start by clearing cache or disabling a protection: those actions destroy evidence and may move the failure to another workload.

bash Compare host categories with a synthetic process view 
grep -E '^(MemAvailable|MemFree|Cached|Dirty|Writeback|SwapFree):' /proc/meminfo
cat /proc/4242/smaps_rollup

The process is a prepared lab fixture. Access and smaps_rollup support vary, and host availability is an estimate rather than a zero-latency guarantee. No caches are changed.

2. Separate occupancy from lost progress

A large cache without stalls can be healthy. Compare memory PSI, reclaim activity, refault evidence, swap movement, and request latency over matching intervals. PSI averages can smooth a short event, so preserve cumulative totals and compute deltas with their documented units. Nonzero used swap alone does not demonstrate current thrashing. If available memory is healthy and there is no harmful reclaim or limit event, investigate the actual slow stage rather than assuming the memory alarm explains it.

3. Establish the OOM or pressure mechanism

Read memory.high, memory.max, relevant ancestor policy, and event deltas for the affected group. A high-boundary event can explain reclaim pressure without a hard-limit kill. Correlate kernel logs, unit result, and application failure at the same time, recognizing that retention and permissions can hide some evidence. The killed process is not necessarily responsible for every byte in the group. Capture other members and retained data before assigning a leak or simply increasing the maximum.

bash Read scope-specific events and active movement 
cat /sys/fs/cgroup/lab-report/memory.events
cat /sys/fs/cgroup/lab-report/memory.pressure
vmstat 1 5

The cgroup path is synthetic and already exists in the lab. Compare repeated observations and distinguish vmstat’s initial summary from later intervals. Nothing is created, killed, or resized.

4. Correct demand while preserving recovery reserve

Test one justified candidate on a disposable copy: bound a cache, reduce batch size, lower admitted concurrency, or repair retained objects. A justified limit increase requires host and sibling headroom, not merely a desire to suppress an OOM message. Keep input and completion semantics fixed and monitor whether less killing turns into prolonged thrashing. Preserve the original policy and a tested reversal. If the cause is uncertain, gather another scoped observation instead of applying a universal swappiness or cache-dropping recipe.

Verify the fix

Repeat equivalent work and compare retained memory, successful completions, tails, pressure deltas, reclaim, swap movement, and OOM events. Confirm the service can restart and recover with its required reserve. Disappearance of an OOM message is not enough if errors, queue age, or time spent stalled become unacceptable.

Keep it from recurring

  • Track workload-normalized growth and pressure alongside usage, document hierarchy and batch limits, and retain pre-failure evidence with appropriate privacy. Test memory behavior during startup and large valid inputs. Keep capacity decisions conditional on the application’s actual working set and recovery contract.

Official references