Linux performance / Advanced: topology, observability, and controlled tuning / Lesson 13

Reason about NUMA locality and page size

Connect CPU placement, memory-node allocation, first touch, huge pages, and topology before proposing locality changes.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Explain local and remote memory without assuming fixed costs.
Distinguish memory policy from cpuset restrictions.
Evaluate huge-page tradeoffs with workload-specific evidence.

Topology changes the meaning of available capacity

A NUMA machine has memory nodes with potentially different access characteristics from each CPU. Logical CPUs can also share physical cores and caches, so a flat core count omits important structure. Locality depends on where threads execute and where their pages reside; the nearest node is not always the best choice if it is already constrained. Virtual machines can expose a topology that differs from physical placement. Record sockets, cores, SMT relationships, nodes, and permitted resources before attributing a memory-bound slowdown to an application algorithm or changing affinity.

Placement policy and allocation history both matter

A thread’s memory policy influences allocation from nodes within the resources it may use. Cpuset restrictions can further limit eligible nodes and take precedence as documented. First-touch behavior means the thread that initializes memory can affect placement, so pinning workers after initialization may not relocate their existing pages. Automatic NUMA balancing, shared mappings, migration, and application policy add complexity. Do not infer page location from thread location alone. Read the application’s allocation phase and the kernel’s policy semantics, then compare node-level usage and access evidence in the same interval.

bash Inspect topology and a synthetic process’s node usage

lscpu -e=CPU,CORE,SOCKET,NODE
numactl --hardware
numastat -p 4242

The tools must exist in a prepared NUMA lab, and PID 4242 is fictional. Node allocation is not a direct remote-access latency measurement; use it to test a placement hypothesis.

Huge pages trade translation work for other costs

Larger pages can reduce translation overhead for suitable working sets, but allocation, compaction, internal waste, and recovery behavior can change. Transparent huge pages and explicitly reserved hugetlb pages are different mechanisms with different administration and failure semantics. Supported sizes and controls depend on architecture and kernel; newer kernels can expose multiple THP sizes. A blanket instruction to enable or disable all huge pages ignores workload differences. Compare actual mapping usage, fault behavior, stalls, and completed work before choosing a lab candidate, and preserve the application’s memory-limit and restart requirements.

Combine placement evidence with useful-work timing

A rise in remote-node allocation can support a hypothesis, but it does not establish that memory locality caused the observed request tail. The data may be rarely accessed or shared intentionally. Compare equivalent workload phases, relevant hardware counters when supported, and application access patterns. Allocation pressure on one node may matter even when total host memory is abundant. Read allowed memory-node masks and effective cgroup constraints. Keep observation overhead bounded because detailed page-mapping collection on a large process can itself be expensive. No page migration or policy write is performed here.

bash Read permitted nodes and existing mapping policy

grep -E '^(Cpus_allowed_list|Mems_allowed_list):' /proc/4242/status
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /proc/4242/numa_maps

These non-runnable reads assume supported interfaces and lab authorization. The bracketed THP mode indicates the current selection in that interface, not a universal recommendation or complete per-size policy.

Keep locality experiments reversible and representative

A locality experiment must preserve application correctness, permitted resources, and shared capacity. Test placement and initialization together on an isolated copy rather than pinning an existing production process. Include startup and steady-state behavior, because an improvement after warm-up may hide unacceptable initialization stalls or reserve consumption. Measure latency, successful throughput, memory pressure, and effects on neighboring workloads. Document the exact topology and page policy so a later move to another machine does not inherit an unsupported conclusion. A useful result is conditional on that measured workload and hardware context.

Common errors and fixes

Pinning after allocation is assumed to move pages

Why it happens: Thread and memory placement were conflated.
How to fix it: Inspect allocation history and node mappings.
Prevent it next time: Design initialization with the locality experiment.

Remote allocation is called proof of a bottleneck

Why it happens: Access frequency and latency correlation were not measured.
How to fix it: Correlate actual demand and supported access evidence.
Prevent it next time: Treat topology counters as hypotheses, not verdicts.

Applying one huge-page rule everywhere

Why it happens: Architecture and workload tradeoffs were ignored.
How to fix it: Test documented policy on a representative copy.
Prevent it next time: Include startup, pressure, and recovery acceptance.

Check your understanding

1. Workers move to node 1 after node 0 initialized the heap. What should be checked?

Show worked solution

Inspect where existing pages remain, allowed node masks, and whether access is actually remote and costly. Moving execution does not automatically migrate every allocation. A controlled test should include the initialization placement and equivalent access pattern.

2. THP improves throughput but worsens startup p99. Can the result be accepted automatically?