Linux performance / Advanced: topology, observability, and controlled tuning / Lesson 16

Trace network scaling through queues and CPUs

Understand receive distribution, softirq work, socket queues, and transport constraints before changing network placement or buffers.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish RSS, RPS, RFS, and XPS roles.
Correlate packet processing with CPU and queue evidence.
Design a reversible network hypothesis without host changes.

Packet processing consumes host resources

Application goodput depends on more than wire capacity. Interrupt handling, softirq processing, virtual switching, protocol work, encryption, copying, and application reads all consume resources. A single busy CPU or receive queue can constrain progress while aggregate CPU and link utilization look modest. Containers add namespace and virtual-device boundaries, and offloads can make packet counts differ between layers. Start with the actual flow and endpoint path. Do not infer that a larger socket buffer or a different congestion algorithm will fix a bottleneck whose location is still unknown.

Scaling mechanisms solve different placement problems

RSS commonly uses NIC hardware queues and a flow hash to distribute receive work. RPS selects receive-processing CPUs in software, while RFS aims to improve flow locality relative to consuming applications. XPS influences transmit queue selection. Their availability and interaction depend on driver, hardware, kernel, and existing policy. A single flow may remain limited even when many queues exist, because distributing packets of one flow has ordering and locality implications. More queues are not automatically better: management cost, cache movement, and NUMA placement can offset added parallelism.

bash Inspect a fictional interface’s queues and counters

ethtool -l lab0
ethtool -S lab0
ip -s link show dev lab0

lab0 is a synthetic prepared-lab interface. ethtool statistics are driver-defined and may not exist for every virtual device; these commands read configuration and counters without changing channels or offloads.

Correlate per-CPU work with the same flow interval

Per-CPU interrupt and softirq counters can reveal concentration, but cumulative totals need deltas and a known interval. Softirq activity is not automatically harmful; it is part of normal packet processing and other kernel work. Compare application latency, queue drops, per-CPU usage, and socket evidence together. High processing cost may come from packet rate rather than bandwidth, especially with small messages. Preserve offload state and payload size in the baseline so a changed packet accounting boundary is not mistaken for a real capacity gain.

Transport and endpoint queues still matter

Even balanced packet processing cannot remove a slow reader, application lock, receiver flow-control limit, or distant congestion. Inspect existing TCP information for retransmissions, RTT, send and receive queues, and congestion-related indicators using the installed tool’s semantics. A burst can fill a queue without sustained saturation, while a larger backlog can preserve packets at the cost of tail delay. Test the exact fresh/reused connection and payload pattern. Avoid global buffer recipes and preserve protocol correctness, management connectivity, and remote endpoint authorization in any later isolated experiment.

bash Read CPU distribution and existing transport state

cat /proc/interrupts
cat /proc/softirqs
ss -tin

These non-runnable reads create no network traffic. Scope shared reports before exporting them because socket endpoints and interrupt/device names may reveal operational details.

Validate the whole path after a candidate change

A candidate can involve request batching, connection reuse, application reads, or an explicitly justified queue-placement policy on a disposable system. Predict the per-CPU and queue effect before testing, keep the workload fixed, and retain the original policy. Measure successful goodput, tails, errors, retransmissions, and neighboring workloads. A lower interrupt count is not sufficient if packets are simply aggregated differently or requests now wait longer. Do not change host IRQ affinity, RPS masks, queue counts, MTU, offloads, or congestion control in this course; the decision process remains observational and reversible.

Common errors and fixes

More queues is assumed always faster

Why it happens: Flow distribution and placement costs were omitted.
How to fix it: Measure queue concentration and useful work first.
Prevent it next time: Record topology and flow mix with the baseline.

Driver counters are treated as universal fields

Why it happens: Different hardware exposes different definitions.
How to fix it: Read the driver and interface documentation.
Prevent it next time: Keep counter definitions and versions with reports.

Balancing CPU is assumed to fix a slow reader

Why it happens: The endpoint application remained the limiting stage.
How to fix it: Inspect socket queues and application consumption.
Prevent it next time: Trace both endpoint and host processing.

Check your understanding

1. One queue and one CPU are busy but link bandwidth is low. What should be compared?

Show worked solution

Compare packet rate, flow count, receive distribution, application latency, and per-CPU work. Small packets or a single-flow boundary may explain the pattern. Aggregate utilization alone cannot establish whether queue redistribution will help.

2. After an offload change packet counts fall. Is that proof of fewer application requests?