Linux performance / Basic: measurement and resource signals / Lesson 5

Separate storage latency from utilization

Map an application operation through filesystems and block devices before interpreting I/O rates or busy percentages.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Map the storage path and measurement layer.
Distinguish bandwidth, IOPS, queue depth, and response time.
Interpret device counters without universal saturation thresholds.

One file can cross several storage layers

An application may wait on its own queue, filesystem metadata, page cache, journal work, block queues, a logical volume, and remote storage. The visible device may represent a virtual disk backed by another machine. Start by mapping the exact path to its mount and backing stack. Do not add statistics from a logical device and its children as if they were unrelated work, because the same I/O can appear at several layers. A filesystem-capacity error and a high-latency read are different incidents even if both mention storage.

Rate, size, and latency answer different questions

IOPS counts operations per second, while bandwidth counts bytes per second. Average operation size links them but does not capture random versus sequential access or durability semantics. Queue depth counts work awaiting or undergoing service. A device can deliver modest bandwidth with expensive small synchronous operations, or high bandwidth with acceptable latency for a batch transfer. Define read/write mix and operation size before comparing two runs. Application request latency can include waiting outside the device counters, so retain both end-to-end timing and lower-layer observations.

bash Map the path before reading interval device reports

findmnt --target /tmp
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
iostat -xz 1 5

These are prepared-lab diagnostics. iostat field names and first-report behavior depend on the installed sysstat version; use interval rows and identify the relevant device rather than treating all rows as additive.

Busy time is not a universal capacity meter

A device utilization percentage describes observed busy time according to its accounting model. On devices that serve many requests concurrently, a high busy percentage does not alone prove no capacity remains. Conversely, a single synchronous stream can suffer unacceptable response time without reaching maximum bandwidth. Read queueing and response-time indicators with device topology and workload parallelism. Virtualized and distributed storage can hide additional queues. Avoid a universal rule such as any particular utilization number means saturation; test whether extra offered work increases useful throughput or mostly increases delay.

Buffered completion is not durable completion

A buffered write can finish after copying data into memory, before persistent media has committed it. Later writeback can consume storage capacity and stall foreground work. A durability request such as fsync has a different completion contract and may include filesystem and device flush costs. Do not compare a cached read benchmark with a cold read or an asynchronous write with a durable transaction as if they were the same workload. Never remove durability guarantees merely to improve a chart; first identify what the application promises to users and what failure recovery requires.

bash Read pending writeback and block accounting

grep -E '^(Dirty|Writeback):' /proc/meminfo
cat /proc/diskstats

Take comparable snapshots in the lab and derive deltas using the documented fields. Raw diskstats values are cumulative, and device sectors in these statistics use their documented accounting size rather than an assumed filesystem block size.

Locate the delay before choosing a remedy

Compare a healthy interval with the affected request mix. If application latency rises while block latency is stable, examine application queues, metadata, remote dependencies, and locking rather than immediately changing a block scheduler. If device response and queue depth rise together under matched demand, investigate contention and the backing service. Preserve errors and availability too: retries can inflate I/O after an initial fault. All examples in this lesson are observational; filesystem repair, cache dropping, scheduler changes, and storage load generation are outside the authorized host activity.

Common errors and fixes

Equating 100 percent busy with universal saturation

Why it happens: Device parallelism and accounting semantics were ignored.
How to fix it: Correlate response, queue depth, throughput, and offered work.
Prevent it next time: Use a workload-specific capacity curve.

Summing stacked-device traffic

Why it happens: The same operations were counted at several layers.
How to fix it: Map logical and physical boundaries before aggregation.
Prevent it next time: Keep topology with every storage report.

Comparing cached and durable operations

Why it happens: The completion contracts differ.
How to fix it: Repeat with matched cache state and durability semantics.
Prevent it next time: Specify persistence requirements before benchmarking.

Check your understanding

1. A service writes 4 KiB synchronously and has low MB/s but high latency. Is bandwidth the only concern?

Show worked solution

No. Small operations, serialization, flushes, and queueing may dominate. Compare operation rate, durable completion latency, and the full storage path under the same workload before assuming a bandwidth upgrade is the right fix.

2. A virtual disk and its backing logical device both show traffic. Should their bytes be added?