Distinguish high I/O latency from saturation

Understand the likely causes

• Small synchronous operations, flushes, filesystem metadata, application locks, deferred writeback, shared-device contention, or remote storage can add delay. A highly parallel device’s busy percentage is not a universal capacity boundary. Logical and physical counters may represent the same operation at different layers and must not be blindly added.

1. Map the operation and backing path

Identify whether completion means copied-to-memory, file close, durable sync, or a committed transaction. Map the destination to its mount, logical device, backing devices, and any remote layer. Record operation size, read/write mix, concurrency, cache state, and recent data-layout changes. Check capacity and errors separately because no-space and failing-device symptoms need their own safe response. Do not start a benchmark, filesystem repair, or cache flush to reproduce the incident on the host. Use the existing evidence and a disposable data copy.

findmnt --target /tmp
lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINTS
iostat -xz 1 5

2. Compare response, queues, rate, and demand

Use the same time window for application latency, device response indicators, operation rate, bandwidth, and queue depth. High latency with low bandwidth can be legitimate for small durable operations or a serialized access pattern. High busy time on a concurrent device may coexist with spare throughput. A stronger saturation hypothesis appears when extra offered work no longer increases useful completion and mostly grows waiting. Do not infer the whole curve from one point; preserve device topology and the exact workload semantics with the comparison.

3. Separate writeback and waits above the device

Compare dirty/writeback memory and I/O pressure with the application’s blocked stages. Buffered acceptance can hide work that later stalls foreground requests. If block response stays stable while application time rises, inspect locks, metadata, remote dependencies, or application queues before blaming the device. If both queueing and response rise at the backing layer, investigate contention and service limits there. Retries after errors can amplify demand, so preserve the first failure rather than only the later busy interval.

grep -E '^(Dirty|Writeback):' /proc/meminfo
cat /proc/pressure/io
pidstat -d -p 4242 1 5

4. Propose the smallest correctness-preserving change

Depending on evidence, test bounded batching, fewer concurrent writers, corrected file layout, reduced repeated reads, or justified backing capacity on an isolated copy. Preserve durable completion requirements and valid data content. A queue-limit change can shift failures or latency elsewhere, so monitor dependencies and recovery behavior. Never disable required synchronization or claim a universal scheduler, readahead, or dirty-memory setting. State the predicted metric change, stop criteria, and rollback before a later authorized lab trial.

Keep it from recurring

• Retain topology and workload metadata with storage charts, monitor deferred queues and tail latency, and bound background data production. Test backup, maintenance, and report workloads together on representative copies. Revisit capacity assumptions after storage, filesystem, virtualization, or application changes instead of retaining old universal thresholds.