Linux performance / Intermediate: mechanism and scoped diagnosis / Lesson 10
Trace writeback and filesystem delay
Relate dirty memory, filesystem metadata, durability boundaries, and block queues without sacrificing correctness for apparent speed.
On this page
Lesson goals
- Explain foreground writes and background writeback.
- Separate metadata, flush, and data-volume costs.
- Design a bounded diagnostic comparison of storage behavior.
Buffered writes move the waiting boundary
Buffered I/O lets an application hand data to memory and defer persistent storage work. This can smooth bursts but cannot create unlimited backing capacity. Dirty memory accumulates when production outpaces writeback, and foreground tasks may later experience throttling or reclaim-related delay. A short fast write interval followed by a long stall is therefore different from sustained durable throughput. Measure the whole cycle, including application queues and recovery requirements. Do not accept a result that ends timing before deferred work becomes visible and then declares the storage problem solved.
Metadata operations can dominate small workloads
Creating many small files, looking up directories, changing attributes, and committing filesystem metadata can cost more than the payload volume suggests. A service writing few bytes can still suffer synchronization or metadata contention. Map whether the application appends one file, replaces files through rename, or creates many independent objects. Filesystem and mount semantics differ, so a result from one layout cannot be generalized to all systems. Check free space, inode availability, and errors too: capacity or corruption symptoms need their own safe operational process rather than performance tuning.
findmnt --target /tmp
df -h /tmp
df -i /tmp
cat /proc/pressure/ioThese lab reads do not repair or remount anything. They separate filesystem identity and capacity from time lost to I/O; no numeric outcome is assumed.
Durability has a precise failure contract
A successful write does not necessarily mean data survived a sudden power loss. Applications use supported synchronization and transaction protocols to establish stronger guarantees. fsync and related operations have filesystem and device implications, and directory updates can require their own durability handling. A database log flush is not interchangeable with a buffered export file. Changing synchronization or mount policy can alter recovery correctness, not just timing. Review the application and filesystem documentation before proposing any such experiment, and never remove a required guarantee to improve a benchmark number.
Correlate dirty data with device completion
Take aligned observations of dirty and writeback memory, filesystem/application latency, block completions, queue depth, and pressure. An increase in dirty pages alone is not proof of a fault; a sustained growth pattern with growing stalls supports a capacity mismatch hypothesis. Device averages can hide a small number of expensive flushes or competing workloads. Read detailed operation types when the installed tool supports them and preserve units. Do not assume every wait is direct device service, because filesystem locks and application synchronization can add delay above the block layer.
grep -E '^(Dirty|Writeback):' /proc/meminfo
iostat -xz 1 5
pidstat -d -p 4242 1 5The synthetic PID belongs only to the prepared lab. Tool fields and process-I/O accounting vary; compare their documented scope and interval rather than expecting byte-for-byte equality with device totals.
Change the producer before guessing global knobs
A smaller batch, bounded queue, fewer concurrent writers, or corrected retention can reduce burst pressure while preserving durability. Test one candidate on a disposable data copy, use the same input distribution, and continue measuring until deferred work settles. Record both throughput and tail latency for foreground and background consumers. Global dirty thresholds can redistribute waiting and memory risk, so they are not universal repairs. This course provides no host-writing sysctl command and runs no storage load generator; any later authorized experiment needs explicit stop criteria, recovery space, and a rollback.
Common errors and fixes
Timing only buffered acceptance
- Why it happens
- Deferred work and durability were excluded.
- How to fix it
- Measure the complete required completion boundary.
- Prevent it next time
- Document persistence semantics with each result.
Assuming low bytes means cheap I/O
- Why it happens
- Metadata and synchronization costs were omitted.
- How to fix it
- Inspect operation types and filesystem behavior.
- Prevent it next time
- Characterize layout and file-creation patterns.
Removing fsync to claim a win
- Why it happens
- The workloadโs correctness contract changed.
- How to fix it
- Restore the guarantee and compare equivalent operations.
- Prevent it next time
- Make recovery correctness an acceptance condition.
Check your understanding
1. A writer is fast for ten seconds, then stalls while Dirty falls. What hypothesis fits?
Show worked solution
It may have accumulated dirty data faster than backing storage could commit it, then paid deferred writeback or throttling cost. Correlate device completions and application waits over the full cycle before concluding; the pattern alone does not name the exact limiter.
2. Why must a database flush benchmark and an export write be compared carefully?
Show worked solution
Their durability, operation size, access pattern, and completion boundaries may differ. Match those semantics and the storage path first. Otherwise a faster export does not demonstrate that durable transaction latency should be equally low.
Key takeaways
- Trace deferred work and preserve durable correctness.
- Correlate filesystem, memory, application, and device evidence.