Linux performance / Advanced: topology, observability, and controlled tuning / Lesson 18

Deliver reversible, evidence-based tuning

Turn a measured mechanism into a controlled change, compare full outcomes, and retain a tested rollback instead of a collection of magic settings.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Write a causal prediction and a counterexample.
Compare paired experiments with explicit acceptance conditions.
Plan staged adoption, rollback, and ongoing verification.

A tuning proposal needs a causal story

State the user-visible failure, the suspected resource mechanism, and the evidence linking them in the same interval. Predict what should change if the hypothesis is true and what observation would weaken it. For example, a batch-size reduction may lower peak retained memory and pressure while preserving completion semantics. A copied sysctl value without a mechanism is not a tuning proposal. Prefer application demand or scoped policy when it addresses the cause more narrowly than a host-wide setting. Keep the original configuration, workload definition, and recovery route before considering any later action.

Control confounders and preserve a counterfactual

Use matched inputs, data state, cache policy, versions, and arrival model. Alternate baseline and candidate trials where practical to reveal environmental drift. Change one meaningful factor at a time unless the experiment explicitly studies an interaction. Retain failed and neutral results, not only the best run. A concurrent deployment, noisy neighbor, or different request mix can create a false improvement. Document uncertainty and repeat enough to characterize normal variation. The baseline should remain reproducible so reverting the candidate restores the expected behavior rather than an undocumented historical state.

Compare complete outcomes and absolute quantities

Calculate per-work cost and preserve successful throughput, latency tails, errors, resource pressure, and effects on dependencies. A percentage can improve because its denominator changed. A mean can improve while a required tail worsens. A service can look faster after rejecting more requests or weakening durability. Acceptance therefore needs the original correctness and outcome contract. Use synthetic arithmetic to check units, but do not invent benchmark conclusions. A real conclusion requires measured matched observations and a clear explanation of how the proposed mechanism changed.

bash Normalize declared synthetic totals by equal work

awk 'BEGIN { baseline_ms=2400; candidate_ms=2100; jobs=100; printf "synthetic_costs=%.1f,%.1f ms/job\n",baseline_ms/jobs,candidate_ms/jobs }'

Expected output

synthetic_costs=24.0,21.0 ms/job

The arithmetic gives 24 and 21 ms per job for the stated totals. It proves only that calculation; it does not establish a real improvement, statistical confidence, or acceptable p99.

Rollback is a tested behavior, not a sentence

Specify the exact previous state, the trigger for reversal, the person or process authorized to act, and the verification after reversal. Some changes affect durable data, caches, or schema compatibility and are not undone by restoring one configuration line. Keep resource reserve for recovery and avoid making the management channel depend on the experimental path. A later staged rollout should begin with a small authorized scope and compare equivalent control traffic, observing neighbors and dependencies. This course performs no rollout or host mutation; it teaches how to prepare a concrete reviewable plan.

text Read a reversible experiment record

hypothesis: bounded synthetic batch reduces retained working set
prediction: lower pressure delta with equal completed work
reject: worse required tail, more errors, or changed correctness
rollback: restore versioned batch policy on the isolated copy
verify: original baseline and recovery behavior return

This non-executable record names mechanism and rejection conditions without prescribing a universal numeric setting. A real plan adds measured thresholds, artifact versions, and authorized recovery details.

A successful change still has a validity boundary

Record the tested workload range, hardware and kernel context, effective limits, evidence, and remaining risks. Recheck after a major workload, application, kernel, or topology change rather than assuming the result lasts forever. Remove unsupported tuning folklore when its original mechanism no longer applies, using the same controlled process. A concise report should allow another engineer to explain why the candidate passed, what would trigger rollback, and which metric detects recurrence. The final product is a maintainable operational decision, not the largest possible collection of custom parameters.

Common errors and fixes

A faster chart is accepted without correctness checks

Why it happens: Rejections or weakened guarantees changed the contract.
How to fix it: Restore equivalent outcomes and test full acceptance.
Prevent it next time: Version correctness requirements with performance goals.

Rollback means changing one line back

Why it happens: Durable effects and recovery dependencies were ignored.
How to fix it: Test the full reversal and verify original behavior.
Prevent it next time: Include state compatibility and reserve in the plan.

One successful run becomes a universal recommendation

Why it happens: Variability and validity boundaries were omitted.
How to fix it: Repeat matched trials and state tested conditions.
Prevent it next time: Revalidate after material environment changes.

Check your understanding

1. A candidate reduces CPU per job but doubles database waiting. What should acceptance consider?

Show worked solution

Evaluate the complete request objective and shared dependency impact. The CPU reduction may merely move work or increase concurrent pressure downstream. Compare tails, successful throughput, errors, and queue age before deciding whether the overall service improved.

2. What observation would weaken the batch-memory hypothesis in the example?