Linux performance / Basic: measurement and resource signals / Lesson 1

Build a trustworthy performance baseline

Define a user-visible goal, record workload context, and compare observations before proposing any Linux tuning change.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Turn a vague slowdown into a measurable question.
Distinguish counters, gauges, rates, and observation scope.
Design a repeatable baseline and a reversible hypothesis.

Start with useful work

Performance describes how a system completes useful work under stated conditions. A fictional report service might require a completion deadline and an acceptable error rate at a particular arrival rate. CPU utilization alone is not that goal. Record the request type, input size, concurrency, success definition, and measurement interval. Decide whose experience matters: a client can wait in a proxy before the application starts timing. A precise question such as why the same report now misses its deadline gives the investigation a target and prevents optimization of an unrelated metric.

Record the experimental context

A baseline is a documented reference, not a screenshot of an idle machine. Include kernel and tool versions, CPU topology, memory size, virtualization, effective container limits, application revision, and relevant configuration. Record cache warmth and background activity because they can alter repeated results. Host observations and container observations may describe different boundaries. Prefer a matching disposable environment for experiments, and retain raw timestamps rather than only rounded dashboard values. Configuration records should omit credentials and unnecessary personal data while preserving the details needed to reproduce the workload.

bash Inspect version, topology, and observation scope

uname -r
lscpu
cat /proc/self/cgroup

These non-runnable study commands read context only. Their real output depends on the prepared lab, and none is presented as a benchmark result.

Match arithmetic to the metric type

A cumulative counter counts events since a defined start, while a gauge describes a current quantity. A rate normally comes from the difference between two counter samples divided by elapsed time. Reboots, process replacement, counter reset, and device replacement invalidate a naïve subtraction. Sample the same object at both ends and use a monotonic interval when computing durations. An average over a long interval can hide a brief stall, so align collection windows with the incident and keep the original resolution before deciding how much aggregation is appropriate.

bash Compute a rate from declared synthetic counters

awk 'BEGIN { before=1200; after=1560; seconds=12; printf "synthetic_rate=%.1f events/s\n", (after-before)/seconds }'

Expected output

synthetic_rate=30.0 events/s

The chosen numbers demonstrate subtraction and units: 360 events over 12 seconds means 30 events per second. They are teaching data, not measurements from this host.

Compare like with like

Collect several representative windows before changing anything. Compare equivalent input distributions and arrival patterns, and separate warm-up from the stated steady interval. Record throughput, latency distribution, errors, and resource evidence together. If work finishes faster because requests were rejected or an expensive feature was disabled, the result does not satisfy the original contract. Where practical, alternate baseline and candidate runs in the lab to expose time drift. Keep enough repetitions to understand normal variability instead of selecting the most flattering single run.

Make one falsifiable prediction

Write a mechanism, a prediction, a small proposed change, a stop condition, and a rollback before experimentation. For example, if queue growth follows CPU-budget exhaustion, the next observation should show throttling in the relevant cgroup during the same window. That is stronger than guessing a kernel parameter from an unrelated article. Prefer read-only evidence first and make no host changes in this course. Preserve an unsuccessful experiment too: it can disprove an explanation and prevent a future engineer from repeating the same uninformative adjustment.

Common errors and fixes

Comparing different workloads

Why it happens: Input size or arrival rate changed between observations.
How to fix it: Repeat matched lab windows and record the difference.
Prevent it next time: Version the workload description with the results.

Treating a counter as a rate

Why it happens: A cumulative total was read without an interval.
How to fix it: Use comparable samples and divide their delta by elapsed time.
Prevent it next time: Label units and reset boundaries explicitly.

Changing several settings at once

Why it happens: The experiment cannot identify which change mattered.
How to fix it: Restore the baseline and test one justified candidate.
Prevent it next time: Require a prediction and rollback for each experiment.

Check your understanding

1. A report is faster after half the requests are rejected. Has performance improved?

Show worked solution

The completion time of accepted work may improve, but the original service contract is not met. Compare successful throughput, errors, and full client latency under the same offered demand before accepting the change.

2. A counter rises from 400 to 580 over nine seconds. Calculate its rate and name a reset check.