Linux performance / Intermediate: mechanism and scoped diagnosis / Lesson 7

Use perf counters to test CPU hypotheses

Choose scoped events, interpret counter ratios, and recognize hardware and permission limits before profiling an application.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Separate event counting from sampled profiling.
Interpret cycles and instructions without treating IPC as a goal.
Account for event support, multiplexing, and scope.

Select events from a mechanism

perf stat counts selected events during a defined observation, while perf record collects samples for later attribution. Counting can test whether more work or different execution behavior accompanies a slowdown; it does not identify every expensive source line. Start with an application-level comparison and a small event set. Record process or thread scope, inheritance, interval, and user/kernel exclusions. A system-wide count includes unrelated work and requires a different permission and privacy review. The examples below describe a synthetic authorized lab target and are never executed against this host.

Events depend on the platform

Hardware counters depend on the CPU model, PMU, virtualization, and kernel support. perf list describes locally recognized events, but an event name can have architecture-specific meaning or constraints. Containers may lack access even when the host supports collection. perf_event_paranoid and capabilities participate in security policy; denial is not a reason to disable host controls. CAP_PERFMON provides a more specific privilege model than granting broad administrative power, subject to installed kernel behavior. Coordinate approved collection rather than changing security policy from a lesson or copying raw event encodings across CPUs.

bash Inspect tool support and access policy

perf --version
perf list
cat /proc/sys/kernel/perf_event_paranoid

These study commands do not relax policy or begin a recording. Available events and permissions are lab-specific; retain their output with any later approved experiment.

Ratios need equivalent populations

Instructions per cycle divides retired instructions by counted cycles under the selected definitions. A higher IPC is not automatically better application performance: the workload may execute more total instructions, omit required work, or run a different code path. Compare completed useful work, elapsed time, instructions per request, and relevant event ratios together. CPU frequency, branch behavior, memory access, and vectorization can change relationships. A generic cache-miss count may not identify the exact cache or explain elapsed delay, so use the CPU’s event documentation and a falsifiable hypothesis rather than a universal target ratio.

Keep the observation bounded

A PID-targeted count should last only the interval needed for the question and must identify the correct process lifetime. Short samples may miss representative work; long samples may mix deployments or demand phases. Selecting more events than hardware can schedule can cause multiplexing, with scaled estimates and reduced time running. Check that related events are measured compatibly before dividing their counts. Record unavailable or unsupported events rather than silently replacing them. Do not publish process paths or trace artifacts without reviewing sensitive context, because profiling data can expose application details.

bash Study a bounded count for a synthetic process

perf stat -e task-clock,cycles,instructions -p 4242 -- sleep 5

PID 4242 is a fictional authorized lab process. The sleep bounds the collection interval; no numerical output is supplied because counts depend on actual workload, PMU support, permissions, and multiplexing.

Normalize by completed work

If a candidate completes twice as many reports, a larger instruction count can still mean fewer instructions per report. Conversely, an improved hardware ratio with unchanged elapsed latency may not address the actual bottleneck. Compare the same request population, completion semantics, and background context. Use a counter result to choose the next investigation, such as a bounded call graph or an application algorithm review. Keep the baseline event definitions and raw report so another engineer can reproduce the reasoning. No event count justifies a host kernel change without a measured causal link.

Common errors and fixes

Using IPC as the acceptance objective

Why it happens: A hardware efficiency ratio replaced useful completion.
How to fix it: Compare elapsed time, errors, and work-normalized counters.
Prevent it next time: Keep the service objective above diagnostic ratios.

Dividing incompatible multiplexed counts

Why it happens: Events did not observe equivalent execution time.
How to fix it: Inspect running proportions and event grouping constraints.
Prevent it next time: Use a small justified event set.

Disabling perf security after denial

Why it happens: An access boundary was treated as a tool fault.
How to fix it: Request scoped approved collection in the lab.
Prevent it next time: Plan privileges and privacy before profiling.

Check your understanding

1. Run A retires 200 million instructions for 100 reports; B retires 300 million for 200 reports. Compare instruction cost.

Show worked solution

A uses two million instructions per report; B uses 1.5 million. B has lower measured instruction cost per completed report, but acceptance still requires equivalent reports, latency, correctness, and errors. Totals alone would suggest the wrong conclusion.

2. Why can an unsupported event produce no useful cache conclusion?