Linux performance / Advanced: topology, observability, and controlled tuning / Lesson 15
Interpret call graphs and off-CPU time
Connect sampled execution, stack quality, blocked waits, and asynchronous request lifetimes without mistaking sample width for total latency.
On this page
Lesson goals
- Separate inclusive and self sample attribution.
- Recognize stack-unwinding and symbolization limits.
- Combine CPU profiles with wait and request evidence.
A profile represents its sampled event population
A CPU sampling profile estimates where execution occurred for the chosen event and scope. It does not directly represent every millisecond of request elapsed time. A function can dominate samples because it runs often, runs for long per call, or belongs to unrelated work in an overly broad collection. Normalize the result to useful work and preserve the sampling configuration. A narrow profile during a representative request phase is usually easier to interpret than an unbounded whole-host capture. Do not claim an exact percentage of application latency from CPU sample share alone.
Self and inclusive attribution answer different questions
Self attribution concerns samples directly at a symbol, while inclusive views include descendant work according to the report’s aggregation. A wrapper can appear broad because it calls expensive children, not because its own instructions are costly. Recursive paths and folded stacks require careful interpretation. A flame graph’s horizontal width commonly represents sample weight rather than chronological duration; adjacent shapes need not be simultaneous or sequential events. Read the capture and visualization conventions before using a picture to propose a rewrite. Keep raw profile data and exact binary identity for later reanalysis.
perf record -F 49 -g -p 4242 -o lab.perf.data -- sleep 5This non-runnable example requires scoped authorization and a prepared target. The frequency is illustrative, not a universal safe setting; stack method, event support, overhead, and output retention must be reviewed.
Unknown frames are missing evidence
Stack reconstruction depends on unwind information, frame pointers or other supported methods, compiler choices, architecture, and tool compatibility. Stripped binaries, missing debug symbols, JIT-generated code, and optimized inlining can reduce source attribution. A broken chain can falsely concentrate samples at the last visible frame. Keep build identifiers and appropriate symbol artifacts under controlled access. Do not guess that an unknown frame is harmless or that enabling a different unwinder has no cost. Validate stack quality with a small known path before interpreting a large application profile.
Wait time needs a different observation
A slow request with little CPU may be blocked on storage, a lock, a connection pool, or a timer. Off-CPU observations can identify where a thread slept, but a sleep stack does not automatically identify the owner or external cause. Runnable scheduling delay is another state and should not be combined blindly with blocking. Asynchronous runtimes can move a request across threads, so thread identity alone may not reconstruct its critical path. Combine application correlation, scheduler evidence, and bounded wait instrumentation to explain elapsed delay rather than forcing every problem into a CPU flame graph.
perf report --stdio --no-children -i ./lab.perf.data
perf report --stdio --children -i ./lab.perf.dataThese commands read an existing synthetic-lab capture and do not start a new recording. Report percentages depend on filters and aggregation; preserve both views and validate symbols before assigning source cost.
Change the measured critical mechanism
After a profile identifies repeated expensive work, propose a narrow application change and compare equivalent results. Reducing copies, avoiding repeated parsing, shortening a lock, or batching within a bounded memory budget may help for different mechanisms. A smaller sample share can simply mean another function became more expensive, so retain absolute time and completed work. Repeat the capture only when a new question requires it and include profiling overhead in the comparison. A justified optimization should improve the stated service objective without breaking correctness, increasing errors, or hiding delay in a downstream queue.
Common errors and fixes
Reading sample share as elapsed request share
- Why it happens
- The profile observes execution rather than all waiting.
- How to fix it
- Combine CPU samples with request and wait timing.
- Prevent it next time
- State event population beside every profile.
Optimizing a broad wrapper’s self code
- Why it happens
- Inclusive child work was mistaken for self cost.
- How to fix it
- Compare inclusive and self attribution.
- Prevent it next time
- Read aggregation conventions before source changes.
Ignoring broken stack chains
- Why it happens
- Missing unwind or symbols distorted attribution.
- How to fix it
- Validate a known path and obtain matching artifacts.
- Prevent it next time
- Record build identity and stack method.
Check your understanding
1. A request takes one second but consumes 20 ms CPU. Can CPU profiling explain all delay?
Show worked solution
No. Most elapsed time lies outside execution under that accounting. Investigate blocking, runnable delay, queues, and asynchronous dependencies, while using the CPU profile only for the execution component and preserving timing-boundary consistency.
2. A function’s percentage falls but its absolute CPU time stays constant. What changed?
Show worked solution
The denominator or other work may have increased. The function did not become cheaper by that evidence alone. Compare completed work and absolute time under matched demand before claiming an optimization.
Key takeaways
- Validate sample population, stack quality, and aggregation.
- Use wait and request evidence for time outside execution.