Linux performance / Advanced: topology, observability, and controlled tuning / Lesson 14

Design bounded eBPF observations

Use kernel event reasoning, filters, histograms, and lifecycle controls to plan an observability experiment without loading programs on the host.

About 4 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish tracepoints, probes, maps, and userspace readers.
Explain verifier, BTF, privilege, and portability boundaries.
Plan bounded collection with loss and overhead checks.

Choose an event that can answer the question

eBPF can execute verified programs at supported attachment points and aggregate observations in maps or emit records to userspace. A tracepoint exposes a defined event interface, while kprobes and uprobes attach to kernel or userspace execution locations with different stability and symbol requirements. Select the event from a specific question, such as the distribution of a scoped operation’s elapsed time. Counting function entries does not automatically measure latency or distinguish success from failure. Identify the start event, completion event, correlation key, and missing-event behavior before writing a probe.

Verification is necessary but not a cost guarantee

The kernel verifier checks program constraints according to the supported BPF model, but an accepted program can still impose meaningful overhead or collect sensitive information. Permissions and allowed program types depend on kernel configuration and security policy. BTF describes types and supports tooling portability, but it does not make every kernel function or argument a stable interface. CO-RE relocation helps adapt supported type layouts; it is not permission to ignore semantic changes. Review the exact kernel, toolchain, attachment interface, and least-privilege collection plan before an authorized isolated experiment.

bash Inspect tool and type-metadata prerequisites

bpftool version
test -r /sys/kernel/btf/vmlinux && printf 'kernel BTF is readable\n'

These study reads neither load nor attach a BPF program. Readable BTF is one prerequisite, not proof that a proposed probe is compatible or authorized.

Filter early and aggregate deliberately

Scope collection to an authorized cgroup, process lifetime, or operation class before exporting data. Per-event printing can overwhelm the reader and perturb a busy path; bounded histograms or counters often answer the question with less data movement. Maps still need bounded keys and a cleanup strategy for exited processes or unmatched operations. A PID can be reused, and threads can overlap requests, so an overly simple key can join unrelated events. Preserve units, bucket boundaries, sample population, and failure outcomes so a histogram remains interpretable rather than merely attractive.

Specify the observation lifecycle

A collection plan defines target, duration, event rate budget, data retention, reader behavior, and guaranteed detach procedure. Include lost records, buffer overflow, map insertion failures, and unmatched starts in the report rather than silently dropping them. If the observation cannot keep up, its apparent latency distribution may be biased. Compare application latency and CPU cost before and during a minimal lab trial. Do not attach to every function or collect payloads by default. This course displays a design sketch only and never loads probes, writes tracefs, or changes BPF access policy.

text Read a non-executable tracing design

scope: synthetic lab-report cgroup
start: remember monotonic timestamp by operation identity
finish: subtract matching timestamp; increment bounded latency bucket
missing: count unmatched events separately
stop: detach after the approved interval; export counts; clear owned state

This pseudocode states correlation and lifecycle obligations without claiming a portable BPF implementation. A real implementation must use the chosen interface’s actual fields and handle concurrency correctly.

Trace evidence narrows a mechanism

A long observed kernel operation can explain part of a request delay only when timestamps and scope align. The event may include waiting, CPU work, or a downstream dependency depending on the interface. A histogram does not by itself identify which application input caused the tail. Correlate with bounded request metadata and independent resource evidence, then test a targeted hypothesis on a copy. Preserve the attachment and type versions with the result. If a simpler counter or application span already answers the question, prefer that lower-cost observation instead of adding a more complex trace.

Common errors and fixes

Verifier acceptance is called zero overhead

Why it happens: Program validity was confused with collection cost.
How to fix it: Measure overhead and bound event volume in the lab.
Prevent it next time: Include perturbation checks in every plan.

Correlating by PID alone

Why it happens: Reuse or overlapping operations join unrelated events.
How to fix it: Use interface-appropriate operation and lifetime identity.
Prevent it next time: Design concurrency and missing-event handling first.

Ignoring dropped trace records

Why it happens: Reader loss biased the observed population.
How to fix it: Report loss and reduce collection scope or volume.
Prevent it next time: Define a loss budget and stop condition.

Check your understanding

1. A histogram looks fast but the reader reports many lost records. What can you conclude?

Show worked solution

Only the retained population looks fast; missing events may hide expensive periods. Reduce collection volume, preserve loss counters, and repeat a bounded authorized lab observation before making a claim about the complete workload.

2. Why is readable BTF insufficient to promise probe portability?