Go / Advanced: Safety, Performance, and Service Design / Lesson 16

Profiling and Runtime Diagnostics

Choose the diagnostic instrument that answers the actual question and preserve a reproducible workload before optimizing.

About 5 min readContent reviewed 2026-10-25

On this page

Lesson goals

Distinguish CPU, heap, goroutine, block, and trace evidence.
Collect a bounded local profile with explicit cleanup.
Interpret samples without inventing universal timings.

Start from a symptom and a hypothesis

High latency, high CPU, retained memory, and blocked goroutines suggest different questions. A CPU profile samples executing work; a goroutine profile shows stack states; heap profiles describe sampled allocation or retained-memory views; execution tracing can reveal scheduling and blocking relationships. Choose a tool that can distinguish your competing explanations.

Record workload size, input distribution, concurrency, toolchain, platform, and build flags. A profile from an idle toy program may be technically valid but irrelevant to a production hot path. First verify outputs against a correctness oracle, then reproduce the resource symptom using bounded synthetic data.

Create and finish a local CPU profile

The first program writes a profile only to a newly created temporary file, stops profiling before inspecting it, and removes the file on return. The known calculation makes the work observable. A nonempty profile file demonstrates collection and finalization; it does not promise that the short workload produced enough samples for a meaningful performance conclusion.

StartCPUProfile can fail, and StopCPUProfile completes writing the profile data. Close the file after stopping. In a real investigation, preserve the profile and matching executable deliberately in an approved local artifact location rather than deleting them, and record the workload. Do not expose a profiling HTTP endpoint merely to collect a local example.

go A temporary, finalized CPU profile

package main

import (
	"fmt"
	"os"
	"runtime/pprof"
)

func main() {
	file, err := os.CreateTemp("", "go-course-cpu-*.pprof")
	if err != nil {
		panic(err)
	}
	defer os.Remove(file.Name())
	defer file.Close()
	if err := pprof.StartCPUProfile(file); err != nil {
		panic(err)
	}
	var total uint64
	for i := uint64(0); i < 1000000; i++ {
		total += i
	}
	pprof.StopCPUProfile()
	info, err := file.Stat()
	if err != nil {
		panic(err)
	}
	fmt.Println("sum", total, "profile written", info.Size() > 0)
}

Expected output

sum 499999500000 profile written true

The known sum is 499999500000; profile finalization writes data without claiming a timing result.

Attach meaningful diagnostic labels

pprof labels can attribute supported profile samples to logical work such as a processing stage. Choose low-cardinality nonsecret labels; individual user identifiers or arbitrary request contents can create noisy profiles and expose sensitive information. Labels assist diagnosis but do not change synchronization, context cancellation, or application ownership.

The second example attaches one label and retrieves it from the derived context inside pprof.Do. It prints a deterministic value rather than a sample count. Actual profile interpretation still requires a representative collection interval and the matching program build. Label support differs among profile kinds, so consult the package contract.

go A scoped diagnostic label

package main

import (
	"context"
	"fmt"
	"runtime/pprof"
)

func main() {
	stage := ""
	pprof.Do(context.Background(), pprof.Labels("stage", "parse"), func(ctx context.Context) {
		stage, _ = pprof.Label(ctx, "stage")
	})
	fmt.Println("stage", stage)
}

Expected output

stage parse

The label is carried by the context supplied to the callback and reads back as parse.

Samples identify candidates, not automatic causes

A hot function can represent useful work, a poor algorithm, repeated allocation, or a symptom of another design choice. Compare flat and cumulative costs and inspect call relationships before editing. A goroutine blocked on a channel may be waiting correctly; many identical owned workers that never finish after cancellation suggest a lifecycle defect.

Profiles add overhead and sampling uncertainty. Compare like workloads and builds, and avoid measuring an instrumented race build as if it were a normal production performance result. For latency, inspect distributions and slow-path traces rather than only averages. Keep a separate record of correctness regressions and performance changes.

Keep diagnostic access bounded and intentional

net/http/pprof registers diagnostic handlers when imported in its documented manner. In a service, route and protect those endpoints intentionally; do not let a default mux accidentally publish them on a public listener. Profiles can contain stack paths, labels, and details useful only to trusted operators. Local command-line profiling avoids that network boundary entirely.

After identifying a candidate, make one justified change, rerun the correctness suite, and collect comparable evidence. Retain the input and build metadata so a future runtime change can be evaluated honestly. Do not fabricate a universal speedup from one sample or assume a function appearing low in one profile is always irrelevant.

Common errors and fixes

A CPU profile is used to explain time spent waiting on I/O.

Why it happens: Executing CPU samples were confused with wall-clock latency.
How to fix it: Use blocking/trace evidence and request timing appropriate to the question.
Prevent it next time: State which resource each measurement represents.

A profile is incomplete or empty after abrupt exit.

Why it happens: Profiling was not stopped and finalized correctly.
How to fix it: Stop collection before closing the destination or terminating.
Prevent it next time: Make profile cleanup part of the owning function.

Debug endpoints become publicly reachable.

Why it happens: Default routing exposed diagnostics unintentionally.
How to fix it: Bind and authorize diagnostic access deliberately.
Prevent it next time: Review listeners and mux ownership during deployment.

Check your understanding

1. A service has high latency and low CPU. Which evidence would you collect before rewriting arithmetic loops?

Show worked solution

Inspect request timing, goroutine stacks, blocking profiles, and execution traces under a representative workload. Look for downstream waits, lock contention, queue admission, and incomplete cancellation. Verify that the CPU profile interval covers the symptom, but do not assume low sampled CPU proves the system is healthy. Choose an intervention only after distinguishing waiting from executing work.

2. Why is a profile label containing every raw customer identifier a poor default?