PyTorch / Storage, restoration, and transformed execution / Lesson 15

Profiling and resource evidence

Separate correctness, analytical storage counts, profiler observations, and measured user-visible performance.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Define a bounded workload and timing boundary before profiling.
Interpret CPU event collection without inventing benchmark results.
Name memory categories and preserve behavior during optimization.

A performance question requires a workload contract

Specify input shapes, dtypes, device, batch and sequence lengths, module and gradient modes, and the output required by the task. Decide whether timing includes loading, validation, data preparation, preprocessing, queueing, forward computation, backward, and optimizer work. A warm isolated forward and a first complete request answer different questions. Keep a correct small reference before measuring, and choose bounded representative inputs. This course performs no PyTorch performance measurement. A tiny mathematical example can establish an expected output but cannot establish throughput, peak memory, or which operator is the real bottleneck.

A profiler records events under supported collection rules

The illustrative program requests CPU activity and shape recording around a labeled arithmetic region. On a compatible supported build, the expected scalar is fifteen and the label appears among grouped events. No duration is printed because it depends on actual execution and collection overhead. Profiler support, event scope, sampling or scheduling choices, and operator aggregation need their documented context. Inclusive time and self time answer different questions, and nested regions should not be summed blindly. A profiler observation can suggest a hypothesis about work; it is not automatically an end-to-end latency benchmark.

python Illustrative CPU PyTorch 2.14 API program; torch unavailable, expected output derived

import torch
from torch.profiler import profile, record_function, ProfilerActivity
x = torch.tensor([1.0,2.0,3.0],dtype=torch.float64,device="cpu")
with profile(activities=[ProfilerActivity.CPU],record_shapes=True) as observation:
    with record_function("lab_forward"):
        value = (2*x+1).sum()
print("result", value.item())
print("label present", any(event.key == "lab_forward" for event in observation.key_averages()))

Expected output

result 15.0
label present True

Expected behavior assumes a build supporting the documented CPU profiler. No profiler, timing, tensor operation, or event collection was executed; no timing value is fabricated.

Memory estimates must name their categories

The standard-library companion counts one linear layer’s weights and biases, equally sized gradients and two assumed optimizer arrays, plus input and output payload. Doubling the batch grows input/output storage but leaves those fixed arrays unchanged. Actual training can retain many intermediate activations, temporary workspaces, extra precision copies, allocator reservations, and object overhead. A view can share storage while a materialized transform allocates another payload, so logical element counts alone can double-count or miss ownership. Label every assumption and omission instead of presenting a model-file size as measured peak training memory.

python Executed deterministic standard-library companion

features, outputs, batch, bytes_per_value = 32,4,12,8
parameters = features*outputs+outputs
parameter_bytes = parameters*bytes_per_value
fixed = parameter_bytes*4
def named_payload(n):
    return fixed+n*(features+outputs)*bytes_per_value
print("parameter count", parameters)
print("assumed payload bytes", named_payload(batch))
print("double-batch payload bytes", named_payload(2*batch))

Expected output

parameter count 132
assumed payload bytes 7680
double-batch payload bytes 11136

The explicit estimate includes weights, gradients, two optimizer arrays, input, and output in eight-byte values. It omits other activations, workspaces, allocator and Python overhead; no tensors are allocated.

Measurement instrumentation can change the workload

Shape, stack, and memory recording can introduce overhead or retain references that affect allocation. Warmup, repeated calls, thread settings, cache state, garbage collection, and surrounding system load can change results. Distinguish first-use compilation or initialization from steady execution if both matter. Report a suitable distribution and sample count rather than one favorable value. Do not copy a device synchronization recipe from another backend into a CPU claim. The runtime, hardware, supported collection method, and exact code boundary belong with any actual measurement; none is supplied as measured evidence here.

Accept an optimization only after behavioral verification

Compare the optimized result with the reference using justified tolerances, names, shapes, valid-length rules, and downstream decisions. Include boundary inputs because a faster path can silently change dtype, reduction, padding, or module state. Then repeat only the measurements relevant to the proposed change under comparable conditions. If the bottleneck remains uncertain, improve the observation rather than editing unrelated layers. The torch snippet is unexecuted, and the executed companion performs integer arithmetic for a declared storage estimate only. No CPU profiler, allocator, benchmark harness, model training, or accelerator behavior was tested.

Common errors and fixes

A file-size ratio is reported as peak training-memory improvement

Why it happens: Only stored parameters were counted.
How to fix it: Estimate and measure the relevant complete memory categories.
Prevent it next time: Keep named assumptions and omissions beside analytical counts.

A trace table is called full request latency

Why it happens: The profiled region excluded relevant work or included instrumentation overhead differently.
How to fix it: Define and measure the intended complete boundary.
Prevent it next time: Report collection method, warmup, workload, environment, and counts.

A faster path changes accepted predictions

Why it happens: An optimization changed semantics without an equivalence check.
How to fix it: Verify the complete output contract before comparing performance.
Prevent it next time: Keep known-input, boundary, and decision-sensitive fixtures.

Check your understanding

1. Why does doubling the batch not double the companion’s total estimate?

Show worked solution

The assumed weights, gradients, and two optimizer arrays are fixed by the 132 parameters. Only input and output payload grow with batch in this narrow formula. Other real allocations are omitted, so the result illustrates category reasoning rather than predicting an observed runtime peak.

2. Why should no duration be inferred from the value result=15?