PyTorch / Storage, restoration, and transformed execution / Lesson 13

Views, contiguity, and in-place updates

Separate logical indexing, shared storage, and gradient history before mutating or reshaping tensor state.

About 5 min readContent reviewed 2026-10-28

On this page

Lesson goals

Read shape and stride as different descriptions.
Distinguish views, possible copies, clone, and detach.
Explain why an in-place change can invalidate a backward calculation.

Shape alone does not determine storage traversal

A dense tensor can describe the same storage through different shapes, offsets, and strides. For a contiguous two-by-three row layout, advancing one row moves three elements and advancing one column moves one. Transposing exchanges logical axes and their strides without necessarily copying elements. The resulting three-by-two view can be noncontiguous even though its values form a perfectly regular table. Contiguity is a storage property relative to a memory format, not evidence that axis meanings or units are correct. Inspect shape, stride, and the intended indexing contract together before choosing an operation that assumes a particular layout.

Copy behavior belongs to the selected operation

The illustrative program transposes a CPU float64 tensor and calls contiguous on that noncontiguous view before changing the original. The transpose sees the new ninety-nine because it shares storage; the materialized copy retains zero. contiguous can return the existing tensor when its condition is already satisfied, so this example does not imply that every call creates independent storage. A view operation requires a compatible relationship between shape and strides. reshape and flatten may return a view or a copy under their documented conditions; code should not depend on an unspecified ownership outcome.

python Illustrative CPU PyTorch 2.14 API program; torch unavailable, expected output derived

import torch
x = torch.arange(6,dtype=torch.float64,device="cpu").reshape(2,3)
transposed = x.t()
owned = transposed.contiguous()
x[0,0] = 99.0
print("strides", x.stride(), transposed.stride())
print("first values", transposed[0,0].item(), owned[0,0].item())
print("contiguous", transposed.is_contiguous(), owned.is_contiguous())

Expected output

strides (3, 1) (1, 3)
first values 99.0 0.0
contiguous False True

The transpose shares storage while contiguous copies this noncontiguous input. The tensors do not require gradients; no tensor layout or mutation was executed in torch.

History and storage independence are separate questions

clone creates copied tensor data while normally retaining a differentiable relationship when grad mode and inputs require it. detach stops the autograd relationship but shares storage with its source. Consequently, a detached tensor is not automatically an independent snapshot that can be changed harmlessly. A deliberate detached independent value often needs both operations, such as detach().clone(), under an explicit purpose. The list companion illustrates storage ownership only and cannot reproduce these gradient semantics. Choose copying for ownership and gradient control for differentiation, rather than using either word as a substitute for the other.

python Executed deterministic standard-library companion

storage = [float(x) for x in range(6)]
def transpose_row(i):
    return [storage[i+j*3] for j in range(2)]
owned = [transpose_row(i) for i in range(3)]
storage[0] = 99.0
print("shared first row", transpose_row(0))
print("copied first row", owned[0])
print("transpose offsets", [[i+j*3 for j in range(2)] for i in range(3)])

Expected output

shared first row [99.0, 3.0]
copied first row [0.0, 3.0]
transpose offsets [[0, 3], [1, 4], [2, 5]]

The executed list-offset model illustrates aliasing and ownership. It does not implement Tensor strides, view eligibility, version counters, or autograd checks.

In-place modification interacts with saved backward values

Operations that end with an underscore or assignment through an index can modify existing tensor storage. Autograd may need an earlier value saved during the forward computation, and version tracking can detect that it was changed before backward. Leaf tensors requiring gradients also have documented restrictions on tracked in-place updates. A no-grad block can be appropriate for a planned optimizer-style update, but does not make arbitrary mutation between forward and backward correct. Do not silence an error by detaching everything or using hidden storage shortcuts; identify the mathematical state the derivative is supposed to use.

Test aliasing at the actual boundary that needs it

Use a tiny numbered tensor, an asymmetric shape, and a controlled change to distinguish value equality from storage independence. Keep both input and expected unchanged snapshots where a test depends on them. Check saved model state separately because a mapping of tensor references can change with the model unless copied appropriately. Optimize memory or layout only after verifying the full computation and measuring the relevant environment. The framework program is unexecuted and uses no gradients; the standard-library companion checks offsets and list copies only, with no tensor kernel, backward pass, or memory-performance claim.

Common errors and fixes

A detached snapshot changes when the source changes

Why it happens: detach removed history but retained shared storage.
How to fix it: Create an intentional independent copy when the snapshot contract requires it.
Prevent it next time: Test a controlled mutation separately from equal initial values.

A reshape is assumed always to copy or always to share

Why it happens: Ownership was inferred from a convenient API name.
How to fix it: Use the documented operation and explicitly request copying when necessary.
Prevent it next time: Keep a noncontiguous example in the ownership checks.

Backward fails after an intermediate value is overwritten

Why it happens: The gradient calculation depended on saved forward state that changed.
How to fix it: Avoid or redesign the in-place change under the intended derivative contract.
Prevent it next time: Test the forward/backward boundary before adding memory optimizations.

Check your understanding

1. Why does the transposed first value become ninety-nine while owned retains zero?

Show worked solution

The transpose indexes the original storage at offset zero, so the source assignment is visible through that view. contiguous materialized independent storage for this noncontiguous input before the assignment. Its copied first element therefore remains zero. The conclusion depends on this layout and timing, not a promise that every contiguous call copies.

2. Why is detach alone insufficient for an independently mutable diagnostic snapshot?