Deep Learning / Tensor and learning foundations / Lesson 1

Tensors, axes, and batches

Read a tensor as a contract about examples, features, and operations before interpreting its numbers.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Distinguish shape from the meaning of an axis.
Calculate affine-layer shapes and parameter counts.
Check batch independence and preserve observation identity.

An array needs a meaning for every axis

A tensor is a multidimensional collection of numbers; in this course the term refers to the mathematical object rather than a particular library class. Shape records axis lengths, while an axis contract records what those lengths mean. A batch with shape (3, 2) can mean three observations with two features each. The same six numbers arranged as (2, 3) represent another contract. Keep feature names, order, units, and observation identifiers beside the array. A successful reshape proves that element counts agree, but cannot prove that examples, channels, or time steps retain their intended meaning.

Dense layers transform the final feature axis

For row-oriented observations, a dense layer computes Y = XW + b. If X has shape (B, F), W has shape (F, H), and b has H entries, Y has shape (B, H). Each observation uses the same parameters. Increasing B changes the work and activation storage, not the number of parameters. Counting FH + H parameters makes this distinction concrete. Other conventions place features on another axis or store a transposed weight; their formulas remain valid when stated consistently. Before writing an expression, annotate the contracted feature axis and the preserved observation axis.

Compute a complete tiny forward pass

The first program treats two lists as observations and multiplies each by one shared matrix. Its explicit loops reveal which index is summed, which is preserved, and where the bias is added. The assertions reject a missing feature rather than allowing zip to silently truncate an input. Such loops are clear for a fixture but are not a recommendation for large numerical workloads. Inspecting the first result by hand, 1 times 1 plus 2 times 3 plus 1 equals 8, gives an independent check of both arithmetic and orientation.

python Deterministic standard-library calculation

X = [[1, 2], [3, 4]]
W = [[1, 0], [3, 2]]
b = [1, -1]
assert all(len(row) == len(W) for row in X)
Y = [[sum(row[k] * W[k][j] for k in range(2)) + b[j]
      for j in range(2)] for row in X]
print(Y)
print("parameters", sum(map(len, W)) + len(b))

Expected output

[[8, 3], [16, 7]]
parameters 6

Rows remain independent observations; the two output features share six trainable scalar parameters.

Reduction must preserve the intended comparison

A reduction combines an axis using a sum, mean, or another rule. Averaging feature values across a batch produces a feature statistic; averaging across features produces one statistic per observation. These operations are not interchangeable even when both return two numbers. The second fixture deliberately uses a non-square batch to make accidental transposition visible. In a real input pipeline, test both shape and selected known values after stacking, padding, flattening, or transposing. A single square test array can hide a wrong-axis bug because dimensions still agree after a mistaken transpose.

python Deterministic standard-library calculation

batch = [[1, 10], [3, 30], [5, 50]]
feature_means = [sum(row[j] for row in batch) / len(batch) for j in range(2)]
row_means = [sum(row) / len(row) for row in batch]
print(feature_means)
print(row_means)

Expected output

[3.0, 30.0]
[5.5, 16.5, 27.5]

The two feature means differ in meaning and length from the three per-observation means.

Validate contracts before optimizing implementation

Tensor libraries add broadcasting, specialized storage, automatic differentiation, and device execution. These conveniences do not remove the need to state axis semantics. Broadcasting a target vector against a column of predictions may construct all pairwise differences instead of one difference per example. Record expected shapes at data, layer, loss, and metric boundaries. Preserve an identifier outside the feature tensor so filtering and shuffling can be audited. The standard-library examples here establish indexing and arithmetic only; later framework tracks explain actual tensor APIs, data loaders, devices, and serialization with their own execution assumptions.

Common errors and fixes

The loss has an unexpected extra axis

Why it happens: A prediction column and target vector were broadcast into pairwise comparisons.
How to fix it: Align both to the documented example shape before subtraction.
Prevent it next time: Assert the unreduced loss shape, including a non-square batch fixture.

A reshape appears to fix training

Why it happens: Only the element count was checked, so axis meaning changed.
How to fix it: Trace the original layout and use an intentional axis permutation if needed.
Prevent it next time: Track named axis contracts and known sentinel values.

A dense layer gains parameters when batch size grows

Why it happens: Activation storage was confused with shared model weights.
How to fix it: Count feature-to-hidden weights and biases independently of examples.
Prevent it next time: Document parameter and activation sizes separately.

Check your understanding

1. For B=4, F=3, H=2, give the input, weight, bias, and output shapes and parameter count.

Show worked solution

The shapes are (4,3), (3,2), (2,), and (4,2). The layer has 3 times 2 plus 2 = 8 parameters. Four observations reuse those parameters. With eight observations the output becomes (8,2), while the count stays eight.

2. Design a fixture that detects a mistaken batch/feature transpose.