Deep Learning / Representation and acceptance evidence / Lesson 13

Embeddings and contrastive objectives

Separate representation geometry, training pairs, and downstream evidence.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Define an embedding and a similarity contract.
Compute a small contrastive classification loss.
Audit positive and negative construction for shortcuts.

An embedding is a learned coordinate system

An embedding maps an object or discrete identifier to a numeric vector used by a subsequent objective. A lookup table assigns one trainable row per vocabulary item; an encoder computes a vector from structured input. Nearby vectors are meaningful only relative to the chosen geometry, training objective, and population. Dot product depends on vector scale, while cosine similarity compares direction for nonzero vectors. Changing normalization changes the model contract. A low-dimensional visualization can distort distances, so it should not replace numerical checks or a downstream evaluation designed for the intended use.

Pair construction defines the learning signal

A contrastive objective can encourage an anchor to match a designated positive relative to designated negatives. Positives might be two permitted transformations of the same synthetic object; negatives come from an explicit sampling policy. These choices encode assumptions about invariance and distinguishability. Two genuinely related objects can become false negatives, and a transformation can remove information the later task needs. An encoder may exploit a shared watermark or padding pattern instead of useful structure. Document how pairs are produced and test whether the desired relationship survives those transformations before considering model size or training duration.

Inspect geometry with fixed vectors

The first fixture compares a vector with a positive rescaling of itself and with an orthogonal vector. Cosine is one and zero respectively. This checks the similarity definition, not the semantic quality of any embedding. A zero vector has undefined direction and requires a documented rejection or stabilization policy; silently dividing by its norm can produce invalid values. Retrieval also needs a tie rule and a candidate-set contract. Measuring nearest neighbors among a tiny selected set can hide failures when the set grows or when confusing but legitimate candidates are included.

python Deterministic standard-library calculation

from math import sqrt
def cosine(a, b):
    na = sqrt(sum(x*x for x in a))
    nb = sqrt(sum(x*x for x in b))
    assert na > 0 and nb > 0
    return sum(x*y for x,y in zip(a,b))/(na*nb)
anchor = [1.0, 0.0]
print(f"positive={cosine(anchor,[2.0,0.0]):.1f}")
print(f"negative={cosine(anchor,[0.0,1.0]):.1f}")

Expected output

positive=1.0
negative=0.0

Cosine ignores positive rescaling of a nonzero vector; zero vectors need an explicit policy.

Temperature changes the contrastive comparison

The second fixture treats the positive as class zero in a two-candidate softmax over similarities divided by temperature. Reducing temperature sharpens this fixed comparison and lowers the displayed loss because the positive already has the larger similarity. If the negative had the larger score, sharpening would increase the penalty instead. The denominator and negative sample set are part of the objective: changing the number or distribution of negatives changes what the loss means. Report those choices when comparing training curves. A smaller contrastive loss under a different candidate set is not automatically a better representation.

python Deterministic standard-library calculation

from math import exp, log
similarities = [1.0, 0.0]
for temperature in [1.0, 0.5]:
    scores = [s/temperature for s in similarities]
    m = max(scores)
    loss = m + log(sum(exp(s-m) for s in scores)) - scores[0]
    print(f"temperature={temperature:.1f}, loss={loss:.6f}")

Expected output

temperature=1.0, loss=0.313262
temperature=0.5, loss=0.126928

This one-positive, one-negative loss changes with temperature for fixed similarities; it is not a trained representation evaluation.

Evaluate the representation under an honest boundary

A frozen linear probe tests one restricted downstream predictor; fine-tuning tests a different procedure that changes the representation. Retrieval metrics assess another behavior and depend on relevance judgments and candidate construction. Keep the selection data separate from the final assessment and prevent related objects from crossing a split in an invalid way. Check slices, nuisance transformations, and simple shortcut baselines. These local examples use fixed vectors without learned encoders, external text, or downloaded models. They establish geometric and loss arithmetic only, so no semantic understanding, fairness property, or transfer performance follows from their outputs.

Common errors and fixes

High similarity is treated as proof of shared meaning

Why it happens: The geometry was interpreted without checking the objective and population.
How to fix it: Validate task-specific relevance with an honest candidate set.
Prevent it next time: Document similarity, normalization, and evaluation assumptions.

Contrastive loss improves after removing difficult negatives

Why it happens: The comparison problem became easier rather than the encoder necessarily better.
How to fix it: Compare the same candidate construction or report the changed task.
Prevent it next time: Version pair-generation and negative-sampling policies.

Cosine computation returns invalid values

Why it happens: A vector had zero norm.
How to fix it: Apply an explicit zero-vector policy before division.
Prevent it next time: Test zero and near-zero representations separately.

Check your understanding

1. What happens to a dot product and cosine when the second vector is multiplied by positive ten?

Show worked solution

The dot product is multiplied by ten, while cosine remains unchanged because the same factor appears in the numerator and vector norm. This assumes both vectors are nonzero. Thus normalized and unnormalized retrieval can rank candidates differently when vector lengths vary.

2. Why might two image crops be an invalid positive pair for a counting task?