Generative AI / Basic: generation and evidence foundations / Lesson 5

Embeddings, similarity, and index identity

Interpret vector similarity as a model-dependent retrieval signal with explicit dimensions, versioning, authorization, and evaluation limits.

About 5 min readContent reviewed 2026-10-26

On this page

Lesson goals

Explain the assumptions and trust boundaries of embeddings, similarity, and index identity.
Interpret both examples and distinguish offline fixture results from model or production evidence.
Diagnose the listed failures and justify the worked decisions while preserving task, data, permission, and uncertainty contracts.

A vector is a representation, not a fact

An embedding maps supported input into a numerical representation under a selected model and preprocessing contract. Similar vectors can be useful for retrieval or clustering, but closeness does not prove factual agreement, relevance to a specific question, or permission to expose a document. The first example uses two hand-authored vectors and a query to demonstrate cosine arithmetic only. No learned representation, semantic encoder, model download, or real corpus is involved. A useful representation depends on task, language, domain, training, pooling, and input handling, so do not infer universal semantic quality from the word embedding.

Define the metric and its assumptions

Cosine similarity compares direction after accounting for vector norms; a dot product and a distance measure have different meanings unless the representation and normalization contract connects them. Zero vectors, incompatible dimensions, and nonfinite values need explicit handling. The first function rejects an empty or mismatched vector and a zero norm before scoring. Its simple input set avoids floating-point ambiguity and prints one and zero. Real similarity thresholds must be calibrated for the chosen model and task; a score of 0.8 is not automatically an eighty-percent probability of correctness or relevance.

python Offline hand-authored vector comparison

import math
def cosine(left, right):
    if not left or len(left) != len(right):
        raise ValueError("incompatible dimensions")
    left_norm = math.sqrt(sum(value * value for value in left))
    right_norm = math.sqrt(sum(value * value for value in right))
    if left_norm == 0 or right_norm == 0:
        raise ValueError("zero vector")
    return sum(a * b for a, b in zip(left, right)) / (left_norm * right_norm)
print(round(cosine([1, 0], [1, 0]), 3))
print(round(cosine([1, 0], [0, 1]), 3))
try:
    cosine([0, 0], [1, 0])
except ValueError as error:
    print(str(error))

Expected output

1.0
0.0
zero vector

Matching directions score one, orthogonal directions zero, and the zero vector is rejected. These vectors carry no learned language semantics.

Version query and document representations together

An index needs the encoder identity, revision, dimensions, preprocessing, chunking, normalization, and distance configuration that produced its vectors. The second synthetic record binds those choices to an index version. Mixing vectors from unrelated encoders or revisions can produce meaningless rankings even when dimensions match. Re-embedding documents may require an index migration and a query-side switch, not merely changing one model name. Keep source and access metadata attached so stale or unauthorized content can be removed or filtered. An old vector can remain searchable after the source changed unless the application explicitly updates its state.

yaml Synthetic index identity record

index_revision: lab-index-3
encoder_revision: synthetic-encoder-v2
query_encoder_revision: synthetic-encoder-v2
dimensions: 2
metric: cosine
chunk_policy: lab-paragraph-v1
access_scope: enforced-by-application-before-exposure

The encoder is fictional and no vectors are generated. Matching version metadata is necessary context, not proof of retrieval quality or privacy.

Combine relevance with access and privacy

Retrieval candidates must respect the trusted caller’s tenant and authorization scope before protected content reaches generation or logs. A nearest neighbor is not an authorization decision. Embeddings can retain information about their inputs and should not be called anonymized data by default. Review storage, external processing, retention, and deletion obligations for both text and vectors. Approximate-nearest-neighbor systems trade resource use and recall under their selected implementation; a successful toy cosine calculation certifies none of those production properties. Use controlled relevance and exclusion cases, including similar but prohibited records, when evaluating the actual retrieval boundary.

Measure the task rather than admire the plot

Evaluate whether relevant authorized passages appear at the required ranks for representative questions and languages. Include out-of-domain queries, rare terms, near duplicates, and changed source versions. Compare lexical, dense, and hybrid approaches under a frozen set instead of assuming one always wins. A two-dimensional visualization can hide the original representation’s geometry and should not substitute for retrieval metrics and error review. Both examples here are offline and public. They contact no embedding service or database and claim no semantic benchmark result. Treat vector similarity as one scored signal inside a governed system, not as proof of understanding or truth.

Common errors and fixes

Similarity is reported as probability of truth

Why it happens: A metric scale was mistaken for calibrated confidence.
How to fix it: Evaluate relevance and factual support separately.
Prevent it next time: Document score interpretation and calibration evidence.

Index and query vectors use incompatible encoders

Why it happens: Representation identity was not versioned together.
How to fix it: Align or migrate the complete representation contract.
Prevent it next time: Store encoder and preprocessing metadata with the index.

A similar protected document is exposed

Why it happens: Ranking bypassed tenant and access controls.
How to fix it: Filter authorized scope before protected content reaches consumers.
Prevent it next time: Test near-match records from excluded tenants.

Check your understanding

1. Two encoders emit the same number of dimensions. Are their vectors automatically comparable?

Show worked solution

No. Coordinates have meaning under the learned representation and preprocessing contract. Equal length is a shape condition only. Align encoder identity and normalization or evaluate a deliberately designed compatibility method before combining indexes.

2. A retrieved document has cosine 0.95 but belongs to another tenant. What should happen?