Generative AI / Intermediate: retrieval and controlled workflows / Lesson 7

Retrieval, chunking, and access scope

Build and evaluate a source-selection boundary that preserves meaning, freshness, provenance, and authorization before generation.

About 5 min readContent reviewed 2026-10-26

On this page

Lesson goals

Explain the assumptions and trust boundaries of retrieval, chunking, and access scope.
Interpret both examples and distinguish offline fixture results from model or production evidence.
Diagnose the listed failures and justify the worked decisions while preserving task, data, permission, and uncertainty contracts.

Retrieval is a decision with its own failures

A retriever chooses candidate evidence from a corpus under a query and scope. It can fail before any generator runs: the right document may be absent, stale, filtered incorrectly, poorly chunked, or ranked below the selected cutoff. Define what relevant means for the user task and keep source identity and version. The first offline example ranks public synthetic English records using token overlap after a tenant filter. It is a deliberately simple lexical baseline, not an embedding model, semantic search service, or production relevance claim. Similar words can be useful evidence for investigation without proving the passage answers the question.

python Offline scope-first lexical ranking

import re
documents = [{"id": "refund-red", "tenant": "red", "text": "Refund window is 14 days."},
             {"id": "refund-blue", "tenant": "blue", "text": "Refund window is 90 days."},
             {"id": "shipping-red", "tenant": "red", "text": "Shipping takes two days."}]
query = {"refund", "window"}
allowed = [doc for doc in documents if doc["tenant"] == "red"]
scored = [(len(query & set(re.findall(r"[a-z]+", doc["text"].lower()))), doc["id"]) for doc in allowed]
print([doc_id for score, doc_id in sorted(scored, key=lambda row: (-row[0], row[1])) if score > 0])

Expected output

['refund-red']

Only refund-red is returned. The blue-tenant near-match is excluded before scoring, and the English regex is not a multilingual tokenizer or semantic model.

Choose chunk boundaries from evidence needs

Chunks should preserve enough context to interpret a claim, including heading, exception, date, subject, and unit where needed. Splitting a condition from its consequence can make an otherwise relevant passage misleading. Overlap can reduce boundary loss but adds duplication and token or storage cost. The second example makes overlapping word windows from a tiny string; words are not model tokens and the algorithm understands no sentence or policy structure. Real chunking should be evaluated with representative questions and document forms rather than a universal fixed size copied from a tutorial. Store parent and neighboring context references when the task needs them.

python Offline overlapping-word windows

def windows(words, width, overlap):
    if width <= 0 or overlap < 0 or overlap >= width:
        raise ValueError("invalid window")
    start = 0
    while start < len(words):
        yield " ".join(words[start:start + width])
        if start + width >= len(words):
            break
        start += width - overlap
print(list(windows("alpha beta gamma delta epsilon".split(), 3, 1)))

Expected output

['alpha beta gamma', 'gamma delta epsilon']

Gamma appears in both chunks. The program preserves a simple overlap and avoids an extra trailing duplicate, but knows nothing about sentences, facts, or model token counts.

Apply trusted access before exposure

The trusted application identity selects tenant and authorization scope; the query text does not grant new permissions. Filter protected candidates before their content reaches generation, caches, or diagnostic logs. A post-generation apology cannot undo disclosure of another tenant’s document. The first example keeps a tempting near-match record outside the allowed scope. In a real vector or hybrid index, validate filtering and metadata behavior under the chosen backend rather than assuming an index namespace is a complete security boundary. Recheck access when a document or membership changes, and include excluded near-neighbors in tests.

Version freshness and ranking together

A document update can require new chunks, embeddings, metadata, and cache invalidation. Keep a corpus snapshot and retrieval configuration with evaluation results so a changed score is interpretable. Lexical, dense, hybrid, and reranking stages have different resource and failure characteristics. Increasing top-k can improve candidate recall while adding distractors, latency, and sensitive exposure risk. A reranker cannot recover an omitted candidate unless the pipeline retrieves it again. Preserve provenance through deduplication and context packing, including which version actually supports the answer rather than only the latest filename in a content store.

Evaluate retrieval before blaming generation

Use known relevant source IDs for representative queries and measure whether they appear at useful ranks. Separate corpus coverage, authorization exclusion, retrieval recall, ranking, and context-budget loss in error records. Test multilingual, rare-name, conflicting-version, and unanswerable cases. The offline examples create no index service, model vectors, network request, customer record, or paid operation. Their results verify only the stated toy rules. A retrieval pipeline earns confidence through task-specific evidence and access tests; neither a vector database label nor a high similarity value makes its selected context automatically sufficient or safe.

Common errors and fixes

Generation is blamed when support was never retrieved

Why it happens: Retrieval and context-selection evidence were not separated.
How to fix it: Trace source coverage and rank before the model boundary.
Prevent it next time: Keep relevant-source fixtures and stage-level metrics.

A chunk omits the policy exception

Why it happens: Fixed splitting ignored semantic boundaries.
How to fix it: Preserve parent context or revise the chunk contract.
Prevent it next time: Test questions requiring qualifiers and neighboring text.

An index filter is assumed secure without testing

Why it happens: Metadata or backend scope was trusted implicitly.
How to fix it: Verify allowed and excluded records under the actual retrieval path.
Prevent it next time: Include cross-tenant near-match cases and deletion updates.

Check your understanding

1. A relevant paragraph ranks eleventh but only five passages are packed. Which stage should the diagnosis name?

Show worked solution

Distinguish retrieval or ranking cutoff from generator behavior. The evidence did not reach generation. Inspect candidate recall, ranking, budget selection, and distractors before changing the answer prompt. Raising top-k is a hypothesis to evaluate, not an automatic remedy.

2. Why can overlap improve recall while harming answer quality?