Generative AI / Basic: generation and evidence foundations / Lesson 2

Tokens, context, and information budgets

Distinguish tokenizer-specific length, effective context use, and deliberate evidence selection rather than assuming a word count measures capacity.

About 5 min readContent reviewed 2026-10-26

On this page

Lesson goals

Explain the assumptions and trust boundaries of tokens, context, and information budgets.
Interpret both examples and distinguish offline fixture results from model or production evidence.
Diagnose the listed failures and justify the worked decisions while preserving task, data, permission, and uncertainty contracts.

Tokens are model-specific units

A tokenizer maps supported input into units used by a model. A token may represent a word, part of a word, punctuation, bytes, or another unit according to the tokenizer. Counts vary across languages, text forms, special markers, and model families. The first example compares whitespace splitting with UTF-8 byte length for public strings; neither is a production tokenizer. Its purpose is to disprove the shortcut that characters, words, bytes, and model tokens are interchangeable. Use the actual selected tokenizer or documented counting mechanism when a real system needs an exact budget. No tokenizer package or model is downloaded in this lesson.

python Offline unit distinction

samples = ["a b", "مرحبا"]
for text in samples:
    print(len(text.split()), len(text.encode("utf-8")))
print("words and bytes are not model-token counts")

Expected output

2 3
1 10
words and bytes are not model-token counts

The English sample has two whitespace words and three bytes; the Arabic sample has one word and ten UTF-8 bytes. Neither result claims a model-token count.

A context window is a constrained input boundary

A particular model and runtime impose a context contract that can include instructions, conversation, retrieved evidence, tool results, and reserved generation space. Exact capacities, accounting, and truncation behavior are unstable vendor and version facts, so this course quotes no current product limit. The second offline example allocates a fictional budget and rejects an overfull pack. Count overhead and output reservation explicitly rather than treating every available unit as document space. A request accepted by a runtime does not prove the model used every included detail or preserved information from earlier turns.

python Offline context-budget boundary

budget = 100
instructions = 20
output_reserve = 30
overhead = 10
available = budget - instructions - output_reserve - overhead
print(available)
for evidence_size in [40, 41]:
    print(evidence_size <= available)
print("fictional budget units only")

Expected output

40
True
False
fictional budget units only

Forty fictional units remain for evidence; forty fits and forty-one does not. These units are not any real model’s tokens or advertised context limit.

Separate presence from effective use

Long-context research shows that information position and task can affect retrieval from the supplied context. Those experimental findings motivate testing, not a universal rule that every current model fails in the middle. Evaluate the chosen model and task with evidence placed in different positions, distractors, and varying lengths. A large advertised window is not a substitute for an information-selection strategy. Remove irrelevant repetition, preserve necessary qualifiers, and keep claim-supporting passages identifiable. Summarization can save space but may omit exceptions, dates, or uncertainty, so compare it against the original evidence for the questions the system must answer.

Manage history and retrieval deliberately

Conversation history can contain stale instructions, private data, rejected tool proposals, and unsupported earlier answers. Decide which state remains authoritative and which content is merely evidence or a prior candidate. A summary of a user request is not new permission to access records. Retrieval should filter by authorization and freshness before packing results. Keep stable source IDs, version, and scope alongside selected passages so downstream validation can trace support. Do not silently drop the only relevant clause to fit a budget and then blame generation for an answer made impossible by the supplied context.

Test budgets as application contracts

Cover empty context, just-fitting input, overflow, multilingual text, large tool results, and insufficient-evidence fallback. Use a recorded tokenizer and model version in a real deployment, and state whether counts are exact or estimated. These examples are tiny deterministic computations over synthetic units, not model capacity tests or cost estimates. They contact no service and allocate no real inference context. A robust application refuses, reduces, or routes an oversized task under an explicit policy while preserving essential evidence and user meaning, rather than silently truncating whatever happened to be last.

Common errors and fixes

Word count is used as an exact token count

Why it happens: Tokenizer and language differences were ignored.
How to fix it: Use the selected tokenizer’s documented counting contract.
Prevent it next time: Test multilingual and special-marker cases.

A large window is treated as guaranteed recall

Why it happens: Capacity was confused with effective task use.
How to fix it: Evaluate evidence position, distractors, and supported answers.
Prevent it next time: Maintain an explicit context-selection strategy.

Truncation removes the supporting exception

Why it happens: Packing optimized length without preserving meaning.
How to fix it: Retain necessary qualifiers or abstain from unsupported conclusions.
Prevent it next time: Test just-fitting and overflow tasks with evidence checks.

Check your understanding

1. A fictional window has 80 units, with 15 instruction, 20 output, and 5 overhead. How much evidence fits?

Show worked solution

Forty units remain: 80 - 15 - 20 - 5. An evidence pack of forty-one must be reduced or rejected under the chosen policy. Do not call those units actual tokens without the selected tokenizer and runtime contract.

2. Why can summarizing old history introduce a factual failure?