Elasticsearch / Basic: document and query contracts / Lesson 3
Analysis, tokens and multilingual text
Inspect how character filters, tokenizers and token filters change searchable vocabulary.
On this page
Lesson goals
- Distinguish index-time and search-time analysis.
- Choose language behavior using token evidence and relevance examples.
Search operates on a representation
A visitor types Bronze Compass, but a full-text field may store indexed terms such as bronze and compass rather than one exact phrase. An analyzer combines character filters, a tokenizer and token filters. Character filters transform the incoming character stream; the tokenizer determines token boundaries; token filters transform or remove tokens. Position information supports phrase behavior, so deleting words is not merely a cosmetic change. Start with the product’s language and retrieval requirements instead of assembling filters because their names sound useful. Every transformation can improve one class of query while damaging another.
Inspect a small known phrase
The Analyze API body below asks the standard analyzer to process a synthetic title. Its logical terms are lowercase bronze and compass; actual API output also includes token types, positions and offsets. This is a teaching expectation, not captured engine output. Inspect tokens for both indexed content and typical user queries. A surprising no-match may come from incompatible representations rather than a missing document. Save a minimal phrase that exhibits the defect, since a full museum description contains too many unrelated words to isolate one transformation reliably.
{"analyzer":"standard","text":"BRONZE compass"}["bronze","compass"]Body for POST /_analyze. Expected term sequence: bronze, compass; no server request was made.
Language processing is a product decision
English stemming can connect inflected forms but can also conflate distinct museum names. Arabic analysis has its own normalization and stemming choices; copying an English analyzer onto Arabic content is not a translation strategy. Keep separate language fields or documented language metadata when the collection is multilingual, and evaluate queries in each language. Proper names, accession codes and quoted titles may need exact representations alongside analyzed text. ASCII folding does not translate language, and removing diacritics or punctuation can alter meaningful names. Review real synthetic variants and known negative matches before adopting a normalization rule.
Keep a custom analyzer’s effects explicit
The index-settings fragment defines a whitespace tokenizer followed by lowercase. It preserves punctuation attached to a token because whitespace alone defines the boundary; Bronze-compass remains one token after lowercasing. This differs from assuming every analyzer splits on punctuation. The fragment is intentionally small so the consequences can be reasoned about. A production analyzer should have a versioned test vocabulary, including hyphens, Unicode, apostrophes and empty input. Changes to index-time analysis usually require rebuilding the index representation, so keep source data and a migration plan rather than expecting a mapping edit to rewrite old terms.
{"settings":{"analysis":{"analyzer":{"museum_words":{"type":"custom","tokenizer":"whitespace","filter":["lowercase"]}}}}}An index-creation settings fragment. Under this analyzer, Bronze-compass becomes bronze-compass as one token.
Query analysis must fit the indexed vocabulary
A match query analyzes its text using the configured search analyzer, while a term query seeks an exact indexed term and does not perform ordinary full-text analysis. Searching an analyzed title with term value Bronze can therefore fail when the indexed term is bronze. Phrase queries also depend on positions and ordering rather than exact source-string equality. Synonyms add another boundary: index-time and search-time approaches have different update and graph behavior. Use supported synonym mechanisms with explicit analyzer compatibility and evaluated examples. Do not promise that adding a thesaurus improves relevance without measuring false positives as well as recovered matches.
Common errors and fixes
A term query misses an obvious title
- Why it happens
- The supplied exact term differs from the analyzed indexed vocabulary.
- How to fix it
- Inspect tokens and choose match for the intended full-text question.
- Prevent it next time
- Test the analyzer and query together with known matching titles.
Normalization merges distinct catalog names
- Why it happens
- An aggressive transformation removed a distinction meaningful to users.
- How to fix it
- Keep an exact representation and narrow the transformation.
- Prevent it next time
- Include negative examples and proper names in relevance evaluation.
Analyzer changes affect only new data
- Why it happens
- Old indexed terms were not rebuilt by a settings or mapping change.
- How to fix it
- Plan a corrected index and reconciled reindex workflow.
- Prevent it next time
- Version analysis rules alongside index templates and query tests.
Check your understanding
1. With whitespace plus lowercase, how many tokens does Bronze-compass create?
Show worked solution
One token, bronze-compass. The whitespace tokenizer does not split the internal hyphen. If separate words are required, choose and evaluate an appropriate tokenizer rather than assuming lowercasing changes boundaries. Confirm positions and offsets through the Analyze API in a separately authorized lab.
2. Does a lowercase filter translate an Arabic artifact name into English search terms?
Show worked solution
No. Case normalization and translation are different operations. Design multilingual fields, language-specific analysis and a translation or cross-language retrieval strategy only where the product requires it. Evaluate Arabic and English query sets independently and preserve exact names for identity-sensitive retrieval.
Key takeaways
- Token evidence explains many apparent search defects.
- Language normalization must preserve the distinctions users need.