Elasticsearch / Intermediate: models, pipelines and relevance evidence / Lesson 12
Relevance judgments and diagnostic evidence
Improve catalog discovery using judged queries, controlled experiments and correctly scoped explanation tools.
On this page
Lesson goals
- Separate matching quality, ranking quality and latency.
- Use judgments and diagnostics without treating scores as probabilities.
A relevant result satisfies a user intent
A visitor searching compass may want a navigation instrument, an artist named Compass or a catalog title containing the word. Choose representative intents and identify useful results before tuning DSL. A query that returns more documents is not automatically better; it may add unrelated objects. Preserve a judged set containing head queries, rare names, spelling variants, Arabic queries and exact accession lookups. Label why each result is useful or irrelevant. Keep the evaluation collection versioned, because changed documents and new acquisitions can alter the meaning of a previously good score.
Boosts express relative preference
The multi_match body searches title and description while boosting title. The boost is a relative contribution within the scoring model, not an eighty-percent probability that the artifact is relevant. Scores are meaningful for ranking under the particular query and index statistics, not as universal values comparable across unrelated queries. Check whether phrase intent, synonyms or exact identifier paths deserve separate clauses. Do not bury a mandatory status or access condition inside an optional relevance boost. Eligibility stays enforced while ranking experiments change.
{"query":{"multi_match":{"query":"navigation compass","fields":["title^3","description"],"type":"best_fields"}}}Title receives a relative boost of 3 in this query shape. Exact scores and ordering require the actual corpus and engine.
Measure a chosen ranking boundary
Precision at k asks what fraction of the first k results are judged relevant under the chosen threshold. Recall asks how much of the known relevant population was recovered; graded ranking metrics reward stronger ordering of more useful results. Unjudged documents need an explicit policy rather than silently being treated as both relevant and irrelevant in different reports. The judgment fragment below contains one useful instrument and one irrelevant administrative page for the same intent. It is evaluation data, not a measured ranking result. Include multiple intents so one improved showcase query cannot hide regressions elsewhere.
{"intent":"navigation-instrument","ratings":[{"artifact_id":"art-101","rating":3},{"artifact_id":"page-admin","rating":0}],"relevant_threshold":1,"evaluation_cutoff":2}If these two documents are the first two hits, precision@2 is 1/2 under this binary threshold. The JSON is a local judgment model, not an Elasticsearch API body.
Explain one score and profile one execution carefully
The Explain API helps inspect why one document matches and how its score is composed. It does not by itself evaluate whether the result satisfies a visitorโs intent. The Profile API provides query execution details with overhead and does not represent every source of request latency, such as all network or client costs. Use these tools on selected synthetic cases in an authorized lab and label their scope. A slow query may need a different retrieval strategy, not simply a larger boost. Avoid enabling expensive diagnostics indiscriminately on a production workload.
Change one justified retrieval hypothesis
A controlled experiment holds corpus, judgments, filters and evaluation cutoff stable while changing an analyzer, field weight or query structure. Record both relevance metrics and latency at an agreed measurement boundary. Keep a separate held-out query set to reduce overfitting to hand-tuned examples. Review Arabic and English outcomes independently so an aggregate average cannot hide a language regression. Search logging must respect privacy and avoid storing sensitive user input unnecessarily. No retrieval benchmark, API explanation or profile execution was performed while authoring this course; its numeric precision example is only a safe local calculation.
Common errors and fixes
A score is displayed as a relevance probability
- Why it happens
- Relative retrieval scores were interpreted as calibrated probabilities.
- How to fix it
- Describe ranking evidence and evaluate against judgments.
- Prevent it next time
- Avoid absolute score thresholds without a validated product contract.
A boost improvement harms Arabic search
- Why it happens
- Evaluation averaged languages and hid a subgroup regression.
- How to fix it
- Compare per-language judged sets and analyzer behavior.
- Prevent it next time
- Keep multilingual and rare-query coverage in acceptance criteria.
Profile timing is treated as end-to-end latency
- Why it happens
- Diagnostic scope and overhead were ignored.
- How to fix it
- Measure the agreed request boundary separately.
- Prevent it next time
- Label every timing source and measurement limitation.
Check your understanding
1. If the first two hits contain one relevant instrument and one irrelevant page, what is precision@2?
Show worked solution
It is 1/2, or 0.5, under the stated binary relevance threshold. That says nothing by itself about recall or the order of a larger result set. Keep the cutoff and unjudged-document policy fixed when comparing query changes.
2. Can Explain prove that an artifact is useful to a visitor?
Show worked solution
No. It explains matching and scoring mechanics for a selected document and query. Human or domain judgments define usefulness. Combine diagnostic evidence with representative relevance evaluation, and verify that mandatory filters remain enforced throughout the experiment.
Key takeaways
- Relevance is measured against intent and judgments, not score size alone.
- Diagnostic tools explain mechanics within a limited observation scope.