Generative AI / Editorial solution guide

RAG answer is wrong: trace evidence before changing the prompt

Locate corpus, access, retrieval, packing, and claim-support failures with safe synthetic traces rather than assuming every error is a model problem.

About 4 min readContent reviewed

Read-only knowledge base · No public comments

Recognize the symptoms

The answer sounds plausible but cites an irrelevant passage, omits an exception, uses the wrong policy version, or invents a number. A source can be present in the corpus yet absent from the final context. Citation presence alone cannot establish that the selected text supports the actual question.

Understand the likely causes

  • Possible causes include missing or stale source data, a correct authorization exclusion, poor chunk boundaries, rank cutoff, duplicate-heavy packing, query rewriting that changed scope, or a candidate that ignored valid evidence. These causes need different corrections. Widening access or increasing top-k blindly can create a second privacy or context problem.

Freeze the question and relevant versions

Record the exact task, trusted caller scope, corpus snapshot, chunk and encoder revisions, retrieval settings, prompt, and candidate bundle. Keep safe source IDs and stage decisions rather than dumping protected documents or user input. Identify which authorized passage should support the requested claim and time period. A current policy is not automatically evidence for a historical question. Use a tiny public reproduction before changing a real system. Neither example here contacts a model, index service, account, or external corpus.

Trace the evidence through every stage

Check corpus presence, authorized eligibility, candidate retrieval, ranking, and final packing independently. The first offline classifier distinguishes policy exclusion, retrieval gap, and packing gap. It does not execute a retriever or prove natural-language relevance. If policy correctly excludes a document, do not treat that as permission to bypass the caller’s scope. If the passage was retrieved but dropped, inspect budget and deduplication before changing generation instructions. Preserve essential heading, date, unit, and exception information when revising chunks.

python Offline retrieval-stage diagnosis 
def stage(gold, authorized, retrieved, packed):
    if not authorized:
        return "excluded_by_policy"
    if gold not in retrieved:
        return "retrieval_gap"
    if gold not in packed:
        return "packing_gap"
    return "candidate_support_check_needed"
print(stage("lab-a", True, ["lab-a", "lab-b"], ["lab-b"]))
print(stage("lab-a", False, [], []))
print(stage("lab-a", True, ["lab-a"], ["lab-a"]))
Expected output
packing_gap
excluded_by_policy
candidate_support_check_needed

The fixture keeps authorization, retrieval, and packing distinct. It uses known synthetic IDs and does not infer relevance or run a generator.

Check what the citation actually supports

Resolve each citation to the exact passage and version used. Compare the claim, qualifier, and question scope, not just a matching title or source ID. The second record shows a historical question paired with current-only evidence; the proper diagnosis is missing applicable support, not a confident answer from the available date. Conflicting sources require an authority or review policy. An existing ID can still cite the wrong sentence, and a faithful quotation can still come from an outdated or unreliable source.

yaml Synthetic source-scope mismatch 
question_scope: historical-policy-period
retrieved_source_scope: current-policy-only
source_id: public-lab-policy-v3
applicable_historical_support: absent
release_decision: insufficient-evidence-or-review
not_justified: answer-historical-question-from-current-rule

The record illustrates missing temporal support. It is not a retrieved document, real policy, model response, or instruction to change any data.

Repair the narrow failed contract

Update missing approved data, repair indexing or freshness propagation, preserve chunk meaning, evaluate ranking changes, or correct candidate validation according to evidence. Keep an explicit insufficient-evidence path instead of filling a gap with plausible text. Freeze comparison cases and include unanswerable and excluded-source examples. A prompt edit can be evaluated when evidence reaches the model, but it cannot supply a document that was never present. Bound retries and preserve privacy when recording failed candidates or attempting a repair.

Verify supported outcomes and negative cases

On held-out representative fixtures, verify authorized recall, packed qualifiers, supported claims, citation accuracy, and useful abstention separately. Compare the same question and corpus versions before declaring improvement. A successful local trace classifier proves its rule only, not model accuracy or production retrieval quality. Use human-reviewed evidence where natural-language support needs judgment. No service configuration, real data, or paid inference is changed by this guide.

Verify the fix

The repaired system should retrieve and preserve applicable authorized evidence, reject unsupported claims, and maintain access exclusions. Check citation existence and support separately, including qualifiers and historical versions. Verify that improvements do not rely on leaked development answers or expose another tenant’s material.

Keep it from recurring

  • Version corpus and retrieval bundles, retain stage-level provenance, and include missing-evidence, conflicting-version, and access-boundary cases in regressions. Evaluate end-to-end usefulness alongside retrieval metrics; a larger context pack is not automatically a better grounded answer.

Official references