Start with one answer whose expected source is known. Preserve the exact query, retrieval configuration, returned passages, scores, prompt context, and output so the first divergence from the expected evidence can be named rather than guessed.
Trace the case in the order the system saw it
Freeze the failed run before changing prompts or indexes. Record the user query, every query rewrite, active filters, retrieval mode, top-k value, returned document identifiers, passage text, scores, and the context sent to the model. Also name the source that a domain reviewer expected. This creates a single comparison between the evidence that should have appeared and the evidence that actually reached the answer step.
Classify the first observable failure. The expected document may be absent from the index, excluded by a filter, split so its useful sentence loses context, outranked by a near-duplicate, retrieved correctly but dropped from the prompt, or present in the prompt while the answer ignores it. Fixing a later stage cannot repair an earlier missing source, so the first divergence determines the next test.
Turn the diagnosis into a repeatable acceptance case
Write the pass condition before tuning. A useful condition names the expected document or passage, an acceptable rank range, the evidence the answer must cite, and any statement the answer must avoid when the record is silent. Keep retrieval relevance separate from answer groundedness: a strong passage can still produce an unsupported answer, while a grounded answer can be based on the wrong passage.
Change one variable and rerun the frozen case. If metadata repair raises the expected passage but harms other cases, retain both results and mark the tradeoff. The output should show configuration, result, and evidence together. That record lets a buyer decide whether the failure was repaired, narrowed, or merely moved to another query.
Where the service stops
Reality Contact, LLC evaluates and repairs the bounded retrieval path, but does not certify model accuracy, source truth, security, compliance, or correctness for queries outside the agreed cases. The buyer approves the source authority and acceptance cases, reviews the reported limitations, and decides whether and how to deploy the changed retrieval configuration. This is software evaluation and implementation; it does not replace legal, financial, medical, security, compliance, or professional advice. No result establishes corpus-wide correctness, source truth, or reliable behavior for queries outside the buyer-approved case set.
Sources: LangSmith RAG evaluation tutorial; Braintrust evaluation documentation.