A useful retrieval evaluation set is a versioned collection of representative queries tied to expected evidence and observable pass conditions. It should contain ordinary cases, known failures, confusing neighbors, and deliberate no-answer cases rather than a pile of unreviewed prompts.
Select cases from real decisions and known failure modes
Begin with the tasks that matter to the buyer: questions that precede an action, recur in support, or already produced a wrong answer. Add nearby queries that use different wording, queries where two documents conflict, and queries for which the corpus contains no support. A compact set of twenty reviewed cases usually teaches more than hundreds of synthetic questions whose expected source nobody has confirmed.
For each case, store the raw query, approved query variants, expected document identifiers, the passage or facts a reviewer considers sufficient, forbidden unsupported claims, and a reason the case matters. Keep the source snapshot or version beside the case. Otherwise a later document change can look like a retrieval regression even when the indexed record itself changed.
Score stages separately and preserve disagreements
Use separate labels for indexing coverage, retrieval relevance, ranking, context assembly, citation match, and answer groundedness. A single overall score hides which component needs repair. Binary pass conditions work for source presence, while graded judgments can capture rank or passage usefulness. Document the judge, rubric, and model version when an automated evaluator contributes a label.
Review disagreements rather than averaging them away. If a subject-matter reviewer expects one policy page while the system retrieves a newer page, resolve source authority before tuning. The final set should be small enough to inspect, stable enough to rerun, and broad enough to expose regressions when a repair helps one family of questions at another's expense.
Where the service stops
Reality Contact, LLC evaluates and repairs the bounded retrieval path, but does not certify model accuracy, source truth, security, compliance, or correctness for queries outside the agreed cases. The buyer approves the source authority and acceptance cases, reviews the reported limitations, and decides whether and how to deploy the changed retrieval configuration. This is software evaluation and implementation; it does not replace legal, financial, medical, security, compliance, or professional advice. No result establishes corpus-wide correctness, source truth, or reliable behavior for queries outside the buyer-approved case set.
Sources: LangSmith evaluation concepts; Braintrust evaluation guide.