October 20, 2026

RAG retrieval evaluation: Compare Benchmarks with Your Own Corpus

RAG retrieval evaluation shows how to screen public benchmarks, then test suppliers on your own corpus with relevance labels, ranked results and fixed settings.

A buyer reading public or supplier retrieval benchmarks needs to identify what those runs actually tested. Match corpus, queries, labels, configuration, and ranked outputs to a buyer-controlled run on representative documents. This guide helps decide whether benchmark evidence supports a shortlist.

Key takeaways

What to compare in supplier retrieval evidence

Screen with public benchmarks: Compare candidates on the named corpus, questions, labels and configuration.
Request inspectable supplier runs: Ask for ranked results, per-query scores, labels and the exact configuration.
Run a buyer-controlled test: Your representative documents and questions provide direct shortlist evidence.
Separate retrieval from answers: Score retrieved evidence, generated answers and agent behavior as distinct results.

What does a public RAG retrieval benchmark actually test?

A retrieval benchmark pairs a document corpus and query set with relevant documents or passages, a retrieval configuration and scoring methods. Its score describes the system’s results for those specific inputs and settings. Changing the corpus, query mix, relevance labels, chunking or retrieval settings changes the evaluation conditions.

In the EnterpriseRAG-Bench paper, the simulated technology company Redwood Inference contains 511,962 documents across nine workplace sources, including Slack, Gmail, Google Drive, GitHub, Jira and Confluence. Its 500 questions cover ten categories, from single-document lookup to multi-document reasoning, constrained retrieval, conflicting information and cases where the answer is absent.

The question set pairs queries with gold answers and, where applicable, gold document identifiers. The High Level and Info Not Found categories have no gold document IDs, so document recall is calculated only for question types with expected IDs.

The paper describes varying lexical overlap: Basic questions have moderate overlap, while Semantic questions use low-overlap, roundabout phrasing; it does not establish that every query avoids keyword overlap.

A question set that includes table lookups, cross-document questions and missing-answer cases tests different retrieval behaviors from a set of short factual lookups. The paper’s scaling experiment reports Recall@10 for BM25 and vector search as the corpus grows; recall declines while top-10 local cosine similarity rises.

The enterprise-rag-gold-standard repository provides a small synthetic mock-company corpus and 23 listed evaluation items. It groups 12 direct-retrieval items, including PDF table traps, six multi-hop items and five synthesis or contradiction items. The repository’s evaluation entries pair questions with context and ground truth, and its runner lists precision, faithfulness and relevance as scoring metrics.

Use EnterpriseRAG-Bench to screen breadth across simulated tasks, then inspect repository examples that match the buyer’s lookup, table, or policy-conflict cases. Both are synthetic references, not measurements on real customer data, guarantees of production suitability, or proof of production permissions; keep benchmark version and run date with every cited score.

How do public, supplier-run and own-corpus results differ?

Evidence becomes more useful for procurement as its data and execution conditions approach the buyer’s intended use. A public score provides a shared reference point. A supplier-run test shows a supplier’s selected system under its stated conditions. A buyer-controlled test measures the candidate on the corpus and questions the organization selected for its decision.

Evidence source	Test scope	Shortlist use
Public benchmark	Results on the benchmark’s published corpus, questions, labels and configuration	Screen for relevant capabilities and compare results reported on the same benchmark and version
Supplier-run test	The supplier’s system on the specific corpus, questions and settings it reports	Review a demonstration of the proposed configuration, then inspect the run artifacts and test ownership
Buyer-run or buyer-controlled test	The candidate’s results on the buyer’s chosen corpus, queries, labels and agreed settings	Decide whether retrieval meets the organization’s defined use cases and acceptance criteria

The versioned 2021 BEIR study evaluates retrieval across heterogeneous task types with binary or graded relevance judgments. Across those experiments, no evaluated retrieval approach consistently outperformed the others, and the authors found no correlation between in-domain performance and zero-shot generalization. Use those public findings as comparative context, then measure fit with the organization’s own documents and questions.

A supplier-run result gains procurement value when the buyer can inspect what was tested. Log who selected the documents and questions, who assigned relevance labels, which configuration ran, and who controlled scoring. A supplier can run a buyer-approved test set, and the buyer can verify the outputs or rerun the scoring. The execution can be supplier-supported while the evidence remains buyer-controlled.

Microsoft Foundry’s RAG evaluator documentation defines document retrieval as process evaluation: it compares returned documents with query-relevance ground-truth labels and reports measures such as NDCG, XDCG, Fidelity, Max Relevance and Holes. Its separate system evaluation measures generated responses, including how well a response aligns with the supplied context.

For a retrieval-focused pilot, measure whether relevant evidence appears in the returned results and how it is ranked. The BEIR authors’ metric discussion describes nDCG@k as a balanced measure for binary and graded relevance judgments; their experiments report nDCG@10.

For no-answer queries and queries with no gold document IDs or no relevant gold documents, recall or nDCG may have no applicable denominator. State the evaluator’s prespecified handling (such as reporting the metric as not applicable) rather than assuming a universal zero or score. Evaluate whether the system makes an unsupported answer or appropriately abstains separately from retrieval metrics.

When relevance labels are incomplete, report coverage of the known labeled evidence as such; it does not guarantee recall across the unjudged corpus.

The RAGAS paper distinguishes context relevance, faithfulness to retrieved context and answer relevance to the question. Keep retrieval and response results in separate columns: one records whether the system found relevant documents, and the other records whether the answer stayed grounded in context and addressed the question.

What evidence should suppliers provide before a benchmark claim affects your shortlist?

Set the test conditions with the buyer’s retrieval lead, security or data owner, and supplier before any scores are compared. Ask for the following evidence in order, then store the answers with the result so the test can be reviewed and repeated.

Define the corpus and its snapshot. Choose representative document collections, formats, languages, dates, metadata and access conditions. Include the document types that drive the buyer’s work, such as scanned forms, tables, policies, product records or conflicting versions. Record the corpus snapshot date, document count and inclusion rules. Use an approved sanitized copy or a controlled test environment when document handling requires it.
Build a representative question set. Derive queries from real user tasks, support cases or subject-matter experts. Tag each query by task, such as direct lookup, multi-document reasoning, constrained search, table extraction or answer-absent detection. Keep a reserved test set separate from examples used to configure or tune the system. State who wrote the questions and how the set reflects the intended users.

For example, a policy-retrieval pilot could test an exact reimbursement lookup, a comparison of two policy versions, a threshold in a table and a case with no matching policy. These examples reveal whether the test covers the buyer’s actual retrieval tasks rather than a collection of easy keyword matches.
Create and document relevance labels. Label the relevant documents or passages for each query, and state whether each label is binary or graded. Define how to handle multiple valid sources, conflicting records and incomplete evidence. Ask who labeled each query, what instructions they followed, and how disagreements were resolved. A second reviewer can label a sample and help surface unclear rules before the full evaluation.

Match the label level to the result the platform returns. When the system returns chunks, label relevant passages directly or map passage judgments consistently to source documents. Record whether a document can be relevant for one part of a question and insufficient for the full answer. This distinction helps explain a retrieval miss separately from an answer-generation gap.

In the RAGAS evaluation study, two annotators judged 50 WikiEval questions for faithfulness, context relevance and answer relevance. They agreed in around 95% of cases for faithfulness and context relevance, and around 90% for answer relevance; discussion resolved disagreements. These figures describe annotator agreement in this study’s sample, not general-purpose accuracy targets.

Request retrieval outputs, not only a final answer or aggregate score. Ask the supplier to return ranked document or chunk identifiers for each query, relevance labels, retrieval depth, metric definitions and per-query results. Ask the supplier to document how the evaluator scores duplicate chunks, filtered documents and empty results. A label-based retrieval evaluator can calculate ranking measures against the buyer’s ground truth. A judge-based relevance score can help when labels are unavailable, but the buyer should record that the score comes from a model judge and inspect representative judgments.
Freeze the configuration and run record. Capture the product and version or edition, corpus snapshot, parser and chunking settings, embedding model, retrieval method, filters, query rewriting, reranker, number of results returned and scoring implementation. Include the run date and any changes made during tuning. The TREC 2024 RAG retrieval appendix records systems that combine stages such as BM25, dense retrieval, query expansion and reranking. A recorded configuration lets the buyer interpret which pipeline produced the score.

When retrieval applies access filters, run the relevant questions under the buyer-approved test identities and include the identity or role with each result. Review the returned document IDs against the expected access conditions as a separate result from relevance. This gives the team a clear record of what the retrieval configuration returned in each tested access context.
Ask for repeatable artifacts and a rerun. Request the query file, expected-relevance file, ranked outputs, evaluator version and a per-query report. Agree on a buyer-witnessed rerun or buyer-side scoring of the returned ranked results. Use tuning and regression sets for iterative changes, recording each configuration and run separately. Evaluate the final holdout only after the configuration is fixed, once under the pre-agreed run policy. If holdout results are used to make further changes, use a new untouched set for a fresh final evaluation or disclose that the exposed holdout is no longer an independent final evaluation.
Check coverage by task, not only by an average. Include the business-critical query types in the evaluation: high-impact policies, table-heavy documents, multi-hop questions and no-answer cases where the use case requires them. Set the acceptance rules in advance with the business owner.

Use the ranked outputs to diagnose a miss before assigning a remedy. A missing source can point to corpus inclusion or parsing; an incorrect rank can point to retrieval or reranking; an unexpected filtered result can point to the access configuration. Record the failure category, make one documented change, and rerun the tuning or regression set. That trace shows whether the change improved the target task and whether it affected other query types.

For a supplier using ARES, request the annotated validation set for that domain adaptation; the RAGBench paper identifies domain-specific validation data as a requirement. When a supplier relies on a tuned evaluator, ask which annotated validation data informed that evaluator and whether it represents the buyer’s domain.

How do you decide whether the evidence is enough to shortlist a platform?

Use the public benchmark to decide which candidates deserve closer evaluation, then make the shortlist decision from the buyer’s representative test. A candidate is ready for the shortlist when its retrieval results meet the acceptance rules set for the important question types, its test conditions are comparable with competing suppliers, and the buyer can inspect enough artifacts to reproduce or audit the result.

Set the decision gate before the supplier sees the held-out questions. The business owner can define the minimum retrieval performance for each critical task and decide which failures require another configuration round. For example, a team may require retrieval of the current policy source for every high-impact query, then use a broader ranking target for routine lookups. Keep the gate tied to the impact of a miss instead of choosing a single universal score.

Apply the same corpus snapshot, questions, relevance labels, result depth and scoring rules to every candidate. For a supplier that offers more than one configuration, preserve results from the agreed baseline and the supplier’s proposed production configuration as separately labeled runs.

To include Seahorse Cloud in the shortlist test, ask for the document or chunk identifiers returned for the buyer's questions and the selected service configuration. Assess the actual offered workflow through the same buyer-controlled result files and scoring rules. Score returned evidence separately from generated answers and any agent tool actions, and keep the ranked-result file and each scoring decision available to the buyer.

The Seahorse Cloud product overview lists on-premises installation and SaaS subscription as delivery options. Hosting location, commercial terms, and operator responsibilities are separate configuration and agreement decisions. During procurement, record the proposed edition, hosting location, operating responsibilities and contract terms as separate details. Then run the same retrieval test on the configuration the supplier proposes, with the buyer retaining the ranked outputs and per-task results for the shortlist decision.

FAQ

How are RAGs evaluated?

Evaluate retrieval by comparing ranked results with labeled relevant documents or passages, then assess generated answers separately for faithfulness to retrieved context and relevance to the question. Add an agent-level evaluation when an agent selects tools or performs multiple retrieval steps.

Which retrieval metric should suppliers report?

Ask for a ranking measure such as NDCG@k alongside task-appropriate recall and precision at the result depth the application uses. Use the same relevance labels and k value across suppliers, and inspect per-query and per-task results as well as the aggregate.

Can a public benchmark determine which supplier to shortlist?

A public benchmark can help screen suppliers on its documented corpus and tasks. Use a buyer-controlled run on representative documents and questions to make the shortlist decision, with relevance labels, settings and outputs recorded.

Keep a buyer-controlled benchmark evidence record

Use comparable, buyer-controlled ranked outputs and a documented acceptance rule to decide whether evidence is sufficient for the shortlist. Keep the corpus snapshot, test set, configuration, run artifacts, and decision record together.

Try Seahorse Cloud

Evaluate this workflow with your own documents and acceptance criteria.

Open Seahorse console