Recall measured against what the pool misses
Your retrieval recall is 98% against a pool your retrievers built
The labelled set of relevant documents came from what BM25 and the embedding index surfaced. Recall against it cannot fall much below 100%, and the documents neither retriever reached are not in it. Recall estimates how many of those there are.
The problem
The pool is the union of what was found, so recall against it is circular
Nobody labels a corpus of two million chunks. They run the retrievers, hand the top results to annotators, and call the relevant ones the pool. Every retrieval metric on the dashboard is then computed against that pool. A document no retriever surfaced was never shown to an annotator, is not in the pool, and does not count as a miss. The dashboard reads 95 to 100% while the answer the user needed sits unretrieved and unlabelled.
The insight
Two retrievers are two captures. The overlap tells you the size of the lake
Ecologists count fish they cannot see by catching twice and counting how many fish turn up both times. A lexical retriever and a dense retriever are two captures of the same relevant population. If most of what dense found was also found by BM25, the population is nearly exhausted. If the overlap is small, there is a lot neither reached. Lincoln-Petersen turns that overlap into an estimate of the relevant population, and therefore into a recall number for each retriever and for the union that does not depend on the pool being complete.
Chapman-corrected Lincoln-Petersen capture-recapture over relevant documents pooled across queries, with a Seber variance interval, treating each retriever as an independent capture and reporting the estimate as a floor because positive dependence between retrievers biases it low.
How it works
Four steps, no data science team
Point Recall at your eval queries and two retrievers that work differently. Lexical and dense is the usual pair.
For each query, which relevant documents both found, which only one found. Pooled across queries so the overlap is large enough to trust.
The pooled overlap gives an estimated relevant population with an interval. Recall per retriever is what it found divided by that, not by the pool.
How many relevant documents the union misses, which retriever is earning its place, and how many queries to label in full to check the estimate.
Who it is for
The engineer who owns retrieval
Teams shipping retrieval-augmented products who measure retrieval against a pool of labels and have started to notice that recall never moves.
Pricing
- –Population estimate
- –Per-retriever recall
- –Dependence caveat
- –Unlimited queries
- –Recall history per release
- –Third-retriever pairwise check
- –Label-sampling plan
- –Self-hosted
- –CI gate on estimated recall
- –Per-slice estimates
- –Audit export
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| RAGAS, LlamaIndex evals, TruLens | Retrieval metrics computed against a set of labelled relevant documents. | Every metric they compute takes the label set as complete. None of them estimates what the label set is missing, which is the number that decides whether the retriever is good. |
| Pooled relevance judgments | The TREC method: pool the top results of many systems and judge those. | Known for thirty years to undercount relevant documents outside the pool. Works at TREC because dozens of systems contribute; a product team has two. |
| Vector database dashboards | Latency, hit rate and recall against a benchmark set. | Recall against an internal benchmark whose relevant set was built with the same index. The number is about the index agreeing with itself. |
| Recall against the pool | What almost every team does today. | Reads 95 to 100% by construction and cannot fall, so it cannot tell you when a retriever is missing the document the user needed. |
The estimate rests on the two retrievers being independent captures, and they are not: a document that shares the query's words is also close in embedding space. That dependence biases the estimate low, so Recall reports a floor on what is missing and says so on every page. If buyers read a floor as a point estimate and stop there, the product has made them feel measured without measuring. The method is a few lines over two sets and any eval framework could add it in a sprint; what they lack is the will to publish a recall number below the one on the dashboard. The bet is that the engineer who owns retrieval would rather know.
Market
Priced against the eval budget teams already spend on retrieval quality, not as a new line
Every team with a production RAG system and an eval set is a candidate, and the moment recall stops moving between releases is the trigger. Three thousand teams at the Team tier is $14M ARR; Scale is priced for the teams whose retrieval gate blocks a release.