Live demo
985 labelled relevant documents in. 230 out.
200 queries against docs-search · BM25 + text-embedding-3. The pool is everything at least one retriever surfaced, so recall against it is 100% and cannot fall. The number after “out” is how many relevant documents the overlap says neither retriever reached.
| Retriever | Found | Only it found | Recall vs pool | Recall vs estimate | |
|---|---|---|---|---|---|
| BM25 | 737 | 355 | 74.8% | 60.7% | |
| Dense | 630 | 248 | 64.0% | 51.9% | |
| Union | 985 | 603 | 100.0% | 81.1% |
BM25 found 737 relevant documents and dense found 630. Only 382 were found by both. If the two were exhausting the population, nearly everything dense found would already be in the BM25 list. It is not.
The estimate treats the two retrievers as independent captures, and they are not: a document that shares the query’s words is also close in embedding space. Documents that are easy for both inflate the overlap, and an inflated overlap makes the population look smaller. So the number above is a floor on what is missing. This data was generated with 25.0% of relevant documents easy for both retrievers and 1,322 relevant documents in total. The estimate lands 8.1% under that, in the direction the caveat predicts, and the true union recall is 74.5% against the 81.1% estimated.
| Query | BM25 | Dense | Both | Estimate | Status | |
|---|---|---|---|---|---|---|
| Q-001 | How do I rotate an API key without downtime | 3 | 5 | 3 | 5.0 | unstable |
| Q-002 | How do I export invoices for a closed billing period | 2 | 2 | 1 | 3.5 | unstable |
| Q-003 | How do I set a per-seat spending limit | 1 | 1 | 1 | 1.0 | complete |
| Q-004 | How do I restore a deleted workspace | 2 | 5 | 1 | 8.0 | unstable |
| Q-005 | How do I configure SSO with an Okta group filter | 0 | 2 | 0 | 2.0 | unstable |
| Q-006 | How do I why webhook retries stop after five attempts | 6 | 5 | 2 | 13.0 | unstable |
| Q-007 | How do I migrate from the v1 to the v2 events endpoint | 1 | 2 | 1 | 2.0 | unstable |
| Q-008 | How do I refund a prorated plan change | 5 | 6 | 4 | 7.4 | unstable |