Tests whether your benchmark leaked
Your benchmark went up two and a half points because the model had seen the answers
You cannot inspect the training data, and you do not need to. If memorised items score better than fresh ones by more than chance, the number you are reporting is inflated — and that is a test, not an opinion.
The problem
Eval credibility rests entirely on trust, and the trust is starting to go
Every model release quotes benchmark numbers nobody can independently check, and every serious practitioner privately assumes some contamination. There is no accepted way to measure it, so the argument stays at the level of suspicion — which is bad for the labs being doubted and worse for the ones being honest.
The insight
Contamination is a two-group comparison, not a search of the corpus
Split items by how much they overlap public text, compare scores between the groups, and ask whether the gap could have arisen by chance. A permutation test answers that with no distributional assumptions and no access to anything private. If the split is uninformative the p-value says so, which makes a clean result meaningful rather than merely unfalsified.
N-gram overlap scoring against public corpora to partition items, mean-score gap between suspect and clean groups, and a permutation null over label assignments for an exact p-value, with a contamination-adjusted score reported alongside the raw one.
How it works
Four steps, no data science team
Every eval item gets an overlap score against public text.
Suspect against clean, on the model you are evaluating.
Thousands of relabellings give the null, and the p-value follows.
What you measured, and what it would be on clean items only.
Who it is for
Whoever puts their name on the eval result
Labs, eval vendors and anyone publishing a benchmark number they expect to be challenged on.
Pricing
- –Overlap scoring
- –Permutation test
- –Adjusted score
- –Continuous checking
- –Version comparison
- –Corpus updates
- –Certificate export
- –Self-hosted
- –Private corpora
- –Custom overlap
- –Direct support
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Canary strings and held-out sets | Benchmark authors mark or withhold data to detect leakage. | Only works if you own the benchmark and planned ahead. Useless for anything already published. |
| Contamination research | An active academic literature with several detection methods. | Papers rather than a service, and no standard a third party can point at when challenged. |
| Private held-out benchmarks | Keep the test set secret. | The strongest defence and it does not scale — the secret set leaks the moment it is used enough to matter. |
| Assuming it is fine | Current practice. | Increasingly untenable, and the first serious public dispute will end it. |
The buyer set is small and unusually sophisticated — labs employ people who have read every contamination paper and can implement this themselves, so the sale is about neutrality rather than capability. That points at being a third-party certifier rather than a vendor, which is a slow, credibility-first business with almost no revenue in year one. And overlap is a proxy: a model can memorise something without lexical overlap, so a clean result is weaker evidence than a dirty one, which must be said plainly or the product is dishonest.
Market
Small in dollars, disproportionate in credibility — it is the reference everyone else cites
A few dozen serious buyers. Better read as an authority-building asset that makes Eval Kits and the Index credible than as a business on its own.