Unknown

Live demo

Two release candidates. One of them is ready.

A defect population was generated and hidden. Three detection methods sample it independently, and the estimator sees only what they found. Every number below is computed in your browser from the overlap between the finding lists, and the true count is revealed afterwards so you can grade it.

The audit most teams actually run

One scanner, one reviewer, and no answer

Two methods found 34 and 28 defects, and agreed on 5 of them. That is not enough shared findings to estimate from, so the honest output is a warning and not a number. Most tools would print the number.

One scanner, one reviewerKeep testingEstimate unstable2 methods · 1 pair

The shape of most security reviews. Two methods, and they barely found the same things.

Static analysis
34
Semgrep, default rulesets, one pass
Manual review
28
One engineer, three days
Found
57
distinct defects, all methods
Estimated total
~168
do not quote this number
Still in there
many
the overlap cannot say how many
Coverage
34%
gate is 90%
Chapman estimate per pair, with the Seber interval
PairFoundOverlapEstimate95% interval
Static analysis x Manual review34 / 28516867 to 270
Round interval 67 to 270, the envelope across pairs. Deliberately the conservative choice.
Overlap warning. Small overlap. Treat the interval as wide and the point estimate as indicative.
Keep testing. The methods barely overlapped, which means the point estimate itself is unreliable. Read it as "a lot is left", not as 168. Either way it is nowhere near the 90% coverage gate.

Ground truth, which the estimator never saw: 160 defects were injected. The estimate came back at 168, off by 5%, and the true value sits inside the interval.

Add a third method

Now the overlap says something

Same product, a longer review and a fuzzing campaign added. Three methods means three independent pairs, and the spread across those pairs is itself the check on whether the independence assumption is holding.

RC1, full security reviewKeep testing3 methods · 3 pairs

Three methods, three weeks. The report lists what was found and says nothing about what was not.

Static analysis
46
Semgrep and CodeQL, default plus in-house rules
Manual review
44
Two engineers, eight days, diff-led
Fuzzing
50
libFuzzer, 40 core-hours across 6 harnesses
What each method found, and what they found together
19272581663Static analysisManual reviewFuzzing
Found
104
distinct defects, all methods
Estimated total
175
median of the pairwise estimates
Still in there
71
estimated remaining
Coverage
59%
gate is 90%
Chapman estimate per pair, with the Seber interval
PairFoundOverlapEstimate95% interval
Static analysis x Manual review46 / 4411175104 to 246
Static analysis x Fuzzing46 / 501911989 to 149
Manual review x Fuzzing44 / 509229121 to 336
Round interval 89 to 336, the envelope across pairs. Deliberately the conservative choice.
The estimated population, one dot per defect
found by all methods found by one nobody found these, and this is the product
Keep testing. Found 104 of an estimated 175, so roughly 71 are still in there. The gate is 90% coverage and this run is at 59%.

Ground truth, which the estimator never saw: 160 defects were injected. The estimate came back at 175, off by 10%, and the true value sits inside the interval.

Three weeks later

What being done looks like

After remediation, against a much smaller residual population. The methods now keep finding the same defects as each other, which is the signal that there is not much left to find. Heavy overlap is the good outcome, and it is the opposite of what it feels like at the time.

RC2, after remediation and a second passShip3 methods · 3 pairs

Same team, same product, three more weeks of testing against a much smaller residual population.

Static analysis
26
Same rulesets, re-run after remediation
Manual review
26
Two engineers, six days, checklist-led
Fuzzing
21
libFuzzer, 300 core-hours, corpus carried forward
What each method found, and what they found together
67464310Static analysisManual reviewFuzzing
Found
40
distinct defects, all methods
Estimated total
41
median of the pairwise estimates
Still in there
1
estimated remaining
Coverage
97%
gate is 90%
Chapman estimate per pair, with the Seber interval
PairFoundOverlapEstimate95% interval
Static analysis x Manual review26 / 26164236 to 49
Static analysis x Fuzzing26 / 21143933 to 46
Manual review x Fuzzing26 / 21134134 to 50
Round interval 33 to 50, the envelope across pairs. Deliberately the conservative choice.
The estimated population, one dot per defect
found by all methods found by one nobody found these, and this is the product
Ship. Every pair of methods overlapped heavily, so the population is well covered. Found 40 of an estimated 41, about 1 still out there against a budget of 3.

Ground truth, which the estimator never saw: 42 defects were injected. The estimate came back at 41, off by 1%, and the true value sits inside the interval.

Where this breaks, and it will break on your data

Unequal findability makes the answer look better than it is

The estimator assumes every defect is roughly as easy to find as every other. Real code does not work like that: some bugs live in the file nobody reads and no method is good at them. Below is the identical 160-defect population from the second panel, with a third of the defects made hard for all three methods. Nothing else changed.

True population
160
same as the RC1 panel
Estimate returned
104
median of three pairs
Shortfall
35%
and it always points this way
Warnings raised
0
the overlaps look healthy
The direction of the bias is the problem

Hard-to-find defects are missed by every method at once, so they never enter the overlap and never enter the estimate. The population looks smaller than it is and coverage looks higher than it is. On the well-behaved RC1 data the estimate landed within 10% of truth. Here it comes back 35% low with no warning attached, because the overlaps are healthy and nothing in the arithmetic can see the difference. Read every number this product produces as a floor on what remains, not as an answer. The true remaining count is usually worse.

Try it on your own numbers

Two methods, two counts, and how many they shared

That is the entire input. Push the overlap up and watch the remaining count collapse. Push it down and watch the interval blow out and the product refuse to give you a number.

Distinct found
50
union of the two lists
Estimated total
~119
Chapman, bias-corrected
Still in there
many
interval 56 to 181
Coverage
42%
gate is 90%
The estimated population, to scale
everything right of here was never foundA only 24both 6B only 20estimated population 119
No number. Small overlap. Treat the interval as wide and the point estimate as indicative. With 6 shared findings out of 50, the only defensible statement is that a lot is left and you should run another method.

Two assumptions hold this up and neither is free. The methods have to be independent, so two reviewers working from the same checklist will overstate your coverage. And defects have to be roughly equally findable, which they are not, and that one biases the answer low.