Live demo
140 eval cases in. 5 out.
A contract-extraction suite of 140 cases, scored against five checks. Then the rebuilt one, scored identically.
Suite today
2.0/10
Rebuilt
8.0/10
Pass rate 95.7% — near the ceiling, which is why the power check fails rather than passes.
failCan it see the effect you care aboutn/a
Every case returns the same verdict, so this suite cannot detect a regression of any size.
warnHow much movement is the judge0%
No repeated judgements, so judge noise is unmeasured — which is itself the finding.
warnEvery slice big enough to report2 thin
assignment (10), renewal (10) are under 30 cases and cannot support a per-slice claim.
failNot saturated or floored96%
Near-ceiling pass rate: this suite has stopped discriminating between versions.
failNo duplicate cases8
8 duplicate cases are double-weighting whatever they test.