The scorecard
Five checks, and the specific reason for each mark
What arrives before any work is agreed. The uncomfortable version, because a scorecard everyone passes is not a scorecard.
Eval scorecard
Contract extraction suite
Score
2.0/10
Cases
140
Slices
5
Pass rate
95.7%
fail
Can it see the effect you care about — Every case returns the same verdict, so this suite cannot detect a regression of any size.
warn
How much movement is the judge — No repeated judgements, so judge noise is unmeasured — which is itself the finding.
warn
Every slice big enough to report — assignment (10), renewal (10) are under 30 cases and cannot support a per-slice claim.
fail
Not saturated or floored — Near-ceiling pass rate: this suite has stopped discriminating between versions.
fail
No duplicate cases — 8 duplicate cases are double-weighting whatever they test.