Live demo
12 environments in. 3 out.
A vendor batch of twelve environments, 840 episodes. Three have a reward a policy can farm without doing the task.
Environments
12
Gameable
3
Clean
9
At $4,000 each
$12,000
| Environment | Reward alignment | Exploit gap | Exploit policy | Verdict |
|---|---|---|---|---|
| api-integrate | 0.99 | 0.26x | — | verified |
| browse-checkout-exploit | -0.16 | 2.80x | reward-loop | gameable |
| calendar-negotiate | 0.99 | 0.26x | — | verified |
| code-refactor | 0.99 | 0.27x | — | verified |
| doc-summarise | 0.99 | 0.26x | — | verified |
| form-complete | 0.99 | 0.25x | — | verified |
| invoice-extract | 0.99 | 0.25x | — | verified |
| log-diagnose | 0.99 | 0.26x | — | verified |
| spreadsheet-fill | 0.99 | 0.27x | — | verified |
| sql-repair-exploit | -0.21 | 2.67x | reward-loop | gameable |
| test-repair | 0.99 | 0.26x | — | verified |
| ticket-triage-exploit | -0.17 | 2.83x | reward-loop | gameable |
Weak alignment alone is explainable by partial credit. A high-scoring non-completing policy alone can be a lucky run. Requiring both is what keeps this report something a vendor argues with rather than dismisses.