Can a model tell whether its change helped?
RE-Bench, MLE-bench, PaperBench and AI4AI-Bench all score whether an agent can produce an improvement. None score whether it can recognise one. That is the thing a self-improvement loop actually runs on. Read +1.2% off two noisy seeds, call it a win, and you walk downhill spending real compute.
240 items. Each is two training arms differing on one axis: depth, width, learning rate, method. You get 2 seeds per arm with loss curves, can buy more at 1 unit each from a budget of 20, then call it helped, no_effect or hurt. Ground truth is up to 216 seeds per arm. The runs are already done, so this replays from cache in a minute on a laptop.
Above the dashed line means a better call from the same seeds.
The t-test almost never does this. The models do it far more often.
Split grok's 240 by label and the failure has a shape. It is good at spotting a real effect and close to a coin flip on recognising that nothing happened.
| grok-4.6 on | acc | n |
|---|---|---|
| hurt | 0.887 | 80 |
| helped | 0.850 | 80 |
| no_effect | 0.537 | 80 |
It returned 178 helped-or-hurt verdicts where 160 exist. For a self-improvement loop that is the wrong direction to be wrong in: it accepts changes that did nothing, and each acceptance is compute spent walking sideways.
On the diagonal is honest. Below it is bluffing.
| policy | acc | spend | edge | brier | fp_helped | missed_help | n |
|---|---|---|---|---|---|---|---|
| claude-haiku-4-5 | 0.691 | 0.6 | +0.234 | 0.190 | 0.091 | 0.364 | 55 |
| claude-sonnet-5 | 0.861 | 4.6 | +0.233 | 0.106 | 0.043 | 0.154 | 36 |
| claude-opus-5 | 0.823 | 4.6 | +0.195 | 0.119 | 0.108 | 0.080 | 62 |
| grok-4.6 | 0.758 | 6.7 | +0.080 | 0.166 | 0.113 | 0.150 | 240 |
| sequential | 0.779 | 8.3 | +0.063 | 0.149 | 0.019 | 0.325 | 240 |
| ttest-k12 | 0.817 | 20.0 | +0.000 | 0.139 | 0.006 | 0.300 | 240 |
| ttest-k2 | 0.421 | 0.0 | +0.000 | 0.230 | 0.000 | 0.900 | 240 |
| ttest-k3 | 0.542 | 2.0 | +0.000 | 0.191 | 0.006 | 0.662 | 240 |
| ttest-k4 | 0.613 | 4.0 | +0.000 | 0.184 | 0.013 | 0.575 | 240 |
| ttest-k6 | 0.713 | 8.0 | +0.000 | 0.167 | 0.006 | 0.412 | 240 |
| ttest-k8 | 0.762 | 12.0 | +0.000 | 0.150 | 0.013 | 0.338 | 240 |
| constant | 0.333 | 0.0 | -0.087 | 0.222 | 0.000 | 1.000 | 240 |
edge is accuracy minus what a Welch t-test gets for the same mean spend. Only grok-4.6 has run all 240; the Claude rows are partial and should not be ranked off. Grok itself sat at 0.852 and +0.270 over its first 28 items and finished at 0.758 and +0.080, so early items flatter.