RSI-Bench

Can a model tell whether its change helped?

RE-Bench, MLE-bench, PaperBench and AI4AI-Bench all score whether an agent can produce an improvement. None score whether it can recognise one. That is the thing a self-improvement loop actually runs on. Read +1.2% off two noisy seeds, call it a win, and you walk downhill spending real compute.

240 items. Each is two training arms differing on one axis: depth, width, learning rate, method. You get 2 seeds per arm with loss curves, can buy more at 1 unit each from a budget of 20, then call it helped, no_effect or hurt. Ground truth is up to 216 seeds per arm. The runs are already done, so this replays from cache in a minute on a laptop.

accuracy vs what it cost

Above the dashed line means a better call from the same seeds.

the error that compounds

The t-test almost never does this. The models do it 7 to 20 times more often.

is the confidence worth anything

On the diagonal is honest. Below it is bluffing.

all of it

policy	acc	spend	edge	brier	fp_helped	missed_help	n
grok-4.6	0.873	4.5	+0.248	0.095	0.098	0.079	79
claude-haiku-4-5	0.691	0.6	+0.234	0.190	0.091	0.364	55
claude-sonnet-5	0.861	4.6	+0.233	0.106	0.043	0.154	36
claude-opus-5	0.823	4.6	+0.195	0.119	0.108	0.080	62
sequential	0.779	8.3	+0.063	0.149	0.019	0.325	240
ttest-k12	0.817	20.0	+0.000	0.139	0.006	0.300	240
ttest-k2	0.421	0.0	+0.000	0.230	0.000	0.900	240
ttest-k3	0.542	2.0	+0.000	0.191	0.006	0.662	240
ttest-k4	0.613	4.0	+0.000	0.184	0.013	0.575	240
ttest-k6	0.713	8.0	+0.000	0.167	0.006	0.412	240
ttest-k8	0.762	12.0	+0.000	0.150	0.013	0.338	240
constant	0.333	0.0	-0.087	0.222	0.000	1.000	240

edge is accuracy minus what a Welch t-test gets for the same mean spend. Baselines run all 240 items; model rows cover what has been scored so far, so small n is provisional.

github.com/teddytennant/rsi-bench