BenchMIRT finds WMDP measures reasoning, not safety

Ai2’s BenchMIRT audit trained on results from 100 open-weight LLMs across 16 benchmarks and more than 34,000 questions, then found WMDP had no significant correlation with safety while tracking general reasoning in the opposite direction. Ai2’s technical release says BBQ, a social-bias test, also aligned much more strongly with general reasoning than safety. Evaluation teams should stop treating one benchmark score as a clean proxy for the construct it names; the archive’s cost study of automated alignment research makes the same case for inspecting what safety tests actually measure.