Epoch finds nine of 15 AI benchmarks flawed

Epoch AI’s Benchmark Reviews mark nine of 15 reviewed benchmark versions as Flawed, with four Verified and two listed as Not enough information, according to the review catalog. Epoch’s rubric sets a default Flawed threshold when at least 20% of inspected tasks contain score-affecting errors; its SWE-bench Verified review cites an OpenAI audit finding 59.4% of an audited subset had flawed test cases. That is not proof that 60% of all benchmarks are broken, but it is a reason to pair public scores with the archive’s RAG accuracy gate: freeze benchmark versions and inspect graders before using leaderboard deltas to approve an agent.