SciCode audit finds 263 benchmark defects

A domain-expert audit found 263 defects across SciCode’s 65 scientific-coding problems, including 192 score-suppressing flaws that touched 91% of the main problems. SciCode-Verified reports that correcting specifications and graders lifted 12 frontier model snapshots from 9–27% to 69–92% accuracy on complete problems; 78% of the consequential defects required physics or mathematics expertise to detect. Operators comparing the frontier should treat benchmark maintenance as part of procurement, not clerical cleanup: the lesson reinforces why Kimi K3’s early leaderboard economics still needed workload-level testing, because a broken instrument can manufacture a capability gap larger than the models themselves.