PatchBench says crash-only tests overstate patching

PatchBench found that crash-only checks overstated vulnerability-patching agents’ solve rates by 1.83× across 11 state-of-the-art systems, while 25% of patches substantially resembled historical developer fixes, according to the new PatchBench paper. It moves the engineering question from “did the PoC stop crashing?” to “did the root cause and program semantics survive?”—a useful companion to the archive’s analysis of AI’s shrinking exploit window for teams deciding whether to trust automated security remediation.