VEX-Bench puts agent exploitability at 75 cases

VEX-Bench evaluates nine models across three agent harnesses on 75 expert-labeled GitHub cases spanning Python, Java, and Go, reporting GPT-5.5 and Claude Opus 4.6 near 80% F1 on binary vulnerability status, according to the benchmark paper. The harder result is the gap between yes/no and explanation: only GPT-5.5 clears 70% macro-F1 on fine-grained exploitability reasons, so a passing classifier is not yet a trustworthy release gate. Security teams should add repository reachability evidence and human review to agent-generated dependency advisories; AISI’s agent-evaluation incident report shows why test harnesses with real permissions belong inside production-grade controls.