Real-SWE tests coding agents on private code

Specific Labs’ Real-SWE benchmark puts Claude Fable 5.1 at 38.8% pass@1 on private production code, ahead of GPT-6 Astra at 33.8% and Gemini 3.8 Flash at 31.2% across eight runs per task. The benchmark methodology and leaderboard show six of 10 tasks below 15% overall, while estimated rollout cost ranges from $2.50 to $6.96; YC’s launch note confirms the private-code premise. For builders, versioned agent benchmark evidence is the right comparison: treat public leaderboard rank as a screening signal, not permission to replace coding workflow.