IBM lifts repeatable agent success to 69%

IBM Research reports that its consistency guidelines raised five-run Pass⁵ from 53.0% to 69.0% on 168 AppWorld tasks, while mean success moved from 77.4% to 81.0%. The evaluation write-up says the method mines one recorded trajectory and tests five fresh runs, shrinking the consistency gap from 24.4 to 12.0 percentage points; results come from IBM’s benchmark setup, not production traffic. Builders evaluating agents for side-effecting workflows should add repeatability to their scorecard, alongside cost and average success, as the search API benchmark shows how headline quality can hide operating cost.