Amazon releases SOP-Bench for real business agents

Amazon Science released SOP-Bench, an open benchmark with more than 2,000 runnable tasks across 12 business domains for testing tool-using agents against ground-truth outcomes. In the benchmark release, Amazon says success nearly halved when six necessary tools were buried among 20 plausible extras, while results ranged from roughly nine in ten on easy email triage to about one in four on hard driving-video annotation. Builders should test their own procedures with a constrained tool set before expanding an agent’s permissions, extending the site’s long-horizon reliability analysis.