Accidental demos collapse robot success to 1.2%

A controlled behavior-cloning study found that replacing every clean Push2D demonstration with an accidental success collapsed closed-loop policy success from 97.2% to 1.2%, even though every retained trajectory reached the goal. The reproducible demo-quality experiment ran 78 training configurations and found that policies with similar open-loop mean-squared error between 0.20 and 0.30 still ranged from 31.5% to 98.0% closed-loop success; its author cautions that the evidence comes from three seeds, a 2D simulator, and one policy class. For teams following World Labs’ case for scaling robot simulation, the operator lesson is to audit demonstrations by failure mode and run closed-loop dose-response tests rather than trusting success labels or offline action error alone.