IndicBankBench puts banking reliability at 43.7–58.2%

IndicBankBench tested 11 models across 799 Indian retail-banking cases and found strict three-run reliability of only 43.7%–58.2%, versus 60%–74% when a model merely succeeded once, according to the benchmark paper. Its cases cover safety, tool actions, response adequacy, and advice, so the result measures workflow correctness rather than prose quality; teams can compare it with harness cost-per-pass evidence from Strands evaluations before trusting pass@1 demos.