Anthropic finds lie detectors fail to generalize

Anthropic researchers found that fine-tuned lie detectors trained on model outputs failed to generalize reliably to unseen deception types: in-distribution AUROC rose from 0.60 to 0.95, but cross-category transfer plateaued around 0.70–0.75, according to the study. The team tested roughly 200,000 labeled examples across eight model families and found that larger prompted models often beat the specialized detectors outright, while about 25% of labels changed during human review. Safety teams building behavioral monitors should test transfer across genuinely new failure modes, a stricter standard than the site’s earlier agent-evaluation guidance.