Model scanners miss 48 malicious families

An artifact-backed benchmark tested ModelScan, ModelAudit, and Fickling across 170 Pickle- and PyTorch-focused artifacts from 145 specimen families; ModelScan failed to complete analysis on 48 malicious families, while ModelAudit produced definitive judgments for all 135 labeled families, according to the study on arXiv. ModelScan still recorded 100% precision, recall, and F1 when it answered, but only 49.6% definitive coverage—an accuracy number without an availability number can flatter a gate. Security teams should track no-answer failures separately and retain redundancy, much as the earlier audit of weak coding-agent correctness gates argues for testing beyond a benchmark pass.