LLM judges fail a 0.90 reproducibility gate

An audit of LLM judges failed its reproducibility gates across 52,988 request attempts: same-window rankings agreed at Spearman 0.400 against a required 0.90, while byte-identical next-day replays reached 0.78 against 0.99. The full preregistered study says transport, schemas, hashes, and metadata were all at ceiling, so clean harness logs did not make the judge a stable instrument. Teams building eval gates should read this with Ai2’s benchmark-question audit: measure judge noise before trusting a model score.