PRECOG cuts edge RAG prefill roughly 4,500x

PRECOG reduced edge-device retrieval prefill from about 27 seconds to under 6 milliseconds, a roughly 4,500-fold speedup, while its authors report matching in-context RAG answer quality. The August 3 preprint pre-encodes documents as fixed hidden states and injects the best match into a 1.2-billion-parameter state-space model with a 192 KB state, avoiding context re-ingestion at query time. For builders comparing retrieval systems after Onton’s search benchmark earned only a pilot, this is a striking edge-RAG direction to track, but not yet a drop-in optimization: the paper says Transformer KV caches cannot use the mechanism and publishes no implementation in the abstract.