Google claims GKE cuts AI wait time 92.8%

Google reports that its GKE Inference Gateway delivered 15.7% higher output-token throughput, 92.8% shorter time to first token, and 62.6% lower inter-token latency than a third-party managed Kubernetes service in a shared-prefix benchmark using eight NVIDIA A100 40GB GPUs. Google’s benchmark write-up attributes the gains to prefix-cache-aware and model-aware routing; the comparison is useful evidence, but it remains a vendor-published account of an external benchmark. Teams serving RAG or multi-turn agents can start with the archive’s Android memory analysis and pilot cache-aware routing before rewriting their serving stack.