AWS cuts HyperPod first-token latency by 82%

AWS launched SageMaker HyperPod Inference Gateway, a Kubernetes-native EKS add-on that the company says cuts first-token latency by up to 82% without changing model servers or client applications. AWS’s benchmark reports a mixed-GPU Qwen3-32B test with 50% higher throughput and a 97% P99 reduction, while uniform fleets under steady traffic showed no improvement. For teams already on HyperPod, this is a routing canary before a serving rewrite: measure P95/P99 and throughput against the archive’s workload-specific latency and cost check instead of budgeting from AWS’s best case.