AWS announced Amazon SageMaker HyperPod Inference Gateway, a Kubernetes-native EKS add-on for existing HyperPod clusters. It uses GPU signals such as KV-cache utilization, queue depth, LoRA-adapter residency, prefix-cache hit rate, and running requests to choose a backend; AWS says an example reduced time to the first token from 4.4 seconds to under 800 milliseconds. The first tier handles routing within a cluster, while global cross-cluster and cross-region routing is planned for a later tier.

Editor's note

This summary is based on the primary source. Open the original for full context, data, and qualifying details.