Google, ByteDance and Red Hat improve AI on Kubernetes

Third, Dynamic Resource Allocation, developed with Intel and others, simplifies and automates how Kubernetes allocates and schedules GPUs, TPUs, and other devices to pods and workloads. When used along with the vLLM inference and serving engine, the community benefits from scheduling efficiency and portability across accelerators. 

“Large-scale inference with scalability and flexibility remains a challenge on Kubernetes. We are excited to collaborate with Google and the community on the Gateway API Inference Extension project to extract common infrastructure layers, creating a more unified and efficient routing system for AI serving — enhancing both AIBrix and the broader AI ecosystem.” – Jiaxin Shan, Staff Engineer at Bytedance, and Founder at AIBrix

“We’ve been collaborating with Google on various initiatives in the Kubernetes Serving working group, including a shared benchmarking tool for gen AI inference workloads. Working with Google, we hope to contribute to a common standard for developers to compare single-node inference performance and scale out to the multi-node architectures that Kubernetes brings to the table.” – Yuan Tang, Senior Principal Software Engineer, Red Hat

“We are partnering with Google to improve vLLM for operationalizing deployments of open-source LLMs for enterprise, including capabilities like LoRA support and Prometheus metrics that enable customers to benefit across the full stack right from vLLM to Kubernetes primitives such as Gateway. This deep partnership across the stack ensures customers get production ready architectures to deploy at scale”  – Robert Shaw, vLLM Core Committer and Senior Director of Engineering Neural Magic (acquired by Red Hat)