Description Most production inference clusters today are single-vendor — not because it is optimal, but because it is the simplest way to set things up. Real fleets are accumulating heterogeneity, through procurement cycles, supply constraints, and the widening cost gap between accelerators. The open question is whether a single Kubernetes-native serving layer can take a heterogeneous GPU fleet a… more

Introducing llm-d llm-d (llm-d.ai) is a high-performance distributed inference serving stack optimized for production deployments on Kubernetes It provides a transparent routing layer that sits between the client and vLLM serving pods, making key scheduling decisions at request granularity: more