Radagast the Brown 2026-05-14 02:12:00 +04

I’ve been watching the LLM inference space settle since KubeCon EU in March. Most discussions center on model quality. For platform teams the bottleneck is operational - not algorithmic. Getting low latency at scale means solving scheduling, caching and capacity problems that Kubernetes wasn’t designed for.

llm-d is the most serious attempt I’ve seen to treat distributed inference as an infrastructure problem rather than a model-serving side effect.

The architecture decomposes into three primitives.

InferencePool is the discovery target - a CRD that groups model server pods sharing the same base model and accelerator configuration. Platform admins define one per model + hardware combination and the Router treats it as the addressable unit for scheduling.

The Router splits into two planes. The Proxy is an Envoy L7 gateway that terminates client requests and delegates placement decisions. The Endpoint Picker (EPP) is where the actual intelligence lives - it scores every candidate pod across pluggable filters and scorers (queue depth, KV cache locality, prefix match ratio, predicted latency) before returning the optimal target to the Proxy. The Proxy is fast and dumb 🐎. The EPP is slower and smart 🐙.

Model servers are the inference engines - vLLM or SGLang running on accelerators. They emit KV cache events and expose utilization data that the EPP consumes. They’re stateless from the Router’s perspective but their internal cache state is the most valuable signal the scheduler has.

Routing to the right InferencePool gets you in the neighborhood. Routing to the right pod - the one that holds the cache prefix, has headroom and can still meet SLOs, is a fundamentally harder problem. A scheduler that ignores that internal state is doing round-robin on expensive hardware.

KV Cache as a first-class routing signal. llm-d maintains a global, near-real-time index of KV block locality across every vLLM pod. The scheduler doesn’t guess where cached prefixes live - it knows. In cache-heavy workloads, p90 TTFT dropped from 90+ seconds to 0.5 seconds 📉. That’s not an optimization. That’s a different operating regime.

Autoscaling that understands cost. The Workload Variant Autoscaler models KV cache pressure, queue depth and request rate across hardware variants then picks the cheapest accelerator that still meets SLOs. Scale-to-zero is built in - not bolted on.

Latency prediction instead of heuristic routing. A sidecar trains online from live traffic: prompt length, cache hit rate, queue depth, KV utilization and predicts TTFT and TPOT per candidate pod. The router picks the pod with headroom, not the one with the shortest queue.

The tradeoff is real: running a predictor sidecar, maintaining the KV event stream, tuning autoscaling policies - this adds operational surface.

llm-d worth a look if you operate inference at scale.