Buoyant, the company behind Linkerd, has open-sourced Kinference, MarkTechPost reports. It sits in front of a fleet of vLLM workers and decides where each request goes.
- Routing policies, token rate limits, KV-cache-aware load balancing and per-tenant metrics in one layer.
- Written in Rust, Apache 2.0, configured through five Kubernetes CRDs. It’s v0.1.0-alpha.7.
- Buoyant’s own benchmark shows median time-to-first-token on shared-prefix traffic dropping from 6.88s to 0.31s, and no gain without shared prefixes.
My take
“vLLM is the data plane, Kinference is the control plane” is a clean way to think about it. It’s alpha and the numbers are self-reported, so test before you trust it.
Review before you install: read the release notes yourself, back up your config, and try new versions somewhere you can afford to break first.
God Bless, and May the Source Be with You!
