Buoyant Releases Kinference: Open Source Inference Control Plane for vLLM and Kubernetes

Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes

We’re excited to announce Kinference [kay-inference], a new open source project for building scalable inference platforms with vLLM.

Kinference by Buoyant

We created Kinference at Buoyant after watching our customers struggle to run open weight models in their environments. We designed Kinference to be the control plane layer for the platform you need to build. It uses vLLM to do the underlying inference, and provides the set of features necessary for building an inference platform in practice. This includes token rate limiting, tenant usage and cost tracking, intelligent load balancing, and inference policies.

Kinference Architecture

Kinference can be paired with your existing stack. Using Kinference is simply a matter of pointing inference clients to its HTTP endpoint and sending OpenAI or Anthropic-compatible inference traffic there.

Kinference serves a heterogeneous, evolving set of models running on a heterogeneous, evolving pool of GPUs, to a heterogenous, evolving set of clients..

Inference load balancing and performance

In addition to policies and a routing layer, Kinference provides inference-specific load balancing that takes into account signals like GPU queue sizes, historical performance, and the contents of the KV cache on the GPU. This KV-cache awareness, in particular, allows Kinference to send requests to the vLLM worker that will do the least work for that specific request.

Modeling the KV cache content of the GPU can produce dramatic gains in performance for distributed inference. For example, in our public Kinference benchmarks, we showed that, when compared to standard Kubernetes load balancing (labeled “service” below), Kinference (labeled “kinference”) was able to cut median time-to-first-token (TTFT) by 95% and median end-to-end latency by 45% of requests on one workload type.

Time to First Token at light load

By incorporating cache awareness into policies, platform owners can create policies that will send simple requests to the cheaper model — but only if that’s less work than sending them to the expensive model in the first place.

In our unloaded latency benchmark above, we sought to measure the effect of Kinference’s KV-cache-aware load balancing in isolation. In our capacity benchmark, below, we measured the requests per minute can each endpoint can serve within a latency SLO.

Goodput by concurrency

In the capacity benchmark, Kinference served 7.4× the goodput on shared-prefix and about 3.5× the goodput on multiturn. It was able to attain our chosen SLO at concurrency 32 (the highest measured) on our shared-prefix workload, and concurrency 8 on the multiturn workload, whereas the kube-proxy approach was not able to attain the SLO at any concurrency level.