vLLM and NVIDIA use coding agents to nearly double Kimi K3 decode speed, then rethink how to review it

Per layer, stock vLLM ran 23 kernels in 130 microseconds; the specialised path runs 13 kernels in 69 microseconds[1]. The post credits four kinds of work: picking different kernels for small and large batches, fusing the attention output projection with its all-reduce so GPUs publish partial results straight into NVLink-accessible buffers, splitting router and latent projection work across GPUs instead of repeating it on each one, and prefetching the next weights into cache while communication runs[1].

It is deliberately narrow. The initial validation covers GB300 with 8 or 16 GPUs, decode batches up to 64, speculative decoding with various draft lengths and decode context parallelism within one NVLink domain[1].

Without speculative decoding, batch-1 decode rises to 193.3 tokens per second on 8 GPUs (1.58x) and 256.4 on 16 GPUs (1.88x), and the 16-GPU setup keeps a 1.62x to 1.71x gain at batches of 16 to 64[1]. For context, vLLM's launch-day Kimi K3 support in July served 118 tokens per second per user on 16 GB300 GPUs without speculation[2].

The 842 tokens per second headline uses DSpark speculative decoding with seven draft tokens and a synthetic acceptance length of 6, and it excludes prefill[1]. That is 1.73x the 488 tokens per second stock vLLM gets with the same setup[1]. In July, with real acceptance rates, vLLM reported around 4.73 accepted tokens per step on coding tasks and around 2.61 on creative writing[2], so real-world speed will depend on the workload.

The more realistic number is AgentX, which replays traffic patterns from real coding-agent sessions with synthetic content. There, P90 interactivity at concurrency 1 went from 223.7 to 403.2 tokens per second per user, though the earlier figure was measured on 8 GPUs and the new one on 16[1].

The candid part of the post is about maintenance. "Agent-optimized inference code is outpacing our review and maintenance workflow," it says, and skipping line-by-line review risks shipping software that fails in production[1].

vLLM's answer is a proposed fast-track model optimisation RFC. Optimisations stay inside the model's own code, target a defined set of hardware and workloads, keep features like full-context serving, temperature sampling and structured output, and must pass both kernel tests and full model evaluations[1]. Agents are expected to write the tests, run the benchmarks and document where the optimisation works, so human reviewers can judge the design and the evidence rather than every line[1].

That validation caught real bugs. An audit found the compiler had moved activation reads ahead of a synchronisation wait, and long-context and long-generation evaluations such as MRCR and AIME25 exposed numerical issues in attention and communication kernels that standalone kernel tests with mock inputs didn't reproduce[1].

The same pattern is coming to DeepSeek-V4.1-Flash behind the same opt-in latency_specialized decode switch, early Muse Spark Open results have passed 950 tokens per second per user, and a GLM5.3 version for low concurrency is due in a follow-up[1].

If you serve Kimi K3 on a GB300 NVL72-class system, this is a large, opt-in latency win worth benchmarking on your own prompts rather than the synthetic headline. For everyone else, the useful lesson is the process: when agents can write kernels faster than people can read them, the bottleneck moves to scoped changes and evaluations that genuinely exercise the new code path.