vLLM sends KV cache back to prefill nodes for faster multi-turn agents

Multi-turn agents repeatedly resend their growing histories: system instructions, earlier prompts, tool results and generated answers. In a disaggregated inference deployment, that can force the prefill side to rebuild key-value cache entries it no longer holds—even though the decode side computed and retained much of the same state during the previous turn.

AWS says it contributed an accelerator-agnostic bidirectional KV-transfer capability to vLLM. The change allows a decode node to return reusable KV blocks to a prefill node on the next turn. For platform teams building long-running assistants, the practical promise is lower time to first token without paying to recompute the whole conversation prefix.

What changed

Conventional prefill-decode disaggregation moves KV state from prefill to decode. The new path adds the reverse direction through vLLM’s NIXL connector. A KV-cache-aware router associates decode-side block metadata with a client-provided conversation ID, then supplies that metadata when the next turn reaches prefill. AWS says the router stores block locations rather than the KV tensors themselves.

The implementation adds two controls that matter operationally. Decode blocks are retained for a configurable lifetime—480 seconds by default—so late turns fall back to normal prefill after expiry. A recompute threshold, defaulting to 64 tokens, avoids paying transfer and coordination costs when rebuilding a short prefix would be cheaper.

What the measurements show

AWS tested one prefill and one decode node on EC2 P5en instances connected with Elastic Fabric Adapter, while disabling local prefix caching on prefill to emulate eviction or routing to a different node. At the longest prompts tested, the company reports time-to-first-token reductions of about 1.6 times for Qwen3-32B and 3.3 times for GLM-5-FP8.

Those are workload-specific measurements, not a universal speedup. They nevertheless identify where the mechanism should pay: larger models, longer histories and deployments where reusable context is likely to live on decode rather than the next selected prefill node. AWS also reports that the transfer itself added tens of milliseconds, but removed substantially more prefill computation in its tests.

What platform teams should do

The capability is available in vLLM 0.21.0 or later with NixlConnector. AWS says operators must enable bidirectional_kv_xfer on both prefill and decode servers and use a KV-cache-aware router; vLLM’s repository includes an example proxy for the coordination flow.

Before adopting it, teams should benchmark realistic turn intervals and context growth. The retention lifetime trades accelerator memory for reuse, while the recompute threshold depends on model cost, KV size and interconnect performance. The important architectural shift is that multi-turn state is no longer assumed to move only toward decode: vLLM can now treat decode-side cache as reusable input to the next prefill phase.