vLLM pushes KV cache off the decode critical path

Disaggregated model serving splits prompt processing from token generation, but the handoff has a cost: the prefill instance must move its key-value cache to the decode instance before generation can start. vLLM’s default NIXL flow has decode discover and pull those blocks after prefill completes, putting allocation, metadata relay and transfer setup directly on the path to the first token.

AWS has contributed a push-based NIXL connector to vLLM. The design lets decode register its destination blocks while prefill is still computing, then has prefill write the finished cache into those blocks from a background thread. The result is a different scheduling tradeoff rather than simply a faster copy.

What changed

The push protocol has three stages. Decode allocates and registers destination blocks before prefill completes. Prefill matches that registration with the request when its cache is ready. A background worker then prepares descriptors and issues a NIXL write directly into decode memory, with completion notifications telling decode when generation can begin.

AWS says the implementation can map different physical block layouts on each side and accounts for retried or preempted requests when matching registrations. Moving descriptor preparation and transfer submission away from decode’s latency-sensitive engine loop is intended to prevent cache-transfer work from starving token generation.

What the measurements show

AWS tested Qwen3-32B with one prefill and one decode node on EC2 P5en instances connected through Elastic Fabric Adapter. With prefix caching disabled to force a full transfer, the company reports that push mode reached roughly 540 milliseconds time to first token at eight queries per second and a 16,000-token input, versus about 1,744 milliseconds for pull mode. It also reports time per output token near 7 milliseconds in push mode, compared with about 10.4 milliseconds for pull mode at the heaviest tested combinations.

That is up to a threefold TTFT improvement and about a 30% TPOT improvement in AWS’s configuration, not a general guarantee. At low load and short prompts, the two modes were close. The reported gap widened as request rate and context length increased because pull-mode setup continued to run inline on the decode worker.

The routing tradeoff

Push mode must select the decode instance early enough to register its destination memory before prefill finishes. Pull mode can defer that choice, preserving more freedom for load-aware routing and quality-of-service decisions. Pull also remains necessary for vLLM’s new bidirectional KV flow, which returns decode-side state to prefill across conversational turns.

Operators can enable the new path by configuring both sides with NixlPushConnector instead of NixlConnector. The decision should follow workload measurements: push is aimed at high request rates where early decode selection is acceptable, while pull remains the more flexible option when routing decisions or reverse-direction cache reuse matter more.