vLLM makes million-token GLM-5.3 serving a two-node memory architecture

vLLM engineers have assembled a 16-GPU serving architecture for GLM-5.3 that can carry a one-million-token context through both prefill and decode. The design combines prefill context parallelism, decode context parallelism, host-memory-backed sparse attention and prefill/decode disaggregation across two eight-GPU NVIDIA H200 nodes.

The useful result is architectural rather than a single kernel speedup: standard tensor or data parallelism cannot fit the model’s one-million-token key-value cache in the tested configuration. The combined design can.

Split queries and cache separately

Prefill context parallelism, or PCP, divides a prompt’s query tokens across eight GPUs. That avoids repeating GLM-5.3’s sparse-attention indexer on every GPU and lets each key-value row serve all 64 attention heads. The team says that changes the indexer’s crossover with sparse attention from roughly 32,000 tokens under eight-way tensor parallelism to about 256,000 tokens.

PCP still leaves a full copy of the cache on every GPU. Decode context parallelism, or DCP, solves the capacity problem by distributing cache tokens round-robin. vLLM calculates that a one-million-token GLM-5.3 request needs 53.4 GiB of FP8 cache; eight-way DCP reduces persistent cache per GPU to 6.7 GiB.

That capacity is not free. The prefill engine must gather the distributed cache for every layer. vLLM reports roughly 6% overhead up to 64,000 cached tokens, rising to 15% at 248,000. Beyond that range, however, the unsharded configuration stops fitting at all.

Two nodes, two memory strategies

The deployment assigns one H200 node to PCP8-by-DCP8 prefill and the other to eight decode ranks. An llm-d router directs requests, while NIXL moves cache state between the engines. Decode uses Hybrid HiSparse so cache can spill into host memory rather than ending when high-bandwidth GPU memory fills.

On the project’s AgentX workload, the combined prefill configuration delivered 1.7× to 3.2× higher per-user speed than data-parallel prefill at the same throughput. At higher concurrency, HiSparse nearly doubled throughput in the reported test. Only the full combination served the one-million-token case.

What platform teams should test

The setup requires vLLM 0.32.0 or later and depends on fast interconnects, substantial host memory and a disaggregated routing layer. Operators should reproduce the results with their own tool-call traces and cache-hit patterns, because long-context performance turns on where time is spent: repeated prefill, cache transfer or decode.

The larger lesson is that million-token serving is no longer just a model-capability checkbox. It is a placement and memory-topology problem spanning the router, both GPU nodes and the host-memory tier.