vLLM turns DeepSeek V4.1 Flash serving into a rack-scale balancing problem

vLLM’s latest DeepSeek V4.1 Flash study is less a victory lap for one benchmark than a map of where large-scale inference loses efficiency. The team reports that a general-purpose vLLM deployment came within 6% of DeepSeek’s model-specific DeepInfer engine on decode throughput across batch sizes 8 through 256, then scaled an online serving configuration across a full NVIDIA GB300 NVL72 rack. The comparison is useful, but the authors are careful not to claim a clean prefill win because the two published setups use different hardware and configurations (vLLM).

What changed

The reported rack-scale result combines prefill/decode disaggregation with wide expert parallelism. On the SemiAnalysis AgentX workload, vLLM reports 954,000 tokens per second per GPU at a P90 rate of 196 tokens per second per user, 7.9 times its earlier single-node result. A separate low-latency decode path reaches a reported 826 P90 tokens per second for one user on four B300 GPUs (vLLM).

The engineering details matter more than the headline number. DeepSeek V4.1 Flash has a relatively small compressed KV-cache footprint, while decode remains dominated by mixture-of-experts weight movement. That led the team to use small DEP2 instances for prefill and wider expert parallelism for decode. In its measurements, DEP4 prefill delivered 83% of DEP2’s per-GPU throughput because data-parallel imbalance grew as ranks fell out of step (vLLM).

For decode, expert-parallel load balancing raised the measured balance ratio from 0.35 to 0.94 for the target model and from 0.21 to 0.89 for its draft model. A new NVLink-backed rebalance path moves expert placement across DEP32 in under a second without stopping decode, while separately tracking the target and draft models’ different routing shapes (vLLM).

Who should care

Teams evaluating disaggregated serving should read this as an architecture result, not a portable price or throughput promise. The study uses GB200, GB300 and B300 systems in different sections, and its DeepInfer prefill comparison relies on values read from a published chart whose hardware is unspecified. vLLM itself says those differences prevent an engine-to-engine prefill speedup claim (vLLM).

The operational lesson is clearer: once GPU execution becomes fast, host scheduling and the slowest rank can control the whole instance. The team found an unoptimized DEP32 configuration slower than DEP16 despite faster GPU execution, because shorter synchronized steps exposed CPU overhead and rank skew (vLLM).

What to try

Platform teams should benchmark prefill and decode separately, compare balanced and deliberately skewed replays, and measure the gap between GPU-only forward time and end-to-end serving time. vLLM’s new bench forward path and DecodeBenchConnector are designed for those two views. The practical decision is not simply how many GPUs can host the model, but which parallelism width minimizes imbalance for each serving phase (vLLM).