Flint pushes vLLM’s single-GPU decode path below two milliseconds

vLLM’s day-zero support for Recursive’s Flint is more than a model-compatibility notice. The implementation targets a particular production constraint: serving a long-context mixture-of-experts model quickly on one GPU, rather than assuming a multi-GPU deployment will absorb inefficient kernels.

The vLLM engineering post says Flint’s NVFP4 and BF16 checkpoints both run on a single GPU. On one NVIDIA GB200, the NVFP4 path serves batch-one decode at 100,000 tokens of context in 2.06 milliseconds per generated token, or about 485 tokens per second end to end.

What changed

Flint is a 40-billion-parameter mixture-of-experts model that activates about 2.5 billion parameters per token. Thirty-five of its 49 layers use a 513-token sliding attention window, while 14 layers attend across the full context. That split limits how much key-value state must span the model’s 262,000-token context window.

The first vLLM implementation still ran 1,096 kernels for each forward pass and performed poorly at small batch sizes. Inferact and Recursive replaced generic operations with Flint-specific paths: fused normalization, rotary-position and cache writes; small-batch expert and dense matrix-vector kernels; a dedicated decode-attention kernel; and programmatic dependent launches that overlap weight loading with the preceding kernel.

The result reduced the forward pass to 346 kernels. vLLM reports NVFP4 decode falling from 4.81 milliseconds to 1.78 milliseconds, BF16 decode from 3.53 milliseconds to 2.34 milliseconds, and an 8,192-token prefill from 207 milliseconds to 145 milliseconds.

Who should care

Teams evaluating sparse agentic models should note the deployment shape as much as the peak number. The optimized low-latency path applies to as many as eight requests and targets one Blackwell GPU with BF16 key-value cache. Other configurations fall back to vLLM’s standard layers, so these results should not be read as a general throughput claim for every accelerator or concurrency level.

The implementation also preserves Flint’s GLM-4.7-style tool calls and reasoning blocks through vLLM parsers. That makes the work relevant to agent-serving stacks where tool compatibility, long prompts and low request concurrency can matter more than aggregate offline throughput.

What to try

A dedicated vllm/vllm-openai:flint image packages the day-zero support for x86-64 and Arm64 systems with CUDA 13.0. The source post provides launch commands for both checkpoints and says support is being merged upstream in vLLM pull request 62938.

Operators should reproduce the result with their own prompt lengths and concurrency before treating 485 tokens per second as a capacity figure. The more durable lesson is architectural: when a sparse model’s real serving shape is small-batch and long-context, kernel count and launch behavior become first-order deployment decisions.