Mistral Large 4 lands in vLLM with serving-stack improvements beyond one model

vLLM has shipped day-zero support for Mistral Large 4, but the integration reaches beyond adding another model architecture. According to the vLLM engineering account, the work produced reusable changes across context parallelism, speculative decoding, cross-layer KV sharing, disaggregated serving and host-memory KV offload.

More than a model definition

Mistral Large 4 is a one-trillion-parameter mixture-of-experts model with 49 billion parameters active per token, plus embeddings and output layers. It combines sparse multi-head latent attention with a top-k indexer, cross-layer KV sharing, NoPE layers and a Pixtral vision encoder. vLLM supports FP8 and NVFP4 checkpoints and provides a dedicated vllm/vllm-openai:mistral-large-4 container image.

The implementation runs on vLLM’s Model Runner V2 and introduces model-agnostic support for attention layers that consume another layer’s KV cache rather than maintaining their own. Nine layers in Mistral Large 4 reuse layer 35’s latent cache, leaving 63 of the model’s 72 layers as cache owners. The vLLM team says that mechanism is being upstreamed separately rather than remaining tied to this model.

Long-context and speculative-serving work

The integration also extends decode context parallelism, or DCP, which shards the latent KV cache by token. On eight GB300 GPUs, vLLM reports that DCP increased FP8 KV capacity from 2.38 million to 18.88 million tokens. Changes to collective operations and fused kernels reduced DCP-specific overhead, while prefill context parallelism can now be combined with DCP and CUDA graphs for long prompts.

Mistral Large 4 includes a built-in DSpark speculative-decoding drafter. In vLLM’s TP8 tests, the drafter accepted about 2.9 tokens per step and raised output throughput roughly twofold. The work adds support for the drafter’s sliding-window attention under DCP and carries both target-model and draft-model state across prefill/decode boundaries.

For disaggregated deployments, the implementation supports NIXL-based transfer between prefill and decode nodes across several parallelism layouts. The published tests cover one prefill and one decode node on four GPUs each, with the team reporting no failed transfers or requests in those runs.

Hardware scope and tradeoffs

The release targets NVIDIA Blackwell systems and AMD Instinct MI350-series GPUs. On AMD, vLLM supports MXFP4 experts, image inputs, expert parallelism and DSpark through its ROCm path. The team also describes a specialized persistent decode kernel for low-concurrency workloads.

The engineering account is unusually explicit about tradeoffs: DCP increases KV capacity but can slow each decode step when cache pressure is not the bottleneck, and speculative decoding consumes some of that capacity. That makes the release most relevant to operators planning long-lived, long-context sessions rather than a blanket speed upgrade for every deployment.