vLLM gives Semantic Router’s Decision 3.0 models a native serving path

vLLM has merged native support for the Decision 3.0 model family used by vLLM Semantic Router and published deployment recipes for all six checkpoints. The change turns those models into a directly served classification workload instead of requiring their checkpoint-specific runtime.

The merged integration detects the models’ decision configuration and serves them through vLLM’s /v1/systemone endpoint. The accompanying recipe collection covers D3, D3 Flash, Mini, Nano, Lite and Edge, ranging from roughly 600 million to 26 billion parameters.

Decisions without generated text

Decision 3.0 is designed for typed questions over a supplied state. A caller can ask the model to choose among options, return a yes-or-no probability, or score an input on a scale. The state can contain text or JSON and can also include images or video. Rather than generate an explanation token by token, the model returns probabilities for the available answers in one forward pass.

That makes this a routing and control-plane capability, not another chat endpoint. A platform team could use the same API shape for request classification, policy decisions or model-selection signals while choosing among six model sizes. The 27B recipe includes launch commands, hardware selectors and client examples for the System One API.

The implementation loads the published checkpoints without converting them. It maps their Qwen 3.5 backbone plus a separate answer-code readout into a vLLM sequence-classification model. The merged pull request also adds handling for the models’ bidirectional full-attention layers and their image and video limits.

The throughput gain shows up under concurrency

The project compared the 27B, 4B and 0.6B variants with their checkpoint runtime on one AMD Instinct MI325X. Across 416 questions per model, vLLM changed no decisions for D3, one for D3 Mini and two for D3 Edge; the changed cases were reported as ties or near-ties.

The more meaningful serving difference appeared under load. At 64 concurrent clients, the project measured 8.5 requests per second for D3 through vLLM versus 4.6 for the checkpoint server. D3 Mini reached 45.0 versus 18.3 requests per second, while D3 Edge reached 110 versus 41.7. Those are project-supplied results from one AMD test environment, not an independent benchmark, and the pull request says CUDA, tensor parallelism and three of the six variants were not measured in that validation.

Nightly builds are the entry point

The support landed on vLLM’s main branch on October 14. The recipes state that it is not present in the current v0.31.0 release, so deployments need a nightly build dated October 15 or later. The 27B recipe calls for at least 63 GB to load its BF16 variant and notes that reducing the maximum model length can shorten cold-start time.

For teams evaluating the feature, the practical sequence is to select a recipe that fits the available accelerator memory, install the appropriate CUDA or ROCm nightly, and test the /v1/systemone behavior against their own routing data. Production adoption should wait for normal compatibility and workload validation rather than treating the project’s single-node measurements as a capacity promise.