vLLM’s strict tool mode turns schema accuracy into a serving-time decision

vLLM 0.31 exposes a server-side strictness control for tool calling, allowing operators to apply schema-constrained decoding when serving models such as Kimi and DeepSeek. The setting addresses a failure mode that model-quality benchmarks can hide: a model may choose the right tool yet emit arguments that the agent runtime cannot invoke.

Ubicloud documented the change after testing Kimi K2.6 on vLLM. Its unconstrained deployment produced tool calls matching the expected schema 92% of the time—roughly two malformed calls in every 25 attempts. Enabling constrained decoding raised schema accuracy to 100% in the K2VV ToolCall benchmark across the model versions the company serves.

What strict decoding changes

A tool declaration gives the model a function name and a JSON schema describing its parameters. Without runtime enforcement, the model samples the entire token vocabulary and may invent a field, use the wrong type or produce malformed JSON. That failed call then becomes an error in the agent’s conversation history and can trigger retries, extra cost and further mistakes.

Constrained decoding narrows the valid next tokens at each stage of generation. vLLM uses structural tags backed by XGrammar to enforce the model-specific tool-call wrapper and the supplied JSON schema. The model still chooses among allowed tokens, but tokens that would make the call syntactically invalid are masked before sampling.

This is different from validating the output after generation. Post-generation validation can identify a bad call; constrained decoding prevents the invalid structure from being generated in the first place.

Why it is not simply the default

vLLM had previously enabled constrained tool decoding more broadly, according to Ubicloud, but the project disabled that default after maintainers encountered model-specific and feature-interaction problems. Some models diverge from the supplied grammar when constrained, grammar translation can contain bugs, and interactions with speculative decoding can reject otherwise useful draft tokens or fail requests.

That leaves inference operators with a deliberate choice. vLLM’s --tool-strict-level flag lets deployments enable the behavior where it has been validated rather than imposing it across a model-agnostic server.

Ubicloud said it initially carried the behavior in a private vLLM fork, then moved to the upstream implementation once the feature landed. It enabled strict decoding for the Kimi and DeepSeek models it serves after rerunning the K2VV benchmark.

What operators should test

Teams serving agentic workloads should measure tool-call correctness independently of general model quality. A useful test suite needs to check both whether the model chooses the intended tool and whether every emitted call conforms to the active schema.

Strict decoding should then be qualified per model, tool parser and serving configuration. In particular, operators using speculative decoding should test the combined path rather than assuming that two independently working features compose safely.

The broader lesson is operational: tool-call reliability is partly an inference-engine property, not solely a model capability. For vLLM deployments, schema enforcement is now a configurable part of the serving contract—and leaving it off is itself a quality decision.