Red Hat AI 3.5 lets interactive and batch inference share GPUs without sharing latency

Red Hat AI 3.5 makes priority-aware request scheduling generally available for model-serving pools, giving platform teams a way to mix latency-sensitive chat traffic with lower-priority batch work on the same GPUs. The mechanism is built on llm-d flow control and leaves application endpoints unchanged; operators express priority and fairness policy in an InferenceObjective Kubernetes resource.

What changed

The flow-control layer reads caller credentials, assigns a priority and tenant, and begins queueing only after the pool reaches a configured saturation threshold. It then serves the highest-priority band first, rotates among tenants within that band, and preserves first-in, first-out ordering within each tenant queue. Below the threshold, lower-priority requests can consume spare capacity rather than leaving it reserved.

Red Hat reports that four traffic tiers sharing one NVIDIA H100 and serving gpt-oss-20b kept the 95th-percentile time to first token below half a second for its two premium tiers with the new concurrency detector: 364 milliseconds for Platinum and 430 milliseconds for Gold. In a separate batch-flood test, flow control reduced rejected requests from 48,224 across three runs to zero by queueing batch work instead of returning HTTP 429 responses. Those figures are Red Hat's internal test results, not an independent benchmark.

Who should care

The design is aimed at platform teams that currently dedicate separate GPU pools to chat, agentic, batch, or service-tier workloads. Consolidation can reclaim idle headroom when demand peaks at different times. Fair rotation also prevents one tenant's burst from consuming an entire priority band; Red Hat says two quiet tenants held near 17 milliseconds of wait while a third tenant surged to 82 requests per second.

The feature matters only under contention. A pool with uniform traffic or ample unused capacity gains little from priority-aware scheduling.

What to test first

Operators must tune the saturation threshold against their own latency target. Set it too high and work backs up inside model servers before flow control intervenes; set it too low and requests queue while GPUs still have room. Red Hat recommends increasing load until premium traffic begins missing its target, then choosing a threshold below that point.

There are two boundaries. Strict priority can postpone batch jobs indefinitely during sustained premium demand. Long generated responses also remain on a GPU after admission, so flow control cannot guarantee premium isolation once many requests are already producing thousands of tokens. Teams serving long-reasoning or agentic outputs should reproduce that case before combining pools.

Red Hat describes service-level-objective-aware scheduling as the next step: operators would state a latency target per tier instead of manually translating that target into a saturation threshold. That capability is not part of the 3.5 feature announced here.