Red Hat documents KEDA autoscaling for llm-d inference workloads

Red Hat has published new guidance for autoscaling distributed inference workloads built around llm-d on OpenShift AI. The Customer Portal article, updated Oct. 14, says LLMInferenceService workloads have two supported capacity patterns: static allocation and KEDA autoscaling. Static allocation keeps capacity fixed; KEDA is the documented route for adjusting replicas as demand changes.

Why the signal matters

The companion llm-d workload autoscaling documentation explains why conventional infrastructure metrics are a poor control signal for large-model serving. CPU and GPU utilization are lagging indicators, and GPU utilization can remain near 100% during active batching even when request pressure differs substantially.

Instead, llm-d exposes service-level signals through its Endpoint Picker, or EPP. The upstream documentation says KEDA’s Prometheus scaler can use queue depth and running-request counts to create and manage a Kubernetes Horizontal Pod Autoscaler. That lets the scaling loop react to queued and active work, then reduce replicas after demand falls. The documented KEDA path requires Prometheus but not Prometheus Adapter.

That distinction is operationally important: the autoscaler is responding to the inference service’s backlog and concurrency rather than waiting for hardware saturation to indicate trouble. It also gives platform teams a more direct way to relate a scaling decision to user-visible latency.

Who should use it

The upstream project describes the KEDA-plus-EPP path as the lower-complexity option for deployments on homogeneous hardware, where each model-server pool can be measured and scaled independently. Its current documentation also describes a separate Workload Variant Autoscaler path for cost-aware allocation across multiple model variants or heterogeneous accelerators; that broader upstream option should not be read as an OpenShift AI support statement beyond Red Hat’s published guidance.

What platform teams should test

Teams evaluating the Red Hat path should first verify that EPP metrics reach Prometheus and that KEDA owns the intended HPA. They should then test both directions of the loop: sustained queue growth must add serving capacity before latency breaches the service objective, while falling demand must remove replicas without destabilizing in-flight work. Scale-to-zero is supported in the upstream KEDA path, but cold-start behavior and model-loading time make it a workload-specific choice rather than a default.

The useful change here is not autoscaling by itself. It is the move from generic accelerator saturation to signals that describe what the inference service is actually trying to serve.