DRA turns GPU–NIC topology into an inference scheduling constraint

Red Hat engineers have published a concrete case for making PCIe topology part of Kubernetes scheduling for distributed inference. Their point is operationally important: a pod can receive both a GPU and an RDMA-capable network interface, start successfully and still inherit a hardware path that cuts transfer performance roughly in half.

The proposed answer uses Kubernetes Dynamic Resource Allocation (DRA) to coordinate devices that traditional plugins allocate independently. In Red Hat’s example, the NVIDIA GPU DRA driver and dra-driver-sriov both publish a standard resource.kubernetes.io/pcieRoot attribute. A ResourceClaimTemplate then uses matchAttribute to require the selected GPU and SR-IOV virtual function to share that PCIe root.

What the measurements show

The team first tested GPUDirect RDMA writes over a nominal 200 Gbit/s InfiniBand link on A100 systems. An aligned GPU and NIC sustained 194.6 Gbit/s in the reported forward direction. A same-NUMA pairing behind different PCIe roots reached 127.1 Gbit/s, while a cross-NUMA pairing reached 110.0 Gbit/s. The preserved results include ten independent runs per case, topology validation and the exact placement details.

The inference test is the more consequential result. Using llm-d on Red Hat OpenShift AI 3.5, the engineers deployed separate prefill and decode pods for Qwen3.8-27B on different A100 nodes. Across 384 successful KV-cache transfers per placement, aligned devices averaged 56.8 ms and 20,709.7 MB/s. Cross-NUMA placement averaged 122.3 ms and 9,612.6 MB/s. That is 2.15 times the effective throughput and 53.6% lower mean transfer latency for the aligned case.

Those ratios are not universal performance promises. The authors explicitly scope them to this model, A100/ConnectX-6 hardware and server topology. The useful result is the failure mode: NUMA awareness alone did not prevent a costly GPU–NIC pairing.

Why DRA changes the failure mode

Traditional device plugins expose opaque resource counts, so GPU and network plugins cannot negotiate a shared PCIe path. DRA exposes individual devices and structured attributes to the scheduler. If no node has a matching GPU and virtual function, the claim remains pending instead of silently producing a runnable but poorly placed pod.

Using an SR-IOV virtual function also leaves the physical network function in the host namespace, preserving host telemetry and administrative control while still giving the workload a GPUDirect-capable path.

What platform teams should do

This is engineering guidance, not a declaration that the complete cross-driver stack is already a supported OpenShift feature. Red Hat says it is working to bring these DRA capabilities and driver ecosystems into OpenShift.

Teams evaluating disaggregated prefill/decode or multi-node expert parallelism should nevertheless inventory PCIe topology now, retain topology evidence with benchmark results and avoid treating “same NUMA node” as proof of an optimal path. The sample manifests and raw results provide a useful validation pattern: compare aligned and deliberately misaligned allocations before attributing inference latency solely to the model server or network fabric.