IBM makes PyTorch the portability layer from Spyre to llm-d

IBM Research says it has made the Spyre inference accelerator a first-class PyTorch backend, pairing torch-spyre with a vLLM hardware plug-in and the multi-accelerator scheduling path in llm-d. The work matters to Red Hat’s AI stack because it tests whether an increasingly GPU-optimized serving ecosystem can keep a portable route for a different execution architecture.

Spyre is a dataflow accelerator for inference on IBM Z, LinuxONE and Power systems. Rather than ask application teams to adopt a separate model catalog and toolchain, IBM is putting PyTorch between familiar model code and the hardware.

The backend gets smaller

torch-spyre compiles ordinary PyTorch code into an executable plan for Spyre. Unsupported or inefficient operations can be replaced by adapters with mathematically equivalent forms that the compiler can lower. IBM also moved the backend to PyTorch Inductor, reducing its custom compiler code by roughly sevenfold, according to the company.

That narrower hardware-specific layer should make upstream improvements easier to absorb. IBM reports that moving graph construction out of the data-transfer path improved Granite 3.3 8B prefill by 9.9% and decode by 13.6% at batch size one and sequence length 1,024. Removing the work from compute dispatch as well raised the total improvement to 1.7 times for prefill and 2.4 times for decode in its demonstration.

vLLM provides the extension seam

Spyre depends on TorchDynamo for graph tracing and TorchInductor for lowering. In vLLM, the plug-in uses extension mechanisms including CustomOp and PluggableLayer, avoiding a wholesale fork of the serving engine.

That extension seam is under pressure as model implementations specialize around frontier GPU hardware. IBM says hardware-agnostic layers now landing in vLLM have been validated through the Spyre plug-in with Gemma 4, Qwen3 and Granite 4.2. On NVIDIA H100 GPUs, those layers produced total token throughput within 3.4% of native implementations across three recent models, the company reports.

The point is not that every path is equally fast. It is that the portable layer can remain close enough to native code that accelerator support does not require a separate model implementation.

llm-d inherits the device path

The architecture then composes upward. torch-spyre exposes the device through PyTorch, the Spyre plug-in exposes it to vLLM, and llm-d can schedule distributed inference onto the resulting backend. IBM describes that chain as the same engineering work serving two goals: making Spyre a PyTorch device and making it usable in llm-d deployments.

IBM, Red Hat and other PyTorch Foundation collaborators are also adding end-to-end model tests, continuous regression checks and release testing. Those checks are essential for an out-of-tree accelerator because a framework or serving-engine change can break downstream hardware support without failing isolated operator tests.

For platform teams, the practical question is now compatibility evidence: which model, backend, plug-in and accelerator combinations have been exercised together. The backend architecture makes hardware choice possible; the test matrix will determine whether it is operationally credible.