Red Hat turns a local AI workstation into a repeatable inference test bench

Red Hat’s Emerging Technologies team has published a practical path for turning one GPU-equipped RHEL machine into a small AI evaluation lab. The walkthrough runs a Red Hat AI Inference container, applies repeatable GuideLLM workloads and then adds Open WebUI for hands-on model checks.

The useful part is not another “run a model locally” command. It is the sequence: choose a model that fits the available VRAM, expose an OpenAI-compatible endpoint, measure it under several concurrency levels and only then judge its conversational behavior.

What the workflow assembles

The guide targets RHEL 10.2 on x86-64 with NVIDIA or AMD GPUs. Its examples pull the Red Hat AI Inference 3.5 CUDA or ROCm image from registry.redhat.io, launch the vLLM server with Podman and retain downloaded model weights in a persistent cache.

Model selection is tied explicitly to memory capacity. Red Hat provides examples ranging from a 1B Llama checkpoint for 8 GB of VRAM through larger Qwen and Gemma variants for 48 GB, 80 GB and multi-GPU systems. The demonstration uses a 35B mixture-of-experts Qwen model on one 48 GB L40S GPU, while warning that context length also consumes memory through the KV cache.

Once the endpoint responds, the workflow moves to GuideLLM rather than treating a successful curl request as evidence of readiness. Its sample profiles represent balanced chat, decode-heavy coding and prefill-heavy summarization workloads. The benchmark sweeps simulated concurrency from one to 100 streams and emits JSON plus an HTML report covering time to first token, inter-token latency and throughput.

Why platform teams should care

This is a compact pre-production decision loop. Teams can compare a candidate model and accelerator against their own latency targets before requesting cluster capacity or exposing an application to a shared service. Because the server speaks the OpenAI API, the same endpoint can feed both the benchmark and a basic chat interface.

There are important boundaries. The published run caps each benchmark stream at three minutes and recommends longer tests for more complete results. Open WebUI is launched with authentication disabled, so that command belongs on a controlled evaluation host, not an exposed production network. The Emerging Technologies site also states that its how-tos are not promises of future product support unless the post says otherwise.

What to try

Start with the smallest validated model that fits the available VRAM, then run the three supplied traffic shapes without changing hardware. Compare tail latency as concurrency rises, not just peak tokens per second. If the workload depends on long prompts or conversations, repeat the test at the intended context length: the guide’s memory recommendations assume a relatively small context window.

The result is not a production architecture. It is a reproducible way to determine whether a local RHEL system is good enough for the next experiment—or whether the workload is ready to justify a larger OpenShift-based deployment.