RH SWE-bench tests coding agents against the repositories enterprise teams actually maintain

Red Hat’s Emerging Technologies team has released RH SWE-bench, an open benchmark designed to test coding agents on the languages, build systems and repositories common in enterprise infrastructure work. The revised v1.1 dataset contains 337 tasks across 24 repositories in Go, Python and C.

That matters because the familiar SWE-bench family is still dominated by Python. RH SWE-bench instead reaches into projects including Podman, Buildah, etcd, kubectl, KubeVirt, Ansible, Molecule, osbuild and Performance Co-Pilot. It asks whether an agent can navigate multi-module Go projects, custom C build systems and repository-specific test conventions—not merely whether it can patch a popular Python package.

The benchmark is also testing the harness

Each task places an agent in a container at a specific repository commit. The agent receives a problem description and must produce a change that makes fail-to-pass tests succeed while preserving existing pass-to-pass tests. Red Hat says it used OpenShift clusters, Docker-in-Docker isolation, prebuilt images and the Harbor evaluation framework to run the workload at scale.

The published results show why the execution harness deserves as much scrutiny as the model. On the original v1.0 task set, Nemotron-120B scored 30.8% with OpenCode but 21.6% with Pi. Qwen3.6-27B-FP8 ranged from 44.0% with OpenCode to 49.3% with Claude Code. A harness that assumes every container uses apt-get, for example, fails outright on Alpine-based PCP tasks.

This is useful operational evidence: an enterprise coding-agent evaluation is partly a model test, but it is also a test of environment detection, timeout policy, package management and test orchestration.

Validation is the real product

The team began with roughly 6,800 generated candidates. Only about 7% survived initial validation, and additional quality filtering removed trivial or unreliable tasks. Its strongest pipeline builds an image at a merged pull request’s base commit, applies only the new tests to reproduce the bug, and then applies the code change to confirm the fix. Red Hat reports that this forward-apply method approached a 100% validation rate for real pull-request tasks, after reverse-patch approaches largely failed.

The benchmark’s limitations are explicit. Version 1.1 reduced the dataset from 357 to 337 tasks after removing flaky or privileged checks and revising prompts, but the full model table has not yet been rerun against those revised tasks. The current numbers should therefore be read as evidence about the benchmark’s shape, not a definitive v1.1 leaderboard.

What teams can do now

Platform and developer-productivity teams can use RH SWE-bench to compare model-and-harness combinations against infrastructure code closer to their own estates. The dataset is available through GitHub, Hugging Face and Harbor Hub, while Red Hat says Java and Rust coverage is planned next. The immediate lesson is simple: choose coding agents with the runtime and repository environment included in the evaluation, rather than treating model scores as portable across toolchains.