IBM and Kaggle turn ScarfBench into a living test for agentic Java migrations

IBM and Kaggle have launched a maintained public leaderboard for ScarfBench, a benchmark that tests whether AI coding agents can migrate enterprise Java applications among Jakarta EE, Quarkus and Spring while preserving their behavior. The move gives application-modernization teams a reproducible comparison surface for model-and-agent-harness combinations rather than a one-time set of vendor results.

A benchmark for working migrations

ScarfBench covers 204 migration tasks and evaluates the agentic system as a whole: the model plus the harness that lets it inspect and modify a repository. Its checks go beyond whether generated code compiles. They also ask whether the migrated application starts and continues to satisfy expected behavior, according to IBM Research's technical note.

That distinction matters for teams considering AI-assisted modernization. IBM reports that 535 of 745 classified failing runs — about 72% — were runtime behavioral failures. Compile or packaging failures accounted for 152 runs, while boot or dependency-injection failures accounted for 58. The recurring problems included lost routes, changed HTTP semantics, source-framework wiring left in a target application and packaging that did not match the target runtime.

The first leaderboard results

On the primary smoke-test metric, Gemini 4 Argon paired with Antigravity led at 75.2%. Claude Opus 5.5 with Claude Code followed at 70.5%, and Claude Opus 5 with Claude Code scored 69.8%. IBM cautions that these are model-and-harness results, not measurements of models in isolation.

The Kaggle leaderboard independently displays the benchmark and its framework scope. It lists compile, deploy and smoke-test results, links to the benchmark paper and source repository, and says the current leaderboard data was updated October 17, 2026.

The ranking also exposes a gap between building and behaving correctly. IBM says eight of the ten evaluated configurations compiled at least 97% of tasks in a sensitivity set that excluded four tasks no system solved, while smoke-test performance remained much more widely spread. No evaluated configuration fully passed more than roughly three-quarters of the migrations.

What changes for Java platform teams

IBM developed ScarfBench and Kaggle independently reproduced and ran the evaluation, sponsored the compute, and will maintain the leaderboard as new models arrive. Public model submission is not available at launch, though IBM says it is on the roadmap.

For Quarkus, Jakarta EE and Spring teams, the useful result is less the current winner than the benchmark's definition of success. A migration that compiles but drops routes or changes business behavior is not a successful modernization. By keeping behavioral tests and harness versions visible, the collaboration gives platform teams a stronger basis for evaluating agent-assisted framework moves — and a public way to see whether future systems improve where today's runs still fail.