AI coding agent benchmarks and frameworks for software engineering

SWE-bench Verified's pass rate climbed from roughly 13% in early 2024 to about 78% by May 2026, a trajectory that looks like progress until you notice it coincided with OpenAI's Frontier Evals team quietly stopping its public reporting of the metric in February 2026, after an internal audit found that many of the hardest remaining problems had flawed or literally unsolvable test cases ^[3]. That saturation-and-leakage problem is exactly what SWE-Bench Pro V2 was built to fix: Scale AI cut the task set from 731 to 642 after finding 89 invalid tasks, added a network-locked evaluation protocol, and sanitized git history specifically to block reward hacking ^[1][2].

The new benchmark's own leaderboard supplies the sharpest evidence that the old gaming problem was real. Claude Opus 5 scores 99.4% pass@1 on the public split of SWE-Bench Pro V2, but that collapses to roughly 82% (222 of 272 tasks) on a held-out private split, a nearly 18-point gap that Scale attributes to training-time exposure to public repositories rather than any evaluation-time leak ^[1]. In other words, a model's headline benchmark score can overstate its real capability by a wide margin simply because the benchmark's public tasks look enough like its training data. FrontierRefactor applies a similarly contract-verified evaluation design to a different task domain, refactoring rather than bug-fixing ^[8].

Running a model through a full agentic benchmark suite is not cheap: a single evaluation run can cost thousands of dollars and take days to complete, a cost structure that limits how often labs can run a full evaluation at all ^[4]. PACE (Proxy for Agentic Capability Evaluation) attacks this by predicting a model's score on expensive benchmarks like SWE-Bench and GAIA from a small, carefully chosen subset of cheap, non-agentic evaluations, hitting under 4% mean absolute error, a Spearman correlation above 0.80, and about 85% pairwise ranking accuracy, all for under 1% of the cost of a full agentic run, validated across 14 models and 23 benchmarks ^[4].

HSPR (Hierarchical Score Posterior Recovery) tackles the same cost problem from a different angle: instead of substituting cheaper non-agentic tasks, it statistically recovers an accurate score from a small subset of the real agentic tasks themselves, combining source-fitted response factors with uncertain group success rates, and it outperforms baseline estimators on mean absolute error across the SWE-bench Lite, Full, and Pro task banks at just a 10% task budget ^[5]. That two groups converged on cost-reduction methods in the same window suggests the field has quietly accepted that full-scale agentic evaluation, as currently built, cannot scale with the pace of model releases.

A cluster of 2026 research converges on one uncomfortable point: agents can clear a benchmark's pass/fail bar while the underlying work is still broken, risky, or useless in practice. AgInt tries to close that gap directly by having a model propose typed evidence records about code behavior that a deterministic harness then validates and grades, producing an auditable certificate rather than a free-form trajectory; on SWE-bench-Live, this lifts reject-class F1 from 0.256 to 0.543 while cutting cost per candidate by 59% ^[6]. The motivation is concrete: a large-scale empirical study of 12,749 real GitHub commits found that AI-generated code carries nearly double the density of high-risk security alerts versus human-written code (0.934 vs 0.464 per thousand lines, worse still in JavaScript) and takes significantly longer to stabilize after merge, 28.28 days versus 11.23, despite being smaller and more localized ^[7].

The same pattern shows up at the benchmark level. On FrontierRefactor, 41.2% of agent submissions are functionally correct but deliver no verified performance improvement, and 19.3% actively regress performance outright ^[8]. CEGame-Bench finds an even starker version of the same gap in a different domain: agents reach a true, playable ending in only 16% of runs ^[9]. Across refactoring, game creation, and raw code generation, the throughline is the same: current coding agents are better at producing artifacts that look finished than at producing ones that actually hold up.

JetBrains' Mellum 2.1, released October 8, 2026, makes the benchmark-overfitting question concrete rather than abstract. The architecture is unchanged from the original Mellum2, a 12-billion-parameter mixture-of-experts model with 2.5 billion active parameters and a 131K context window, but reinforcement-learning training on real repositories pushed its SWE-bench Verified score from 2.0 to 47.0, its SWE-bench Pro score from 0.0 to 28.0, and its LiveCodeBench v6 score from 69.4 to 82.0 ^[10]. JetBrains frames it explicitly as an efficient 'worker' model meant to run privately inside IDE agent workflows rather than compete head-on with frontier planning models ^[11].

Community reaction split along a familiar fault line. Enthusiasm centered on raw throughput and the model's suitability for local, privacy-preserving execution on consumer hardware, consistent with JetBrains' own efficiency pitch. But hands-on testers pushed back hard: some reported the model underperforming a year-old mid-size open model on coding tasks without matching benchmark fanfare to justify it, others described it failing basic project scaffolding or shipping half-finished code with broken tests when used as a sub-agent, and an early broken chat template reportedly caused faulty tool-calling before a fix landed. That tension, official benchmark numbers moving sharply upward while some practical use cases stumble, is the same dynamic SWE-Bench Pro V2 surfaced at the frontier-model level, just playing out in miniature around an open, self-hostable release. It sits alongside other 2026 infrastructure plays aimed at making agentic coding more trustworthy or more coordinated, including Arctic-SWE's asynchronous reinforcement-learning framework for training long-horizon coding agents ^[12]and SandboxAQ's Switch, which coordinates agents across rival frameworks in shared chat workspaces ^[13]. Zooming out to market-wide testing, one survey of six coding agents found a combined functional-and-secure code pass rate topping out at just 25.1%, underscoring that the benchmark-versus-reality gap is an industry-wide pattern, not a Mellum-specific one ^[14].