New AI Agent Evaluation Benchmarks Reveal Where Agents Still Fail

In late September and early October 2026, four new agent benchmarks and one evaluation-methodology paper arrived within days of each other, joining SRE-Bench - an earlier 2026 reverse-engineering benchmark still being actively tested and discussed - to form a cluster of narrow, domain-specific agent evaluations. CompMat-Bench tests whether agents can execute real steps from published computational materials science studies without needing to rerun expensive simulations ^[1]. EurekaBench pushes agents through 26 long-horizon scientific-discovery tasks spanning neuroscience, geophysics, astrophysics, computer science, plasma physics, and chemistry, tied to 306 target insights ^[2]. Company Knowledge Bench, built by Kapa.ai from 1,000 real enterprise retrieval cases, measures how well agents find exactly the right internal documentation without drowning the context window in noise ^[3]. KaliBench grades 8,504 natural-language-to-command translations across 1,642 Kali Linux tools and 23 capability dimensions ^[4], with code and data published openly ^[5]. And SRE-Bench forces agents to reverse-engineer compiled binaries from 19 private programs built from scratch specifically to avoid training-data contamination ^[6]. The common thread is a shift away from generic coding and chat leaderboards toward narrow, expert-curated domains where correctness is harder to fake.

Every benchmark in this cluster reproduces the same shape of result. On CompMat-Bench, agents score a respectable 66.0-90.4% pass rate on single tasks under full methodological guidance, but performance drops as workflows lengthen and guidance is withdrawn, with most failures traced to scientific reasoning errors rather than software mistakes ^[1]. On EurekaBench, agents already exceed human scientists at predictive accuracy but fall substantially short at deriving the scientific insight the task is built around ^[2]. On Company Knowledge Bench, a naive 'agentic grep' baseline pulls roughly four times the tokens of a fixed retrieval pipeline while only needing about one in every twenty-five chunks it returns, showing that being agentic alone does not guarantee efficiency; the benchmark's best-performing system was instead a separately engineered agentic retriever, which achieved the top accuracy score ^[3]. On KaliBench, no open-weight model clears 42% exact-command accuracy once tool hints are removed ^[4]. And on SRE-Bench, even frontier models that excel at source-level vulnerability research - the skill reverse engineering is supposed to build on - post full-solve rates of only 26.9-31.5% ^[6]. Across five very different domains, the pattern holds: narrow, well-scaffolded steps are solvable; open-ended, reduced-guidance reasoning is not.

SRE-Bench's creators built it specifically to be contamination-free, using over 5,000 expert-hours of clean-room program development so no binary could already be baked into a model's training data ^[6]. Despite that design, a related leaderboard reported a frontier model reaching a near-perfect pass@4 score on the benchmark with no budget or safety constraints ^[7], even though the paper's own controlled evaluation put the best full-solve rate at just 31.5% ^[6]. Community discussion elsewhere noted that the same top-scoring model separately performed far worse when asked to reproduce a program from documentation rather than explain what an existing binary does, prompting some observers to argue that knowing what code does and reproducing it look like almost separate skills, and that a single headline score under loosened conditions can overstate general capability. That split mirrors a broader skepticism visible across discussion of agent benchmarks generally: practitioners point to prior cases where simplistic or idle baseline agents have outscored genuinely capable models on tool-use leaderboards, and where fixing grading bugs in established coding benchmarks reshuffled a meaningful share of the rankings - evidence that a single aggregate score can reflect benchmark artifacts as much as real capability.

The most consequential paper in this batch may not be a benchmark at all. 'Agents Are Systems, Not Models' argues that scoring an LLM-based agent as if it were a single fixed model obscures what is actually driving performance: the task information it receives, whether it has a dedicated self-verification tool versus just being prompted to check its own work, the time budget it is given, and the backbone model itself ^[8]. The paper's starkest finding is that about 54% of outcome variance in its released 18,000-plus agent trajectories comes from simply re-running the identical configuration, not from changing the model or the task ^[8]. It also shows these levers interact rather than add up: extra time budget only helps an agent that already has adequate task information or a sufficiently capable backbone model ^[8]. That framing lines up with what practitioners trying to benchmark coding agents have been converging on independently - freezing the harness and swapping only the model, logging every tool call, and scoring against an evaluator the agent never saw - precisely so that harness and configuration differences stop masquerading as model capability differences.