AI Agent Benchmarks, Harness Evolution, and Memory Unlearning Failures

Across this week's crop of agent papers, the dominant innovation isn't a bigger model - it's smarter scaffolding bolted onto an existing one. Harmony Harness treats the harness itself as a multi-objective optimization target, separating general-purpose mechanisms from benchmark-specific adaptations ^[1]. Harness^2 recursively refines prompts and workflows through a contrastive propose-probe-compose loop with no task-specific verifier required ^[2]. ClawLoop trains entirely inside a sandbox, synthesizing its own tasks and feedback with no external teacher model ^[3], and HarnessNet shows harness-level gains survive a full swap of the underlying foundation model - evidence the scaffolding, not the weights, is where the capability now lives ^[4]. A companion theory paper formalizes this as a distinct object of study, proving a generalization bound for persistent, feedback-driven revisions to prompts and tools while model weights stay frozen ^[5]. Independent reporting backs the scale of the effect: rewriting an agent's scaffolding alone lifted Terminal-Bench from 69.7% to 77%, and the same style of approach produced 15-50% relative gains on SWE-bench Verified, Terminal-Bench 2, GAIA, and AppWorld without retraining the model at all ^[6]. The catch, voiced independently on r/LocalLLaMA, is that almost no benchmark isolates harness effects from model effects - so a leaderboard win still can't tell you whether the model or its scaffolding deserves the credit.

The corpus' starkest negative result: when researchers built the first dedicated benchmark for memory unlearning in LLM agents, not one of six mainstream memory systems correctly honored more than 1.5% of user forget-requests end-to-end - agents either stubbornly retained what users asked them to delete, or deleted far more than intended ^[7]. A second, independently constructed benchmark extending the TOFU unlearning suite into tool-use and persistent-memory territory found the same structural gap between model-level forgetting and system-level agent unlearning ^[8]. That gap compounds a separate finding on privacy leakage: more capable agents leak more private information in multi-turn dialogue, not less, with leakage rising up to 3x between a one-turn and a six-turn conversation, and reasoning models with visible chain-of-thought leaking more than non-reasoning counterparts ^[9]. Put together, the field has spent this cycle teaching agents to accumulate semantic and episodic memory - exactly the kind IBM Technology's own explainer content argues agents can't function without - while the parallel machinery to selectively erase that memory on request remains, by these benchmarks, essentially unsolved.

Three papers converge on the same uncomfortable pattern: multi-agent failures look worse the closer you inspect them. A trivial rule - blame whichever step happened last - is already correct 67-96% of the time on typical failure logs, meaning most attribution claims compete against a weak baseline; performance collapses toward chance once failures are restricted to cases independently verifiable by replay ^[10]. BLADE responds by auditing how information is used at each handoff rather than just where it came from, beating seven rival strategies by 2.2-11.1 points of step accuracy on 1,228 failed runs without needing ground-truth answers ^[11], a framing echoed by outside research arguing attribution needs the full execution trace, not just final outputs, to get anywhere close to reliable ^[15]. A large empirical study of communication faults across roughly 29,000 executions found 20.7% of failures leave no immediate detection signal (rising to 76.2% for faults that corrupt content rather than drop it outright), and that flat peer-to-peer workflows survive these faults far better than hierarchical or sequential ones (53.0% success under fault versus 3.2-3.3%) ^[12]. Even when agents don't fail outright, they often solve the wrong problem: a benchmark built from real coding-agent transcripts found 56% of initial instructions omit necessary requirements, and frontier models lose 53-67 points of success rate once required specifications go missing from the prompt ^[13], while a replay-attack study found agents misapply correct tool results to the wrong execution context in roughly two-thirds of cross-task trials ^[14]. This is the same tension IBM Technology's own benchmark-explainer content names directly - the gap between a leaderboard score and real-world reliability.

A cluster of training papers this week attacks an identical bottleneck - sparse, delayed rewards in multi-turn agent reinforcement learning - with five unrelated mechanisms, which is a stronger signal than any one of them alone. PEARL clusters recurring decision patterns from past successes and failures to sharpen credit assignment without an extra critic model, reaching 97.9% mean success on ALFWorld at 4B scale ^[16]. PTGPO builds prefix trees over trajectory suffixes and assigns credit from the gap between a child and parent branch's empirical return, again critic-free ^[17]. VH-GPO takes a teacher-student approach instead: a teacher rescues a stuck student to a recoverable state, then hands control back so the eventual success is attributed to the student's own policy, lifting ALFWorld success from 59.38% to 94.79% on a 1.5B model ^[18]. Adaptive-Group Process Rewards splits GUI trajectories into subgoal groups and scores actions at both the action and group level to address sparse rewards in that domain specifically ^[19]. And EAU goes a level further up the stack, treating the training environment, reward design, and user simulator themselves as components to co-evolve mid-run, improving overall score by 14.7-15.0% over prior RL methods ^[20]. Together, these five independently developed mechanisms are convergent evidence that outcome-only rewards are now considered an inadequate training signal across the field.

While academia argues over how to measure agent reliability, commercial tooling is already packaging the unsolved parts as a product. CopilotKit's CLiF turns live production agent traces and user actions directly into skills, evals, and fine-tuning data ^[21], explicitly pitched at teams whose agents work in demos but struggle to execute consistent end-to-end tasks once deployed ^[22]- the same reliability gap this week's academic papers are trying to diagnose and fix from the research side. The market appetite behind that pitch is large: analyst estimates cited this cycle put only 17% of organizations as having actually deployed an AI agent so far, against a forecast that 40% of enterprise applications will ship with task-specific agents by the end of 2026 ^[23]. That gap between adoption ambition and deployment reality mirrors the gap the rest of this corpus is measuring from the research side - harnesses whose gains don't isolate cleanly, memory that can't be selectively un-taught, and failures that evade detection until they're already in production. Sakana AI's own public framing of multi-agent self-supervision as a path to recursive collective improvement, and a parallel push to orchestrate coding agents for scientific co-discovery, suggest at least one major lab is betting the fix is more autonomy layered on top, not less.