GPT-5.6 Sol ARC-AGI-3 score triples via API settings

The official ARC-AGI-3 evaluation wipes a model's working memory after every single move: it discards the model's private reasoning and, once the running history exceeds the context window, drops earlier actions too, forcing the model to reorient in the game environment from scratch on each turn^[2]. OpenAI's fix was two Responses API settings: Retained Reasoning, which keeps chain-of-thought persisting across turns when a developer passes along the previous response ID, and Compaction, which summarizes older context instead of truncating or discarding it^[1]. With both switched on, GPT-5.6 Sol's ARC-AGI-3 public-set score rose from an official baseline in the 7.8-13.3% range to 38.3%, while output tokens per game fell roughly sixfold^[1][2].

ARC-AGI-3 discards reasoning between moves specifically so a score reflects a model's own problem-solving, not accumulated context or harness engineering^[2]. OpenAI's settings reintroduce exactly the continuity the standardized evaluation was built to strip out, then use the resulting score to argue GPT-5.6 Sol edges past Claude Opus 5. ARC Prize co-founder Francois Chollet's response drew a distinction between an unfair 'custom-made' harness and a legitimate 'general-purpose API setting...available to all API users' - but he conditioned that acceptance on the settings and cost being clearly disclosed^[3]. That the acceptance needed a disclosure caveat at all is itself a sign the two evaluations are not measuring the same thing.

Timing matters here. GPT-5.6 Sol's officially verified ARC-AGI-3 score was 7.78% under Max reasoning effort^[4]. Only days before OpenAI's settings post, Claude Opus 5 set a new official record of 30.16% (30.2%) under High effort, verified by ARC Prize as the highest score on the benchmark to date^[5]. OpenAI's 38.3% figure, published after that record, is framed as beating Opus 5 - but Opus 5 was never run with the same retained-reasoning-plus-compaction configuration, so the comparison sets one model's custom-tuned harness against another model's standardized one^[3].

The same model weights produced scores nearly five times apart - 7.8% to 13.3% under the standard harness versus 38.3% under OpenAI's own settings - which means harness and API configuration can matter as much as which model is being tested^[1][6]. As one outlet summarized the underlying dynamic: 'Evaluations rarely test models in isolation. They are also influenced by API settings and harness design.'^[6]Any future lab claim of 'model X beats model Y' on a benchmark is worth reading as a claim about model plus harness plus settings, not the model alone.