GPT-6 Astra launch and benchmark manipulation controversy

OpenAI's headline claim that GPT-6 Astra hit 99.9% on the ARC-AGI-3 benchmark looks far less clean once measured against ARC Prize's own numbers. Under OpenAI's proprietary Provider Adapter - a wrapper that preserves the model's opaque reasoning state across requests and compacts long conversations - Astra scored 99.9% at a cost of $18,817. Under ARC Prize's standard, model-agnostic harness at maximum reasoning, the same model scored 62.7% for $26,098^[1]. That is not a rounding difference; it is two different tests wearing the same benchmark's name. OpenAI's own leadership leaned into AGI language around the launch - President Greg Brockman said 'I think it's not unreasonable to feel that we are now in the AGI era'^[4]- which makes what came next pointed rather than incidental: ARC Prize itself, the benchmark's creator, explicitly declined to endorse that framing, with co-founder Mike Knoop writing that 'we lack evidence to call this AGI yet'^[1]. Co-creator Francois Chollet took a more optimistic read, arguing Astra shows genuine internal symbolic-modeling capability that previously required an external harness to unlock, and moved up his own AGI timeline forecast as a result^[2]. Both views can be true at once: the model may be genuinely stronger, but the reported number is a harness artifact, not an apples-to-apples capability score, and it remains well short of the AGI framing OpenAI's own president floated.

Internet Archive snapshots of OpenAI's own launch page show the published numbers moving during and after release, not just once. Astra's headline hallucination rate read 4.2% in the first snapshot at 2:23 p.m. ET, was revised down to 2% in a later version, then reverted to 4.2% - with the GPT-5.6 Sol comparison figure also shifting before settling back at 12.2%^[3]. GPT-5.6 Sol's ExploitBench cybersecurity score doubled mid-launch, from 5.5% to 11.5%; OpenAI later said it was investigating reverting the number because it reflects a reasoning tier not commercially available^[3]. The ARC-AGI-3 score sent to press under embargo was 98.6%, then published live at 99.99%^[3], and even a competitor's number moved: Anthropic's Fable 5.1 FrontierMath comparison score dropped nearly 10 points, from 87.8% to 78%, between snapshots^[3]. Snorkel AI's Vincent Sunn Chen offered a more charitable read, characterizing the edits as typical of final launch logistics rather than manipulation^[3], while Stanford researchers Anka Reuel and Mike Hardy argued the system card lacked adequate documentation for how internal evaluations like hallucination rate are produced, and suggested the edits conveniently favored the marketing narrative^[3].

OpenAI's launch blog post claims Astra is the company's first model to reach the 'Critical' level of cybersecurity capability under its own Preparedness Framework^[4]- the internal framework OpenAI uses to gate additional safety review before a model ships. That classification is not a marketing footnote; it is meant to be the trigger for extra scrutiny on a model's real-world risk. Yet the primary evidence behind Astra's cybersecurity story, the ExploitBench score, is exactly the kind of number this launch has already shown to be unstable: GPT-5.6 Sol's ExploitBench figure doubled between versions of OpenAI's own blog post, from 5.5% to 11.5%, with OpenAI attributing the jump to a reasoning tier not commercially available^[3]. When the underlying benchmark feeding a safety-relevant capability threshold moves that much within the same launch cycle, the 'Critical' designation built on top of it inherits the same credibility problem as every other disputed figure in this launch: it is difficult to know whether the classification would have held at the benchmark's earlier, lower number, or whether it was set against the version that happened to clear the bar.

Set against third-party testing, Astra's story looks far less dramatic than OpenAI's own charts suggest. Artificial Analysis's Intelligence Index puts Astra at 61.2, essentially tied with its predecessor GPT-5.6 Sol at 60.9, and behind Anthropic's Claude Fable 5.1 at 65.7^[5]- a picture hard to reconcile with a launch pitched as a generational leap. On hallucination specifically, Artificial Analysis's own testing found Astra hallucinating in 51% of cases at maximum effort, down from 92% for Sol - a real improvement, but nowhere near the sub-5% figures OpenAI published^[6]. Independent measurement and OpenAI's own published charts are telling two different stories about the same launch, and only one of them is reproducible outside OpenAI's own infrastructure.

GPT-6 Astra's benchmark dispute is not an isolated incident for OpenAI. In late 2024, its o3 model's claimed 25% score on the FrontierMath benchmark could not be reproduced independently, with Epoch AI's own testing landing closer to 10%^[7]. It later emerged that OpenAI had secretly funded and had advance access to the FrontierMath dataset before that launch, without disclosing the arrangement to the academic contributors involved^[8]. More recently, GPT-5.5 drew criticism when users discovered its premium Extended Thinking mode was being silently downgraded for many queries^[9]. Reaction to Astra split hard along platform lines once the pattern became visible. On X, an early hype-driven tweet touting the raw headline numbers (near-perfect ARC-AGI-3, ExploitBench, and SRE Bench scores) gave way within about a day to a more skeptical mood, as commentators amplified reporting on the quietly revised metrics. Reddit threads devoted extensive debate to what commenters called 'benchmaxxing' - whether OpenAI's proprietary-harness ARC-AGI-3 score or ARC Prize's neutral-harness score is the fairer measure of real-world agent capability - with accusations of misleading marketing sitting alongside defenders of harness-augmented scoring as a legitimate capability signal. YouTube's coverage landed calmer: analysts offered measured takes defending the benchmarks' underlying rigor while acknowledging the real hallucination gains were more modest than headline figures implied, and floated architectural theories for the score jumps - a tone distinct from the hype-then-backlash swing on X and Reddit. Even OpenAI's rollout mechanics fed the skepticism - the launch post was delayed almost two hours past its scheduled time, and the company later apologized and offered compensation to paid subscribers who lacked day-one access^[10].