OpenAI o1 outperforms ER physicians in diagnosis

The headline number — 67.1% diagnostic accuracy versus 55.3% and 50.0% for human attendings — is striking, but the methodological choice underneath it is the more consequential finding. Unlike previous medical AI evaluations, the Harvard team did not curate, summarize, or vignette-ify the cases. Each ER patient was handed to o1-preview exactly as the chart appeared in the electronic health record: vital signs, intake nurse notes, free-text fragments, and all. That is the closest thing the literature has produced to an in-the-wild physician comparison.

The second mechanical pillar is the model itself. o1-preview was OpenAI's first 'step-by-step reasoning' system, and the authors attribute the leap over GPT-4o — its non-reasoning predecessor — to that architectural change. On a separate 70-case NEJM head-to-head, o1-series scored ~89% exact-or-very-close versus ~73% for GPT-4. In other words, the jump in real ER triage tracks the jump on hard published cases, suggesting it isn't an artifact of the 76-case sample. The story is: a reasoning model, fed raw clinical text, beat physicians who had access to the same record. That is a different claim than 'AI passes the USMLE,' which the field has heard for two years.

The comparator group was not ER specialists. It was two attending internal medicine physicians being asked to make ER triage calls, and emergency physician Kristen Panthagani has been pointed about why that matters: 'If we're going to compare AI tools to physicians' clinical ability, we should start by comparing to physicians who actually practice that specialty.' Her deeper objection is that ER triage isn't really a guess-the-diagnosis task — it's a rule-out-lethal-conditions task, with very different success criteria than the study's exact-match scoring.

Reddit's medical commenters surfaced a parallel critique: doctors in real triage have information that isn't in the chart — the patient's color, breathing pattern, demeanor — and the study only handed the model text. The top community read across r/EverythingScience and r/Futurology was that the model was effectively producing a 'second opinion based on paperwork,' not replacing a clinician at the bedside. Even the senior authors agree the text-only modality is a real limitation, since real ER care also depends on imaging, exam findings, and patient interaction. The result is best read as a ceiling on what unprocessed-text reasoning can do, not as a verdict on ER medicine.

The most uncomfortable thread running through both X and Reddit reactions is structural: a hospital can now have a system that demonstrably catches more diagnoses, and its general counsel will tell it not to use that system. There is no formal accountability framework for an AI-influenced diagnosis that turns out wrong — no clear malpractice standard, no FDA pathway tuned for general-purpose LLM-as-second-opinion, no precedent for who pays when the model is right and the doctor overrules it (or vice versa). One widely shared X thread distilled the asymmetry to a sentence: an AI that is right 67% of the time gets called dangerous, while a doctor who is right 55% of the time gets called board-certified.

This is why the authors' insistence on randomized controlled trials matters beyond academic hygiene. The first health system that figures out how to deploy an LLM overlay — passively flagging differential diagnoses against the EHR, with a clear decision-rights and indemnity model — captures a generational advantage. Yujin Potter at Berkeley reframes it as an AI safety problem rather than a regulatory one: outperforming doctors on aggregate doesn't tell you what the model does on the rare adversarial case, and the study didn't formally measure hallucination rates. Arya Rao's warning is the same in clinical language: LLM 'reasoning' is brittle precisely where uncertainty and nuance matter most. The deployment question is therefore less 'is the model good enough' and more 'what failure mode does the institution agree to own.'

o1-preview shipped in September 2024. The Harvard paper landed in Science in late April 2026 — roughly nineteen months later, multiple frontier model generations downstream. The implicit question that drove the loudest social response, including from The Rundown AI's audience, was: what do the numbers look like for systems that aren't already two years behind? The study isn't a measurement of frontier capability; it's a measurement of a generation-old system that already exceeded the comparator.

That reframing connects to Thomas Buckley's other observation: multiple-choice medical exams are saturated near 100%, so the field can't use them to track progress anymore. The real research frontier is moving toward unprocessed-EHR evaluation of the kind this paper pioneered — harder, messier, closer to deployment reality, and resistant to the kind of memorization that inflated earlier benchmark scores. Combined with parallel results from Microsoft's MAI-DxO orchestrator (better diagnoses at ~20% lower cost) and the Elsevier survey finding ~20% of clinicians already quietly using LLMs for second opinions, the Harvard study reads less as a single startling result and more as confirmation that the bottleneck has migrated. It's no longer can the model do it. It's whether the legal, regulatory, and clinical workflow infrastructure can catch up before the technology overtakes it twice over.