OpenAI's o1 outperforms ER doctors in Harvard Science study on real emergency cases

Most coverage led with the headline triage number — 67.1% for o1-preview versus 55.3% and 50.0% for two attending physicians on 76 Beth Israel Deaconess cases. That gap is real but modest, and with richer clinical detail the spread compressed to 82% for o1 versus 70-79% for physicians, a difference the authors note was not statistically significant. The astonishing result lives one decision point deeper: on management reasoning using expert-scored clinical vignettes, o1 scored a median of 89% while 46 physicians with conventional resources scored 34%. As an emergency physician noted in published commentary on the result, 'That is not a typo.'

Management reasoning is the chain of choices after diagnosis — what to image, what to admit, which therapy to start, what to monitor. It is where attending physicians earn their salary, and it is where benchmark-style multiple-choice tests have historically failed to discriminate between strong and weak clinicians. The Science paper also reports that o1 received a perfect clinical-reasoning score on 98% of cases versus 35% for attending physicians. Whatever the eventual deployment story, the implication for medical AI evaluation is structural: the locus of automation pressure is shifting from 'name the disease' to 'plan the next 24 hours of care,' and that is the dimension where current attendings look least defensible on paper.

An emergency physician writing about the result captured the shift bluntly: 'This was published in Science. Not a preprint. Not a company blog post. Peer-reviewed, in one of the two most prestigious scientific journals in the world.' Until April 30, 2026, the strongest AI-vs-physician numbers came either from journal pieces using clean vignettes (the JAMA Network Open GPT-4 study at 90% versus 74-76%), from AI lab releases (Microsoft's MAI-DxO at 85% versus 20% for unaided physicians on 304 NEJM cases), or from preprints on NEJM clinicopathological conference cases. Each could be discounted by skeptics as either too clean, too marketed, or too narrow.

The Harvard-BIDMC paper closes those escape hatches. It uses raw, unstructured electronic health records exactly as they appear in clinical practice. It evaluates the model at three sequential decision points — initial triage on arrival, first contact with a physician, and admission to the medical floor or ICU — rather than at a single artificial moment. It uses real attendings as the comparator, not residents or web-search baselines. And it ran the gauntlet of Science peer review. As co-author Peter Brodeur warns, existing multiple-choice and vignette benchmarks are saturating near 100%, which is precisely why this study's choice to anchor evaluation on attending physicians and raw EHRs matters: it is the new ceiling test, and the ceiling is no longer above the ground floor.

The paper's authors are unusually loud about how the result should not be used. Adam Rodman has explicitly proposed a 'triadic care model … the doctor, the patient, and an artificial intelligence system' rather than replacement, and Arjun Manrai — while saying o1 'eclipsed both prior models' — pushes back on framing the result as AI replacing doctors. Coverage notes that both researchers publicly criticize companies marketing AI to 'cut doctors out of the loop.' Peter Brodeur reinforces the guardrail: 'Humans should be the ultimate baseline when it comes to evaluating performance and safety.' The accountability picture is also exposed: reporters note 'there is not a formal framework right now for accountability,' and Rodman warns a regulatory posture of 'just trust me' would limit adoption.

Reader communities split along a different axis. Reddit splits along an accessibility-cheerleader vs. dystopia-skeptic line: top voices in r/singularity frame the result as a path to cheaper care for people with no doctor at all — 'getting care from an error-prone AI is better than dying with 0 care' — while r/artificial commenters worry the next step is insurer-mandated AI gates implemented 'in the worst and cheapest way possible,' and flag that the model still cannot prescribe. Practitioner comments cut a third way: a pediatrics resident and a physician describe using o1-class models as 'instant professional consultants,' but one commenter advises walking the human doctor step-by-step toward the AI's conclusion without revealing AI was used, suggesting clinicians already navigate a quiet de facto deployment. Yujin Potter at UC Berkeley adds the alignment dimension that the accuracy numbers do not address: hallucinations, deception risk, and the gap between benchmark performance and safety in the loop.

Every result in the Science paper rests on text. The study notes that real-life clinical medicine is 'multifaceted and awash with non-text inputs,' and that auditory cues such as patient distress and visual inputs such as medical imaging were not part of the evaluation. Thomas Buckley, a doctoral candidate at HMS Biomedical Informatics, is direct about the consequence: 'They're underperforming on most medical imaging benchmarks.' Coverage confirms parallel studies on image and signal performance are underway. Until those land, any reading of the paper that claims o1 can do an attending's full job is going further than the data.

The text-only constraint also explains why the management reasoning gap is so wide. Management plans on a written EHR are heavily verbal artefacts — admit/discharge, order this lab, start this antibiotic — and a step-by-step reasoning model with no time pressure, no shift fatigue, and no patient interaction friction is well matched to that task. Strip the model of physical exam, imaging, and the noisy social environment of an ER, and you have removed the dimensions where physicians most clearly outperform. The press release flag that the model may recommend 'unnecessary testing that could expose patients to harm' is the obvious failure mode of a text-strong, context-poor recommender. Future work — and the prospective trials the authors are calling for — has to test whether the gap survives when imaging, EKGs, and bedside cues come back into the loop.

Three forces converged to make this study land hard. First, reasoning models architecturally fit the task: o1's step-by-step chain-of-thought design was engineered for multi-step problem solving, which maps onto how clinicians construct differentials and management plans. Second, the existing benchmark stack is saturating — top models score near 100% on multiple-choice and vignette tests, which is exactly why Brodeur and colleagues anchored on raw EHRs and attending-physician baselines. Without that pivot, the result would not have been distinguishable from prior preprints. Third, ER workflow itself is the highest-leverage entry point for text-strong AI: triage relies on parsing messy, noisy electronic health records under time pressure with sparse early data, the regime where LLM second-readers can plausibly add value.

The demand side is also already in place. Coverage cites that by 2025 roughly 20% of clinicians were consulting LLMs for second opinions, with UK survey figures at 16% daily and 15% weekly use. That is de facto deployment without trials, without accountability frameworks, and without integration into liability and reimbursement. The Harvard-BIDMC authors are reading the room: they want their result interpreted as a call for prospective randomized trials, human-computer interaction studies, and equity/cost/safety evaluation rather than as a green light. Hopkins and Cornelisse at Flinders University frame the same point externally: 'Accuracy on a defined task is only one dimension of deployment readiness. Clinical AI must also deliver equitable, cost-effective, and safe outcomes.' The takeaway for tech-adjacent readers is that the headline number (89% vs 34%) and the policy posture (no, do not deploy this in your ER) are coming from the same authors, on purpose, and the gap between the two is exactly what the next two years of medical AI regulation will be about.