🔍 Read the full analysis: How The Research Framed Microsoft’s AI Comparison With Doctors on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A study published in Science on October 22 reported that Microsoft’s MAI-DxO system diagnosed selected complex cases more accurately than generalist physicians in a text-based simulation. The results do not establish how the system would perform with real patients or against specialists and clinical teams; Microsoft says a patient study is planned at Beth Israel Deaconess Medical Center.
Microsoft’s MAI-DxO diagnostic AI scored higher than generalist physicians on two sets of historical medical cases in a study published in Science on October 22, according to results reported by the company. The comparison used simulated, text-based cases rather than real patients, and pitted an AI system against individual generalists—not specialist doctors or clinical teams—so the findings do not establish how it would perform in routine care.
On 304 complex cases drawn from the New England Journal of Medicine, Microsoft reported 83.2% diagnostic accuracy for MAI-DxO. The physician comparison averaged 36.1% when doctors could consult textbooks and the internet, and 22.3% without those resources. On a separate set of 325 emergency department cases, the reported scores were 80.3% for the system and 46.8% for physicians. These are results reported by Microsoft for its own system and study.
The paper, “Sequential diagnosis with language models,” appeared in Science, volume 394, pages 423–428. The benchmark gave each participant a short case summary; additional information, such as test results, had to be requested from a simulated gatekeeper. MAI-DxO used a panel-like process in which language-model agents proposed diagnoses, chose questions or tests, and evaluated options over multiple rounds. The benchmark counted both diagnostic performance and estimated testing costs.
The AI’s score was not solely a result of that orchestration method. Microsoft’s chart showed GPT-5 alone scored 78.9% on the complex cases, compared with 83.2% for MAI-DxO. The source account says the added system layer improved the score by about four percentage points and reduced estimated testing costs. Those cost figures are benchmark estimates, not observed savings in a hospital.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Scores Can—and Cannot—Show
The results matter because they put a sequential diagnostic task—deciding what to ask or test next—at the center of an AI evaluation, rather than simply asking a model to choose a diagnosis after receiving a complete case file. That makes the benchmark relevant to questions about how diagnostic systems might gather information and use tests, not just how well they answer a medical quiz.
But the comparison is narrower than a claim that AI can replace doctors. The physician participants were primary care doctors and hospital generalists working individually. The cases were selected for difficulty, and, as the source account notes, such cases may be referred to specialists or handled by teams in practice. A system’s performance against one generalist does not show how it compares with those real-world alternatives.
The benchmark also cannot show whether using the system improves patient outcomes, reduces delays, or lowers actual costs. Its test prices were estimates based on billing codes and a 2023 U.S. hospital price list, alongside a fixed physician-visit cost. For hospitals, insurers and patients, the results are a research signal—not evidence that the tool is ready for clinical use or that it will save money.
AI diagnostic tools for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How Microsoft Built the Test
Microsoft’s Sequential Diagnosis Benchmark begins with a brief case summary. Participants request further details from a model holding the case file, and tests carry assigned costs. This design aims to represent a feature of diagnosis that standard question-and-answer tests often leave out: a clinician must decide which information to seek and when to stop.
Several methodological details described in the source account come from the study’s 2025 arXiv preprint, rather than a full reading of the paywalled Science paper, and could differ in the final version. In that earlier description, a separate model graded diagnoses against the published final diagnosis. When a requested result was absent from the record, the gatekeeper generated a synthetic result. The authors also described test prices as first-order estimates.
The final paper is peer-reviewed, but that status is not the same as independent replication. The source account says all 15 authors were affiliated with Microsoft, including Microsoft AI chief executive Mustafa Suleyman. An earlier preprint, posted in June 2025, reported a different model and a larger gap between AI and physicians; those preprint figures should not be confused with the later Science results.
“MAI-DxO achieved 83.2% diagnostic accuracy on 304 complex NEJM cases.”
— Microsoft, in the study’s reported results
Limits of the Patient Evidence
The reported benchmark does not show how MAI-DxO performs during live clinical encounters, where symptoms may be incomplete, patients may respond unpredictably, and examination findings or imaging may shape decisions. The source account says the test used historical text cases, without real patients, physical examinations or imaging.
It is also unclear from the supplied material whether the final Science version changed details described in the earlier preprint. The account says the preprint’s physician sample included 21 generalists, who were assessed on 56 held-out cases, while the AI was assessed on all 304. The final publication reports the physician scores and adds an open-book comparison, but the source does not provide a full account of every change between versions.
The cases reportedly did not include healthy patients or harmless explanations, meaning the study could not measure false alarms in the usual sense. The benchmark also relied on model-generated grading and estimated costs. Independent replication, performance on a broader range of cases, and evidence of benefit or harm in patient care remain outstanding.
The Planned Patient Study
Microsoft’s next stated step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients will interact with the system alongside their normal care. That setting could provide evidence unavailable from historical case records, although the supplied material does not specify the study’s start date, size, endpoints or publication timeline.
Readers should watch for how the study measures diagnostic accuracy, test use, false alarms and patient outcomes, and whether clinicians can override or correct the system. Until those details and results are available, the Science paper supports a limited conclusion: MAI-DxO performed strongly on the reported benchmark, while its safety, effectiveness and costs in clinical practice remain unestablished.
Key Questions
What did Microsoft report about MAI-DxO?
Microsoft reported that MAI-DxO scored 83.2% on 304 complex cases and 80.3% on 325 emergency department cases, outperforming the generalist physician averages in those benchmark comparisons.
Were these results measured with real patients?
No. The reported evaluation used historical, text-based cases in a simulation. The source account says it did not include real patient interactions, physical examinations or imaging.
Was MAI-DxO compared with specialists?
The comparison described in the source material was against generalist physicians working individually. It did not include specialist doctors or clinical teams.
Does the study prove the system is ready for hospitals?
No. Peer review indicates the paper was reviewed for publication, but the benchmark does not establish clinical safety, patient benefit or real-world cost savings. Those questions require evidence from patient care settings.
What is the next reported step?
Microsoft says it plans a prospective study at Beth Israel Deaconess Medical Center, where consenting patients will interact with the system alongside their usual care. Details such as timing and study size were not provided in the source material.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
