🔍 Read the full analysis: Microsoft’s AI Doctor Results: What The Study Set Out To Measure on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A paper published in Science on October 22 reports that Microsoft’s MAI-DxO system reached 83.2% diagnostic accuracy on 304 complex cases, compared with 36.1% for generalist physicians using reference materials. The study tested a text-based simulator built from historical cases, not diagnosis or treatment of live patients; the authors’ next step is a prospective study alongside usual care.
Microsoft’s MAI-DxO achieved 83.2% diagnostic accuracy on 304 complex medical cases in a study published October 22 in Science, compared with 36.1% for generalist physicians given textbooks and internet access. The peer-reviewed paper reports results from a text-based simulation of historical cases, not from treating patients, so it does not establish how the system would perform in routine clinical care.
The researchers assessed diagnosis using the Sequential Diagnosis Benchmark, in which each case begins with a short summary. A doctor or AI system must request additional details, such as answers to questions or test results, from a simulated case file. The evaluation records the final diagnosis and estimates the cost of the requested tests.
Microsoft reported a second comparison using 325 emergency department cases: MAI-DxO scored 80.3%, against 46.8% for physicians. On the complex-case benchmark, doctors scored 22.3% without reference materials and 36.1% with textbooks and internet access. Those numbers are results reported by the paper’s Microsoft-affiliated authors; peer review does not mean that an independent group has reproduced them.
MAI-DxO coordinates a language model through a panel of virtual physicians that suggest diagnoses, choose questions or tests, and review one another’s reasoning. On the complex cases, GPT-5 alone scored 78.9%, close to the system’s 83.2%. That comparison suggests most of the reported accuracy came from the underlying model, while Microsoft’s orchestration added about four percentage points, according to its results.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
Why the Benchmark Matters
The results point to a possible use for AI in organizing diagnostic investigations: deciding what information to seek next, rather than simply selecting an answer from a complete case description. The benchmark also attempts to account for the cost of tests, a practical concern for hospitals and patients if such systems are eventually used in care.
But the measured advantage is specific to the study setup. The comparison was between a coordinated AI system and individual generalist physicians, not specialist teams or doctors working together. The cases were selected for difficulty and came from published records. The figures therefore cannot establish that MAI-DxO is more accurate than the clinicians who would normally handle comparable cases, or that it would improve patient outcomes, reduce spending, or avoid unnecessary tests in practice.
For health systems, insurers and patients, the distinction is material: a high score on a curated diagnostic benchmark can support further research, but does not by itself show that a system is safe, effective or cost-saving in real clinical workflows.
Products Worth Considering
Apple Watch Series 11 GPS + Cellular, 46mm Jet Black Aluminum Case with Black Sport Band (M/L)
Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds with Immersive Audio and Wireless Charging Case, Black
Bose QuietComfort Wireless Noise-Cancelling Headphones, White Smoke
As an affiliate, we earn on qualifying purchases.
How the Cases Were Tested
The paper, Sequential diagnosis with language models, appeared in Science, volume 394, pages 423–428, on October 22, 2026. All 15 listed authors are affiliated with Microsoft, including Microsoft AI chief executive Mustafa Suleyman. An earlier, non-peer-reviewed version appeared on arXiv in June 2025 and reported a different comparison using an earlier model. The journal paper’s results should be distinguished from that earlier preprint.
The benchmark differs from a conventional medical exam because the participant does not receive every case detail at once. A simulated “gatekeeper” provides information when requested. According to details described in the earlier preprint, a model graded diagnoses against the published answer, and missing test results could be synthetically generated. Test prices were estimated using billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit cost. The preprint described these figures as approximations; the final paper’s full text is behind a paywall, so some implementation details may have changed.
The physician comparison also has limits. In the preprint, 21 U.S. and UK generalists took part, and doctors were scored on a subset of held-out cases while the AI was assessed across all 304 complex cases. The Science paper reports the final comparison figures, including an open-book group, but the benchmark remains a test of historical, text-only case records rather than bedside practice.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch
Limits of the Reported Results
The study did not involve live patients, physical examinations or imaging. It also did not test whether recommendations would change treatment, improve health outcomes, or lead to harmful delays. The case set contained difficult published cases and no healthy patients or harmless explanations, according to the source material; the authors noted that this prevents measurement of false alarms in that benchmark.
It remains unclear how the system would compare with specialists, multidisciplinary teams or clinicians using their usual tools in a real hospital. The reported test costs are estimates rather than observed spending, and the study’s reported results have not been independently reproduced. The paper’s peer-reviewed status supports scrutiny of its methods and reporting, but is not proof of clinical benefit.
Testing Alongside Patient Care
The authors say a prospective study at Beth Israel Deaconess Medical Center in Boston is planned. Consenting patients are expected to interact with the system alongside their normal care. That setting could provide evidence about how MAI-DxO functions with real clinical information and workflows, though the source material does not give a start date, study size or expected publication date.
Until those details and results are available, the current evidence is best understood as a benchmark result: promising performance on selected historical cases, with open questions about generalizability, safety, costs and patient outcomes.
Key Questions
What did Microsoft’s MAI-DxO study measure?
It measured diagnostic accuracy and estimated test costs as an AI system and physicians requested information from simulated records of historical cases.
Did MAI-DxO diagnose real patients in the study?
No. The reported tests used text-only historical cases in a simulator. The study did not involve live patient diagnosis or treatment.
How did the physicians score?
On 304 complex cases, the paper reports physician accuracy of 36.1% with textbooks and internet access and 22.3% without those resources. The comparison involved generalists working individually, not specialist teams.
Does peer review confirm that MAI-DxO works in hospitals?
No. Peer review means the paper was evaluated for publication; it does not establish real-world effectiveness or independent replication. A prospective study alongside normal care is planned.
Source: ThorstenMeyerAI.com
