Microsoft’s Diagnostic AI Beat Doctors in Retrospective Tests, Not a Hospital

A medical AI system from Microsoft has moved beyond the unusually difficult case reports that powered its 2025 debut. In a paper published in Science on October 22, 2026, researchers evaluated the Microsoft AI Diagnostic Orchestrator, or MAI-DxO, on both complex teaching cases and a second set of more typical emergency-department encounters.

The new dataset matters. It contains 325 encounters selected from MIMIC-IV-ED, a deidentified research database whose emergency module includes triage assessments, validated vital signs, medication reconciliation and treatment information. Microsoft says MAI-DxO reached 80% diagnostic accuracy on these cases, compared with 47% for participating physicians.

Why the system is different

Most medical question benchmarks reveal the whole case at once and ask for one answer. SDBench instead starts with limited information. A participant must decide what to ask, examine or test next, receive another piece of evidence and revise the likely diagnosis.

MAI-DxO coordinates several model roles that propose diagnoses, request information and challenge one another. The orchestrator repeats this process over several rounds while weighing the expected value and estimated cost of each test.

The headline result is not that a chatbot knew more facts. It is that an orchestrated system chose a diagnostic path, step by step, inside a controlled simulation.

On 304 complex cases adapted from The New England Journal of Medicine, Microsoft reports 83% accuracy. Physicians scored 22% without outside references and 36% when allowed textbooks and the internet. One cost-focused configuration retained 81% accuracy while using tests estimated to cost less than half as much as comparison systems.

What has actually changed

The difficult NEJM benchmark and an earlier version of MAI-DxO were disclosed in June 2025. The peer-reviewed paper adds the emergency-department cohort, updated comparisons and a clearer account of the sequential method. It therefore supplies broader evidence, rather than introducing the system for the first time.

The paper also reports that the approach improved when newer underlying models were substituted without changing the orchestration method. That suggests some of the gain may come from the workflow around the models, not only from one model generation.

What comes next

Microsoft says it will work with Beth Israel Deaconess Medical Center in Boston on a prospective study in which patients interact with the system and its performance is compared with independent physician assessments. That planned evaluation is the key bridge between retrospective benchmark success and evidence from live clinical workflows.