What Did Microsoft’s AI Actually Do Better Than Doctors?

🔍 Read the full analysis: What Did Microsoft’s AI Actually Do Better Than Doctors? on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get smart everyday buys delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI scored higher than generalist physicians on two sets of historical medical cases, according to a study published in Science on October 22. The system reached 83.2% accuracy on 304 difficult case records, compared with 36.1% for physicians using textbooks and the internet, but the research did not test diagnosis or care for real patients.

On a separate benchmark of 325 emergency department cases, MAI-DxO scored 80.3%, while physicians scored 46.8%. The results are Microsoft’s reported findings from its study, titled Sequential diagnosis with language models. All 15 listed authors have Microsoft affiliations, including Microsoft AI chief executive Mustafa Suleyman.

The benchmark was designed to test how a participant gathers information, not just whether it can choose a diagnosis when a complete case file is already available. Each case began with a short summary. The AI or physician had to request further details, such as a test result, from a simulated gatekeeper. The study assigned costs to tests and evaluated both the final diagnosis and the information-gathering process.

Microsoft describes MAI-DxO as a system that uses a language model in a panel of virtual physicians. The agents propose diagnoses, request questions or tests, and challenge one another over multiple rounds. The study’s chart also reports that GPT-5 alone scored 78.9% on the difficult cases. That suggests much of the reported accuracy came from the underlying model, with the MAI-DxO orchestration layer adding a smaller gain; Microsoft also reported lower estimated test costs for its system.

What the Scores Can—and Cannot—Show

The results matter because they offer a test of a process closer to clinical reasoning than a simple multiple-choice exam: deciding what to ask, which test to request, and when to stop. If similar performance were demonstrated in clinical settings, such tools could help clinicians organize diagnostic information or identify possibilities that warrant further investigation.

But the reported lead is not evidence that the system provides better care. The comparison was between a panel of AI agents and individual generalist doctors working alone, not against specialists or multidisciplinary teams. The case records were selected for diagnostic difficulty, and the benchmark contained no healthy patients or harmless explanations. As a result, it could not establish how often the AI would raise unnecessary alarms in a broader patient population.

The study also did not establish whether use of MAI-DxO changes treatment decisions, reduces time to diagnosis, improves health outcomes, or creates new risks. Its accuracy figures describe performance on specified case records under test conditions. They should not be read as a measure of how the system would perform across routine hospital or primary-care visits.

How Microsoft Tested the System

The paper appeared in Science, volume 394, pages 423–428, on October 22, 2026. An earlier version was published as a non-peer-reviewed preprint in June 2025. That version reported a more dramatic gap, with an earlier model scoring 80% against 20% for doctors. The published paper reports different figures and adds a physician comparison using textbooks and internet access.

For the earlier benchmark details, the supplied source describes 21 generalists from the United States and United Kingdom: 17 primary-care doctors and four hospital generalists, with a median of 12 years’ experience. They worked without search engines or language models. The preprint’s physician sample covered 56 held-out cases, while the AI was evaluated on all 304. The source notes that some methodological details come from the preprint because the full Science article is behind a paywall; those details may differ in the final paper.

The benchmark also relied on estimates rather than direct clinical costs. In the preprint methodology, tests were mapped to billing codes and priced using a 2023 U.S. hospital price list, with a fixed $300 charge per physician visit. When a requested test result was not in a case file, a gatekeeper model generated a synthetic result. A separate model graded diagnoses against the published answer; physician review of a sample showed good, but not perfect, agreement. The authors characterized the cost figures as first-order approximations.

“More than twice as accurate as physicians using textbooks and the internet on complex cases”

— Microsoft, in its blog post about the study

Limits of the Patient Evidence

No real patients were tested in the reported benchmark. The cases were historical and text-based, without physical examinations or real-time imaging, and the study did not measure treatment choices or patient outcomes. It is also not clear from the supplied account whether all benchmark and scoring details in the earlier preprint carried over unchanged into the peer-reviewed paper.

The comparison does not show how MAI-DxO would perform against specialists, clinician teams, or doctors using the same case set under the same conditions. Nor does the test establish how well the system recognizes common, low-risk explanations, since its cases contained difficult diagnoses and no healthy patients. The figures therefore cannot show its rate of false alarms in ordinary practice.

Peer review means the paper underwent journal review; it does not mean independent researchers have reproduced the results. The study’s authors are Microsoft-affiliated, and the supplied material does not report external replication. How the system would handle incomplete or unusual records, and what safeguards would be needed before clinical use, remain open questions.

A Planned Study Alongside Care

Microsoft’s next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the supplied source, consenting patients are expected to interact with the system alongside their normal care. That setting could provide evidence about how the tool performs when patients and clinicians are involved, rather than when a model works through a simulated case file.

The available account does not give a start date, enrollment target, study duration, or planned outcome measures. Until those results are reported, the central evidence remains benchmark performance on selected historical cases. Readers should look for whether the next study reports safety, diagnostic accuracy in a broader patient group, effects on clinicians’ decisions, and patient outcomes—not only how often the system matches a recorded diagnosis.

Key Questions

What did Microsoft’s AI score on the difficult cases?

Microsoft reported that MAI-DxO reached 83.2% diagnostic accuracy on 304 difficult case records. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without those resources.

Did the study show that MAI-DxO is better than doctors in hospitals?

No. It compared the system with individual generalists on historical case benchmarks, not with specialists or clinical teams treating patients. It did not test whether the AI improves care in hospitals.

Were patients diagnosed using the system?

No patients were tested in the reported benchmark. The cases were historical and text-based, with additional information requested through a simulated gatekeeper.

What is the next study?

The supplied source says Microsoft plans a prospective study at Beth Israel Deaconess Medical Center in Boston, where consenting patients will interact with the system alongside their usual care. Timing and detailed study measures were not provided.

Source: ThorstenMeyerAI.com