Revolutionizing Clinical Diagnosis: SDBench and MAI-DxO Enable Cost-Effective, Real-Time AI Reasoning

Bridging the Gap in Clinical AI Reasoning

Artificial intelligence holds great promise in making expert medical reasoning accessible to more people. However, traditional AI evaluations often rely on simplified, static scenarios that don’t capture the dynamic nature of real clinical practice. Physicians continuously refine their diagnoses by asking targeted questions and interpreting new information step by step, balancing the costs and benefits of tests to avoid premature conclusions and unnecessary procedures.

Limitations of Current AI Evaluation Methods

Many current AI models perform well on structured exams, but these tests lack the complexity of real-world clinical environments. Earlier AI systems employed Bayesian frameworks for sequential diagnoses but required extensive expert input, limiting scalability. More recent approaches utilize language models assessed mainly through static, multiple-choice benchmarks or fixed case vignettes, which do not fully represent the cost-sensitive, iterative decision-making process doctors use.

Introducing SDBench: An Interactive Diagnostic Benchmark

To address these challenges, Microsoft AI researchers developed SDBench, a benchmark using 304 real diagnostic cases from the New England Journal of Medicine. SDBench transforms cases into interactive simulations where AI or physicians must ask questions and order tests before making a final diagnosis. A language model acts as a Gatekeeper, providing information only when specifically requested, mimicking real clinical interactions.

Enhancing AI with MAI-DxO

Alongside SDBench, the MAI-DxO system was introduced as an orchestrator co-designed with physicians. It simulates a virtual medical panel to prioritize high-value, cost-effective diagnostic tests. When paired with advanced language models like OpenAI’s o3, MAI-DxO achieved up to 85.5% diagnostic accuracy while significantly reducing costs.

Performance and Impact

The Sequential Diagnosis Benchmark covers a wide range of clinical conditions and uses realistic cost estimates based on CPT codes. Evaluations showed that MAI-DxO consistently outperforms standard models and even physicians by delivering higher accuracy at lower diagnostic costs. For example, it reached 81.9% accuracy at $4,735 per case, compared to 78.6% accuracy at $7,850 by the off-the-shelf o3 model. This demonstrates improved efficiency, reducing unnecessary tests through smarter information gathering.

Future Directions

SDBench and MAI-DxO represent a significant step toward realistic, cost-aware AI clinical reasoning. However, current limitations include a focus on complex cases with less representation of everyday conditions and real-world constraints. Future research aims to validate these systems in clinical settings and low-resource environments, potentially transforming global health and medical education.