Diagnosis and Triage Performance of Contemporary Large Language Models on Short Clinical Vignettes.

ABSTRACT: General-purpose large language models (LLMs) are increasingly proposed for diagnostic and triage decision support, yet their reliability relative to humans remains unclear. We evaluated eight contemporary LLMs (ChatGPT-4, ChatGPT-o1, DeepSeek-V3, DeepSeek-R1, Gemini-2.0, Copilot, Grok-2, Llama-3.1) on 48 single-turn clinical vignettes spanning four triage levels (Emergent, 1-day, 1-week, Self-care). Models were tested without prompts and with structured prompts comprising exemplar cases. Primary outcomes were diagnostic and triage accuracy. Secondary measures included confusion matrices, over-triage, safety of advice, and the Capability Comparison Score (CCS). Structured prompting improved performance across models: mean diagnostic accuracy increased from 89.84% to 91.67%, and mean triage

SUBMITTER: Xu L 

PROVIDER: S-EPMC12535515 | biostudies-literature | 2025 Oct

REPOSITORIES: biostudies-literature