Evaluating the performance of large language models versus human researchers on real world complex medical queries.

ABSTRACT: Whether large language models (LLMs) can resolve real-world dilemmas faced by clinicians remains unclear and physician quality assessment is often used as a measure of LLM output quality. We compared reports - defined as answers to clinical queries generated by LLMs or written by human researchers - generated by GPT-4o, Gemini 2.0, and Claude Sonnet 3.5 in response to such dilemmas (n = 20) to reports written by trained human researchers and studied whether physician satisfaction correlates with objective report quality. Twenty human reports and fifty-six LLM-reports were analyzed. Human reports met physicians' expectations more frequently (p = 0.044), were considered more reliable (p = 0.032), professionally written (p = 0.003), and time-saving (p = 0.003). Human reports cited more source

SUBMITTER: Idan D 

PROVIDER: S-EPMC12572390 | biostudies-literature | 2025 Oct

REPOSITORIES: biostudies-literature