The pitfalls of multiple-choice questions in generative AI and medical education.

ABSTRACT: The performance of Large Language Models (LLMs) on multiple-choice question (MCQ) benchmarks is frequently cited as proof of their medical capabilities. We hypothesized that LLM performance on medical MCQs may in part be illusory and driven by factors beyond medical content knowledge and reasoning capabilities. To assess this, we created a novel benchmark of free-response questions with paired MCQs (FreeMedQA). Using this benchmark, we evaluated three state-of-the-art LLMs (GPT-4o, GPT-3.5, and LLama-3-70B-instruct) and found an average absolute deterioration of 39.43% in performance on free-response questions relative to multiple-choice (p = 1.3 * 10^-5) which was greater than the human performance decline of 22.29%. To isolate the role of the MCQ format on performance, we perfo

SUBMITTER: Singh S 

PROVIDER: S-EPMC12658246 | biostudies-literature | 2025 Nov

REPOSITORIES: biostudies-literature