Evaluating the Accuracy of Responses by Large Language Models for Information on Disease Epidemiology.

ABSTRACT:

Purpose

Accurate background epidemiology of diseases are required in pharmacoepidemiologic research. We evaluated the performance of large language models (LLMs), including ChatGPT-3.5, ChatGPT-4, and Google Bard, when prompted with questions on disease frequency.

Methods

A total of 21 questions on the prevalence and incidence of common and rare diseases were developed and submitted to each LLM twice on different dates. Benchmark data were obtained from literature searches targeting "gold-standard" references (e.g., government statistics, peer-reviewed articles). Accuracy was evaluated by comparing LLMs' responses to the benchmark data. Consistency was determined by comparing the responses to the same query submitted on different dates. The relevance and authenticity of refe

SUBMITTER: Zhu K 

PROVIDER: S-EPMC11791122 | biostudies-literature | 2025 Feb

REPOSITORIES: biostudies-literature