Evaluation of large language model-generated medical information on idiopathic pulmonary fibrosis.

ABSTRACT:

Background

Idiopathic Pulmonary Fibrosis (IPF) information from AI-powered large language models (LLMs) like ChatGPT-4 and Gemini 1.5 Pro is unexplored for quality, reliability, readability, and concordance with clinical guidelines.

Research question

What is the quality, reliability, readability, and concordance to clinical guidelines of LLMs in medical and clinically IPF-related content?

Study design and methods

ChatGPT-4 and Gemini 1.5 Pro responses to 23 ATS/ERS/JRS/ALAT IPF guidelines questions were compared. Six independent raters evaluated responses for quality (DISCERN), reliability (JAMA Benchmark Criteria), readability (Flesch-Kincaid), and guideline concordance (0-4). Descriptive analysis, Intraclass Correlation Coefficient, Wilcoxon signed-rank test, and ef

SUBMITTER: Cherrez-Ojeda I 

PROVIDER: S-EPMC12504196 | biostudies-literature | 2025

REPOSITORIES: biostudies-literature