Large language models are less effective at clinical prediction tasks than locally trained machine learning models.

ABSTRACT:

Objectives

To determine the extent to which current large language models (LLMs) can serve as substitutes for traditional machine learning (ML) as clinical predictors using data from electronic health records (EHRs), we investigated various factors that can impact their adoption, including overall performance, calibration, fairness, and resilience to privacy protections that reduce data fidelity.

Materials and methods

We evaluated GPT-3.5, GPT-4, and traditional ML (as gradient-boosting trees) on clinical prediction tasks in EHR data from Vanderbilt University Medical Center (VUMC) and MIMIC IV. We measured predictive performance with area under the receiver operating characteristic (AUROC) and model calibration using Brier Score. To evaluate the impact of data privacy prote

SUBMITTER: Brown KE 

PROVIDER: S-EPMC12012369 | biostudies-literature | 2025 May

REPOSITORIES: biostudies-literature