Not the Models You Are Looking For: Traditional ML Outperforms LLMs in Clinical Prediction Tasks.

ABSTRACT:

Objectives

To determine the extent to which current Large Language Models (LLMs) can serve as substitutes for traditional machine learning (ML) as clinical predictors using data from electronic health records (EHRs), we investigated various factors that can impact their adoption, including overall performance, calibration, fairness, and resilience to privacy protections that reduce data fidelity.

Materials and methods

We evaluated GPT-3.5, GPT-4, and ML (as gradient-boosting trees) on clinical prediction tasks in EHR data from Vanderbilt University Medical Center and MIMIC IV. We measured predictive performance with AUROC and model calibration using Brier Score. To evaluate the impact of data privacy protections, we assessed AUROC when demographic variables are generalized.

SUBMITTER: Brown KE 

PROVIDER: S-EPMC11643212 | biostudies-literature | 2024 Dec

REPOSITORIES: biostudies-literature