Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment.

ABSTRACT:

Introduction

The inability of large language models (LLMs) to communicate uncertainty is a significant barrier to their use in medicine. Before LLMs can be integrated into patient care, the field must assess methods to estimate uncertainty in ways that are useful to physician-users.

Objective

Evaluate the ability for uncertainty proxies to quantify LLM confidence when performing diagnosis and treatment selection tasks by assessing the properties of discrimination and calibration.

Methods

We examined confidence elicitation (CE), token-level probability (TLP), and sample consistency (SC) proxies across GPT3.5, GPT4, Llama2, and Llama3. Uncertainty proxies were evaluated against 3 datasets of open-ended patient scenarios.

Results

SC discrimination outperformed TLP

SUBMITTER: Savage T 

PROVIDER: S-EPMC11648734 | biostudies-literature | 2025 Jan

REPOSITORIES: biostudies-literature