A study of calibration as a measurement of trustworthiness of large language models in biomedical natural language processing.

ABSTRACT:

Objectives

To assess the calibration of 9 large language models (LLMs) within biomedical natural language processing (BioNLP) tasks, furthering understanding of trustworthiness and reliability in real-world settings.

Materials and methods

For each LLM, we collected responses and corresponding confidence scores for all 13 datasets (grouped into 6 tasks) of the Biomedical Language Understanding & Reasoning Benchmark (BLURB). Confidence scores were assigned using 3 strategies: Verbal, Self-consistency, and Hybrid. For evaluation, we introduced Flex-ECE (Flexible Expected Calibration Error), a novel adaptation of ECE that accounts for partial correctness in model responses, allowing for a more realistic assessment of calibration in language-based settings. Two post-hoc calibrati

SUBMITTER: de Oliveira R 

PROVIDER: S-EPMC12249208 | biostudies-literature | 2025 Aug

REPOSITORIES: biostudies-literature