Large language models for data extraction from unstructured and semi-structured electronic health records: a multiple model performance evaluation.

ABSTRACT:

Objectives

We aimed to evaluate the performance of multiple large language models (LLMs) in data extraction from unstructured and semi-structured electronic health records.

Methods

50 synthetic medical notes in English, containing a structured and an unstructured part, were drafted and evaluated by domain experts, and subsequently used for LLM-prompting. 18 LLMs were evaluated against a baseline transformer-based model. Performance assessment comprised four entity extraction and five binary classification tasks with a total of 450 predictions for each LLM. LLM-response consistency assessment was performed over three same-prompt iterations.

Results

Claude 3.0 Opus, Claude 3.0 Sonnet, Claude 2.0, GPT 4, Claude 2.1, Gemini Advanced, PaLM 2 chat-bison and Llama 3-70b exhi

SUBMITTER: Ntinopoulos V 

PROVIDER: S-EPMC11751965 | biostudies-literature | 2025 Jan

REPOSITORIES: biostudies-literature