Lightweight open-source large language models versus cTAKES for information extraction from discharge summaries: tobacco smoking status test case.

ABSTRACT:

Objectives

To compare lightweight open-source large language models (LLMs) with cTAKES, a state-of-the-art natural language processing (NLP) system, in an information extraction task from hospitalization discharge summaries.

Materials and methods

Two readers annotated 250 randomly sampled adult discharge summaries (BJC HealthCare, 2018-2023) for tobacco smoking status as "Smoker," "Never smoker," "Unknown." Six LLMs (Llama-3 [1B-70B], gpt-oss-20B, MedGemma-27B) and cTAKES extracted smoking status from summaries. Performance was benchmarked against consensus annotations using weighted F1-score, macro F1-score, and per-class F1-scores and a noninferiority test.

Results

Inter-reader agreement was excellent (κ = 0.91). LLM size (2.3-47.3 GB) and inference time (2.5-14.5 s

SUBMITTER: Davila-Garcia DM 

PROVIDER: S-EPMC12803785 | biostudies-literature | 2026 Feb

REPOSITORIES: biostudies-literature