Analyzing evaluation methods for large language models in the medical field: a scoping review.

ABSTRACT:

Background

Owing to the rapid growth in the popularity of Large Language Models (LLMs), various performance evaluation studies have been conducted to confirm their applicability in the medical field. However, there is still no clear framework for evaluating LLMs.

Objective

This study reviews studies on LLM evaluations in the medical field and analyzes the research methods used in these studies. It aims to provide a reference for future researchers designing LLM studies.

Methods & materials

We conducted a scoping review of three databases (PubMed, Embase, and MEDLINE) to identify LLM-related articles published between January 1, 2023, and September 30, 2023. We analyzed the types of methods, number of questions (queries), evaluators, repeat measurements, additional ana

SUBMITTER: Lee J 

PROVIDER: S-EPMC11606129 | biostudies-literature | 2024 Nov

REPOSITORIES: biostudies-literature