Preliminary assessment of large language models' performance in answering questions on developmental dysplasia of the hip.

ABSTRACT:

Objective

To evaluate the performance of three large language models in answering questions regarding pediatric developmental dysplasia of the hip.

Methods

We formulated 18 open-ended clinical questions in both Chinese and English and established a gold standard set of answers to benchmark the responses of the large language models. These questions were presented to ChatGPT-4o, Gemini, and Claude 3.5 Sonnet. The responses were evaluated by two independent reviewers using a 5-point scale. The average score, rounded to the nearest whole number, was taken as the final score. A final score of 4 or 5 indicated an accurate response, whereas a final score of 1, 2, or 3 indicated an inaccurate response.

Results

The raters demonstrated a high level of agreement in scoring the

SUBMITTER: Li S 

PROVIDER: S-EPMC11999979 | biostudies-literature | 2025 Apr

REPOSITORIES: biostudies-literature