Concordance between humans and GPT-4 in appraising the methodological quality of case reports and case series using the Murad tool.

ABSTRACT:

Background

Assessing the methodological quality of case reports and case series is challenging due to human judgment variability and time constraints. We evaluated the agreement in judgments between human reviewers and GPT-4 when applying a standard methodological quality assessment tool designed for case reports and series.

Methods

We searched Scopus for systematic reviews published in 2023-2024 that cited the appraisal tool by Murad et al. A GPT-4 based agent was developed to assess the methodological quality using the 8 signaling questions of the tool. Observed agreement and agreement coefficient were estimated comparing published judgments of human reviewers to GPT-4 assessment.

Results

We included 797 case reports and series. The observed agreement ranged between

SUBMITTER: Tarakji Z 

PROVIDER: S-EPMC11533388 | biostudies-literature | 2024 Nov

REPOSITORIES: biostudies-literature