Interrater reliability estimators tested against true interrater reliabilities.

ABSTRACT:

Background

Interrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (a_o), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three

SUBMITTER: Zhao X 

PROVIDER: S-EPMC9426226 | biostudies-literature | 2022 Aug

REPOSITORIES: biostudies-literature