A comprehensive guide to study the agreement and reliability of multi-observer ordinal data.

ABSTRACT:

Background

A recent systematic review revealed issues in regard to performing and reporting agreement and reliability studies for ordinal scales, especially in the presence of more than two observers. This paper therefore aims to provide all necessary information in regard to the choice among the most meaningful and most used measures and the planning of agreement and reliability studies for ordinal outcomes.

Methods

This paper considers the generalisation of the proportion of (dis)agreement, the mean absolute deviation, the mean squared deviation and weighted kappa coefficients to more than two observers in the presence of an ordinal outcome.

Results

After highlighting the difference between the concepts of agreement and reliability, a clear and simple interpretation

SUBMITTER: Vanbelle S 

PROVIDER: S-EPMC11660713 | biostudies-literature | 2024 Dec

REPOSITORIES: biostudies-literature