Dataset Information

The interrater reliability of a pain mechanisms-based classification for patients with nonspecific neck pain.

ABSTRACT:

Objective

To examine the interrater reliability and agreement of a pain mechanisms-based classification for patients with nonspecific neck pain (NSNP).

Methods

Design - Observational, cross-sectional reliability study with a simultaneous examiner design.

Setting

University hospital-based outpatient physical therapy clinic.

Participants

A random sample of 48 patients, aged between 18 and 75 years old, with a primary complaint of neck pain was included.

Interventions

Subjects underwent a standardized subjective and clinical examination, performed by 1 experienced physical therapist. Two assessors independently classified the participants' NSNP on 3 main outcome measures.

Main outcome measures

The Cohen kappa, percent agreement, and 95% confidence intervals (CIs) were calculated to determine the interrater reliability for (1) the predominant pain mechanism; (2) the predominant pain pattern; and (3) the predominant dysfunction pattern (DP).

Results

There was almost perfect agreement between the 2 physical therapists' judgements on the predominant pain mechanism, kappa=.84 (95% CI, .65-1.00), p<.001. There was substantial agreement between the raters' judgements on the predominant pain pattern and predominant DP with respectively kappa=.61 (95% CI, .42-.80); and kappa=.62 (95% CI, .44-.79), p<.001.

Conclusion(s)

The proposed classification exhibits substantial to almost perfect interrater reliability. Further validity testing in larger neck pain populations is required before the information is used in clinical settings.

Clinical trial registration number

NCT03147508 (https://clinicaltrials.gov/ct2/show/NCT03147508).

SUBMITTER: Dewitte V

PROVIDER: S-EPMC6823704 | biostudies-literature | 2019 Sep - Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The interrater reliability of a pain mechanisms-based classification for patients with nonspecific neck pain.

Dewitte Vincent V De Pauw Robby R Danneels Lieven L Bouche Katie K Roets Arne A Cagnie Barbara B

Brazilian journal of physical therapy 20181029 5

<h4>Objective</h4>To examine the interrater reliability and agreement of a pain mechanisms-based classification for patients with nonspecific neck pain (NSNP).<h4>Methods</h4>Design - Observational, cross-sectional reliability study with a simultaneous examiner design.<h4>Setting</h4>University hospital-based outpatient physical therapy clinic.<h4>Participants</h4>A random sample of 48 patients, aged between 18 and 75 years old, with a primary complaint of neck pain was included.<h4>Intervention ...[more]

PMID: 30389348

Similar Datasets

Project description:BackgroundInterrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.MethodsWe conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.ResultsThe most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.ConclusionThe authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.

Dataset Information

The interrater reliability of a pain mechanisms-based classification for patients with nonspecific neck pain.

Objective

Methods

Setting

Participants

Interventions

Main outcome measures

Results

Conclusion(s)

Clinical trial registration number

Publications

The interrater reliability of a pain mechanisms-based classification for patients with nonspecific neck pain.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets