People make mistakes: Obtaining accurate ground truth from continuous annotations of subjective constructs.

ABSTRACT: Accurately representing changes in mental states over time is crucial for understanding their complex dynamics. However, there is little methodological research on the validity and reliability of human-produced continuous-time annotation of these states. We present a psychometric perspective on valid and reliable construct assessment, examine the robustness of interval-scale (e.g., values between zero and one) continuous-time annotation, and identify three major threats to validity and reliability in current approaches. We then propose a novel ground truth generation pipeline that combines emerging techniques for improving validity and robustness. We demonstrate its effectiveness in a case study involving crowd-sourced annotation of perceived violence in movies, where our pipeline achieves

SUBMITTER: Booth BM 

PROVIDER: S-EPMC11525321 | biostudies-literature | 2024 Dec

REPOSITORIES: biostudies-literature