CurmElo: The theory and practice of a forced-choice approach to producing preference rankings.

ABSTRACT: We introduce CurmElo, a forced-choice approach to producing a preference ranking of an arbitrary set of objects that combines the Elo algorithm with novel techniques for detecting and correcting for (1) preference heterogeneity induced polarization in preferences among raters, and (2) intransitivity in preference rankings. We detail the application of CurmElo to the problem of generating approximately preference-neutral identifiers, in this case four-letter and five-letter nonsense words patterned on the phonological conventions of the English language, using a population of Amazon Mechanical Turk workers. We find evidence that human raters have significant non-uniform preferences over these nonsense words, and we detail the consequences of this finding for social science work that utilize

SUBMITTER: Sankaran S 

PROVIDER: S-EPMC8158949 | biostudies-literature | 2021

REPOSITORIES: biostudies-literature