Reliability of crowdsourcing as a method for collecting emotions labels on pictures.

ABSTRACT:

Objective

In this paper we study if and under what conditions crowdsourcing can be used as a reliable method for collecting high-quality emotion labels on pictures. To this end, we run a set of crowdsourcing experiments on the widely used IAPS dataset, using the Self-Assessment Manikin (SAM) emotion collection instrument, in order to rate pictures on valence, arousal and dominance, and explore the consistency of crowdsourced results across multiple runs (reliability) and the level of agreement with the gold labels (quality). In doing so, we explored the impact of targeting populations of different level of reputation (and cost) and collecting varying numbers of ratings per picture.

Results

The results tell us that crowdsourcing can be a reliable method, reaching excellent le

SUBMITTER: Korovina O 

PROVIDER: S-EPMC6822440 | biostudies-literature | 2019 Oct

REPOSITORIES: biostudies-literature