Unknown

Dataset Information

Using crowdsourcing to evaluate published scientific literature: methods and example.


ABSTRACT: Systematically evaluating scientific literature is a time consuming endeavor that requires hours of coding and rating. Here, we describe a method to distribute these tasks across a large group through online crowdsourcing. Using Amazon's Mechanical Turk, crowdsourced workers (microworkers) completed four groups of tasks to evaluate the question, "Do nutrition-obesity studies with conclusions concordant with popular opinion receive more attention in the scientific community than do those that are discordant?" 1) Microworkers who passed a qualification test (19% passed) evaluated abstracts to determine if they were about human studies investigating nutrition and obesity. Agreement between the first two raters' conclusions was moderate (κ = 0.586), with consensus being reached in 96% of abstr

SUBMITTER: Brown AW 

PROVIDER: S-EPMC4079692 | biostudies-literature | 2014

REPOSITORIES: biostudies-literature