Luca de Alfaro, Vassilis Polychronopoulos, Michael Shavlovsky
08/10/2015 07:12 PM
Computer Science
We propose novel algorithms for the problem of crowdsourcing binary labels. Such binary labeling tasks are very common in crowdsourcing platforms, for instance, to judge the appropriateness of web content or to flag vandalism. We propose two unsupervised algorithms: one simple to implement albeit derived heuristically, and one based on iterated bayesian parameter estimation of user reputation models. We provide mathematical insight into the benefits of the proposed algorithms over existing approaches, and we confirm these insights by showing that both algorithms offer improved performance on both synthetic and large-scale real-world datasets obtained via Amazon Mechanical Turk.
