Quizz: Targeted Crowdsourcing with a Billion (Potential) Users
Event details
| Date | 23.06.2015 |
| Hour | 15:00 › 16:00 |
| Speaker |
Panos Ipeirotis is an Associate Professor and George A. Kellner Faculty Fellow at the Department of Information, Operations, and Management Sciences at Leonard N. Stern School of Business of New York University, and he is also a visiting scientist at Google. His recent research interests focus on crowdsourcing and on mining user-generated content on the Internet. He received his Ph.D. degree in Computer Science from Columbia University in 2004. He has received six “Best Paper” awards (IEEE ICDE 2005, ACM SIGMOD 2006, WWW 2011, ICIS 2012, HCOMP 2014, Management Science 2011-14), three “Best Paper Runner Up” awards (JCDL 2002, ACM KDD 2008, INFORMS Data Mining Contest 2014), and is also a recipient of a CAREER award from the National Science Foundation and of several other grants. |
| Location | |
| Category | Conferences - Seminars |
We describe Quizz, a gamified crowdsourcing system that simultaneously
assesses the knowledge of users and acquires new knowledge from them.
Quizz operates by asking users to complete short quizzes on specific
topics; as a user answers the quiz questions, Quizz estimates the
user’s competence. To acquire new knowledge, Quizz also incorporates
questions for which we do not have a known answer; the answers given
by competent users provide useful signals for selecting the correct
answers for these questions. Quizz actively tries to identify
knowledgeable users on the Internet by running advertising campaigns,
effectively leveraging “for free” the targeting capabilities of
existing, publicly available, ad placement services. Quizz quantifies
the contributions of the users using information theory and sends
feedback to the advertising system about each user. The feedback
allows the ad targeting mechanism to further optimize ad placement.
Our experiments, which involve over ten thousand users, confirm that
we can crowdsource knowledge curation for niche and specialized
topics, as the advertising network can automatically identify users
with the desired expertise and interest in the given topic. We present
controlled experiments that examine the effect of various incentive
mechanisms, highlighting the need for having short-term rewards as
goals, which incentivize the users to contribute. Finally, our
cost-quality analysis indicates that the cost of our approach is below
that of hiring workers through paid-crowdsourcing platforms, while
offering the additional advantage of giving access to billions of
potential users all over the planet, and being able to reach users
with specialized expertise that is not typically available through
existing labor marketplaces.
assesses the knowledge of users and acquires new knowledge from them.
Quizz operates by asking users to complete short quizzes on specific
topics; as a user answers the quiz questions, Quizz estimates the
user’s competence. To acquire new knowledge, Quizz also incorporates
questions for which we do not have a known answer; the answers given
by competent users provide useful signals for selecting the correct
answers for these questions. Quizz actively tries to identify
knowledgeable users on the Internet by running advertising campaigns,
effectively leveraging “for free” the targeting capabilities of
existing, publicly available, ad placement services. Quizz quantifies
the contributions of the users using information theory and sends
feedback to the advertising system about each user. The feedback
allows the ad targeting mechanism to further optimize ad placement.
Our experiments, which involve over ten thousand users, confirm that
we can crowdsource knowledge curation for niche and specialized
topics, as the advertising network can automatically identify users
with the desired expertise and interest in the given topic. We present
controlled experiments that examine the effect of various incentive
mechanisms, highlighting the need for having short-term rewards as
goals, which incentivize the users to contribute. Finally, our
cost-quality analysis indicates that the cost of our approach is below
that of hiring workers through paid-crowdsourcing platforms, while
offering the additional advantage of giving access to billions of
potential users all over the planet, and being able to reach users
with specialized expertise that is not typically available through
existing labor marketplaces.
Practical information
- General public
- Free