Harini Alagarai Sampath
ABSTRACT
Recent research in human computation has focused on improving the quality of work done by crowd workers on crowdsourcing platforms. Multiple approaches have been adopted like filtering crowd workers through qualification tasks, and aggregating responses from multiple crowd workers to obtain consensus. We investigate here how improving the presentation of the task itself by using cognitively inspired features affects the performance of crowd workers. We illustrate this with a case-study for the task of extracting text from scanned images. We generated six task-presentation designs by modifying two parameters - visual saliency of the target fields and working memory requirements - and conducted experiments on Amazon Mechanical Turk (AMT) and with an eye-tracker in the lab setting. Our results identify which task-design parameters (e.g. highlighting target fields) result in improved performance, and which ones do not (e.g. reducing the number of distractors). In conclusion, we claim that the use of cognitively inspired features for task design is a powerful technique for maximizing the performance of crowd workers.
Supplemental Material
- Baddeley, A. Working memory: Looking back and looking forward. Nature Reviews Neuroscience 4, 10 (2003), 829--839.Google Scholar
Cross Ref
- Ballard, D. H., Hayhoe, M. M., Pook, P. K., and Rao, R. P. Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences 20, 04 (1997), 723--742.Google ScholarCross Ref
- Bates, D., Maechler, M., and Bolker, B. lme4: Linear mixed-effects models using s4 classes.Google Scholar
- Biewald, L., and Van Pelt, C. Distributing a task to multiple workers over a network for completion while providing quality control, 2011. WO Patent 2,011,159,434.Google Scholar
- Chen, K., Kannan, A., Yano, Y., Hellerstein, J. M., and Parikh, T. S. Shreddr: pipelined paper digitization for low-resource organizations. In Proceedings of the 2nd ACM Symposium on Computing for Development, ACM (2012), 3. Google ScholarDigital Library
- Clark, A. Supersizing the Mind: Embodiment, Action, and Cognitive Extension: Embodiment, Action, and Cognitive Extension. Oxford University Press, USA, 2008.Google Scholar
- Deshpande, Y. Adaptivity and Interface Design: A Human-Computer Interaction Study in E-Learning Applications. PhD thesis, Indian Institute of Technology, Guwahati, January 2014.Google Scholar
- Fisher, D. L., and Tan, K. C. Visual displays: The highlighting paradox. Human Factors: The Journal of the Human Factors and Ergonomics Society 31, 1 (1989), 17--30. Google ScholarDigital Library
- Gamow, G., and Stern, M. Puzzle-math. Macmillan, 1958.Google Scholar
- Gray, W. D., Sims, C. R., Fu, W.-T., and Schoelles, M. J. The soft constraints hypothesis: a rational analysis approach to resource allocation for interactive behavior. Psychological review 113, 3 (2006), 461.Google Scholar
- Green, S., Heer, J., and Manning, C. D. The efficacy of human post-editing for language translation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2013), 439--448. Google ScholarDigital Library
- Heer, J., and Bostock, M. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2010), 203--212. Google ScholarDigital Library
- Huang, E., Zhang, H., Parkes, D. C., Gajos, K. Z., and Chen, Y. Toward automatic task design: A progress report. In Proceedings of the ACM SIGKDD workshop on human computation, ACM (2010), 77--85. Google ScholarDigital Library
- Ipeirotis, P. G., Provost, F., and Wang, J. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, ACM (2010), 64--67. Google ScholarDigital Library
- Kittur, A., Chi, E. H., and Suh, B. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, ACM (2008), 453--456. Google ScholarDigital Library
- Komarov, S., Reinecke, K., and Gajos, K. Z. Crowdsourcing performance evaluations of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2013), 207--216. Google ScholarDigital Library
- Korf, R. E. Toward a model of representation changes. Artificial intelligence 14, 1 (1980), 41--78.Google Scholar
- Landauer, T. K. Latent semantic analysis. Encyclopedia of Cognitive Science (2006).Google Scholar
- Law, E., and Ahn, L. v. Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning 5, 3 (2011), 1--121. Google ScholarDigital Library
- Little, G., Chilton, L. B., Goldman, M., and Miller, R. C. Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD workshop on human computation, ACM (2010), 68--76. Google ScholarDigital Library
- Mason, W., and Watts, D. J. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter 11, 2 (2010), 100--108. Google ScholarDigital Library
- Miller, G. The magical number seven, plus or minus two: Some limits on our capacity for processing information. The psychological review 63 (1956), 81--97.Google Scholar
- Morris, R., Dontcheva, M., and Gerber, E. Priming for better performance in microtask crowdsourcing environments. Internet Computing, IEEE 16, 5 (2012), 13--19. Google ScholarDigital Library
- Olesen, P. J., Westerberg, H., and Klingberg, T. Increased prefrontal and parietal activity after training of working memory. Nature neuroscience 7, 1 (2003), 75--79.Google Scholar
- Paolacci, G., Chandler, J., and Ipeirotis, P. Running experiments on amazon mechanical turk. Judgment and Decision Making 5, 5 (2010), 411--419.Google ScholarCross Ref
- Parent, G., and Eskenazi, M. Speaking to the crowd: Looking at past achievements in using crowdsourcing for speech and predicting future challenges. In INTERSPEECH (2011), 3037--3040.Google Scholar
- Toomim, M. Economic utility of interaction in crowdsourcing. In Workshop on Crowdsourcing and Human Computation at CHI (2011).Google Scholar
- Treisman, A. Perceptual grouping and attention in visual search for features and for objects. Journal of Experimental Psychology: Human Perception and Performance 8, 2 (1982), 194--214.Google ScholarCross Ref
- Wickens, C. D., Ambinder, M. S., Alexander, A. L., and Martens, M. The role of highlighting in visual search through maps. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 48 (2004), 1895--1899.Google ScholarCross Ref
- Wolfe, J. M. Guided search 2.0 a revised model of visual search. Psychonomic bulletin and review 1, 2 (1994), 202--238.Google Scholar
Index Terms
- Cognitively inspired task design to improve user performance on crowdsourcing platforms
Recommendations
The Influences of Task Design on Crowdsourced Judgement: A Case Study of Recidivism Risk Evaluation
WWW '22: Proceedings of the ACM Web Conference 2022Crowdsourcing is widely used to solicit judgement from people in diverse applications ranging from evaluating information quality to rating gig worker performance. To encourage the crowd to put in genuine effort in the judgement tasks, various ways to ...
Software Crowdsourcing Design: An Experiment on the Relationship Between Task Design and Crowdsourcing Performance
HCI International 2020 – Late Breaking Papers: Interaction, Knowledge and Social MediaAbstractSoftware crowdsourcing platforms allow for wide task accessibility and self-selection, which require participants to interact with a wide range of task options. This increased accessibility enables crowd workers to freely choose tasks based on ...
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsUnderstanding perception is critical to effective visualization design. With its low cost and scalability, crowdsourcing presents an attractive option for evaluating the large design space of visualizations; however, it first requires validation. In ...
Comments