Sang Won Lee
Skip Abstract Section
Abstract
Real-time collaboration between a requester and crowd workers expands the scope of tasks that crowdsourcing can be used for by letting requesters and crowd workers interactively create various artifacts (e.g., a sketch prototype, writing, or program code). In such systems, it is increasingly common to allow requesters to verbally describe their requests, receive responses from workers, and provide immediate and continuous feedback to enhance the overall outcome of the two groups' real-time collaboration. This work is motivated by the lack of a deep understanding of the challenges that end users of such systems face in their communication with workers and the need of design implications that can address such challenges for other similar systems. In this paper, we investigate how requesters verbally communicate and collaborate with crowd workers to solve a complex task. Using a crowd-powered UI design tool, we conducted a qualitative user study to explore how requesters with varying expertise communicate and collaborate with crowd workers. Our work also identifies the unique challenges that collaborative crowdsourcing systems pose: potential expertise differences between requesters and crowd workers, the asymmetry of two-way communication (e.g., speech versus text), and the shared artifact's concurrent modification by two disparate groups. Finally, we make design recommendations that can inform the design of future real-time collaboration processes in crowdsourcing systems.
- 2017. Amazon's Mechanical Turk. http://www.mturk.com.Google Scholar
- 2017. Voice Qualities. http://www.ncvs.org/ncvs/tutorials/voiceprod/tutorial/quality.html.Google Scholar
- Salvatore Andolina, Hendrik Schneider, Joel Chan, Khalil Klouche, Giulio Jacucci, and Steven Dow. 2017. Crowdboard:Augmenting In-Person Idea Generation with Real-Time Crowds. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (C&C '17). ACM, New York, NY, USA, 106--118. Google ScholarDigital Library
- Adam Bergkvist, Daniel C Burnett, Cullen Jennings, and Anant Narayanan. 2012. Webrtc 1.0: Real-time communication between browsers. Working draft, W3C 91 (2012).Google Scholar
- Michael S. Bernstein, Joel Brandt, Robert C. Miller, and David R. Karger. 2011. Crowds in Two Seconds: Enabling Realtime Crowd-powered Interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 33--42. Google ScholarDigital Library
- Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell ,and Katrina Panovich. 2015. Soylent: A Word Processor with a Crowd Inside. Commun. ACM 58, 8 (July 2015), 85--94. Google ScholarDigital Library
- Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. 2010. VizWiz: nearly real-time answers to visual questions .In User Interface Software and Technology (UIST). 333--342. Google ScholarDigital Library
- Yan Chen, Sang Won Lee, Yin Xie, Yiwei Yang, Walter S. Lasecki, and Steve Oney. 2017. Codeon: On-Demand Software Development Assistance. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 6220--6231. Google ScholarDigital Library
- Domenic V Cicchetti. 1994. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. In Psychological Assessment, 6(4). 284--290.Google ScholarCross Ref
- Herbert H Clark, Susan E Brennan, et al. 1991. Grounding in communication. Perspectives on socially shared cognition13, 1991 (1991), 127--149.Google Scholar
- Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the Crowd Yields Better Work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY,USA, 1013--1022. Google ScholarDigital Library
- Clarence A. Ellis, Simon J. Gibbs, and Gail Rein. 1991. Groupware: Some Issues and Experiences. Commun. ACM 34, 1(Jan. 1991), 39--58. Google ScholarDigital Library
- Susan R. Fussell, Robert E. Kraut, and Jane Siegel. 2000. Coordination of Communication: Effects of Shared Visual Context on Collaborative Work. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00). ACM, New York, NY, USA, 21--30. Google ScholarDigital Library
- Darren Gergle, Robert E. Kraut, and Susan R. Fussell. 2004. Action As Language in a Shared Visual Space. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW '04). ACM, New York, NY, USA, 487--496. Google ScholarDigital Library
- Darren Gergle, Robert E. Kraut, and Susan R. Fussell. 2004. Language Efficiency and Visual Technology: Minimizing Collaborative Effort with Visual Information. Journal of Language and Social Psychology 23, 4 (2004), 491--517. arXiv:https://doi.org/10.1177/0261927X04269589Google ScholarCross Ref
- Mitchell Gordon, Jeffrey P. Bigham, andWalter S. Lasecki. 2015. LegionTools: A Toolkit + UI for Recruiting and Routing Crowds to Synchronous Real-Time Tasks. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15 Adjunct). ACM, New York, NY, USA, 81--82. Google ScholarDigital Library
- Sai R. Gouravajhala, Jinyeong Yim, Karthik Desingh, Yanda Huang, Odest Chadwicke Jenkins, and Walter S. Lasecki.2018. EURECA: Enhanced Understanding of Real Environments via Crowd Assistance. In Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018, Zürich, Switzerland, July 5--8, 2018. 31--40. https://aaai.org/ocs/index.php/HCOMP/HCOMP18/paper/view/17933Google Scholar
- Saul Greenberg and David Marwood. 1994. Real Time Groupware As a Distributed System: Concurrency Control and Its Effect on the Interface. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work (CSCW'94). ACM, New York, NY, USA, 207--217. Google ScholarDigital Library
- Carl Gutwin. 2001. The effects of network delays on group work in real-time groupware. In ECSCW 2001. Springer,299--318. Google ScholarDigital Library
- Carl Gutwin and Saul Greenberg. 2002. A descriptive framework of workspace awareness for real-time groupware. Computer Supported Cooperative Work (CSCW) 11, 3--4 (2002), 411--446. Google ScholarDigital Library
- Carl Gutwin and Reagan Penner. 2002. Improving Interpretation of Remote Gestures with Telepointer Traces. In Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work (CSCW '02). ACM, New York, NY,USA, 49--57. Google ScholarDigital Library
- Oliver L. Haimson and John C. Tang. 2017. What Makes Live Events Engaging on Facebook Live, Periscope, and Snapchat. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 48--60. Google ScholarDigital Library
- William A. Hamilton, Oliver Garretson, and Andruid Kerne. 2014. Streaming on Twitch: Fostering Participatory Communities of Play Within Live Mixed Media. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 1315--1324. Google ScholarDigital Library
- Pamela J Hinds. 1999. The curse of expertise: The effects of expertise and debiasing methods on prediction of novice performance. Journal of Experimental Psychology: Applied 5, 2 (1999), 205.Google ScholarCross Ref
- Pamela J Hinds, Michael Patterson, and Jeffrey Pfeffer. 2001. Bothered by abstraction: The effect of expertise on knowledge transfer and subsequent novice performance. Journal of applied psychology 86, 6 (2001), 1232.Google ScholarCross Ref
- Simo Hosio, Jorge Goncalves, Niels van Berkel, Simon Klakegg, Shin Ichi Konomi, and Vassilis Kostakos. 2017. Facilitating Collocated Crowdsourcing on Situated Displays. Human-Computer Interaction 0, 0 (2017), 1--37. arXiv:https://doi.org/10.1080/07370024.2017.1344126Google Scholar
- Ting-Hao Kenneth Huang, Walter S. Lasecki, Amos Azaria, and Jeffrey P Bigham. 2016. "Is There Anything Else I Can Help You With?" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. In Fourth AAAI Conference on Human Computation and Crowdsourcing.Google Scholar
- Ellen A. Isaacs and Herbert H. Clark. 1987. References in conversation between experts and novices. Journal of experimental psychology: general 116, 1 (1987), 26.Google ScholarCross Ref
- Srinivasan Janarthanam and Oliver Lemon. 2010. Learning to adapt to unknown users: referring expression generation in spoken dialogue systems. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 69--78. Google ScholarDigital Library
- Laurent Karsenty. 1999. Cooperative work and shared visual context: An empirical study of comprehension problems in side-by-side and remote help dialogues. Human-Computer Interaction 14, 3 (1999), 283--315. Google ScholarDigital Library
- Harmanpreet Kaur, Mitchell Gordon, Yiwei Yang, Jeffrey P. Bigham, Jaime Teevan, Ece Kamar, and Walter S. Lasecki.2017. Crowdmask: Using crowds to preserve privacy in crowd-powered systems via progressive filtering. In Proceedings of the AAAI Conference on Human Computation (HCOMP 2017)., HCOMP, Vol. 17.Google Scholar
- Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 1301--1318. Google ScholarDigital Library
- Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. CrowdForge: Crowdsourcing Complex Work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 43--52. Google ScholarDigital Library
- Robert E. Kraut, Susan R. Fussell, and Jane Siegel. 2003. Visual Information as a Conversational Resource in Collaborative Physical Tasks. Human-Computer Interaction 18, 1--2 (2003), 13--49. Google ScholarDigital Library
- Walter S. Lasecki and Jeffrey P Bigham. 2013. Interactive crowds: Real-time crowdsourcing and crowd agents. In Handbook of human computation. Springer, 509--521.Google Scholar
- Walter S. Lasecki, Mitchell Gordon, Danai Koutra, Malte F. Jung, Steven P. Dow, and Jeffrey P. Bigham. 2014. Glance: Rapidly Coding Behavioral Video with the Crowd. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 551--562. Google ScholarDigital Library
- Walter S. Lasecki, Mitchell Gordon, Winnie Leung, Ellen Lim, Jeffrey P. Bigham, and Steven P. Dow. 2015. Exploring Privacy and Accuracy Trade-Offs in Crowdsourced Behavioral Video Coding. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1945--1954. Google ScholarDigital Library
- Walter S. Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P. Bigham, and Michael S. Bernstein. 2015. Apparition: Crowdsourced User Interfaces That Come to Life As You Sketch Them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1925--1934. Google ScholarDigital Library
- Walter S. Lasecki, Kyle I. Murray, Samuel White, Robert C. Miller, and Jeffrey P. Bigham. 2011. Real-time Crowd Control of Existing Interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology(UIST '11). ACM, New York, NY, USA, 23--32. Google ScholarDigital Library
- Walter S. Lasecki, Phyo Thiha, Yu Zhong, Erin Brady, and Jeffrey P. Bigham. 2013. Answering Visual Questions with Conversational Crowd Assistants. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '13). ACM, New York, NY, USA, Article 18, 8 pages. Google ScholarDigital Library
- Walter S. Lasecki, Rachel Wesley, Jeffrey Nichols, Anand Kulkarni, James F. Allen, and Jeffrey P. Bigham. 2013. Chorus:A Crowd-powered Conversational Assistant. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 151--162. Google ScholarDigital Library
- Matthew Lease, Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim,Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C Miller. 2013. Mechanical turk is not anonymous. (2013).Google Scholar
- Sang Won Lee. 2018. Improving User Involvement through live collaborative creation. Ph.D. Dissertation. University of Michigan, Ann Arbor, MI, USA.Google Scholar
- Sang Won Lee, Yan Chen, and Walter S Lasecki. 2017. The Need for Real-Time Crowd Generation of Task Lists from Speech. In Work-in-Progress, the AAAI Conference on Human Computation (HCOMP 2017).Google Scholar
- Sang Won Lee, Y. Zhang, I. Wong, Yang Y., S. O'Keefe, and W.S. Lasecki. 2017. SketchExpress: Remixing Animations For More Effective Crowd-Powered Prototyping Of Interactive Interfaces. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST). ACM. Google ScholarDigital Library
- Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2009. TurKit: Tools for Iterative Tasks on Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '09). ACM, New York, NY, USA,29--30. Google ScholarDigital Library
- Alan Lundgard, Yiwei Yang, Maya L. Foster, and Walter S. Lasecki. 2018. Bolt: Instantaneous Crowdsourcing via Just-in-Time Training. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18).ACM, New York, NY, USA, Article 467, 7 pages. Google ScholarDigital Library
- Meredith Ringel Morris, Kathy Ryall, Chia Shen, Clifton Forlines, and Frederic Vernier. 2004. Beyond "Social Protocols": Multi-user Coordination Policies for Co-located Groupware. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW '04). ACM, New York, NY, USA, 262--265. Google ScholarDigital Library
- Michael Nebeling, Alexandra To, Anhong Guo, Adrian A. de Freitas, Jaime Teevan, Steven P. Dow, and Jeffrey P. Bigham.2016. WearWrite: Crowd-Assisted Writing from Smartwatches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3834--3846. Google ScholarDigital Library
- Jakob Nielsen and Thomas K. Landauer. 1993. A Mathematical Model of the Finding of Usability Problems. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (CHI '93). ACM, New York, NY,USA, 206--213. Google ScholarDigital Library
- Sharon Oviatt, Phil Cohen, LizhongWu, Lisbeth Duncan, Bernhard Suhm, Josh Bers, Thomas Holzman, Terry Winograd, James Landay, Jim Larson, and David Ferro. 2000. Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions. Human-Computer Interaction 15, 4(2000), 263--322. Google ScholarDigital Library
- Daniel Recktenwald. 2017. Toward a transcription and analysis of live streaming on Twitch. Journal of Pragmatics 115(2017), 68 -- 81.Google ScholarCross Ref
- Fraser J. M. Reid and Susan E. Reed. 2007. Conversational grounding and visual access in collaborative design. CoDesign3, 2 (2007), 111--122.Google Scholar
- Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014. Expert Crowdsourcing with Flash Teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 75--85. Google ScholarDigital Library
- Niloufar Salehi, Andrew McCabe, Melissa Valentine, and Michael Bernstein. 2017. Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 1700--1713. Google ScholarDigital Library
- Niloufar Salehi, Jaime Teevan, Shamsi Iqbal, and Ece Kamar. 2017. Communicating Context to the Crowd for Complex Writing Tasks. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing(CSCW '17). ACM, New York, NY, USA, 1890--1901. Google ScholarDigital Library
- Stacey D. Scott, M. Sheelagh T. Carpendale, and Kori M. Inkpen. 2004. Territoriality in Collaborative Tabletop Workspaces. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW '04). ACM, New York, NY, USA, 294--303. Google ScholarDigital Library
- Anthony Tang, Michel Pahud, Kori Inkpen, Hrvoje Benko, John C. Tang, and Bill Buxton. 2010. Three's Company: Understanding Communication Channels in Three-way Distributed Collaboration. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (CSCW '10). ACM, New York, NY, USA, 271--280. Google ScholarDigital Library
- John C Tang. 1989. Listing, drawing and gesturing in design: A study of the use of shared workspaces by design teams. Ph.D. Thesis (1989).Google Scholar
- John C Tang. 1991. Findings from observational studies of collaborative work. International Journal of Man-machine studies 34, 2 (1991), 143--160. Google ScholarDigital Library
- Melissa A. Valentine, Daniela Retelny, Alexandra To, Negar Rahmati, Tulsee Doshi, and Michael S. Bernstein. 2017.Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3523--3537. Google ScholarDigital Library
- Alvin Yuan, Kurt Luther, Markus Krause, Sophie Isabel Vennix, Steven P Dow, and Bjorn Hartmann. 2016. Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. In Proceedings oft he 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York ,NY, USA, 1005--1017. Google ScholarDigital Library
Index Terms
- Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool
Recommendations
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
How many crowdsourced workers should a requester hire?
Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...
Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web
WWW '15 Companion: Proceedings of the 24th International Conference on World Wide WebWithin the scope of this PhD proposal, we set out to investigate two pivotal aspects that influence the effectiveness of crowdsourcing: (i) microtask design, and (ii) workers behavior. Leveraging the dynamics of tasks that are crowdsourced on the one ...
Comments