Liwei Chan
Yi-Chi Liao
John J Dudley
Per Ola Kristensson
Antti Oulasvirta
ABSTRACT
Designers reportedly struggle with design optimization tasks where they are asked to find a combination of design parameters that maximizes a given set of objectives. In HCI, design optimization problems are often exceedingly complex, involving multiple objectives and expensive empirical evaluations. Model-based computational design algorithms assist designers by generating design examples during design, however they assume a model of the interaction domain. Black box methods for assistance, on the other hand, can work with any design problem. However, virtually all empirical studies of this human-in-the-loop approach have been carried out by either researchers or end-users. The question stands out if such methods can help designers in realistic tasks. In this paper, we study Bayesian optimization as an algorithmic method to guide the design optimization process. It operates by proposing to a designer which design candidate to try next, given previous observations. We report observations from a comparative study with 40 novice designers who were tasked to optimize a complex 3D touch interaction technique. The optimizer helped designers explore larger proportions of the design space and arrive at a better solution, however they reported lower agency and expressiveness. Designers guided by an optimizer reported lower mental effort but also felt less creative and less in charge of the progress. We conclude that human-in-the-loop optimization can support novice designers in cases where agency is not critical.
Supplemental Material
- Marine Agogué, Nicolas Poirel, Arlette Pineau, Olivier Houdé, and Mathieu Cassotti. 2014. The impact of age and training on creativity: A design-theory approach to study fixation effects. Thinking Skills and Creativity 11 (2014), 33–41.Google Scholar
Cross Ref
- Ferran Argelaguet Sanz and Carlos Andujar. 2013. A Survey of 3D Object Selection Techniques for Virtual Environments. Computers and Graphics 37, 3 (May 2013), 121–136. https://doi.org/10.1016/j.cag.2012.12.003Google ScholarDigital Library
- Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sabrina Hoppe. 2013. MenuOptimizer: interactive optimization of menu systems. In Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST ’13. ACM Press, St. Andrews, Scotland, United Kingdom, 331–342. https://doi.org/10.1145/2501988.2502024Google ScholarDigital Library
- Xiaojun Bi, Barton A. Smith, and Shumin Zhai. 2010. Quasi-Qwerty Soft Keyboard Optimization. Association for Computing Machinery, New York, NY, USA, 283–286. https://doi.org/10.1145/1753326.1753367Google ScholarDigital Library
- Doug A. Bowman, Donald B. Johnson, and Larry F. Hodges. 1999. Testbed Evaluation of Virtual Environment Interaction Techniques. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (London, United Kingdom) (VRST ’99). Association for Computing Machinery, New York, NY, USA, 26–33. https://doi.org/10.1145/323663.323667Google ScholarDigital Library
- Eric Brochu, Tyson Brochu, and Nando de Freitas. 2010. A Bayesian interactive optimization approach to procedural animation design. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 103–112.Google ScholarDigital Library
- Géry Casiez and Nicolas Roussel. 2011. No More Bricolage! Methods and Tools to Characterize, Replicate and Compare Pointing Transfer Functions. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, USA) (UIST ’11). Association for Computing Machinery, New York, NY, USA, 603–614. https://doi.org/10.1145/2047196.2047276Google ScholarDigital Library
- Yeonjoo Cha and Rohae Myung. 2013. Extended Fitts’ law for 3D pointing tasks using 3D target arrangements. International Journal of Industrial Ergonomics 43, 4(2013), 350 – 355. https://doi.org/10.1016/j.ergon.2013.05.005Google ScholarCross Ref
- Xiang ’Anthony’ Chen, Ye Tao, Guanyun Wang, Runchang Kang, Tovi Grossman, Stelian Coros, and Scott E. Hudson. 2018. Forte: User-Driven Generative Design. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3174070Google ScholarDigital Library
- Erin Cherry and Celine Latulipe. 2014. Quantifying the Creativity Support of Digital Tools through the Creativity Support Index. ACM Trans. Comput.-Hum. Interact. 21, 4, Article 21 (June 2014), 25 pages. https://doi.org/10.1145/2617588Google ScholarDigital Library
- Shanna R Daly, Robin S Adams, and George M Bodner. 2012. What does it mean to design? A qualitative investigation of design professionals’ experiences. Journal of Engineering Education 101, 2 (2012), 187–219.Google ScholarCross Ref
- Alena Denisova and Paul Cairns. 2015. Adaptation in Digital Games: The Effect of Challenge Adjustment on Player Performance and Experience. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (London, United Kingdom) (CHI PLAY ’15). Association for Computing Machinery, New York, NY, USA, 97–101. https://doi.org/10.1145/2793107.2793141Google ScholarDigital Library
- Kees Dorst. 2004. On the problem of design problems-problem solving and design expertise. Journal of design research 4, 2 (2004), 185–196.Google ScholarCross Ref
- Peitong Duan, Casimir Wierzynski, and Lama Nachman. 2020. Optimizing User Interface Layouts via Gradient Descent. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376589Google ScholarDigital Library
- John J. Dudley, Jason T. Jacques, and Per Ola Kristensson. 2019. Crowdsourcing Interface Feature Design with Bayesian Optimization. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems(CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300482Google ScholarDigital Library
- John J. Dudley and Per Ola Kristensson. 2018. A Review of User Interface Design for Interactive Machine Learning. ACM Trans. Interact. Intell. Syst. 8, 2, Article 8 (June 2018), 37 pages. https://doi.org/10.1145/3185517Google ScholarDigital Library
- Mark Dunlop and John Levine. 2012. Multidimensional pareto optimization of touchscreen keyboards for speed, familiarity and improved spell checking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’12). Association for Computing Machinery, New York, NY, USA, 2669–2678. https://doi.org/10.1145/2207676.2208659Google ScholarDigital Library
- Anna Maria Feit, Shane Williams, Arturo Toledo, Ann Paradiso, Harish Kulkarni, Shaun Kane, and Meredith Ringel Morris. 2017. Toward Everyday Gaze Input: Accuracy and Precision of Eye Tracking and Implications for Design. Association for Computing Machinery, New York, NY, USA, 1118–1130. https://doi.org/10.1145/3025453.3025599Google ScholarDigital Library
- Gregory Francis. 2000. Designing multifunction displays: An optimization approach. International Journal of Cognitive Ergonomics 4, 2 (2000), 107–124.Google ScholarCross Ref
- Scott Frees, G Drew Kessler, and Edwin Kay. 2007. PRISM interaction for enhancing control in immersive virtual environments. ACM Transactions on Computer-Human Interaction (TOCHI) 14, 1(2007), 2–es.Google ScholarDigital Library
- Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: Automatically Generating User Interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces (Funchal, Madeira, Portugal) (IUI ’04). Association for Computing Machinery, New York, NY, USA, 93–100. https://doi.org/10.1145/964442.964461Google ScholarDigital Library
- Katerina Gorkovenko, Daniel J Burnett, James K Thorp, Daniel Richards, and Dave Murray-Rust. 2020. Exploring the future of data-driven product design. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Guang-Zhong Yang. 2019. XAI—Explainable artificial intelligence. Science Robotics (2019).Google Scholar
- Shunan Guo, Zhuochen Jin, Fuling Sun, Jingwen Li, Zhaorui Li, Yang Shi, and Nan Cao. 2021. Vinci: An Intelligent Graphic Design System for Generating Advertising Posters. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445117Google ScholarDigital Library
- Matthew Guzdial, Nicholas Liao, Jonathan Chen, Shao-Yu Chen, Shukan Shah, Vishwa Shah, Joshua Reno, Gillian Smith, and Mark O. Riedl. 2019. Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300854Google ScholarDigital Library
- Gregory M Hallihan, Hyunmin Cheong, and LH Shu. 2012. Confirmation and cognitive bias in design cognition. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 45066. American Society of Mechanical Engineers, 913–924.Google ScholarCross Ref
- Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9Google ScholarCross Ref
- Vincent Hayward, Jehangir Choksi, Gonzalo Lanvin, and Christophe Ramstein. 1994. Design and Multi-Objective Optimization of a Linkage for a Haptic Interface. In Advances in Robot Kinematics and Computational Geometry, Jadran Lenarčič and Bahram Ravani (Eds.). Springer Netherlands, Dordrecht, 359–368. https://doi.org/10.1007/978-94-015-8348-0_36Google ScholarCross Ref
- Daniel Hernandez-Lobato, Jose Hernandez-Lobato, Amar Shah, and Ryan Adams. 2016. Predictive Entropy Search for Multi-objective Bayesian Optimization. In International Conference on Machine Learning. PMLR, 1492–1501. http://proceedings.mlr.press/v48/hernandez-lobatoa16.html ISSN: 1938-7228.Google Scholar
- David G Jansson and Steven M Smith. 1991. Design fixation. Design studies 12, 1 (1991), 3–11.Google Scholar
- Florian Kadner, Yannik Keller, and Constantin Rothkopf. 2021. AdaptiFont: Increasing Individuals’ Reading Speed with a Generative Font Model and Bayesian Optimization. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 585, 11 pages. https://doi.org/10.1145/3411764.3445140Google ScholarDigital Library
- Ashish Kapoor, Bongshin Lee, Desney Tan, and Eric Horvitz. 2010. Interactive Optimization for Steering Machine Classification. Association for Computing Machinery, New York, NY, USA, 1343–1352. https://doi.org/10.1145/1753326.1753529Google ScholarDigital Library
- Mohammad M. Khajah, Brett D. Roads, Robert V. Lindsey, Yun-En Liu, and Michael C. Mozer. 2016. Designing Engaging Games Using Bayesian Optimization. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(CHI ’16). Association for Computing Machinery, New York, NY, USA, 5571–5582. https://doi.org/10.1145/2858036.2858253Google ScholarDigital Library
- J. Knowles. 2006. ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation 10, 1 (Feb. 2006), 50–66. https://doi.org/10.1109/TEVC.2005.851274 Conference Name: IEEE Transactions on Evolutionary Computation.Google ScholarDigital Library
- Werner A König, Jens Gerken, Stefan Dierdorf, and Harald Reiterer. 2009. Adaptive pointing–design and evaluation of a precision enhancing technique for absolute pointing devices. In IFIP Conference on Human-Computer Interaction. Springer, 658–671.Google ScholarDigital Library
- Yuki Koyama, Issei Sato, and Masataka Goto. 2020. Sequential gallery for interactive visual design optimization. ACM Transactions on Graphics 39, 4 (July 2020), 88:88:1–88:88:12. https://doi.org/10.1145/3386569.3392444Google ScholarDigital Library
- Yuki Koyama, Issei Sato, Daisuke Sakamoto, and Takeo Igarashi. 2017. Sequential line search for efficient visual design optimization by crowds. ACM Transactions on Graphics 36, 4 (July 2017), 48:1–48:11. https://doi.org/10.1145/3072959.3073598Google ScholarDigital Library
- Daniel Lange, Tim Claudius Stratmann, Uwe Gruenefeld, and Susanne Boll. 2020. HiveFive: Immersion Preserving Attention Guidance in Virtual Reality. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376803Google ScholarDigital Library
- Yang Li, Samy Bengio, and Gilles Bailly. 2018. Predicting Human Performance in Vertical Menu Selection Using Deep Learning. Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/3173574.3173603Google ScholarDigital Library
- Antonios Liapis, Gillian Smith, and Noor Shaker. 2016. Mixed-initiative Content Creation. In Procedural Content Generation in Games: A Textbook and an Overview of Current Research, Noor Shaker, Julian Togelius, and Mark J. Nelson (Eds.). Springer, 195–214.Google Scholar
- J. Derek Lomas, Jodi Forlizzi, Nikhil Poonwala, Nirmal Patel, Sharan Shodhan, Kishan Patel, Ken Koedinger, and Emma Brunskill. 2016. Interface Design Optimization as a Multi-Armed Bandit Problem. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 4142–4153. https://doi.org/10.1145/2858036.2858425Google ScholarDigital Library
- Shouichi Matsui and Seiji Yamada. 2008. Genetic Algorithm Can Optimize Hierarchical Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 1385–1388. https://doi.org/10.1145/1357054.1357271Google ScholarDigital Library
- David E Meyer, Richard A Abrams, Sylvan Kornblum, Charles E Wright, and JE Keith Smith. 1988. Optimality in human motor performance: ideal control of rapid aimed movements.Psychological review 95, 3 (1988), 340.Google Scholar
- Brad A. Myers and William Buxton. 1986. Creating Highly-Interactive and Graphical User Interfaces by Demonstration. In Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’86). Association for Computing Machinery, New York, NY, USA, 249–258. https://doi.org/10.1145/15922.15914Google ScholarDigital Library
- Mathieu Nancel, Emmanuel Pietriga, Olivier Chapuis, and Michel Beaudouin-Lafon. 2015. Mid-Air Pointing on Ultra-Walls. ACM Trans. Comput.-Hum. Interact. 22, 5, Article 21 (Aug. 2015), 62 pages. https://doi.org/10.1145/2766448Google ScholarDigital Library
- Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. DesignScape: Design with Interactive Layout Suggestions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems(CHI ’15). ACM, New York, NY, USA, 1221–1224. https://doi.org/10.1145/2702123.2702149Google ScholarDigital Library
- Antti Oulasvirta, Niraj Ramesh Dayama, Morteza Shiripour, Maximilian John, and Andreas Karrenbauer. 2020. Combinatorial optimization of graphical user interface designs. Proc. IEEE 108, 3 (2020), 434–464.Google ScholarCross Ref
- Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst, and Tadao Ichikawa. 1996. The Go-Go Interaction Technique: Non-Linear Mapping for Direct Manipulation in VR. In Proceedings of the 9th Annual ACM Symposium on User Interface Software and Technology(Seattle, Washington, USA) (UIST ’96). Association for Computing Machinery, New York, NY, USA, 79–80. https://doi.org/10.1145/237091.237102Google ScholarDigital Library
- IVAN POUPYREV and TADAO ICHIKAWA. 1999. Manipulating Objects in Virtual Worlds: Categorization and Empirical Evaluation of Interaction Techniques. Journal of Visual Languages and Computing 10, 1 (1999), 19 – 35. https://doi.org/10.1006/jvlc.1998.0112Google ScholarDigital Library
- Joon Gi Shin, Doheon Kim, Chaehan So, and Daniel Saakes. 2020. Body Follows Eye: Unobtrusive Posture Manipulation Through a Dynamic Content Position in Virtual Reality. Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376794Google ScholarDigital Library
- Srinath Sridhar, Anna Maria Feit, Christian Theobalt, and Antti Oulasvirta. 2015. Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI ’15. ACM Press, Seoul, Republic of Korea, 3643–3652. https://doi.org/10.1145/2702123.2702136Google ScholarDigital Library
- Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and Explore with a Layout Optimiser. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems(DIS ’16). ACM, New York, NY, USA, 543–555. https://doi.org/10.1145/2901790.2901817Google ScholarDigital Library
- Johann Wentzel, Greg d’Eon, and Daniel Vogel. 2020. Improving Virtual Reality Ergonomics Through Reach-Bounded Non-Linear Input Amplification. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376687Google ScholarDigital Library
- Georgios N. Yannakakis and John Hallam. 2008. Real-time adaptation of augmented-reality games for optimizing player satisfaction. In 2008 IEEE Symposium On Computational Intelligence and Games. 103–110. https://doi.org/10.1109/CIG.2008.5035627Google ScholarCross Ref
- Georgios N Yannakakis, Pieter Spronck, Daniele Loiacono, and Elisabeth André. 2013. Player modeling. (2013).Google Scholar
- Robert J. Youmans and Thomaz Arciszewski. 2014. Design fixation: Classifications and modern methods of prevention. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 28, 2 (2014), 129–137. https://doi.org/10.1017/S0890060414000043Google ScholarCross Ref
- Jaesik Yun, Youn-kyung Lim, Kee-Eung Kim, and Seokyoung Song. 2015. Interactivity Crafter: An Interactive Input-Output Transfer Function Design Tool for Interaction Designers. Archives of Design Research 28 (08 2015), 21–37. https://doi.org/10.15187/adr.2015.08.28.3.21Google ScholarCross Ref
- Shumin Zhai, Michael Hunter, and Barton A. Smith. 2000. The Metropolis Keyboard - an Exploration of Quantitative Techniques for Virtual Keyboard Design. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (San Diego, California, USA) (UIST ’00). Association for Computing Machinery, New York, NY, USA, 119–128. https://doi.org/10.1145/354401.354424Google ScholarDigital Library
- Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021. GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 748–760. https://doi.org/10.1109/ICSE43902.2021.00074Google ScholarDigital Library
- Yijun Zhou, Yuki Koyama, Masataka Goto, and Takeo Igarashi. 2021. Interactive Exploration-Exploitation Balancing for Generative Melody Composition. In 26th International Conference on Intelligent User Interfaces (College Station, TX, USA) (IUI ’21). Association for Computing Machinery, New York, NY, USA, 43–47. https://doi.org/10.1145/3397481.3450663Google ScholarDigital Library
Index Terms
- Investigating Positive and Negative Qualities of Human-in-the-Loop Optimization for Designing Interaction Techniques
Recommendations
Practical approaches to group-level multi-objective Bayesian optimization in interaction technique design
Designing interaction techniques for end-users often involves exploring vast design spaces while balancing many objectives. Bayesian optimization offers a principled human-in-the-loop method for selecting designs for evaluation to efficiently explore such ...
Cooperative Multi-Objective Bayesian Design Optimization
Computational methods can potentially facilitate user interface design by complementing designer intuition, prior experience, and personal preference. Framing a user interface design task as a multi-objective optimization problem can help with ...
User-centered design, activity-centered design, and goal-directed design: a review of three methods for designing web applications
SIGDOC '09: Proceedings of the 27th ACM international conference on Design of communicationWhen conducting research with users in order to design web applications, the practitioner has a variety of methods from which to choose. This paper examines three such methods'User-Centered Design (UCD), Goal-Directed Design (GDD), and Activity-Centered ...
Comments