Anil Shankar
Honray Lin
Hans-Frederick Brown
ABSTRACT
Lab based user testing with participants quickly becomes a bottleneck for UX teams in the industry that exist in an Agile software development environment characterized by frequent release cycles and continually changing requirements. For such teams to reduce this testing time and to quickly glean usability insights we leverage human performance modeling via CogTool. Our work compares CogTool's expert user model's task time with actual user time from lab sessions in two user studies. In these two studies CogTool's task time estimates were statistically significantly lower when compared to lab based user times but the two task times were positively correlated. We leverage this correlation between CogTool task times and lab user times to build a predictive user model. Next, we apply this user model to rapidly evaluate two new designs without lab based user testing. Based on these results we provide recommendations for Agile UX teams to harness CogTool for enhancing user research efforts and thereby reduce the bottleneck of lab based user testing.
- Cogtool. http://cogtool.com. Accessed: 2014-09--30.Google Scholar
- Platfora. http://www.platfora.com. Accessed: 2014-09--30.Google Scholar
- Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R. C., Mallor, S., Shwaber, K., and Sutherland, J. The Agile Manifesto. 2001.Google Scholar
- Bellamy, R., John, B., and Kogan, S. Deploying cogtool: integrating quantitative usability assessment into real-world software development. Software Engineering, International Conference on (2011), 691--700. Google ScholarDigital Library
- Blackmon, M. H., Kitajima, M., and Polson, P. G. Tool for accurately predicting website navigation problems, non-problems, problem severity, and effectiveness of repairs. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '05, ACM (New York, NY, USA, 2005), 31--40. Google ScholarDigital Library
- Brooke, J. Sus: A quick and dirty usability scale, 1996.Google Scholar
- Card, S. K., Newell, A., and Moran, T. P. The Psychology of Human-Computer Interaction. L. Erlbaum Associates Inc., Hillsdale, NJ, USA, 1983. Google ScholarDigital Library
- Gothelf, J., and Seiden, J. Lean UX: Applying Lean Principles to Improve User Experience, 1st edition ed. No. ISBN-10: 1449311652. O'Reilly Media, 2013.Google Scholar
- John, B. E. Using predictive human performance models to inspire and support ui design recommendations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, ACM (New York, NY, USA, 2011), 983--986. Google ScholarDigital Library
- John, B. E., Prevas, K., Salvucci, D. D., and Koedinger, K. Predictive human performance modeling made easy. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '04, ACM (New York, NY, USA, 2004), 455--462. Google ScholarDigital Library
- Kuniavsky, M. Observing the User Experience: A Practitioner's Guide to User Research (Morgan Kaufmann Series in Interactive Technologies) (The Morgan Kaufmann Series in Interactive Technologies). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2003. Google ScholarDigital Library
- Luo, L., and John, B. E. Predicting task execution time on handheld devices using the keystroke-level model. In CHI '05 Extended Abstracts on Human Factors in Computing Systems, CHI EA '05, ACM (New York, NY, USA, 2005), 1605--1608. Google ScholarDigital Library
- Miller, C. S., and Remington, R. W. Modeling information navigation: Implications for information architecture. Hum.-Comput. Interact. 19, 3 (Sept. 2004), 225--271. Google ScholarDigital Library
- Montgomery, D. C., Peck, E. A., and Vining, G. G. Introduction to Linear Regression Analysis (4th ed.). Wiley & Sons, Hoboken, July 2006. Google ScholarDigital Library
- Oluwakemi, A. Assessing the accuracy of task time prediction of an emerging human performance modeling software - CogTool. PhD thesis, Louisiana State University, Baton Rouge, LA, USA, December 2013.Google Scholar
- Teo, L., and John, B. E. Cogtool-explorer: Towards a tool for predicting user interaction. In CHI '08 Extended Abstracts on Human Factors in Computing Systems, CHI EA '08, ACM (New York, NY, USA, 2008), 2793--2798. Google ScholarDigital Library
Index Terms
- Rapid Usability Assessment of an Enterprise Application in an Agile Environment with CogTool
Recommendations
Deploying CogTool: integrating quantitative usability assessment into real-world software development
ICSE '11: Proceedings of the 33rd International Conference on Software EngineeringUsability concerns are often difficult to integrate into real-world software development processes. To remedy this situation, IBM research and development, partnering with Carnegie Mellon University, has begun to employ a repeatable and quantifiable ...
Supporting cognitive models as users
Special issue on human-computer interaction in the new millennium, Part 2Cognitive models are computer programs that simulate human performance of cognitive skills. They have been useful to HCI by predicting task times, by assisting users, and by acting as surrogate users. If cognitive models could interact with the same ...
CogTool+: Modeling Human Performance at Large Scale
Cognitive modeling tools have been widely used by researchers and practitioners to help design, evaluate, and study computer user interfaces (UIs). Despite their usefulness, large-scale modeling tasks can still be very challenging due to the amount of ...
Comments