Adam Fourney
Richard Mann
ABSTRACT
People routinely rely on Internet search engines to support their use of interactive systems: they issue queries to learn how to accomplish tasks, troubleshoot problems, and otherwise educate themselves on products. Given this common behavior, we argue that search query logs can usefully augment traditional usability methods by revealing the primary tasks and needs of a product's user population. We term this use of search query logs CUTS - characterizing usability through search. In this paper, we introduce CUTS and describe an automated process for harvesting, ordering, labeling, filtering, and grouping search queries related to a given product. Importantly, this data set can be assembled in minutes, is timely, has a high degree of ecological validity, and is arguably less prone to self-selection bias than data gathered via traditional usability methods. We demonstrate the utility of this approach by applying it to a number of popular software and hardware systems.
Supplemental Material
Available for Download
- Akers, D., Simpson, M., Jeffries, R., and Winograd, T. Undo and erase events as indicators of usability problems. In Proc CHI '09, ACM (New York, NY, USA, 2009), 659--668. Google Scholar
Digital Library
- Aula, A., Khan, R. M., and Guan, Z. How does search behavior change as search becomes more difficult? In Proc CHI '10, ACM (New York, NY, USA, 2010), 35--44. Google ScholarDigital Library
- Baeza-Yates, R., and Tiberi, A. Extracting semantic relations from query logs. In Proc KDD '07, ACM (New York, NY, USA, 2007), 76--85. Google ScholarDigital Library
- Bar-Yossef, Z., and Gurevich, M. Mining search engine query logs via suggestion sampling. Proc. VLDB Endow. 1, 1 (2008), 54--65. Google ScholarDigital Library
- Barrett, R., Kandogan, E., Maglio, P. P., Haber, E. M., Takayama, L. A., and Prabaker, M. Field studies of computer system administrators: analysis of system management tools and practices. In Proc CSCW '04, ACM (New York, NY, USA, 2004), 388--395. Google ScholarDigital Library
- Brandt, J., Guo, P. J., Lewenstein, J., Dontcheva, M., and Klemmer, S. R. Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. In Proc CHI '09, ACM (New York, NY, USA, 2009), 1589--1598. Google ScholarDigital Library
- Broder, A. A taxonomy of web search. SIGIR Forum 36, 2 (2002), 3--10. Google ScholarDigital Library
- Dou, Z., Song, R., and Wen, J.-R. A large-scale evaluation and analysis of personalized search strategies. In Proc WWW '07, ACM (New York, NY, USA, 2007), 581--590. Google ScholarDigital Library
- Gikas, M. Lab tests: Why Consumer Reports can't recommend the iPhone 4. Consumer Reports (July 2010).Google Scholar
- Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., and Brilliant, L. Detecting influenza epidemics using search engine query data. Nature 457 (February 2009), 1012--1014.Google ScholarCross Ref
- Google Corperation. Features: Google suggest. http://www.google.com/support/websearch/bin/answer.py?hl=en\&answer=106230, 2010.Google Scholar
- Google Corperation. Google AdWords keyword tool. https://adwords.google.com/select/KeywordToolExternal?forceLegacy=true, 2010.Google Scholar
- Google Corperation. Google suggest : FAQ. https://labs.google.com/intl/en/suggestfaq.html, 2010.Google Scholar
- Hilbert, D. M., and Redmiles, D. F. Extracting usability information from user interface events. ACM Computing Surveys 32, 4 (2000), 384--421. Google ScholarDigital Library
- Huang, J., and Efthimiadis, E. N. Analyzing and evaluating query reformulation strategies in web search logs. In Proc CIKM '09, ACM (New York, NY, USA, 2009), 77--86. Google ScholarDigital Library
- Hurst, A., Hudson, S. E., and Mankoff, J. Dynamic detection of novice vs. skilled use without a task model. In Proc CHI '07, ACM (New York, NY, USA, 2007), 271--280. Google ScholarDigital Library
- Kellar, M., Watters, C., and Shepherd, M. A field study characterizing web-based information-seeking tasks. J. Am. Soc. Inf. Sci. Technol. 58, 7 (2007), 999--1018. Google ScholarDigital Library
- Lafreniere, B., Bunt, A., Whissell, J. S., Clarke, C. L. A., and Terry, M. Characterizing large-scale use of a direct manipulation application in the wild. In Proc GI '10 (Toronto, Canada, 2010), 11--18. Google ScholarDigital Library
- Landis, J. R., and Koch, G. G. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), pp. 159--174.Google ScholarCross Ref
- Miller, G. A. Wordnet: a lexical database for english. Commun. ACM 38 (November 1995), 39--41. Google ScholarDigital Library
- Nielsen, J., and Molich, R. Heuristic evaluation of user interfaces. In Proc CHI '90, ACM (New York, NY, USA, 1990), 249--256. Google ScholarDigital Library
- Pang, B., and Lee, L. Opinion mining and sentiment analysis. Found. Trends Inf. Retr. 2 (January 2008), 1--135. Google ScholarDigital Library
- Quesenbery, W., Jarrett, C., Roddis, I., Allen, S., and Stirling, V. Search Is Now Normal Behavior. What Do We Do about That. In UPA 2008 (Baltimore, Maryland, USA, June 2008).Google Scholar
- Richardson, M. Learning about the world through long-term query logs. ACM Trans. Web 2, 4 (2008), 1--27. Google ScholarDigital Library
- Rose, D. E., and Levinson, D. Understanding user goals in web search. In Proc WWW '04, ACM (New York, NY, USA, 2004), 13--19. Google ScholarDigital Library
- Saraiva, P. C., Silva de Moura, E., Ziviani, N., Meira, W., Fonseca, R., and Riberio-Neto, B. Rank-preserving two-level caching for scalable search engines. In Proc SIGIR '01, ACM (New York, NY, USA, 2001), 51--58. Google ScholarDigital Library
- Strauss, A., and Corbin, J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, 3rd edition ed. Sage Publications, 2008.Google Scholar
- Zeller, T. AOL technology chief quits after data release. The New York Times (August 2006).Google Scholar
Index Terms
- Characterizing the usability of interactive applications through query log analysis
Recommendations
When do people use query suggestion? A query suggestion log analysis
AbstractQuery suggestion, which enables the user to revise a query with a single click, has become one of the most fundamental features of Web search engines. However, it has not been clear what circumstances cause the user to turn to query suggestion. In ...
Evaluation of phrasal query suggestions
CIKM '07: Proceedings of the sixteenth ACM conference on Conference on information and knowledge managementThis paper evaluates the uptake and efficacy of a unified approach to phrasal query suggestions in the context of a high-precision search engine. The search engine performs ranked extended-Boolean searches with the proximity operator <scp>NEAR</scp> ...
Query suggestions in the absence of query logs
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information RetrievalAfter an end-user has partially input a query, intelligent search engines can suggest possible completions of the partial query to help end-users quickly express their information needs. All major web-search engines and most proposed methods that ...
Comments