Skip to main content
SpringerLink
Log in

Automatic query reformulation for code search using crowdsourced knowledge

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Traditional code search engines (e.g., Krugle) often do not perform well with natural language queries. They mostly apply keyword matching between query and source code. Hence, they need carefully designed queries containing references to relevant APIs for the code search. Unfortunately, preparing an effective search query is not only challenging but also time-consuming for the developers according to existing studies. In this article, we propose a novel query reformulation technique–RACK–that suggests a list of relevant API classes for a natural language query intended for code search. Our technique offers such suggestions by exploiting keyword-API associations from the questions and answers of Stack Overflow (i.e., crowdsourced knowledge). We first motivate our idea using an exploratory study with 19 standard Java API packages and 344K Java related posts from Stack Overflow. Experiments using 175 code search queries randomly chosen from three Java tutorial sites show that our technique recommends correct API classes within the Top-10 results for 83% of the queries, with 46% mean average precision and 54% recall, which are 66%, 79% and 87% higher respectively than that of the state-of-the-art. Reformulations using our suggested API classes improve 64% of the natural language queries and their overall accuracy improves by 19%. Comparisons with three state-of-the-art techniques demonstrate that RACK outperforms them in the query reformulation by a statistically significant margin. Investigation using three web/code search engines shows that our technique can significantly improve their results in the context of code search.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

Notes

  1. http://data.stackexchange.com/stackoverflow

  2. https://jsoup.org/

  3. https://goo.gl/A6gEqA

  4. https://code.google.com/p/reflections

  5. http://stats.stackexchange.com/questions/132652

  6. https://www.javatpoint.com

  7. https://www.tutorialspoint.com

  8. https://code.google.com/p/stop-words

  9. http://jsoup.org

  10. http://kodejava.org

  11. http://www.javadb.com

  12. http://java2s.com

  13. https://sourceforge.net/

  14. https://code.google.com/

  15. http://lucene.apache.org/

  16. http://homepage.usask.ca/~masud.rahman/rack/

  17. http://homepage.usask.ca/~masud.rahman/rack/

  18. https://lucene.apache.org/core

  19. https://lucene.apache.org/

  20. https://developers.google.com/custom-search

  21. https://developers.google.com/custom-search

  22. https://api.stackexchange.com

  23. https://developer.github.com/v3

  24. https://jsoup.org/

  25. https://github.com/javaparser

References

  • Bacchelli A, Lanza M, Robbes R (2010) Linking e-mails and source code artifacts. In: Proceedings ICSE, pp 375–384

  • Bajracharya S, Lopes C (2012a) Analyzing and mining a code search engine usage log. Empirical Softw. Engg. 17(4-5):424–466

    Article  Google Scholar 

  • Bajracharya S, Lopes C (2012b) Analyzing and mining a code search engine usage log. EMSE 17(4-5):424–466

    Google Scholar 

  • Bojanowski P, Grave E, Joulin A, Mikolov T (2016) Enriching word vectors with subword information. arXiv:1607.04606

  • Brandt J, Guo PJ, Lewenstein J, Dontcheva M, Klemmer SR (2009) Two Studies of opportunistic programming interleaving web foraging, learning, and writing code. In: Proceedings SIGCHI, pp 1589–1598

  • Brin S, Page L (1998) The anatomy of a large-scale hypertextual web search engine. Comput Netw ISDN Syst 30(1-7):107–117

    Article  Google Scholar 

  • Campbell BA, Treude C (2017) Nlp2code: code snippet content assist via natural language tasks. In: Proceedings ICSME, pp 628–632

  • Capobianco G, Lucia AD, Oliveto R, Panichella A, Panichella S (2013) Improving IR-based traceability recovery via noun-based indexing of software artifacts. J Softw Evol Process 25(7):743–762

    Article  Google Scholar 

  • Carmel D, Yom-Tov E (2010) Estimating the query difficulty for information retrieval. Morgan & Claypool, San Rafael

    MATH  Google Scholar 

  • Carmel D, Yom-Tov E, Darlow A, Pelleg D (2006) What makes a query difficult?. In: Proceedings SIGIR, pp 390–397

  • Chan W, Cheng H, Lo D (2012) Searching connected API subgraph via text phrases. In: Proceedings FSE, pp 10:1–10:11

  • Chaparro O, Florez JM, Marcus A (2017) Using observed behavior to reformulate queries during text retrieval-based bug localization. In: Proceedings ICSME, page to appear

  • Dagenais B, Robillard MP (2012) Recovering traceability links between an API and its learning resources. In: Proceedings ICSE, pp 47–57

  • Furnas GW, Landauer TK, Gomez LM, Dumais ST (1987) The vocabulary problem in human-system communication. Commun ACM 30(11):964–971

    Article  Google Scholar 

  • Gay G, Haiduc S, Marcus A, Menzies T (2009) On the use of relevance feedback in IR-based concept location. In: Proceedings ICSM, pp 351–360

  • Gosling J, Joy B, Steele G, Bracha G (2012) The java language specification: Java SE 7th edn

  • Gvero T, Kuncak V (2015) Interactive synthesis using free-form queries. In: Proceedings ICSE, pp 689–692

  • Haiduc S, Marcus A (2011) On the effect of the query in IR-based concept location. In: Proceedings ICPC, pp 234–237

  • Haiduc S, Bavota G, Marcus A, Oliveto R, De Lucia A, Menzies T (2013) Automatic query reformulations for text retrieval in software engineering. In: Proceedings ICSE, pp 842–851

  • Harris Z (1968) Mathematical structures in language contents

  • Hellendoorn VJ, Devanbu P (2017) Are deep neural networks the best choice for modeling source code?. In: Proceedings ESEC/FSE, pp 763–773

  • Hill E, Pollock L, Vijay-Shanker K (2009) Automatically capturing source code context of NL-queries for software maintenance and reuse. In: Proceedings ICSE, pp 232–242

  • Howard MJ, Gupta S, Pollock L, Vijay-Shanker K (2013) Automatically mining software-based semantically-similar words from comment-code mappings. In: Proceedings MSR, pp 377–386

  • Järvelin K., Kekäläinen J (2002) Cumulated gain-based evaluation of ir techniques. ACM Trans Inf Syst 20(4):422–446

    Article  Google Scholar 

  • Keivanloo I, Rilling J (2011) Internet-scale java source code data set. http://aseg.cs.concordia.ca/codesearch/~#IJaDataSet

  • Keivanloo I, Rilling J, Zou Y (2014) Spotting working code examples. In: Proceedings ICSE, pp 664–675

  • Kevic K, Fritz T (2014a) Automatic search term identification for change tasks. In: Proceedings ICSE, pp 468–471

  • Kevic K, Fritz T (2014b) A dictionary to translate change tasks to source code. In: Proceedings MSR, pp 320–323

  • Kimmig M, Monperrus M, Mezini M (2011) Querying source code with natural language. In: Proceedings ASE, pp 376–379

  • Li Z, Wang T, Zhang Y, Zhan Y, Yin G (2016) Query reformulation by leveraging crowd wisdom for scenario-based software search. In: Proceedings internetware, pp 36–44

  • Lin J, Liu Y, Guo J, Cleland-Huang J, Goss W, Liu W, Lohar S, Monaikul N, Rasin A (2017) Tiqi: a natural language interface for querying software project data. In: Proceedings ASE, pp 973–977

  • Linares-Vásquez M., Bavota G, Di Penta M, Oliveto R, Poshyvanyk D (2014) How do api changes trigger stack overflow discussions? a study on the android sdk. In: Proceedings ICPC, pp 83–94

  • Lopes C, Bajracharya S, Ossher J, Baldi P (2010) UCI source code data sets. http://www.ics.uci.edu/~lopes/datasets/

  • Mamykina L, Manoim B, Mittal M, Hripcsak G, Hartmann B (2011) Design lessons from the fastest q & a site in the west. In: Proceedings CHI, pp 2857–2866

  • McMillan C, Grechanik M, Poshyvanyk D, Xie Q, Fu C (2011) Portfolio: finding relevant functions and their usage. In: Proceedings ICSE, pp 111–120

  • Mihalcea R, Tarau P (2004) Textrank: bringing order into texts. In: Proceedings EMNLP, pp 404–411

  • Moreno L, Treadway JJ, Marcus A, Shen W (2014) On the use of stack traces to improve text retrieval-based bug localization. In: Proceedings ICSME, pp 151–160

  • Moreno L, Bavota G, Haiduc S, Di Penta M, Oliveto R, Russo B, Marcus A (2015) Query-based configuration of text retrieval solutions for software engineering tasks. In: Proceedings ESEC/FSE, pp 567–578

  • Nakasai K, Tsunoda M, Hata H (2016) Web search behaviors for software development. In: Proceedings CHASE, pp 125–128

  • Nie L, Jiang H, Ren Z, Sun Z, Li X (2016) Query expansion based on crowd knowledge for code search. TSC 9(5):771–783

    Google Scholar 

  • Ponzanelli L, Bavota G, Di Penta M, Oliveto R, Lanza M (2014) Mining stackOverflow to turn the IDE into a self-confident programming prompter. In: Proceedings MSR, pp 102–111

  • Rahman MM, Roy CK (2014) On the use of context in recommending exception handling code examples. In: Proceedings SCAM, pp 285–294

  • Rahman MM, Roy CK (2016) QUICKAR: automatic query reformulation for concept location using crowdsourced knowledge. In: Proceedings ASE, pp 220–225

  • Rahman MM, Roy CK (2017) STRICT: information retrieval based search term identification for concept location. In: Proceedings SANER, pp 79–90

  • Rahman MM, Roy CK (2018) Effective reformulation of query for code search using crowdsourced knowledge and extra-large data analytics. In: Proceedings ICSME, p 12

  • Rahman MM, Yeasmin S, Roy CK (2014) Towards a context-aware IDE-based meta search engine for recommendation about programming errors and exceptions. In: Proceedings CSMR–WCRE, pp 194–203

  • Rahman MM, Roy CK, Lo D (2016) RACK: automatic API recommendation using Crowdsourced knowledge. In: Proceedings SANER, pp 349–359

  • Rahman MM, Barson J, Paul S, Kayani J, Lois FA, Quezada SF, Parnin C, Stolee KT, Ray Baishakhi (2018) Evaluating how developers use general-purpose web-search for code retrieval. In: Proceedings MSR, p 10

  • Rigby PC, Robillard MP (2013) Discovering essential code elements in informal documentation. In: Proceedings ICSE, pp 832–841

  • Rocchio JJ (1971) The SMART retrieval system—experiments in automatic document processing. Prentice-Hall, Inc, Upper Saddle River

    Google Scholar 

  • Romano J, Kromrey JD, Coraggio J, Skowronek J (2006) Appropriate statistics for ordinal level data Should we really be using t-test and Cohen’sd for evaluating group differences on the NSSE and other surveys?. In: Annual meeting of the Florida Association of Institutional Research, pp 1–3

  • Roy CK, Cordy JR (2008) NICAD: accurate detection of near-Miss Intentional clones using flexible pretty-printing and code normalization. In: Proceedings ICPC, pp 172–181

  • Sadowski C, Stolee KT, Elbaum S (2015) How developers search for code A case study. In: Proceedings ESEC/FSE, pp 191–201

  • Sirres R, Bissyandé TF, Kim D, Lo D, Klein J, Kim K, Traon YL (2018) Augmenting and structuring user queries to support efficient free-form code search. EMSE

  • Sisman B, Kak AC (2013) Assisting code search with automatic query reformulation for bug localization. In: Proceedings MSR, pp 309–318

  • Svajlenko J, Roy CK (2018) Fast, scalable and user-guided clone detection. In: Proceedings ICSE-c, pp 352–353

  • Svajlenko J, Islam JF, Keivanloo I, Roy CK, Mia MM (2014) Towards a big data curated benchmark of inter-project code clones. In: Proceedings ICSME, pp 476–480

  • Thongtanunam P, Kula RG, Yoshida N, Iida H, Matsumoto K (2015) Who Should Review my Code?. In: Proceedings SANER, pp 141–150

  • Thummalapenta S, Xie T (2007) Parseweb: a programmer assistant for reusing open source code on the web. In: Proceedings ASE, pp 204–213

  • Thung F, Lo D, Lawall J (2013a) Automated library recommendation. In: Proceedings WCRE, pp 182–191

  • Thung F, Wang S, Lo D, Lawall J (2013b) Automatic recommendation of API methods from feature requests. In: Proceedings ASE, pp 290–300

  • Toutanova K, Manning CD (2000) Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. In: Proceedings EMNLP, pp 63–70

  • Vassallo C, Panichella S, Di Penta M, Canfora G (2014) Codes: mining source code descriptions from developers discussions. In: Proceedings ICPC, pp 106–109

  • Wang S, Lo D (2014) Version history, similar report, and structure: Putting them together for improved bug localization. In: Proceedings ICPC, pp 53–63

  • Wang S, Lo D, Jiang L (2014) Active code search: incorporating user feedback to improve code search relevance. In: Proceedings ASE, pp 677–682

  • Wang Y, Wang L, Li Y, He D, Liu T (2013) A theoretical analysis of NDCG type ranking measures. In: Proceedings COLT, pp 25–54

  • Warr FW, Robillard MP (2007) Suade: topology-based searches for software investigation. In: Proceedings ICSE, pp 780–783

  • Wong E, Yang J, Tan L (2013) AutoComment: mining question and answer sites for automatic comment generation. In: Proceedings ASE, pp 562–567

  • Xia X, Bao L, Lo D, Kochhar PS, Hassan AE, Xing Z (2017) What do developers search for on the web? EMSE 22(6):3149–3185

    Google Scholar 

  • Xie T, Pei J (2006) MAPO: mining api usages from open source repositories. In: Proceedings MSR, pp 54–57

  • Yang J, Tan L (2012) Inferring semantically related words from software context. In: Proceedings MSR, pp 161–170

  • Yuan T, Lo D, Lawall J (2014) Automated construction of a software-specific word similarity database. In: Proceedings CSMR-WCRE, pp 44–53

  • Zhang F, Niu H, Keivanloo I, Zou Y (2017) Expanding queries for code search using semantically related api class-names. TSE, page to appear

Download references

Acknowledgements

This research was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant.

Author information

Authors and Affiliations

  1. University of Saskatchewan, Saskatoon, Canada

    Mohammad M. Rahman & Chanchal K. Roy

  2. Singapore Management University, Singapore, Singapore

    David Lo

Authors
  1. Mohammad M. Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chanchal K. Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Lo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad M. Rahman.

Additional information

Communicated by: Andrea De Lucia

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rahman, M.M., Roy, C.K. & Lo, D. Automatic query reformulation for code search using crowdsourced knowledge. Empir Software Eng 24, 1869–1924 (2019). https://doi.org/10.1007/s10664-018-9671-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-018-9671-0

Keywords

Search

Navigation