Skip to main content
SpringerLink
Log in

An empirical study of non-reproducible bugs

  • Original Article
  • Published:
International Journal of System Assurance Engineering and Management Aims and scope Submit manuscript

Abstract

A large number of bug reports received in bug tracking system causes a negative impact on software’s growth and maintenance. Non-Reproducible bugs are an additional overhead on bug fixing process as certain percentage of Non-Reproducible bugs get fixed and these are difficult to detect. Their fixation can enhance the efficiency of bug fixing process. Past studies had ignored Non-Reproducible bugs while investigating bugs with varied perspectives. In this paper, we investigate characteristics of Non-Reproducible bug reports to understand their behaviour so that overall bug fixing process can be improved. We have also build machine learning models to predict fixable bugs from the set of Non-Reproducible marked bugs. We mine four large, open-source projects of Bugzilla repository and analyze Non-Reproducible bugs with different dimensions: frequency distribution, time-series analysis, active fix time, developer discussions, severity, priority and component. This study shows that there exists a significant number of Non-Reproducible bugs in each project and their occurrence follows a normal distribution with project longevity. They possess higher severity and lower priority as compared to reproducible bugs. Also, there exist specific components which are more prone to have non-reproducible bugs. We found that Non-Reproducible bugs consume more fixation time thereby affecting bug fixing processes’ efficiency. The overall results suggested that efforts in Non-Reproducible bugs could be minimized by developing a fixability prediction model that can build confidence among developers. Hence, we learned 52 classifiers for prediction of fixability among NR bugs. Our results show that prediction models achieved F1-Score up to 89.2%, 93.7%, 93.8% and 98.4% for NetBeans, Eclipse, Open Office and Mozilla Firefox projects respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. https://netbeans.org/bugzilla/.

  2. https://bugs.eclipse.org/bugs/.

  3. https://bz.apache.org/ooo/.

  4. https://bugzilla.mozilla.org/.

  5. www.bugzilla.org.

  6. https://bugzilla.mozilla.org/show_bug.cgi?id=170984.

  7. https://www.bugzilla.org/docs/2.18/html/lifecycle.html.

  8. https://en.wikipedia.org/wiki/Eclipse_(software).

  9. https://en.wikipedia.org/wiki/OpenOffice.org.

  10. https://en.wikipedia.org/wiki/NetBeans.

  11. https://en.wikipedia.org/wiki/Firefox.

References

  • Akbarinasaji S, Caglayan B, Bener A (2017) Predicting bug-fixing time: a replication study using an open source software project. J Syst Softw 136:173–186

    Article  Google Scholar 

  • An open letter to github from the maintainers of open source projects. https://github.com/dear-github/dear-github, 2016. Accessed 30 March 2018

  • Anvik J, Murphy GC (2011) Reducing the effort of bug report triage: recommenders for development-oriented decisions. ACM Trans Softw Eng Methodol (TOSEM) 20(3):10:1–10:35

    Article  Google Scholar 

  • Anvik J, Hiew L, Murphy GC (2006) Who should fix this bug? In: Proceedings of the 28th international conference on Software engineering. ACM, pp 361–370

  • Artzi S, Kim S, Ernst MD (2008) Recrash: making software failures reproducible by preserving object states. In: Ecoop, vol 8. Springer, pp 542–565

  • Basili VR, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456–473

    Article  Google Scholar 

  • Bhattacharya P, Neamtiu I, Shelton CR (2012) Automated, highly-accurate, bug assignment using machine learning and tossing graphs. J Syst Softw 85(10):2275–2292

    Article  Google Scholar 

  • Bhattacharya P, Neamtiu I (2010) Fine-grained incremental learning and multi-feature tossing graphs to improve bug triaging. In: 2010 IEEE international conference on software maintenance (ICSM). IEEE, pp 1–10

  • Boehm BW (1988) Understanding and controlling software costs. J Parametr 8(1):32–68

    Article  Google Scholar 

  • Challet D, DU YL (2005) Microscopic model of software bug dynamics: closed source versus open source. Int J Reliab Qual Saf Eng 12(06):521–534

    Article  Google Scholar 

  • Chaparro O, Lu J, Zampetti F, Moreno L, Di Penta M, Marcus A, Bavota G, Ng V (2017) Detecting missing information in bug descriptions. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering. ACM, pp 396–407

  • Chen T-H, Nagappan M, Shihab E, Hassan AE (2014) An empirical study of dormant bugs. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 82–91

  • Chen T-H, Thomas SW, Hassan AE (2016) A survey on the use of topic models when mining software repositories. Empir Softw Eng 21(5):1843–1919

    Article  Google Scholar 

  • Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MathSciNet  MATH  Google Scholar 

  • Destefanis G, Ortu M, Counsell S, Swift S, Marchesi M, Tonelli R (2016) Software development: do good manners matter? PeerJ Comput Sci 2:e73

    Article  Google Scholar 

  • Di Sorbo A, Spillner J, Canfora G, Panichella S (2019) Won’t we fix this issue? Qualitative characterization and automated identification of wontfix issues on github. arXiv preprint arXiv:1904.02414

  • DiStasi A (2019) Improving developer profiling and ranking to enhance bug report assignment. Master Thesis, Rochester Institute of Technology

  • Dixon WJ, Massey Frank J (1950) Introduction to statistical analsis. McGraw-Hill, New York

    Google Scholar 

  • Efendioglu M, Sen A, Koroglu Y (2018) Bug prediction of systemc models using machine learning. IEEE Trans Comput Aided Des Integr Circuits Syst 38(3):419–429

    Article  Google Scholar 

  • Erfani Joorabchi M, Mirzaaghaei M, Mesbah A (2014) Works for me! characterizing non-reproducible bug reports. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 62–71

  • Fountaine A, Sharif B (2017) Emotional awareness in software development: theory and measurement. In: Proceedings of the 2nd international workshop on emotion awareness in software engineering. IEEE Press, pp 28–31

  • Goyal A, Sardana N (2017) Nrfixer: sentiment based model for predicting the fixability of non-reproducible bugs. e-Inform Softw Eng J 11(1):109–122

    Google Scholar 

  • Goyal A, Sardana N (2017) Optimizing bug report assignment using multi criteria decision making technique. Intell Decis Technol 11(3):307–320

    Article  Google Scholar 

  • Graziotin D, Wang X, Abrahamsson P (2014) Happy software developers solve problems better: psychological measurements in empirical software engineering. PeerJ 2:e289

    Article  Google Scholar 

  • Guo PJ, Zimmermann T, Nagappan N, Murphy B (2010) Characterizing and predicting which bugs get fixed: an empirical study of microsoft windows. In: 2010 ACM/IEEE 32nd international conference on software engineering, vol 1. IEEE, pp 495–504

  • Guzman E, Azócar D, Li Y (2014) Sentiment analysis of commit comments in github: an empirical study. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 352–355

  • Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. ACM SIGKDD Explor Newslett 11(1):10–18

    Article  Google Scholar 

  • Hann I-H, Roberts J, Slaughter S, Fielding R (2002) Why do developers contribute to open source projects? First evidence of economic incentives. In: 2nd workshop on open source software engineering, Orlando, FL, pp 1–4

  • Hemetsberger A, Reinhardt C (2006) Learning and knowledge-building in open-source communities: a social-experiential approach. Manag Learn 37(2):187–214

    Article  Google Scholar 

  • Jeong G, Kim S, Zimmermann T (2009) Improving bug triage with bug tossing graphs. In: Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering. ACM, pp 111–120

  • Jin W, Orso A (2012) Bugredux: reproducing field failures for in-house debugging. In: 2012 34th international conference on software engineering (ICSE). IEEE, pp 474–484

  • Jonsson L, Borg M, Broman D, Sandahl K, Eldh S, Runeson P (2016) Automated bug assignment: ensemble-based machine learning in large scale industrial contexts. Empir Softw Eng 21(4):1533–1578

    Article  Google Scholar 

  • Kaur A, Jindal SG (2019) Text analytics based severity prediction of software bugs for apache projects. Int J Syst Assur Eng Manag 1–18

  • Koponen T (2006) Life cycle of defects in open source software projects. Open Source Syst 195–200

  • Kumari M, Sharma M, Singh V (2018) Severity assessment of a reported bug by considering its uncertainty and irregular state. Int J Open Source Softw Process 9(4):20–46

    Article  Google Scholar 

  • Lal S, Sardana N, Sureka A (2017) Eclogger: cross-project catch-block logging prediction using ensemble of classifiers. e-Inform Softw Eng J 11(1):7–38

    Google Scholar 

  • Lamkanfi A, Demeyer S, Giger E, Goethals B (2010) Predicting the severity of a reported bug. In: 2010 7th IEEE working conference on mining software repositories (MSR). IEEE, pp 1–10

  • Lamkanfi A, Demeyer S, Soetens QD, Verdonck T (2011) Comparing mining algorithms for predicting the severity of a reported bug. In: 2011 15th European conference on software maintenance and reengineering (CSMR). IEEE, pp 249–258

  • Lazar A, Ritchey S, Sharif B (2014) Improving the accuracy of duplicate bug report detection using textual similarity measures. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 308–311

  • Marks L, Zou Y, Hassan AE (2011) Studying the fix-time for bugs in large open source projects. In: Proceedings of the 7th international conference on predictive models in software engineering. ACM, p 11

  • Mockus A, Fielding RT, Herbsleb J (2000) A case study of open source software development: the apache server. In: Proceedings of the 22nd international conference on software engineering. ACM, pp 263–272

  • Moustafa S, ElNainay MY, El Makky N, Abougabal MS (2018) Software bug prediction using weighted majority voting techniques. Alex Eng J 57(4):2763–2774

    Article  Google Scholar 

  • Nguyen HA, Nguyen AT, Nguyen TT, Nguyen TN, Rajan H (2013) A study of repetitiveness of code changes in software evolution. In: Proceedings of the 28th IEEE/ACM international conference on automated software engineering. IEEE Press, pp 180–190

  • Perry DE, Stieg CS (1993) Software faults in evolving a large, real-time system: a case study. In: European software engineering conference. Springer, pp 48–67

  • Pressman RS (2005) Software engineering: a practitioner’s approach. Palgrave Macmillan, London

    MATH  Google Scholar 

  • Pushpalatha M, Mrunalini M (2019) Predicting the severity of open source bug reports using unsupervised and supervised techniques. Int J Open Source Softw Process 10(1):1–15

    Article  Google Scholar 

  • Python NLTK library. http://www.nltk.org/. Accessed 30 March 2018

  • Qin F, Zheng Z, Qiao Y, Trivedi KS (2018) Studying aging-related bug prediction using cross-project models. IEEE Trans Reliab 1–20

  • Qu Y, Zheng Q, Chi J, Jin Y, He A, Cui D, Zhang H, Liu T (2019) Using k-core decomposition on class dependency networks to improve bug prediction model’s practical performance. IEEE Trans Softw Eng

  • Rastkar S, Murphy GC, Murray G (2014) Automatic summarization of bug reports. IEEE Trans Softw Eng 40(4):366–380

    Article  Google Scholar 

  • Rastogi A, Gupta A, Sureka A (2013) Samiksha: mining issue tracking system for contribution and performance assessment. In: Proceedings of the 6th India software engineering conference. ACM, pp 13–22

  • Saha RK, Khurshid S, Perry DE (2015) Understanding the triaging and fixing processes of long lived bugs. Inf Softw Technol 65:114–128

    Article  Google Scholar 

  • Sbih A, Akour M (2018) Towards efficient ensemble method for bug triaging. J Multiple Valued Log Soft Comput 31(5–6):567–590

    Google Scholar 

  • Schröter A, Zimmermann T, Zeller A (2006) Predicting component failures at design time. In: Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering. ACM, pp 18–27

  • Shihab E, Ihara A, Kamei Y, Ibrahim WM, Ohira M, Adams B, Hassan AE, Matsumoto K-I (2013) Studying re-opened bugs in open source software. Empir Softw Eng 18(5):1005–1042

    Article  Google Scholar 

  • Shokripour R, Anvik J, Kasirun ZM, Zamani S (2015) A time-based approach to automatic bug report assignment. J Syst Softw 102:109–122

    Article  Google Scholar 

  • Sun J (2011) Why are bug reports invalid? In: 2011 IEEE fourth international conference on software testing, verification and validation (ICST). IEEE, pp 407–410

  • Sureka A (2012) Learning to classify bug reports into components. In: Objects, models, components, patterns, pp 288–303

    Chapter  Google Scholar 

  • Sureka A, Jalote P (2010) Detecting duplicate bug report using character n-gram-based features. In: 2010 17th Asia Pacific software engineering conference (APSEC). IEEE, pp 366–374

  • Tamrawi A, Nguyen TT, Al-Kofahi J, Nguyen TN (2011) Fuzzy set-based automatic bug triaging: Nier track. In: 2011 33rd international conference on software engineering (ICSE). IEEE, pp 884–887

  • Tian Y, Lo D, Xia X, Sun C (2015) Automated prediction of bug report priority using multi-factor analysis. Empir Softw Eng 20(5):1354–1383

    Article  Google Scholar 

  • Tian Y, Lo D, Sun C (2012) Information retrieval based nearest neighbor classification for fine-grained bug severity prediction. In: 2012 19th working conference on reverse engineering (WCRE). IEEE, pp 215–224

  • Tian Y, Lo D, Sun C (2013) Drone: predicting priority of reported bugs by multi-factor analysis. In: 2013 29th IEEE international conference on software maintenance (ICSM). IEEE, pp 200–209

  • Umer Q, Liu H, Sultan Y (2019) Sentiment based approval prediction for enhancement reports. J Syst Softw 155:57–69

    Article  Google Scholar 

  • Valdivia Garcia, H, Shihab E (2014) Characterizing and predicting blocking bugs in open source projects. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 72–81

  • White M, Linares-Vásquez M, Johnson P, Bernal-Cárdenas C, Poshyvanyk D (2015) Generating reproducible and replayable bug reports from android application crashes. In: 2015 IEEE 23rd international conference on program comprehension (ICPC). IEEE, pp 48–59

  • Witten IH, Frank E, Hall MA, Pal CJ (2016) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, San Francisco

    Google Scholar 

  • Xia X, Lo D, Shihab E, Wang X, Yang X (2015) Elblocker: predicting blocking bugs with ensemble imbalance learning. Inf Softw Technol 61:93–106

    Article  Google Scholar 

  • Xia X, Lo D, Wang X, Zhou B (2015) Dual analysis for recommending developers to resolve bugs. J Softw Evol Process 27(3):195–220

    Article  Google Scholar 

  • Xia X, Lo D, Ding Y, Al-Kofahi JM, Nguyen TN, Wang X (2017) Improving automated bug triaging with specialized topic model. IEEE Trans Softw Eng 43(3):272–297

    Article  Google Scholar 

  • Yadav A, Singh SK, Suri JS (2019) Ranking of software developers based on expertise score for bug triaging. Inf Softw Technol 112:1–17

    Article  Google Scholar 

  • Yang G, Min K, Lee J-W, Lee B (2019) Applying topic modeling and similarity for predicting bug severity in cross projects. KSII Trans Internet Inf Syst 13(3):1583–1598

    Google Scholar 

  • Yin Z, Ma X, Zheng J, Zhou Y, Bairavasundaram LN, Pasupathy S (2011) An empirical study on configuration errors in commercial and open source systems. In: Proceedings of the twenty-third ACM symposium on operating systems principles. ACM, pp 159–172

  • Yin Z, Yuan D, Zhou Y, Pasupathy S, Bairavasundaram L (2011) How do fixes become bugs? In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on foundations of software engineering. ACM, pp 26–36

  • Zaman S, Adams B, Hassan AE (2012) A qualitative study on performance bugs. In: 2012 9th IEEE working conference on mining software repositories (MSR). IEEE, pp 199–208

  • Zhang T, Chen J, Yang G, Lee B, Luo X (2016) Towards more accurate severity prediction and fixer recommendation of software bugs. J Syst Softw 117:166–184

    Article  Google Scholar 

  • Zhao Y, He T, Chen Z (2019) A unified framework for bug report assignment. Int J Softw Eng Knowl Eng 29(04):607–628

    Article  Google Scholar 

  • Zhao X, Xia X, Kochhar PS, Lo D, Li S (2014) An empirical study of bugs in build process. In: Proceedings of the 29th annual ACM symposium on applied computing. ACM, pp 1187–1189

Download references

Acknowledgements

The authors would like to acknowledge Late Prof. Ashish Sureka whose contribution to this work was of great significance. He conceived and designed the experiments of this empirical study.

Author information

Authors and Affiliations

  1. Department of Computer Science Engineering and Information Technology, Jaypee Institute of Information Technology, Sector-62, Noida, Uttar Pradesh, India

    Anjali Goyal & Neetu Sardana

Authors
  1. Anjali Goyal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Neetu Sardana
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anjali Goyal.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goyal, A., Sardana, N. An empirical study of non-reproducible bugs. Int J Syst Assur Eng Manag 10, 1186–1220 (2019). https://doi.org/10.1007/s13198-019-00850-5

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13198-019-00850-5

Keywords

Search

Navigation