Abstract
This paper presents multiple empirical experiments that investigate the impact of fault quantity and fault type on statistical, coverage-based fault localization techniques and fault-localization interference. Fault-localization interference is a phenomenon revealed in earlier studies of coverage-based fault localization that causes faults to obstruct, or interfere, with other faults’ ability to be localized. Previously, it had been asserted that a fault-localization technique’s effectiveness was negatively correlated to the quantity of faults in the program. To investigate these beliefs, we conducted an experiment on six programs consisting of more than 72,000 multiple-fault versions. Our data suggests that the impact of multiple faults exerts a significant, but slight influence on fault-localization effectiveness. In addition, faults were categorized according to four existing fault-taxonomies and found no correlation between fault type and fault-localization interference. In general, even in the presence of many faults, at least one fault was found by fault localization with similar effectiveness. Additionally, our data exhibits that fault-localization interference is prevalent and exerts a meaningful influence that may cause a fault’s localizability to vary greatly. Because almost all real-world software contains multiple faults, these results affect the practical use and understanding of statistical fault-localization techniques.
Similar content being viewed by others
References
Abreu R, Zoeteweij P, Van Gemund AJ (2007) On the accuracy of spectrum-based fault localization. In: Testing: academic and industrial conference practice and research techniques-MUTATION, 2007. TAICPART-MUTATION 2007, pp 89–98. IEEE
Ali S, Andrews JH, Dhandapani T, Wang W (2009) Evaluating the accuracy of fault localization techniques. In: Proceedings of the 2009 IEEE/ACM international conference on automated software engineering, pp 76–87. IEEE Computer Society
Arumuga Nainar P, Liblit B (2010) Adaptive bug isolation. In: Proceedings of the 32nd ACM/IEEE international conference on software engineering, vol 1, pp 255–264. ACM
Clark S, Cobb J, Kapfhammer GM, Jones JA, Harrold MJ (2011) Localizing sql faults in database applications. In: Proceedings of the 26th IEEE/ACM international conference on automated software engineering (ASE), pp 213–222
Cleve H, Zeller A (2005) Locating causes of program failures. In: Proceedings of the 27th international conference on software engineering, pp 342–351. ACM
Debroy V, Wong WE (2009) Insights on fault interference for programs with multiple bugs. In: 20th international symposium on software reliability engineering, 2009. ISSRE’09, pp 165–174. IEEE
Denmat T, Ducassé M, Ridoux O (2005) Data mining and cross-checking of execution traces: a re-interpretation of jones, harrold and stasko test information. Tech. rep.
Dickinson W, Leon D, Podgurski A (2001) Finding failures by cluster analysis of execution profiles. In: Proceedings of the international conference on software engineering. http://portal.acm.org/citation.cfm?id=.
Dickinson W, Leon D, Podgurski A (2001) Pursuing failure: the distribution of program failures in a profile space. In: Proceedings of the international symposium on foundations of software engineering . http://doi.acm.org/10.1145/503209.503243
DiGiuseppe N, Jones JA (2011a) Fault interaction and its repercussions. In: 2011 27th IEEE international conference on software maintenance (ICSM), pp 3–12. IEEE
DiGiuseppe N, Jones JA (2011b) On the influence of multiple faults on coverage-based fault localization. In: Proceedings of the 2011 international symposium on software testing and analysis, pp 210– 220. ACM
DiGiuseppe N, Jones JA (2012a) Concept-based failure clustering. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering, p 29. ACM
DiGiuseppe N, Jones JA (2012b) Software behavior and failure clustering: an empirical study of fault causality. In: 2012 IEEE fifth international conference on software testing, verification and validation (ICST), pp 191–200. IEEE
Do H, Elbaum S, Rothermel G (2005) Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empir Softw Eng 10(4):405–435
Hayes JH (1994) Testing of object-oriented programming systems (oops): a fault-based approach. In: Object-oriented methodologies and systems, pp 205–220. Springer
Hayes JH, Chemannoor IR, Holbrook EA (2011) Improved code defect detection with fault links. Softw Test Verification and Reliab 21(4):299–325
Jones JA (2008) Semi-automatic fault localization. Georgia Institute of Technology, Ph.D. thesis
Jones JA, Bowring JF, Harrold MJ (2007) Debugging in parallel. In: Proceedings of the 2007 international symposium on software testing and analysis, pp 16–26. ACM
Jones JA, Harrold MJ (2005) Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th IEEE/ACM international conference on automated software engineering, pp 273–282. ACM
Jones JA, Harrold MJ, Stasko J (2002) Visualization of test information to assist fault localization. In: Proceedings of the 24th international conference on Software engineering, pp 467–477. ACM
Kung DC, Gao J, Kung CH (1998) Testing object-oriented software. Tech. rep.
Liblit B, Naik M, Zheng AX, Aiken A, Jordan MI (2005) Scalable statistical bug isolation, pp 15–26. ACM
Liu C, Han J (2006) Failure proximity: a fault localization-based approach. In: Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, pp 46–56. ACM
Liu C, Yan X, Fei L, Han J, Midkiff SP (2005) Sober: statistical model-based bug localization, pp 286–295. ACM
Offutt AJ, Lee A, Rothermel G, Untch RH, Zapf C (1996) An experimental determination of sufficient mutant operators. ACM Trans Softw Eng Methodol (TOSEM) 5(2):99–118
Parnin C, Orso A (2011) Are automated debugging techniques actually helping programmers? In: Proceedings of the 2011 international symposium on software testing and analysis, pp 199–209. ACM
Podgurski A, Leon D, Francis P, Masri W, Minch M, Sun J, Wang B (2003) Automated support for classifying software failure reports. In: Proceedings 25th international conference on software engineering, 2003, pp 465–475. IEEE
Renieres M, Reiss SP (2003) Fault localization with nearest neighbor queries. In: Proceedings. 18th IEEE international conference on automated software engineering, 2003, pp 30–39. IEEE
Santelices R, Jones JA, Yu Y, Harrold MJ (2009) Lightweight fault-localization using multiple coverage types. In: Proceedings of the 31st international conference on software engineering, ICSE ’09, pp 56–66
Smith M, Robson D (1992) A framework for testing object-oriented programs. J Object-Oriented Program 5(3):45–53
Srivastav M, Singh Y, Gupta C, Chauhan DS (2010) Complexity estimation approach for debugging in parallel. In: 2010 2nd international conference on computer research and development, pp 223–227. IEEE
Vessey I (1985) Expertise in debugging computer programs: a process analysis. Int J Man-Machine Stud 23(5):459–494
Voas J (1992) Pie: a dynamic failure-based technique. IEEE transactions on software engineering 18(8):717–727. doi:10.1109/32.153381
Wong WE, Debroy V (2009) A survey of software fault localization. University of Texas at Dallas. Tech. Rep. UTDCS-45-09
Yu Y, Jones JA, Harrold MJ (2008) An empirical study of the effects of test-suite reduction on fault localization. In: Proceedings of the 30th international conference on software engineering, pp 201–210. ACM
Zeller A (2002) Isolating cause-effect chains from computer programs. In: Proceedings of the 10th ACM SIGSOFT symposium on foundations of software engineering, pp 1–10. ACM
Zeller A (2009) In: why programs fail: a guide to systematic debugging. Morgan Kaufmann
Zheng AX, Jordan MI, Liblit B, Naik M, Aiken A (2006) Statistical debugging: simultaneous identification of multiple bugs. In: Proceedings of the 23rd international conference on machine learning, pp 1105–1112. ACM
Acknowledgments
This material is based upon work supported by the National Science Foundation, through Award CCF-1116943 and through Graduate Research Fellowship under Grant No. DGE-0808392.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: James Miller
Appendix A: Fault-Type to Number Index
Appendix A: Fault-Type to Number Index
Smith92
-
1.
Inter-routine conceptual
-
2.
Inter-routine actual
-
3.
Intra-routine conceptual
-
4.
Intra-routine actual
Firesmith92
-
1.
Incorrect visibility
-
2.
Missing component
-
3.
Inconsistent component
-
4.
Incorrect allocation/deallocation of resources
-
5.
Does not meet requirements
Hayes94
-
1.
Abstraction
-
2.
Encapsulation
-
3.
Modularity
-
4.
Hierarchy
Hayes11
-
1.
Data Declaration
-
2.
Data Initialization
-
3.
Data representation
-
4.
Data accessing
-
5.
Incorrect equation
-
6.
Wrong manipulation
-
7.
Incorrect/missing processing
-
8.
Unnecessary processing
-
9.
Rampaging go to
-
10.
Incorrect labels
-
11.
Dead-end code
-
12.
Duplicate logic
-
13.
Unachievable path
-
14.
Incorrect initial value
-
15.
Incorrect terminal value
-
16.
Incorrect control value processing
-
17.
Incorrect exception exit processing
-
18.
Illogical conditions or impossible cases
-
19.
Incorrect module interaction
-
20.
Incorrect module-external data structure
-
21.
Incorrect input parameters
-
22.
Large response time
-
23.
Lack of naturalness
-
24.
Inconsistency
-
25.
Redundancy
-
26.
Complexity
-
27.
Lack of flexibility
-
28.
Non-supportiveness
-
29.
Unpredictable flows
-
30.
Visual stimulation
-
31.
Platform
-
32.
Wrong file included
-
33.
Incorrect environment variable setting
-
34.
Documentation
1.1 A.1 Expense for Prominent Fault
1.2 A.2 Median Expense for All Faults
1.3 A.3 Median Suspiciousness for All Faults
1.4 A.4 Expense Distribution per Fault Type
1.5 A.5 Fault Interference Distribution per Fault Type
Rights and permissions
About this article
Cite this article
DiGiuseppe, N., Jones, J.A. Fault density, fault types, and spectra-based fault localization. Empir Software Eng 20, 928–967 (2015). https://doi.org/10.1007/s10664-014-9304-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-014-9304-1