Lingming Zhang
Lu Zhang
ABSTRACT
This paper presents a novel methodology for localizing faults in code as it evolves. Our insight is that the essence of failure-inducing edits made by the developer can be captured using mechanical program transformations (e.g., mutation changes). Based on the insight, we present the FIFL framework, which uses both the spectrum information of edits (obtained using the existing FaultTracer approach) as well as the potential impacts of edits (simulated by mutation changes) to achieve more accurate fault localization. We evaluate FIFL on real-world repositories of nine Java projects ranging from 5.7KLoC to 88.8KLoC. The experimental results show that FIFL is able to outperform the state-of-the-art FaultTracer technique for localizing failure-inducing program edits significantly. For example, all 19 FIFL strategies that use both the spectrum information and simulated impact information for each edit outperform the existing FaultTracer approach statistically at the significance level of 0.01. In addition, FIFL with its default settings outperforms FaultTracer by 2.33% to 86.26% on 16 of the 26 studied version pairs, and is only inferior than FaultTracer on one version pair.
- R. Abreu, P. Zoeteweij, and A. J. Van Gemund. On the accuracy of spectrum-based fault localization. In Testing: Academic and Industrial Conference Practice and Research Techniques-MUTATION, pages 89--98, 2007. Google Scholar
Digital Library
- E. Alves, M. Gligoric, V. Jagannath, and M. d'Amorim. Fault-localization using dynamic slicing and change impact analysis. In Proc. of ASE, pages 520--523, 2011. Google ScholarDigital Library
- P. Ammann and J. Offutt. Introduction to software testing. Cambridge University Press, 2008. Google ScholarDigital Library
- T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Theoretical and empirical studies on using program mutation to test the functional correctness of programs. In Proc. of POPL, pages 220--233, 1980. Google ScholarDigital Library
- O. Chesley, X. Ren, B. Ryder, and F. Tip. Crisp - A Fault Localization Tool for Java Programs. In Proc. of ICSE, pages 775--779, 2007. Google ScholarDigital Library
- B. Daniel, V. Jagannath, D. Dig, and D. Marinov. Reassert: Suggesting repairs for broken unit tests. In Proc. of ASE, pages 433--444, 2009. Google ScholarDigital Library
- B. Daniel, T. Gvero, and D. Marinov. On test repair using symbolic execution. In Proc. of ISSTA, pages 207--218, 2010. Google ScholarDigital Library
- R. DeMillo and A. Offutt. Constraint-based automatic test data generation. IEEE Transactions on Software Engineering, 17(9):900--910, 1991. Google ScholarDigital Library
- R. DeMillo, R. Lipton, and F. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11 (4):34--41, 1978. Google ScholarDigital Library
- H. Do, S. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering, 10 (4):405--435, 2005. ISSN 1382-3256. Google ScholarDigital Library
- P. G. Frankl, S. N. Weiss, and C. Hu. All-uses vs mutation testing: an experimental comparison of effectiveness. Journal of Systems and Software, 38(3):235--253, 1997. Google ScholarDigital Library
- G. Fraser and A. Zeller. Mutation-driven generation of unit tests and oracles. In Proc. of ISSTA, pages 147--158, 2010. Google ScholarDigital Library
- M. Gligoric, L. Zhang, C. Pereira, and G. Pokam. Selective mutation testing for concurrent code. In Proc. of ISSTA, pages 224--234, 2013. Google ScholarDigital Library
- R. Hamlet. Testing programs with the aid of a compiler. IEEE Transactions on Software Engineering, (4):279--290, 1977. Google ScholarDigital Library
- D. Hao, L. Zhang, Y. Pan, H. Mei, and J. Sun. On similarity-awareness in testing-based fault localization. Automated Software Engineering, 15(2):207--249, 2008. Google ScholarDigital Library
- D. Hao, L. Zhang, T. Xie, H. Mei, and J.-S. Sun. Interactive fault localization using test information. Journal of Computer Science and Technology, 24(5):962--974, 2009. Google ScholarDigital Library
- D. Hao, T. Xie, L. Zhang, X. Wang, J. Sun, and H. Mei. Test input reduction for result inspection to facilitate fault localization. Automated Software Engineering, 17(1):5--31, 2010. Google ScholarDigital Library
- M. Harman, Y. Jia, and W. Langdon. Strong higher order mutation-based test data generation. In Proc. of FSE, pages 212--222, 2011. Google ScholarDigital Library
- M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi. Regression test selection for Java software. In Proc. of OOPSLA, pages 312--326, 2001. Google ScholarDigital Library
- W. Howden. Weak mutation testing and completeness of test sets. IEEE Transactions on Software Engineering, (4):371--379, 1982. Google ScholarDigital Library
- R. Ihaka and R. Gentleman. R: A language for data analysis and graphics. Journal of computational and graphical statistics, 5(3):299--314, 1996.Google Scholar
- D. Jeffrey, N. Gupta, and R. Gupta. Fault localization using value replacement. In Proc. of ISSTA, pages 167--178, 2008. Google ScholarDigital Library
- J. Jones, M. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In Proc. of ICSE, page 477, 2002. Google ScholarDigital Library
- S. Kim, E. J. Whitehead, and Y. Zhang. Classifying software changes: Clean or buggy? IEEE TSE, 34(2):181--196, 2008. Google ScholarDigital Library
- A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung. An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE TSE, 32(12):971--987, 2006. Google ScholarDigital Library
- E. Krauser, A. Mathur, and V. Rego. High performance software testing on simd machines. IEEE Transactions on Software Engineering, 17(5):403--423, 1991. Google ScholarDigital Library
- C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In Proc. of ICSE, pages 3--13, 2012. Google ScholarDigital Library
- B. Liblit, M. Naik, A. Zheng, A. Aiken, and M. Jordan. Scalable statistical bug isolation. In Proc. of PLDI, pages 15--26, 2005. ISBN 1595930566. Google ScholarDigital Library
- R. Lowry. Concepts and applications of inferential statistics. R. Lowry, 1998.Google Scholar
- C. Mayer, S. Hanenberg, R. Robbes, É. Tanter, and A. Stefik. An empirical study of the influence of static type systems on the usability of undocumented software. In Proc. of OOPSLA, pages 683--702, 2012. Google ScholarDigital Library
- H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel. A static approach to prioritizing junit test cases. TSE, 38 (6):1258--1275, 2012. Google ScholarDigital Library
- G. Misherghi and Z. Su. Hdd: hierarchical delta debugging. In Proc. of ICSE, pages 142--151, 2006. Google ScholarDigital Library
- E. Murphy-Hill, T. Zimmermann, C. Bird, and N. Nagappan. The design of bug fixes. In Proc. of ICSE, pages 332--341, 2013. Google ScholarDigital Library
- A. Offutt, A. Lee, G. Rothermel, R. Untch, and C. Zapf. An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology (TOSEM), 5(2):99--118, 1996. Google ScholarDigital Library
- A. Offutt, Z. Jin, and J. Pan. The dynamic domain reduction procedure for test data generation. Software-Practice and Experience, 29(2):167--194, 1999. Google ScholarDigital Library
- M. Papadakis and Y. L. Traon. Using mutants to locate unknown faults. In Proc. of ICST Workshop on Mutation Analysis, pages 691--700, 2012. Google ScholarDigital Library
- C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In Proc. of ISSTA, pages 199--209, 2011. Google ScholarDigital Library
- Y. Qi, X.Mao, Y. Lei, and C.Wang. Using automated program repair for evaluating the effectiveness of fault localization techniques. In Proc. of ISSTA, pages 191--201, 2013. Google ScholarDigital Library
- X. Ren and B. Ryder. Heuristic ranking of Java program edits for fault localization. In Proc. of ISSTA, pages 239--249, 2007. Google ScholarDigital Library
- X. Ren, F. Shah, F. Tip, B. Ryder, and O. Chesley. Chianti: A tool for change impact analysis of Java programs. In Proc. of OOPSLA, 2004. Google ScholarDigital Library
- G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27(10):929--948, 2001. Google ScholarDigital Library
- D. Schuler and A. Zeller. Javalanche: Efficient mutation testing for Java. In Proc. of FSE, pages 297--298, 2009. Google ScholarDigital Library
- D. Schuler, V. Dallmeier, and A. Zeller. Efficient mutation testing by checking invariant violations. In Proc. of ISSTA, pages 69--80, 2009. Google ScholarDigital Library
- S. S. Shapiro and M. B.Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591--611, 1965.Google ScholarCross Ref
- M. Stoerzer, B. Ryder, X. Ren, and F. Tip. Finding failure-inducing changes in Java programs using change classification. In Proc. of FSE, pages 57--68, 2006. Google ScholarDigital Library
- R. Tzoref, S. Ur, and E. Yom-Tov. Instrumenting where it hurts: an automatic concurrent debugging technique. In Proc. of ISSTA, pages 27--38, 2007. Google ScholarDigital Library
- R. Untch, A. Offutt, and M. Harrold. Mutation analysis using mutant schemata. In ACM SIGSOFT Software Engineering Notes, volume 18, pages 139--148, 1993. Google ScholarDigital Library
- W.Weimer, T. Nguyen, C. Le Goues, and S. Forrest. Automatically finding patches using genetic programming. In Proc. of ICSE, pages 364--374, 2009. Google ScholarDigital Library
- F. Wilcoxon. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80--83, 1945.Google ScholarCross Ref
- G. Xu and A. Rountev. Regression test selection for aspectj software. In Proc. of ICSE, pages 65--74, 2007. Google ScholarDigital Library
- K. Yu, M. Lin, J. Chen, and X. Zhang. Practical isolation of failure-inducing changes for debugging regression faults. In Proc. of ASE, pages 20--29, 2012. Google ScholarDigital Library
- Y. Yu, J. Jones, and M. Harrold. An empirical study of the effects of test-suite reduction on fault localization. In Proc. of ICSE, pages 201--210, 2008. Google ScholarDigital Library
- A. Zeller. Yesterday, my program worked. today, it does not. why? In Proc. of FSE, pages 253--267, 1999. Google ScholarDigital Library
- A. Zeller. Automated debugging: Are we close? Computer, 34(11):26--31, 2001. Google ScholarDigital Library
- L. Zhang, S.-S. Hou, J.-J. Hu, T. Xie, and H. Mei. Is operator-based mutant selection superior to random mutant selection? In Proc. of ICSE, pages 435--444, 2010. Google ScholarDigital Library
- L. Zhang, T. Xie, L. Zhang, N. Tillmann, J. De Halleux, and H. Mei. Test generation via dynamic symbolic execution for mutation testing. In Proc. of ICSM, pages 1--10, 2010. Google ScholarDigital Library
- L. Zhang, M. Kim, and S. Khurshid. Localizing failure-inducing program edits based on spectrum information. In Proc. of ICSM, pages 23--32, 2011. Google ScholarDigital Library
- L. Zhang, D.Marinov, L. Zhang, and S. Khurshid. Regression mutation testing. In Proc. of ISSTA, pages 331--341, 2012. Google ScholarDigital Library
- L. Zhang, D. Hao, L. Zhang, G. Rothermel, and H. Mei. Bridging the gap between the total and additional test-case prioritization strategies. In Proc. ICSE, pages 192--201, 2013. Google ScholarDigital Library
- L. Zhang, D. Marinov, and S. Khurshid. Faster mutation testing inspired by test prioritization and reduction. In Proc. of ISSTA, pages 235--245, 2013. Google ScholarDigital Library
- S. Zhang. Practical semantic test simplification. In Proc. of ICSE, pages 1173--1176, 2013. Google ScholarDigital Library
- S. Zhang, Y. Lin, Z. Gu, and J. Zhao. Effective identification of failure-inducing changes: a hybrid approach. In Proc. of PASTE, pages 77--83, 2008. Google ScholarDigital Library
- S. Zhang, C. Zhang, and M. D. Ernst. Automated documentation inference to explain failed tests. In Proc. of ASE, pages 63--72, 2011. Google ScholarDigital Library
- X. Zhang, N. Gupta, and R. Gupta. Locating faults through automated predicate switching. In Proc. of ICSE, pages 272--281, 2006. Google ScholarDigital Library
- X. Zhang, N. Gupta, and R. Gupta. Pruning dynamic slices with confidence. In Proc. of PLDI, pages 169--180, 2006. Google ScholarDigital Library
- T. Zimmermann, N. Nagappan, P. J. Guo, and B. Murphy. Characterizing and predicting which bugs get reopened. In Proc. of ICSE, pages 1074--1083. IEEE, 2012. Google ScholarDigital Library
Index Terms
- Injecting mechanical faults to localize developer faults for evolving software
Recommendations
Transforming programs and tests in tandem for fault localization
Localizing failure-inducing code is essential for software debugging. Manual fault localization can be quite tedious, error-prone, and time-consuming. Therefore, a huge body of research e orts have been dedicated to automated fault localization. ...
Injecting mechanical faults to localize developer faults for evolving software
OOPSLA '13This paper presents a novel methodology for localizing faults in code as it evolves. Our insight is that the essence of failure-inducing edits made by the developer can be captured using mechanical program transformations (e.g., mutation changes). Based ...
Are faults localizable?
MSR '12: Proceedings of the 9th IEEE Working Conference on Mining Software RepositoriesMany fault localization techniques have been proposed to facilitate debugging activities. Most of them attempt to pinpoint the location of faults (i.e., localize faults) based on a set of failing and correct executions and expect debuggers to ...
Comments