Abstract
Pseudo-tested methods are defined as follows: they are covered by the test suite, yet no test case fails when the method body is removed, i.e., when all the effects of this method are suppressed. This intriguing concept was coined in 2016, by Niedermayr and colleagues, who showed that such methods are systematically present, even in well-tested projects with high statement coverage. This work presents a novel analysis of pseudo-tested methods. First, we run a replication of Niedermayr’s study with 28K+ methods, enhancing its external validity thanks to the use of new tools and new study subjects. Second, we perform a systematic characterization of these methods, both quantitatively and qualitatively with an extensive manual analysis of 101 pseudo-tested methods. The first part of the study confirms Niedermayr’s results: pseudo-tested methods exist in all our subjects. Our in-depth characterization of pseudo-tested methods leads to two key insights: pseudo-tested methods are significantly less tested than the other methods; yet, for most of them, the developers would not pay the testing price to fix this situation. This calls for future work on targeted test generation to specify those pseudo-tested methods without spending developer time.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Compared to Niedermayr et al. (2016), we add two new transformations, one to return null and another to return an empty array. These additions allow to expand the scope of methods to be analyzed.
The computation of the Pearson coefficient and the Wilcoxon test were performed using the features of the R language.
The violin plot for pseudo-tested methods of commons-cli and jopt-simple are not displayed, as they have too few methods in this category.
References
Andrews JH, Briand LC, Labiche Y (2005) Is Mutation an Appropriate Tool for Testing Experiments? In: Proceedings of the 27th International Conference on Software engineering, ICSE ’05. ACM, New York, pp 402–411
Androutsopoulos K, Clark D, Dan H, Hierons RM, Harman M (2014) An analysis of the relationship between conditional entropy and failed error propagation in software testing. In: Proceedings of the 36th international conference on software engineering. ACM, pp 573–583
Coles H, Laurent T, Henard C, Papadakis M, Ventresque A (2016) PIT: A Practical Mutation Testing Tool for Java (Demo). In: Proceedings of the 25th International Symposium on Software Testing and analysis, ISSTA 2016. ACM, New York, pp 449–452
Daran M, Thévenod-Fosse P (1996) Software Error Analysis: A Real Case Study Involving Real Faults And Mutations. In: Proceedings of the 1996 ACM SIGSOFT International Symposium on Software Testing and analysis, ISSTA ’96. ACM, New York, pp 158–171
Delahaye M, Bousquet LD (2013) A Comparison of Mutation Analysis Tools for Java. In: 2013 13th International, Conference on Quality Software, pp 187–195
Delamaro ME, Offutt J, Ammann P (2014) Designing deletion mutation operators. In: 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, pp 11–20
DeMillo RA, Lipton RJ, Sayward FG (1978) Hints on test data selection Help for the practicing programmer. Computer 11(4):34–41
Deng L, Offutt J, Li N (2013) Empirical evaluation of the statement deletion mutation operator. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, pp 84–93
Durelli VHS, Souza NMD, Delamaro ME (2017) Are deletion mutants easier to identify manually? In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp 149–158
Gopinath R, Ahmed I, Alipour MA, Jensen C, Groce A (2017) Does choice of mutation tool matter? Softw Qual J 25(3):871–920
Gopinath R, Jensen C, Groce A (2014) Mutations: How close are they to real faults?. In: 2014 IEEE 25th international symposium on Software reliability engineering (ISSRE). IEEE, pp 189–200
Gourlay JS (1983) A mathematical framework for the investigation of testing. IEEE Transactions on software engineering SE-9(6):686–709
Jahangirova G, Clark D, Harman M, Tonella P (2016) Test oracle assessment and improvement. In: Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, pp 247–258
Just R, Schweiggert F, Kapfhammer GM (2011) MAJOR An efficient and extensible tool for mutation analysis in a Java compiler. In: Proceedings of the International Conference on Automated Software Engineering (ASE), pp 612–615
Just R, Jalali D, Inozemtseva L, Ernst MD, Holmes R, Fraser G (2014) Are Mutants a Valid Substitute for Real Faults in Software Testing? In: Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software engineering, FSE 2014. ACM, New York, pp 654–665
Kampenes VB, Dybå T, Hannay JE, Sjøberg DIK (2007) A systematic review of effect size in software engineering experiments. Inf Softw Technol 49(11-12):1073–1086
Kintis M, Papadakis M, Papadopoulos A, Valvis E, Malevris N (2016) Analysing and Comparing the Effectiveness of Mutation Testing Tools: A Manual Study. In: 2016 IEEE, 16th International Working Conference on Source Code Analysis and Manipulation (SCAM), pp 147–156
Laurent T, Papadakis M, Kintis M, Henard C, Traon YL, Ventresque A (2017) Assessing and Improving the Mutation Testing Practice of PIT. In: 2017 IEEE, International Conference on Software Testing, Verification and Validation (ICST), pp 430–435
Niedermayr R, Juergens E, Wagner S (2016) Will my tests tell me if I break this code?. In: Proceedings of the International Workshop on Continuous Software Evolution and delivery. ACM Press, New York, pp 23–29
Petrovic G, Ivankovic M (2018) State of mutation testing at google. In: Proceedings of the 40th International Conference on Software Engineering 2017 (SEIP)
Schuler D, Zeller A (2013) Checked coverage: an indicator for oracle quality. Softw Test Verification Reliab 23(7):531–551
Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in Empirical Software Engineering. Empir Softw Eng 13(2):211–218
Staats M, Whalen MW, Heimdahl MP (2011) Programs, tests, and oracles: the foundations of testing revisited. In: Proceedings of the 33rd international conference on software engineering. ACM, pp 391–400
Untch RH (2009) On reduced neighborhood mutation analysis using a single mutagenic operator. In: Proceedings of the 47th Annual Southeast Regional Conference, ACM-SE 47. ACM, New York, pp 71:1–71:4
Vera-Pérez OL, Monperrus M, Baudry B (2018) Descartes: A pitest engine to detect pseudo-tested methods. In: Proceedings of the 2018 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE ’18), pp 908–911
Acknowledgments
We would like to acknowledge the invaluable help and feedback provided by the development teams of authzforce, spoon and pdfbox. We also express our appreciation to Simon Urli, Daniel Le Berre, Arnaud Blouin, Marko Ivanković, Goran Petrovic and Andy Zaidman for their feedback and their very accurate suggestions. This work has been partially supported by the EU Project STAMP ICT-16-10 No.731529 and by the Wallenberg Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Paolo Tonella
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix A: Source code for the Study Subjects
Appendix A: Source code for the Study Subjects
Legend
Column “Project” lists the projects included in the present study. Column “URL” contains links to the available source code. Column “Commit ID” contains the SHA-1 hash identifying the commit with the source code state that was used in this study.
Project | URL/Commit ID |
|---|---|
authzforce | |
81ae56671bc343eabf2bc99ee0c51ba6ae28d649 | |
aws-sdk-java | |
b5ae6ce44f4b5053a9a0255c9648f3073fafcf55 | |
commons-cli | |
c246bd419ee0efccd9a96f9d33486617d5d38a56 | |
commons-codec | |
e9da3d16ae67f2940a0bbdf982ecec19a0481981 | |
commons-collections | |
db189926f7415b9866e76cd8123e40c09c1cc67e | |
commons-io | |
e36d53170875d26d59ca94bd376bf40bc5690ee6 | |
commons-lang | |
e8f924f51be5bc8bcd583ea96e5ef25f9b2ca72a | |
flink-core | |
740f711c4ec9c4b7cdefd01c9f64857c345a68a1 | |
gson | |
c3d17e39f1cb6ec41496e639ab42f7e7cca3b465 | |
jaxen | |
a8bd80599fd4d1c9aa1248d3276198535a30bfc5 | |
jfreechart | |
a7156d4595ff7f6a7c8dac50625295c284b86732 | |
jgit | |
1513a5632dcaf8c6e2d6998427087e11ba35566d | |
joda-time | |
6ad133837a4c4f8199d00a05c3c16267dbf6deb8 | |
jopt-simple | |
b38b70d1e7685766ab400d8b57ef9ca9c010e0bb | |
jsoup | |
35e80a779b7908ddcd41a6a7df5f21b30bf999d2 | |
pdfbox | |
09e9173e400ea6c1794354ca54c36607c53391ff | |
sat4j-core | https://gitlab.ow2.org/sat4j/sat4j/tree/master/org.sat4j.core |
1a0127645bf98b768ee3628076d0246596dd15eb | |
scifio | |
2760af6982ad18aab400e9cd99b9f63ef2495333 | |
spoon | |
fd878bc71b73fc1da82356eaa6578f760c70f0de | |
urbanairship | |
aafc049cc1cd3971c62a3dfc1d72cfe61160f32c | |
xwiki-rendering | |
cb3c444fb743e073eefbac2b44351a6166d94ac1 |
Rights and permissions
About this article
Cite this article
Vera-Pérez, O.L., Danglot, B., Monperrus, M. et al. A comprehensive study of pseudo-tested methods. Empir Software Eng 24, 1195–1225 (2019). https://doi.org/10.1007/s10664-018-9653-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-018-9653-2