Paul Robinson
ABSTRACT
Executable unit tests are a key component of many software engineering methodologies. A “green” test is one that is reported as passing (successfully testing some software feature). However, it is common for a test harness to assume that a test has passed when, in fact, it has merely not reported a failure. In this gap, where the “excluded middle” lives and thrives, we find the Rotten Green Test: A test that looks like it does something useful, but in fact does not.
Google Test, a popular open-source test framework for C++ software, has been enhanced to detect rotten green tests. This enhancement follows the lead of similar work done for the Pharo language, but in a framework more applicable in industry, and with no requirement for test modifications or an external tool. The enhanced Google Test has detected 183 rotten assertions in the LLVM project’s unit tests, and even found one in Google Test’s own internal test suite. The enhancement may report false positives from parameterized tests where assertions are conditioned on the parameter, and currently does not detect rotten assertions in helper methods.
- G. Mezaros, S. M. Smith and J. Andrea, "The test automation manifesto," in Proceedings of the Third XP and Second Agile Universe Conference, 2003.
Google Scholar
- D. Astels, Test-Driven Development--A Practical Guide, Prentice-Hall, 2003.
Google Scholar
- J. Delplanque, S. Ducasse, G. Polito, A. P. Black and A. Etien, "Rotten Green Tests," in IEEE/ACM 41st International Conference on Software Engineering, Montréal, Québec, Canada, 2019.
Google Scholar
- A. P. Black, S. Ducasse, O. Nierstrasz, D. Pollet, D. Cassou and M. Denker, Pharo by Example, Kehrsatz, Switzerland: Square Bracket Associates, 2009.
Google Scholar
- Google, "GoogleTest ," 2023. https://github.com/google/googletest.
Google Scholar
- The LLVM Project, "Clang: a C language family frontend for LLVM," 2023. [Online]. Available: https://clang.llvm.org.
Google Scholar
- The LLVM Project, "The LLVM Compiler Infrastructure," 2023. [Online]. Available: https://llvm.org.
Google Scholar
- Google, "Bazel," https://bazel.build/.
Google Scholar
Index Terms
- Rotten Green Tests in Google Test
Recommendations
Rotten green tests
ICSE '19: Proceedings of the 41st International Conference on Software EngineeringUnit tests are a tenant of agile programming methodologies, and are widely used to improve code quality and prevent code regression. A green (passing) test is usually taken as a robust sign that the code under test is valid. However, some green tests ...
Comparing test quality measures for assessing student-written tests
ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software EngineeringMany educators now include software testing activities in programming assignments, so there is a growing demand for appropriate methods of assessing the quality of student-written software tests. While tests can be hand-graded, some educators also use ...
A Static Approach to Prioritizing JUnit Test Cases
Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most ...
Comments