Authors:
Samia Al Blwi
1
;
Amani Ayad
2
;
Besma Khaireddine
3
;
Imen Marsit
4
and
Ali Mili
1
Affiliations:
1
NJIT, Newark NJ, U.S.A.
;
2
Kean University, Union NJ, U.S.A.
;
3
University of Tunis El Manar, Tunis, Tunisia
;
4
University of Sousse, Sousse, Tunisia
Keyword(s):
Software Testing, Test Suite Effectiveness, Syntactic Coverage, Mutation Coverage, Semantic Coverage.
Abstract:
Several syntactic measures have been defined in the past to assess the effectiveness of a test suite: statement coverage, condition coverage, branch coverage, path coverage, etc. There is ample analytical and empirical evidence to the effect that these are imperfect measures: exercising all of a program’s syntactic features is neither necessary nor sufficient to ensure test suite adequacy; not to mention that it may be impossible to exercise all the syntactic features of a program (re: unreachable code). Mutation scores are often used as reliable measures of test suite effectiveness, but they have issues of their own: some mutants may survive because they are equivalent to the base program not because the test suite is inadequate; the same mutation score may mean vastly different things depending on whether the killed mutants are distinct from each other or equivalent; the same test suite and the same program may yield different mutation scores depending on the mutation operators tha
t we use. Fundamentally, whether a test suite T is adequate for a program P depends on the semantics of the program, the specification that the program is tested against, and the property of correctness that the program is tested for (total correctness, partial correctness). In this paper we present a formula for the effectiveness of a test suite T which depends exactly on the semantics of P, the correctness property that we are testing P for, and the specification against which this correctness property is tested; it does not depend on the syntax of P, nor on any mutation experiment we may run. We refer to this formula as the semantic coverage of the test suite, and we investigate its properties.
(More)