Mutant reduction based on dominance relation for weak mutation testing

Abstract

Context: As a fault-based testing technique, mutation testing is effective at evaluating the quality of existing test suites. However, a large number of mutants result in the high computational cost in mutation testing. As a result, mutant reduction is of great importance to improve the efficiency of mutation testing.

Objective: We aim to reduce mutants for weak mutation testing based on the dominance relation between mutant branches.

Method: In our method, a new program is formed by inserting mutant branches into the original program. By analyzing the dominance relation between mutant branches in the new program, the non-dominated one is obtained, and the mutant corresponding to the non-dominated mutant branch is the mutant after reduction.

Results: The proposed method is applied to test ten benchmark programs and six classes from open-source projects. The experimental results show that our method reduces over 80% mutants on average, which greatly improves the efficiency of mutation testing.

Conclusion: We conclude that dominance relation between mutant branches is very important and useful in reducing mutants for mutation testing.

Introduction

Software testing, which is used to seek existing defects or faults in software before it is released to the market, is an important way to improve software quality. Mutation testing is commonly used to evaluate the quality of existing test suites to guide testers how they might be able to improve them [1]. Compared with other structural coverage criteria, test suites that are mutation adequate can reveal more faults [2]. It is noticeable that mutation testing has attracted widespread attention from researchers and developers in both academia and industry.

Mutation testing is a fault-based technique [3], [4], and the related concepts are given as follows. By making a simple syntactic change to the original program, a mutant is generated. A rule used to perform the syntactic changes is called a mutation operator. If a test datum can distinguish the outputs between a mutant and its original program, the mutant is said to be killed. A mutant is equivalent, if it cannot be killed by any test datum. Generally, the adequacy of mutation testing named mutation score is defined as the ratio of the number of killed mutants to the total number of non-equivalent mutants.

In order to optimize the execution of the traditional mutation testing, Howden first proposed weak mutation testing [5]. Instead of checking a mutant after executing the whole program, weak mutation testing checks a mutant immediately after executing the mutated statement.

In mutation testing, mutants are employed to reflect possible real faults in software under test [6], [7], [8]. Many lines of code (LOCs), complicated statements, and a variety of data types [9] in software greatly increase the number of mutants. It results in the high computational cost in mutation testing, therefore the mutant reduction is of great importance and necessity. Although there have been several techniques for mutant reduction [10], [11], [12], [13], [14], [15], [16], their efficiency needs to be further improved.

Just et al. focused on the COR and ROR mutation operators to identify redundant mutants [13]. Kaminski et al. sought a subset of relational operators that subsumes the others to reduce mutants [14]. Focusing only on a subset of mutation operators opens new research directions [13], [14]. Papadakis and Malevris transformed the problem of killing mutants into the problem of covering mutant branches in the new program, and generated test data by conventional approaches [17]. Although it is relatively efficient, a large number of mutants without reduction, will inevitably add high complexity to the new program.

In the previous work on dominance analysis, Marre and Bertolino employed the subsumption relation between entities in a ddgraph (a simplified control flow graph) to seek the minimal set of entities named the spanning set, so as to reduce the number of entities needed to cover [18]. In addition, Ghiduk and Girgis identified the non-dominated nodes in a control flow graph (CFG) by analyzing the dominance relation between nodes [19]. Both the above methods are performed among the original entities (nodes) for structural coverage testing. Different from the above work, we analyze the dominance relation between mutant branches, which are instrumented branches transformed from mutants based on the method presented by Papadakis and Malevris for weak mutation testing, with the aim to reduce the number of mutants, and to improve the efficiency of testing.

Considering all the traditional (method level) mutation operators, we first construct mutant branches based on the statements before and after mutation, and form the new program by fusing all mutant branches into the original program using the method proposed by Papadakis and Malevris [17]. Then, we identify redundant mutants according to the dominated mutant branches after manual analysis with the aid of the dominance relation graph. Mutants associated with the non-dominated ones will remain. The test data that cover the non-dominated mutant branches can also cover all the mutant branches, i.e., kill all the mutants before reduction in weak mutation testing.

The basic idea of defining the dominance relation between mutant branches and applying the dominance relation to reduce mutants was initially reported, with examples on several small programs, at the 2nd Chinese Search Based Software Engineering (CSBSE’2013) workshop [20]. Given the fact that the two-page abstract is preliminary, we have extended the idea in the following four new directions:

• defining four concepts, mutant branch, dominance relation, non-dominated branch, and dominance relation graph;

• presenting two theorems on how to form the non-dominated mutant branch set and identify the non-dominated mutants;

• providing an example throughout the whole paper to intuitively demonstrate the above work;

• evaluating the proposed method by applying it to ten benchmark programs and six classes from open-source projects with various sizes and complexities.

The main contributions of this paper are as follows:

• A method of reducing mutants is proposed for weak mutation testing, which is conducted by analyzing the dominance relation between mutant branches in the new program.

• Four definitions of identifying the dominance relation between mutant branches are provided, and the dominance relation graph is given to describe all the dominance relations in the new program.

• Two theorems of determining the non-dominated mutant branches are given, so as to reduce redundant mutants.

• The proposed method is applied to ten benchmark programs and six classes from open-source projects, and the experimental results suggest that our method reduces over 80% mutants.