iBiR: Bug-report-driven Fault Injection

2.1 Assessment of Testing Techniques

Fault injection is used extensively by researchers as a tool to evaluate the fault-revealing capability of automated test techniques such as automated test generation techniques. This approach was found to be used by approximately 19% of all software testing studies published in major SE conferences by a bibliometric analysis performed in 2016 [59]. This is because real and diverse bug datasets are hard to collect and make it hard to perform controlled studies as they usually result in faulty versions including single faults. Fault injection is thus a fast and convenient way to perform control studies since it avoids the costly and tedious work of creating fault datasets. In such cases, the realism of the injected faults is a major validity question that may impact the results of the experiments. Recent studies [62] have shown that conventional mutation testing doesn’t perform well in this regard as it introduces many faults that are unrealistic. To deal with such cases, we develop iBiR and show that it injects more semantically similar faults than traditional mutation testing.

2.2 Fault Tolerance Assessment

Fault injection is also frequently used to evaluate the system’s performance under faulty test executions. In such a case, the injected faults simulate the effects of real ones by performing arbitrary code changes everywhere. To this end, iBiR guides the injection toward specific error-prone targets/features and fault types. This is particularly important in order to improve the realism of the analysis. Interestingly, previous research on fault tolerance assessment [51] has shown that fault injection realism can be improved by appropriately controlling the locations and types of the injected faults. We therefore propose a way to do so by leveraging information from bug reports.

3.1 Fault Localization

Fault localization is the activity of identifying the suspected fault locations, which will be transformed to generate patches. Several automated fault localization techniques have been proposed [80], such as slice based [79], spectrum based [2], statistics based [37], mutation based [61], and so forth.

Fault localization techniques based on Information Retrieval (IR) [12, 18, 44, 67] exploit textual bug reports to identify code chunks relevant to the bug, without relying on test cases. IR-based fault localization tools extract tokens from the bug report to formulate a query to be matched with the collection of documents formed by the source code files [42, 65, 75, 77, 81, 84]. Then, they rank the documents based on their relevance to the query, such that source files ranked higher are more likely to contain the fault. Recently, automated program repair methods have been designed on top of IR-based fault localization [30]. They achieve comparable performance to methods using spectrum-based fault localization, yet without relying on the assumption that test cases are available.

We leverage IR-based fault localization to achieve a different goal; instead of localizing the reported bug, we aim at injecting faults at code locations that implement a functionality similar to the one described by the bug report.

3.2 Mutation Testing

Mutation testing is a popular fault-based testing technique [60]. It operates by inserting artificial faults into a program under test, thereby creating many different versions (named mutants) of the program. The artificial faults are injected through syntactic changes to all program locations in the original program, based on predefined rules named mutation operators. Such operators can, for instance, invert relational operators (e.g., replacing \(\ge\) with \(\lt\)).

Mutants can be used to indicate the strengths of test suites, based on their ability to distinguish the mutants from the original program. If there exists a test case distinguishing the original program from a particular mutant, then the mutant is said to be killed. Then, we term a mutant to be “coupled” with respect to a particular fault if the test cases that kill it are a subset of the test cases that can also detect that fault (make the program fail by exerting the fault).

Previous research has shown that the choice of mutation operators and location can affect the fault-revealing ability of the produced mutants [5, 35]. Thus, it is important to select appropriate mutation testing strategies. Nevertheless, previous research has shown that random mutant sampling achieves comparable results with the mutation testing state of the art [9, 32], making the random mutant sampling a natural baseline to compare with.

Another issue with mutation testing regards its application cost. The problem stems from the vast number of faults that are injected, which need to be executed with large test suites, thereby requiring expensive computational resources [60]. Unfortunately, the mutant execution problem becomes intractable when test execution is expensive or the test suites involve system-level tests, thereby often limiting mutation testing application to unit level. This is a major problem when performing fault tolerance [51] or large-scale testing campaigns. Recent studies aim at reducing the computational demands of the mutant execution through a combination of static and dynamic metrics [82], but these methods cannot be applied for fault tolerance assessment and do not identify which mutants are realistic and which are not. Thus, it remains an open question to identify the few but realistic mutants.

We fill this gap by using bug-report-driven fault injection. In essence, we leverage IR-based fault localization techniques to identify the locations where fault injection should happen, i.e., locations relevant to the targeted functionality described in the bug report, and apply frequent fault patterns to produce mutants that behave similarly to real faults.

3.3 Fix Patterns

In automated program repair [22], a common way to generate patches is to apply fix patterns [26] (also named fix templates [38] or program transformation schemes [23]) in suspicious program locations (detected by fault localization). Patterns used in the literature [15, 23, 26, 31, 38, 38, 39, 46, 66] have been defined manually or automatically (mined from bug fix datasets).

Instead of fix patterns, we use fault patterns that are fix patterns inverted. Since fix patterns were designed using recurrent faults, their related fault patterns introduce them. This helps injecting faults that are similar to those described in the bug reports. iBiR inverts and uses the patterns implemented by TBar [40] as we detail in the following section.

4.1 Bug-report-driven Fault Localization

IR-based fault localisation (IRFL) [63, 74] leverages potential similarity between the terms used in a bug report and the program source code to identify relevant buggy code locations. It typically starts by extracting tokens from a given bug report to formulate a query to be matched in a search space of documents formed by the collections of source code files and indexed through tokens extracted from source code [42, 65, 75, 77, 78, 84]. IRFL approaches then rank the documents based on a probability of relevance. Top-ranked files are likely to contain the buggy code.

We follow the same principle to identify promising locations where to inject realistic faults. We rely on the information contained in the bug report to localize the code location with the highest similarity score. Most IRFL techniques have focused on file-level localization, which is too coarse-grained for our purpose of fault injecting. Thus, we rather use a statement-level IRFL approach that has been successfully applied to support program repair [30].

It is to be noted that, contrary to program repair, we do not aim to identify the exact bug location. We are rather interested in locations that allow injecting realistic faults (similar to the bug). This means that IRFL may pinpoint multiple locations of interest for fault injection even if those were not buggy code locations.

To identify fault injection locations that are relevant to the targeted bug report, we leverage an existing IRFL tool that was originally developed as part of the iFixR [30] tool. The IRFL works by matching words of a bug report with source code file(s) using 17 features. These features are extracted from the bug report (7 features) and the source code git repository (10 features) and are listed in Table 1.

For every feature, the tokenizer applies a lexical analysis where (1) it extracts tokens from the retrieved text; (2) then drops stopwords to reduce the noise, i.e., caused by the programming language keywords; and (3) applies stemming on all tokens to create homogeneity with the root of the token. The tokens are extracted by considering both white space and source-code-specific separators, such as punctuation and camel case splitting; i.e., \(calculateMaximum\) is split to \(calculate\) and \(Maximum\). The tokens are then checked against the WordNet [16] dictionary to discard all unknown ones. An additional sanity check is then applied to detect stack-traces and source code elements using specific regular expressions.

The IRFL calculates then the similarity coefficient (\(Cosine\) [64]) between the bug report and a source code file using a revised Vector Space Model (rVSM) [85] based on the occurrence frequency of the extracted tokens in the preprocessing tokenization step (the vectors are calculated using \(tf-idf\) [47]).

Next, an ensemble of classification models provided by D&C [29] was used in order to rank the source code files according to their suspiciousness. This ensemble takes as input the calculated \(7\times 10\) weights of all pairs <bug report, source code file> and outputs their averaged prediction results. This ensemble was used as it has been shown to work well on a diverse set of bug reports [29] since every classifier of the ensemble model was trained on a different set of data.

In a last step, as iFixR [30], the IRFL localizes suspicious statements from the 20 most suspicious files based on their rVSM cosine-similarity [64] with the given bug report (the vectors are calculated using \(tf-idf\) [47]) and outputs these statements in a list of statements ranked according to their suspiciousness. Further details on the IRFL can be found in the D&C work [29] and our implementation [1].

4.2 Fault Patterns

We start from the fix patterns developed in TBar [40], a state-of-the-art pattern-based program repair tool. Any pattern is described by a context, i.e., an AST node type to which the pattern applies, and a recipe, a syntactical modification to be performed similar to program repair techniques [76]. For each pattern, we define a related fault injection pattern that represents the inverse of that pattern. For instance, inverting the fix pattern that consists of adding an arbitrary statement yields a remove statement fault pattern. Interestingly, some fix patterns are symmetric in the sense that their inverse pattern is also a fix pattern, e.g., inverting a Boolean connector. These patterns can thus be used for both bug fixing and fault injection. Table 2 enumerates the resulting set of fault injection patterns used by our approach.

Given a location (code statement) to inject a fault into, we identify the patterns that can be applied to the statement. To do so, our method starts from the AST node of the statement and visits it exhaustively, in a breadth-first manner. Each time it meets an AST node that matches the context of a fault pattern, it memorizes the node and the pattern for later application. Then the method continues until it has visited all AST nodes under the statement node. This way, we enumerate all possible applications of all fault patterns onto the location.

Since more than one pattern may apply to a given location, we prioritize them by leveraging heuristic priority rules previously defined in automated program repair methods (these were inferred from real-world bug occurrences [40]). This means that every fault injection pattern gets the priority order of its inverse fix pattern.

4.3 Fault Injection

The last step consists of applying, one by one, the fault patterns to inject faults at the program locations identified by IRFL. Locations of higher ranking are considered first. Within a location, pattern applications are ordered based on the pattern priority. By applying a pattern to a corresponding AST node of the location, we inject a fault within the program before recompiling it. If the program does not compile, we discard the fault and restart with the next one. We continue the process until it reaches the desired number of (compilable) injected faults or all locations and patterns have been considered.

4.4 Demonstration Example

Figures 2 and 3 illustrate the execution steps of iBiR when injecting faults in commons-math project, based on the content of the bug report MATH-329.²

Fig. 2.

View Figure

Fig. 3.

View Figure

iBiR starts by parsing the bug report and extracting its relevant information: the summary (1), the summary hints (2), the description (3), the description hints (4), code elements (5), and the raw bug report. This example bug report does not contain any stack-trace as the corresponding bug causes a misbehavior but does not trigger any crash or throw any exception.

iBiR loads also all the required information from the project’s repository (6) and then uses all of these features to find the code locations that are the most likely related to the input bug report. This search happens in two steps—file-level then statement-level localization—and ends by the output of a sorted list of source-code lines (7), as detailed in Section 4.1.

iBiR parses these lines one by one starting with the highest rank. In this example, the first rank is attributed to the line number 303 of the file src/main/java/org/apache/ commons/math/stat/Frequency.java (8), which corresponds to a return statement that invokes the method getPct with a variable v, which is cast to the type Comparable. iBiR selects all compatible fault patterns with this statement’s AST and applies them one by one on the source code, inducing multiple faults. In Figure 3 we illustrate the modified source code corresponding to five faults injected in the line 303 of the Frequency.java file (9): Faults 1 and 2 are injected by invoking respectively the methods getCumPct and getCumFreq instead of getPct. In fault 3, the method getPct is invoked with the field this.freqTable as variable instead of v. Faults 4 and 5 are injected by inserting additional method calls before the return statement, respectively addValue(v); and clear();.

iBiR continues parsing the sorted source code locations by the IRFL until all of them are treated or the requested number of faults has been injected.

6.1 Dataset and Benchmark

To evaluate iBiR we needed a set of benchmark programs, faults, and bug reports. We decided to use Defects4J [24] since it is a benchmark that includes real-world bugs and it is quite popular in software engineering literature.

6.2 Experimental Procedure

To compare the fault injection techniques we need to set a common basis for comparison. We set this basis as the number of injected faults since it forms a standard cost metric [53] that puts the studied methods under the same cost level. We used sets of 5, 10, 30, and 100 injected faults since our aim is to equip researchers with few representative faults, per targeted fault, in order to reach reasonable execution demands. To reduce the arbitrariness due to the stochastic nature of mutation testing, we reproduced the injection 15 times, and then we sorted the executions by their average Ochiai coefficient (for every bug separately) and we reported the mean execution. On the other hand, we run iBiR only once as its approach does not depend on random decisions.

To measure how well the injected faults imitate the real ones (answer RQ1 and RQ2) we use a semantic similarity metric (Ochiai coefficient) between the test failures on the injected and real (targeted) faults. Precisely, let \(fTS_M\) and \(fTS_B\) be the sets of failing tests when executing a test suite \(TS\) correspondingly on a mutant \(M\) and a buggy project \(B\); the Ochiai coefficient is 0 if any of \(fTS_M\) or \(fTS_B\) is empty, or else is calculated as \(Ochiai(M,B) = \frac{|fTS_M \cap fTS_B|}{\sqrt {|fTS_M|. |fTS_B|}}\), where \(|set|\) denotes the set size. In our study, as we’re executing the fixed-version test suites provided by Defects4J, every targeted bug breaks at least one test, and thus, \(fTS_B\) is never empty. This coefficient quantifies the similarity level of the program behaviors exercised by the test suites and is often used in mutation testing literature [62]. The metric takes values in the range [0, 1], with 0 indicating complete difference and 1 exact match. We treated the injected faults that were not detected by any of the test suites as equivalent mutants [6, 58]. This choice does not affect our results since we approximate the program behaviors through the projects test suites; i.e., they are never killed.

To measure whether the injected faults couple with the existing ones (answer RQ3), we followed the process suggested by Just et al. [25] and identified whether there were any injected faults that were killed by at least one failing test (test that detects the real fault) and not by any passing test (test that does not detect the real fault). In RQ4 we randomly sampled 50 test suites, random subsets of the accompanied test suites, that included between 10% and 30% test cases of the original test suite (provided by Defects4J). Thus, we ensure that the selected samples (1) are smaller than the original test suite, (2) have different sizes, and (3) have different ratios of killing the mutants and detecting the targeted bug. Then we recorded the ratios of the injected faults that are detected when injecting 5, 10, 30, and 100 faults. We also recorded binary variables indicating whether or not each test suite detects the targeted fault. This process simulates cases where test suites of different strengths are compared. Based on these data, we computed two statistical correlation coefficients, the Kendall and Pearson.

To further validate whether the two approaches provide sufficient indicators on the effectiveness of the test suites, we check whether the detection ratios of the injected faults are statistically higher when test suites detect the targeted faults than when they do not.

To reduce the influence of stochastic effects we used the Wilcoxon test with a significance level of 0.05. This helped in deciding whether the differences we observe can be characterized as statistically significant. Statistical significance does not imply sizable differences, and thus, we also used the Vargha Delaney effect size \(\hat{\text{A}}_{12}\) [71]. In essence, the \(\hat{\text{A}}_{12}\) values quantify the level of the differences. For instance, a value \(\hat{\text{A}}_{12} = 0.5\) can be interpreted as a tendency of equal value of the two samples. \(\hat{\text{A}}_{12} \gt 0.5\) suggests that the first set has higher values, while \(\hat{\text{A}}_{12} \lt 0.5\) suggests the opposite.

6.3 Implementation

To perform our experiments, we implemented iBiR’s approach as described in Section 4: we have used the IRFL implementation proposed in iFixR [30] and implemented the mutator component that is responsible for injecting faults in specific locations, as a java standalone application. Second, for the mutation testing, denoted as “Mutation” in our experiments, we used randomly sampled mutants from those produced by typical mutation operators, coming from the mutation testing literature. In particular, we implemented the muJava intra-method mutation operators [43], which are the most frequently used [28]. Third, to reduce the noise from stillborn mutants, i.e., mutants that do not compile, we discarded without taking into any consideration, i.e., prior to our experiment, every mutant that did not compile or whose execution with the test suite exceeded a timeout of 5 minutes. Fourth, when answering the RQ1, we found out that there were many cases where iBiR injected fewer than 100 faults. To perform a fair comparison, we discarded these cases (for both approaches). This means that we always report results where both studied approaches manage to inject the same number of faults.

7.1 RQ1: Semantic Similarity between iBiR Injections and the Targeted Real Faults

To check whether the injected faults imitate well the targeted ones, we measured their behavior (semantic) similarity w.r.t. the project test suites (please refer to Section 6 for details). Figure 4 shows the distribution of the similarity coefficient values that were recorded in our study. As can be seen, iBiR injects hundreds of faults that are similar to real ones, whereas mutation testing (denoted as “ Mutation” in Figure 4) did not manage to generate any. At the same time, as typically happens in mutation testing [62], a large number of injected faults have low similarity. This is evident in our data, where mutations have 0 similarity.

Fig. 4.

View Figure

To investigate whether iBiR successfully injects any fault that is similar (semantically) to the targeted ones, we collected the best similarity coefficients, per targeted fault, when injecting 5, 10, 30, and 100 faults. Figure 5 shows the distribution of these results. For more than half of the targeted faults, iBiR yields a best similarity value higher than 0.5 when injecting 100 faults, indicating that iBiR’s faults imitate relatively well the targeted ones. We also observe that in many faults the best similarity values are above 0 by injecting just 10 faults. This is important since it indicates that iBiR successfully identifies relevant locations for fault injection.

Fig. 5.

View Figure

To establish a baseline and better understand the value of iBiR, we need to contrast iBiR’s performance with that of mutation testing when injecting the same number of faults. Mutation testing forms the current SoA of fault injection and thus a related baseline. As can be seen from Figure 5, the similarity values of mutation testing are significantly lower than those of iBiR.

Figure 6 shows the distribution of the semantic similarities, between real and injected faults, when injecting 5, 10, 30, and 100 faults. As can be seen from the boxplots, the trend is that a large portion of faults injected by iBiR have positive similarity scores with the targeted ones.

Fig. 6.

View Figure

Interestingly, in mutation testing, only outliers have their similarity above 0. In particular, mutation testing injected faults with similarity values higher than 0 in 87, 112, 145, and 189 of the targeted faults (when injecting 5, 10, 30, and 100 faults), while iBiR injected in 130, 156, 190, and 226 of the targeted faults, respectively.

To validate this finding, we performed a statistical test (Wilcoxon paired test) on the data of both Figures 5 and 6 to check for significant differences. Our results showed that the differences are significant, indicating the low probability of this effect to be happening by chance. The size of the difference is also big, with iBiR yielding \(\hat{\text{A}}_{12}\) values between 0.64 and 0.68, indicating that iBiR injects faults with higher semantic similarity to real ones in the great majority of the cases. Due to the many cases with 0 similarity values, the average similarity value of iBiR’s faults is 0.163, while for mutation it is 0.010, indicating the superiority of iBiR.

7.2 RQ2: iBiR vs. Mutation Testing at Particular Classes

To check the performance of iBiR at the class level of granularity we repeated our analysis by discarding, from our priority lists, every mutant that is not located on the targeted classes, i.e., classes where the targeted faults have been fixed. Figure 7 shows the distribution of the semantic similarities when injecting 5, 10, 30, and 100 faults at a particular class. As expected, mutation testing scores are higher than those presented before, but still mutation testing falls behind.

Fig. 7.

View Figure

To validate this finding, we performed a statistical test and found that the differences are significant. The size of the difference is between 0.62 and 0.65, meaning that iBiR scores more than 60% times higher than mutation testing. The average similarity values of the iBiR faults is 0.217, while for mutation it is 0.066, indicating that iBiR is better.

7.3 RQ3: Fault Coupling

The coupling between the injected and the real faults forms a fundamental assumption of the fault-based testing approaches [24]. An injected fault is coupled to a real one when a test case that reveals the injected fault also reveals the real fault [24]. This implies that revealing these coupled injected faults results in revealing potential real ones. We therefore check this property in the faults we inject and contrast it with the baseline mutation testing approach.

Figure 8 shows the percentage of targeted faults where there is at least one injected fault that is coupled to a real one. This is shown for the scenarios where 5, 10, 30, and 100 faults per target are injected. As we can see from these data, iBiR injects coupled faults for approximately 16% of the target faults when it aims at injecting 5 faults. This percentage increases to 36% when the number of injected faults is increased to 100.

Fig. 8.

View Figure

Perhaps surprisingly, mutation testing did not perform well (it injected coupled faults for around 4% of the target, when injecting 100 faults per target). These results differ from those reported by previous research [25, 62], because (1) previous research only injected faults at the faulty classes and not the entire project and (2) previous research injected all possible mutant instances and not 100 as we do.

7.4 RQ4: Fault Detection Estimates

The results presented so far provide evidence that some of the injected faults imitate well the targeted ones, though the question of whether the injections provide representative results of real faults remains, especially since we observe a large number of faults with low similarity values. Therefore, we check the correlations between the failure rates of the sets of injected faults and the real faults when executed with different test suites (please refer to Section 6 for details).

Figure 9 shows the distribution of the correlation coefficients when injecting different numbers of faults. Interestingly, the results on both figures show a trend in favor of iBiR. This difference is statistically significant, shown by a Wilcoxon test, with an effect size of approximately 0.6. Table 3 records the effect size values, \(\hat{\text{A}}_{12}\), for the examined strategies. In essence, these effect sizes mean that iBiR outperforms the mutant injection in 60% of the cases, suggesting that iBiR could be a much better choice than mutation testing, especially in cases of large test suites with expensive test executions.

Fig. 9.

View Figure

To further validate whether iBiR’s faults provide good indicators (estimates) of test effectiveness (fault detection) we split our test suites between those that detect the targeted faults and those that do not. We then tested whether detection ratios of the injected faults in the test suite group that detects the real faults are significantly (statistically) higher than those in the group that does not detect it. In case this happens, we have evidence that our injected faults favor test suites capable of detecting real faults. This is important when comparing test generation techniques, where the aim is to identify the most effective (at detecting faults) technique.

Figure 10 records the number of faults where (real) fault-detecting test suites detect a statistically higher number of injected faults than those test suites that do not detect them. As can be seen by these results, iBiR has a big difference from mutation; i.e., it distinguishes between passing and failing test suites in 126 faults, while mutation does so in 55 faults. We also measured the Vargha and Delaney \(\hat{\text{A}}_{12}\) effect size values on the same data, recorded in Figure 11. Of course it does not make sense to contrast insignificant cases, so we only performed that on the results where iBiR has a statistically significant difference. Interestingly, big differences are recorded (in approximately 80% of the cases) in favor of our approach.

Fig. 10.

View Figure

Fig. 11.

View Figure

8.1 Injecting Large Number of Faults

Figure 8 shows that iBiR injects many more faults that couple with the real ones than conventional mutation testing. In fact, it achieves a higher coupling percentage when injecting only 10 faults than the percentage achieved by conventional mutation testing when injecting 1,000 faults. We can see also that when injecting 1,000 faults we achieve the coupling percentages of 61.1% and 18.2% for, respectively, iBiR and mutation testing. This is obviously because the more faults we inject, the more chances we have to inject faults that couple with the real ones. Considering that injecting more faults comes with a considerable consequent cost increase, as the practitioners will need more time to analyze the produced mutants, this option is often not favored in practice, where it is better to have few relevant faults than many.

To have a better understanding of the impact of injecting multiple faults, we illustrate in Table 4 the averaged faults coupling success rates when injecting 5, 10, 30, 100, 200, 500, and 1,000 faults with iBiR and mutation testing. We define the success rate as the percentage of coupled faults among all the injected ones. As an example, a coupling success rate of 5% corresponds to 5 coupled faults when injecting 100 faults. In our study, iBiR achieves a much higher success rate than mutation testing: 20, 87, 49, and 36.7 times higher when injecting 5, 100, 500, and 1,000 faults. Even if the coupling percentage increases by injecting more faults (Figure 8), we can see that the more we inject faults, the more the success rate decreases for iBiR. This is a direct consequence of the decrease of the injection-locations likelihood to be related to the targeted bug report. As we explain further in Section 4, iBiR starts by injecting faults in the highly ranked code locations found by its IRFL, then iterates further until all locations are treated or the requested number of faults has been injected. So the higher the requested number of injected faults is, the more faults in lower-ranked locations are injected. On the other hand, we see that the success rate of conventional mutation testing remains relatively low and far behind the one of iBiR. For instance, it remains at 0.07% even when doubling the number of injected faults from 500 to 1,000. In Table 4, we notice that injecting five faults with mutation testing achieves a success rate of 0.29%, which is much higher than the ratios achieved when injecting more faults by the same technique. This is caused by the randomness in the conventional mutation testing results.

8.2 Distribution of the Patterns Inducing Most Effective Injections

To understand better the impact of the used patterns in injecting faults that are similar to real ones, we grouped the faults by their creating patterns and compared the sizes of each group. Figure 12 illustrates the proportion of every pattern’s induced faults that have high Ochiai Coefficients (more than 0.8) when injecting 1,000 faults by IBIR in the current dataset. Clearly, more than 70% of the faults with high similarity coefficients have been generated by patterns that are not commonly used in conventional mutation testing techniques: mainly by adding conditional expressions (42.3%) or by mutating variables (29.5%). This is significantly higher than the 15.3% generated with the commonly used conventional mutation operators (10.4% by mutating operators and 4.9% by removing statements). This highlights the fact that iBiR’s patterns are bringing a clear advantage over mutation testing.

Fig. 12.

View Figure

These percentages and the general performance of every pattern depend on the targeted bug report and the project nature. For instance, the low percentages of multiple patterns in Figure 12 can be the consequence of multiple factors, such as (1) the fact that some faults are occurring less frequently in the current dataset, (2) the fact that some patterns are only applicable on a few specific statement ASTs, or (3) that some patterns produce relatively more mutants in the same location and thus have higher percentages (i.e., the “Mutate Method Invocation,” which induced Fault 1 and Fault 2 in the same statement in Figure 3 in Section 4.4).

8.3 iBiR vs. Typical Mutation Operators

Early research on mutation testing defined mutation operators based on all possible simple removals or replacements of programming language elements [3, 27]. This practice was then adopted when defining mutation operators for other languages, such as Java, and in defining object-oriented related mutants [43, 54]. To reduce the number of mutants involved, many tool developers applied a restrictive set of mutation operators, usually referred to as the 5-operator set, based on the selective mutation testing studies performed by Offutt et al. [53, 55] with the result that the majority of modern mutation testing tools implement a version of this 5-operator set together with some deletion operators [34, 60].

In view of the above, all the iBiR injections that involve addition of code elements, i.e., “Insert Statement” and “Mutate Return Staement” categories of Table 2, are fundamentally different from what has been used in mutation testing studies over the years. The “Mutation Literal Expression” category is also something that has not been used by mutation testing studies. The rest of the operators have some similarities with operators used in some studies overall differing significantly from the operators used by any single tool or study. In the following we provide a detailed list of iBiR operators and their related similarities (or novelties) with respect to other studies.

Operators that have not been used by other studies:

Operators that have similarities with those used by other studies:

Operators that are frequently used by other studies:

8.4 Project Size and iBiR’s Effectiveness

Considering the fault injection as a search task where the target is injecting faults similar to real ones and the search space is the combination of the source code locations and mutation possibilities, we were interested in assessing iBiR’s performance for different project sizes. Figures 13(a) and 13(b) show the scatter plots of the semantic similarity by the project size in terms of number of classes. Figures 13(a) and 13(b) consider respectively all the injected faults and the faults having an Ochiai coefficient higher than zero. We can see that the project size has no impact on the effectiveness of iBiR.

Fig. 13.

View Figure