Constrained locating arrays for combinatorial interaction testing

This paper introduces the notion of Constrained Locating Arrays (CLAs), mathematical objects which can be used for fault localization in a testing process for information systems. CLAs extend ordinary locating arrays to make them applicable to testing of systems that have constraints on test parameters. Such constraints are common in real-world systems; thus CLA enhances the applicability of locating arrays to practical testing problems.


Introduction
Combinatorial interaction testing is a well-known strategy for software testing. In the strategy, a System Under Test (SUT) is modeled as a finite set of factors and every interaction of interest is exercised by at least one test. Empirical results suggest that testing interactions involving a fairly small number of factors, typically two or three, suffices to reveal most of latent faults. Many studies have been developed to construct a small test set for combinatorial interaction testing. Such test sets are often called Covering Arrays (CAs). Surveys on these studies can be found in, for example, [4,9,15].
An important direction of extending the capability of combinatorial interaction testing is to add fault localization capability to it. An enabling technique for this is using a Locating Array (LA) as a test suite [6]. In [6] LAs of a few different types are defined. For example, a (d, t)-LA enables to locate a set of d failure-triggering t-way interactions using the test outcome.
The purpose of this paper is to extend the notion of LAs to expand the applicability to practical testing problems. Specifically, we propose Constrained Video camera requires camera and color display Video ringtones cannot occur with No video camera The combination of 16 Million colors, Text email viewer and 2 Megapixel camera will not be supported Figure 1: Example of an SUT [3].
Locating Arrays (CLAs) which can be used to detect failure-triggering interactions in the presence of constraints. Constraints, which prohibit some particular tests, are common in real-world systems. Constraint handling in combinatorial interaction testing has been well studied [1]. The main focus of the previous studies is on constructing a test set, often called a Constrained Covering Array (CCA), that consists only of constraint-satisfying tests and covers all interactions that can occur in constraint-satisfying tests.
CLAs requires additional considerations about constraints. Specifically, constraints may make it impossible to distinguish a failure-triggering interaction or set of such interactions from another; hence a special treatment must be needed to deal with such an inherently indistinguishable pair. By extending LAs with the concept of distinguishability, we provide the formal definition of CLAs. We also show some non-trivial examples of CLAs.
The rest of the paper is organized as follows. Section 2 describes the SUT model and the definition of locating arrays, as well as some related notions. Section 3 presents the definition of CLAs and some basic theorems about them. Section 4 presents a few examples of CLAs together with a conceptual description of a possible approach of mechanical CLA constructions. Section 5 briefly summarizes related work. Section 6 concludes the paper with possible future directions of work.

Preliminaries
An SUT is modeled as F , S, φ where F = {F 1 , F 2 , ..., F k } is a set of factors, S = {S 1 , S 2 , ..., S k } is a set of domains for the factors, and φ : S 1 × ... × S k → {true, f alse} is a mapping that represents constraints. Each domain S i consists of two or more consecutive integers ranging from 0; i.e., is a t-way interaction or an interaction of strength t. Hence a test contains (covers) ( k t ) t-way interactions. (The only 0-way interaction is ∅.) Note that a k-way interaction {(1, σ 1 ), ..., (k, σ k )} and a test σ σ σ = (σ 1 , ..., σ k ) can be treated interchangeably. Thus we write T ⊆ σ σ σ iff a test σ σ σ covers an interaction T . An interaction T is valid iff T ⊆ σ σ σ for some valid test σ σ σ.
As a running example, consider a classic cell-phone example taken from [3] (Fig. 1). This SUT model has five factors which have three or two values in their domains. The constraints consist of seven parts. Test (1, 0, 1, 1, 1), for example, is valid, whereas test (1, 0, 0, 0, 1) is not valid (invalid) because it violates the third and fourth constraints. Similarly, two-way interaction {(1, 1), (2, 0)} is valid, since it occurs in valid test (1, 0, 1, 1, 1). On the other hand, A test suite is a (possibly empty) collection of tests and thus can be represented as an N × k array A when the number of tests is N . For such an array A and interaction T , we let ρ A (T ) denote the set of tests (rows) of A in which the interaction is covered. For a set of interactions T , we define An interaction is either failure-triggering or not. The result of executing a test σ σ σ is fail iff σ σ σ covers at least one failure-triggering interaction; otherwise the result is pass. Hence the result of executing a test suite A is a vector of size N , each element being either pass or fail.
When there are no constraints, i.e., φ(σ σ σ) = true for any σ σ σ, a CA can be used to detect the existence of fault-triggering interactions of a given strength t or less. Let I t be the set of all t-way interactions. Formally, a t-CA is defined by the following condition: On the other hand, an LA can be used as a test suite that locates the set of failure-triggering interactions. Colbourn and McClary introduced a total of six types of LAs in [6]. The definitions of the two most basic types of LAs are shown below.
The definition of other two types of LAs, namely (d, t)-LAs and (d, t)-LAs, requires the notion of independence [6]. Let I t be the set of all interactions of strength at most t, i.e., I t = I 0 ∪ I 1 ∪ ... ∪ I t . A set of interactions (interaction set) T ⊆ I t is independent iff there do not exist two interactions T, T ′ ∈ T with T ⊂ T ′ . For example, consider a set of two interactions {{(1, 1)}, {(1, 1), (2, 0)}} (⊆ I 2 ) for the running example. This interaction set is Figure 2: (1,2)-LA for the running example. Constraints are not taken into account.
Note that if two interactions T, T ′ are both failure-triggering and T ⊂ T ′ , then the failure triggered by T always masks the failure triggered by T ′ . Because of this, it is natural to limit the scope of fault localization to independent interaction sets. Based on I t and the notion of independent interaction sets, the two types of LAs are defined as follows.
We do not consider the remaining two types of locating arrays, namely (d, t)-LAs and (d, t)-LAs, because they either exist in trivial cases or otherwise are equivalent to (d, t)-and (d, t)-LAs. Figure 2 shows a (1, 2)-LA for the running example shown in Fig. 1. Let A be the LA and a a a i (1 ≤ i ≤ N = 15) be the ith row. If the pass/fail result were obtained for all these tests, any failure-triggering single two-way interaction could be identified. For example, if only the first test a a a 1 failed, then the failuretriggering interaction would be determined to be {(2, 0), (3, 0)}, because ρ(T ) = {a a a 1 } holds only for T = {{(2, 0), (3, 0)}}, provided that |T | = 1 and |T | = 2 for T ∈ T . However, this array cannot be used for testing the system because of the constraints. For example, a a a 1 is not valid and thus cannot be executed in reality.

Constrained locating arrays
In the presence of constrains, a test suite must consist of only valid tests. Clearly, this means that invalid interactions cannot and need not be located. From now on, we assume that an array A representing a test suite consists of only valid tests or no rows.
In the field of practical software testing, this problem has been circumvented by, instead of CAs, using Constrained Covering Arrays (CCAs). Let VI t be the set of all valid t-way interactions. Then a CCA of strength t, denoted as t-CCA, is defined as follows.

t-CCA
∀T ∈ VI t : ρ A (T ) = ∅ In words, a t-CCA is an array that covers all valid interactions of strength t. When incorporating constraints into LA, it is crucial to take into consideration, in addition to the presence of invalid interactions, the fact that constraints may make it impossible to identify some set of failure-triggering interactions, which could be identified if no constraints existed. This requires us the notion of distinguishability to formally define CLAs.
Definition 1 A pair of sets of valid interactions, T 1 and T 2 , are distinguishable iff ρ A (T 1 ) = ρ A (T 2 ) for some array A consisting of valid tests. Definition 2 Let d ≥ 0 and 0 ≤ t ≤ k. Let VI t be the set of all valid t-way interactions and VI t be the set of all valid interactions of strength at most t. An array A that consists of valid tests or no rows is a (d, t)-, (d, t)-, (d, t)-or (d, t)-CLA iff the corresponding condition shown below holds.
(d, t)-CLA ∀T 1 , T 2 ⊆ VI t such that |T 1 | = |T 2 | = d and T 1 , are independent and distinguishable: ρ A (T 1 ) = ρ A (T 2 ) (In extreme cases where no two such interaction sets T 1 , T 2 exist, any A is a CLA.) The intuition of the definition is that if the SUT has a set of d (or ≤ d) failure-triggering interactions, then the test outcome obtained by executing all tests in A will be different from the one that would be obtained when the SUT had a different set of d (or ≤ d) failure-triggering interactions, unless the two interaction sets are not distinguishable.
The following theorem follows from the definition. Theorem 2 Suppose that the SUT has no constraints, i.e., φ(σ σ σ) = true for all σ σ σ. If an LA A exists, then 1) A is a CLA with the same parameters, and 2) any CLA with the same parameters as A is an LA (which is possibly different from A) with the same parameters.
Proof. Suppose that A is a (d, t)-LA. Let T 1 , T 2 ⊆ I t (= VI t ) be any two interaction sets such that and thus they are distinguishable, which in turn implies ρ A ′ (T 1 ) = ρ A ′ (T 2 ). If T 1 = T 2 , then they are not distinguishable and trivially ρ A ′ (T 1 ) = ρ A ′ (T 2 ). Hence A ′ is a (d, t)-LA. The same argument applies to the other three types of LAs.
It should be noted that a CLA always exists whether there are constraints or not, as will be shown in Theorem 3. On the other hand, LAs do not always exist. For example, no (2, 1)-LAs exist for the running example: Consider T 1 = {{(4, 1)}, {(4, 2)}} and T 2 = {{(5, 1)}, {(5, 2)}}. Then ρ A (T 1 ) and ρ A (T 2 ) both include all rows; thus ρ A (T 1 ) = ρ A (T 2 ) for any A. The guarantee of the existence of CLAs comes from the definition which exempts indistinguishable pairs of interaction sets from fault localization. In that sense, CLAs can be viewed as a "best effort" variant of LAs.

Theorem 3 If
A is an array consisting of all valid tests, then A is a CLA with any parameters.
Although Theorem 3 guarantees that a test suite consisting of all valid tests is a CLA, it is desirable to use a smaller test suite in practice. In the next section, we show some CLAs for the running SUT example, which are the smallest we have obtained for particular parameters.
In the remaining part of the section, we briefly describe how to obtain such CLAs. In the first step, all valid interactions of strength t or ≤ t, i.e., VI t or VI t , are enumerated. This can be done by enumerating all valid tests and then enumerating all interactions of strength t or ≤ t that are contained in at least one of the valid tests. For the running example, all interactions of strength ≤ 2 are valid, except ten two-way interactions listed below.  Figure 7 shows pairs of interaction sets that are not distinguishable for the other parameters d(d), t(t).
Finally, an array consisting of valid tests is somehow created as a candidate for a CLA and then checked to see if the array meets the condition of a CLA. If the array is not a CLA, then a new array is repeatedly created and checked until a CLA is found.
In CCA generation algorithms, the repeated creation of candidate arrays is often performed by meta-heuristic algorithms, such as simulated annealing [8]  or tabu search [11], or by systematic search based on combinatorial problem solvers, such as Boolean satisfiability (SAT) solvers [14,20]. We believe that both approaches can also be used to generate CLAs. Indeed we obtained the two CLAs with the aid of a combinatorial problem solver. Although the details of how we used it are beyond the scope of the paper, we plan to report them in future.

Related Work
Constraint handling has been an important issue in combinatorial interaction testing, even before the name of this testing approach was coined. Early work includes, for example, [19,2]. A recent systematic literature review on constrained interaction testing listed 103 research papers addressing this particular problem [1]. In contrast, research on LAs is still in the early stage [7]. Mathematical properties and constructions of LAs are presented in [17,18,5], but only for some specific cases. Computational constructions of LAs are proposed in [13,10]. None of these previous studies consider constraints. Mathematical objects similar to LAs include Detecting Arrays [6,16] and Error Locating Arrays [12]. To our knowledge, no attempts have been reported to incorporate constraints into these arrays, either.

Conclusions
In this paper, we introduced the notion of Constrained Locating Arrays (CLA), which generalize locating arrays by incorporating constraints on test parameters into them. The extension enables to apply locating arrays to testing of real-world systems which usually have such constraints. We proved some basic properties of CLAs and showed a few nontrivial examples of CLAs. Possible future research directions include, for example, devising systematic constructions of CLAs and investigating the usefulness of CLAs in real-world testing.