Automating test case design within the classification tree editor

This paper describes how the proven test design technique of the classification tree method is extended within the classification tree editor in order to contribute to current test design matters. The classification tree editor not only provides the tooling to use the method but also to apply new and helpful features in the test design process. This includes the automatic generation of test cases and test sequences according to desired test depth and focus, automated boundary value analysis, various tool couplings to integrate in each individual test process and supporting features like test evaluation or test coverage analysis amongst others.


I. INTRODUCTION
T HE classification tree method [1] as well as the editor [2]   have been developed at Daimler's research department for software technology in the 90ties.In the last 20 years both, method and tool became proven in practice around the world.
The classification tree method as a black box test design technique was introduced by Matthias Grochtmann and Klaus Grimm in 1993 [1].The basic idea of the classification tree method is to separate the input data characteristics of the system under test into different classes that directly reflect the relevant test scenarios.The descriptive method covers the categories of test case design needed in industry, such as equivalence class tests, boundary value analysis [3], interface testing, as well as combinatorial [4] and statistical testing [5].
In the last 6 years where Berner and Mattner took over the further development of the classification tree method and the editor, it got extended by academic research results and industrial input.This includes additional statistical testing methods [5], a professional requirements tracing [6], a synchronization to test management, test evaluation, coverage analysis or support for website testing [7].The classification tree editor supports thus the tester in the test design phase.It offers multiple functions to structure the test problem, to systematically generate test scenarios and to do all this within the scope of the testers environment.
The outline of this paper is as follows: Section II introduces the classification tree method, while Section III shows how it will prove to serve testers needs for a more automated way in designing test cases within the classification tree editor.Section IV the integration into the test process with requirements and test management tools are introduced.Section V details Excel import and combinatorial test coverage analysis, test evaluation is given in Section VI.Related work can be found in Section VII, while conclusion is drawn in Section VIII.

A. How to create a classification tree
The basic idea of the classification tree method is to separate the input data characteristics of the system under test into different classes that directly reflect the relevant test scenarios (classifications) [1].The main source of information is the specification of the system under test or a functional understanding of the system should no specification exist.Two significant steps must be performed to create a classification tree: • The identification of relevant factors involves the determination and structuring of the relevant test scenarios and their interrelations to other parts of the system under test.
• The test specifications combine the relevant factors needed in order to achieve the desired test coverage.The first phase begins with the identification of the classifications that are relevant for testing based on a functional description and understanding of the system under test.For each classification, there may be several input data that are all to be considered during testing, the classes of a classification.
Each classification should have a limited number of clearly defined input scenarios for the system under test.The input data characteristics are used to define the input ranges required for the relevant test scenarios.This is a classification in the mathematical sense: The set of all possible inputs is disjointly and completely classified into subsets -the classes.The separation based on the input data characteristics is done independently for each test scenario, and can therefore be done easily.
This approach applies the concept of equivalence class testing [3]: Testing with a data item that is representative of an equivalence class makes tests with all other elements of the same class redundant and therefore unnecessary because there should be no difference in how they are handled during execution of the system under test.
Example. Figure 1 shows a classification tree for a database management system.Three aspects of interest (Access Method, Operation and Privileges) have been identified for the system under test.The classifications are partitioned into classes which represent the partitioning of the concrete input values.In our example the refinement aspect JavaScript is identified for the class Browser and it is divided further into two classes Yes and No.

B. Dependency Rules
Often, there are dependencies or constraints [8] between some classes of the classification tree.To overcome this design gap, the CTE offers to describe the relationships between the elements of the classification tree by defining dependency rules.The classification tree editor provides two mechanisms for defining dependency rules: 1) Logical dependency rules between the classes of a classification tree using propositional logic.The result is that test cases that would not fulfill the previously defined rules are not generated and vice versa [9].2) Numerical dependencies between the classifications using logical and numerical operators.They serve to express mathematical dependencies between the elements of the classification tree [10].

C. Boundary Value Analysis
The boundary value analysis [3] is a helpful tool to run an analysis automatically with user specified input.There are two ways of applying the boundary value analysis within the CTE.The first is for an initial analysis, the second is to analyze and expand a classification tree with more possible parameters and intervals see Fig. 2.
For the initial analysis, the CTE supports to create parameters and intervals.For each interval it is possible to set the boundary borders so that CTE will create the corresponding boundary values.The classification tree is created from the specified values.
For the analysis or expansion of the tree, the CTE loads the existing data into its own data model.Then it is possible to add new parameters and intervals.The CTE then generates the boundary values and adds additional elements to the tree.In practice, not every test generation mechanism is applicable to every test problem.Thus, there are numerous facilities available within the classification tree editor (CTE) in order to serve testers in many ways.The following test case generation mechanisms are implemented in the CTE.
• combinatorial test case generation • prioritized test case generation • test sequence generation

A. Combinatorial Test Case Generation
For combinatorial testing, the question is on how to achieve adequate test coverage without having to test too much.Complex software testing problems easily reach a maximum number of test cases of several billions.This is not feasible to handle.So a wise, structured and reproducible selection of test cases according to the current test problem saves effort, time and helps to actually better understand the testing focus.For this, combinatorial testing with all of its aspects is ideal.It brings variation to a test suite with a clear definition of the test intensity.The CTE offers combinatorial test case generation facilities that support the structured variation [5], [9].This includes pairwise combination, threewise combination, minimal test coverage and individual combinatorial rules.
The minimal combination A + B ensures that each class of classification A and each class of classification B is considered in at least one test case.
The pairwise combination (A 1 , . . ., A n ) ensures that each class of the classifications A 1 to A n is combined pairwise with every other class in at least one test case.
The threewise combination (A 1 , . . ., A n ) ensures that every possible combination of three classes of the different classifications A 1 to A n is generated in at least one test case.
Higher strength interaction coverage can be achieved by practical testing approaches [11].The key to unlocking better performance for higher strengths of interaction, seems to rely upon the use of constraints, which reduce the number of possibilities to be considered.In CTE, constraints are defined in terms of dependency rules.
Individual combination rules may be designed upon the needs of each test problem.All standard operators are available for that.
Logical expressions are used to formulate dependency rules.The CTE XL allows to specify any kind of logical dependency rules, containing {AND, OR, NOT, ⇒, ⇔, XOR, NOR, NAND }.Parentheses are used to formulate more complex expressions [9].

B. Prioritized Test Case Generation
In addition to the above mentioned classical combinatorial test case generation, the CTE offers the possibility to add weights to classes for a prioritized test case generation [5].
Weights on classes can be used to create test cases in an order corresponding to their relevance.For example, those tests that have revealed most of the failures in previous runs can be executed more frequently.
Within CTE, weights are distributed to classes in order to generate prioritized test cases according to occurrence, error or risk probability.The result is a test suite of tests covering the pairwise combination criterion and being sorted according to their relevance most relevant test cases at the top of the test suite, least relevant at the bottom [12].By using the optimize menu, the tester can adjust the test suite individually by selecting the weighted coverage see Fig. 3.

C. Test Sequence Generation
Many software-based systems are state-based.Thus, test data used in test steps of one test sequence must provide a logical sequence to run the desired state transitions.The manual modeling of test sequences important for testing is a challenging task for the tester.Within the CTE this can be done automatically [13].
The tester defines simple state machines [14], modeling the behavior of the system see Fig. 4. Test cases are then derived from that.
A possible application for Hardware-in-the-loop testing has been discussed recently [15].

A. Requirements Tracing within CTE
In the development process, test design is not the first action to take.Before even thinking about testing, the function must be specified.For this, there are several requirements management tools on the market which serve to define and monitor specifications.
For test design, a connection to those requirements might be a huge advantage with regard to traceability.Also, when linking requirements to test cases, gaps can be detected.
The CTE offers the tester to link tree elements and test cases to requirements from MS Access or IBM Rational DOORS [6].The result is a requirements matrix that directly shows where there is still work to do (Fig. 5).Linked and not yet used requirements can be visualized.Often, requirements change in the process.For this, synchronizing the changed database shows the tester in CTE where to modify the classification tree in order to be conform to the specification [6].
This connection is currently one way to get information from DOORS to use in CTE.In several practical projects, a bidirectional connection has been established, where the results from CTE were exported to the corresponding DOORS module.

B. Test Management in CTE
Parallel to all activities in the test process, test management is essential.For this, CTE synchronizes with HP ALM to commit or submit test cases from and to the test management tool.All information relevant to the tester will be downloaded from the server and also uploaded with updated data.This is supported by a comprehensive mapping mechanism in CTE (Fig. 6).The classification tree will be saved to the central folder on the server in order to make it available for other users for systematic testing.Currently, the synchronization is made with the test plan module.For future work, it is planned to extend that connection to the requirements information in order to have a consistent and traceable process, and to extend it to read the information from the test lab which shows information to evaluate the performed tests.

A. Excel Import
The most common tool for test design is Excel.There are many different ways in practice to use it within the testing process.With this background, an Excel import to continue with a systematic testing approach in essential.
All Excel sheets no matter if the column or the rows represent the test cases are imported into CTE, as well as the already defined test cases.So the classification tree is built, the test cases are imported and further combinatorial testing can be performed.

B. Coverage Analysis
In practice, the need to give a value for coverage increases.The tester needs to know how good the defined test cases are.Within the CTE, the coverage analysis gives a basic overview of the covered tuples see Fig. 7.If we refer to the Excel import above, it is hard to tell how good or bad the manually defined test cases that have been imported are.With the coverage analysis function, these imported test cases can be compared to specific generation criteria like the pairwise rule, or the minimal coverage rule.According to the result, test suites can then be complemented in order to reach the desired coverage criteria.

VI. TEST EVALUATION
Throughout the test process, test evaluation is the final step to determine the relevance of the test results.Within CTE, test results for test cases can be either imported or added and then evaluated.The CTE can track test results in terms of Passed, Failed, Error, Not Executed.By means of a root cause analysis, the error rate of single classes can be detected and thus gives important information on possible defects of the system under test.
Elbaum et al. provide good overviews of existing prioritization approaches [26].There has been some work on test case prioritization that considered limited resources [27], [28].There are some known algorithm supporting prioritized test case generation.The first is an algorithm published in [29], which is an extension to [30].For efficiency reasons, this algorithm does not consider constraints.The other approach supports constraints and is based on Binary Decision Diagram (BDD) [31].CTE XL Professional also uses the latter [5], [12].Search based solutions for test case generations have been presented for both, conventional [8], [32] and prioritized [33] test case generation.
A body of work on the application of the classification tree method and CTE can be found in recent work [34]- [37].
An introduction to root cause analysis can be found in [38], while there are several case studies available, e.g.[39].The idea of combining the field of (combinatorial) testing with (root) cause analysis is not new, e.g.Cleve and Zeller try to find failure causes through automated testing [40].The reduction of test suites down to failure-causing combinations is the background of [41], [42], so this approach might not fit too well for regression testing, where new errors can occur.The adaptation of combinatorial testing has also been discussed more recently in [43].Beside widely positive works, Ramler et al. see limitations with the integration of cause analysis to combinatorial testing, although there is customer demand [44].
VIII.CONCLUSION Regardless of the system under test, level of integration, test phase and domain, the classification tree editor is universally applicable [34]- [37].This method has found a worldwide acceptance over the last two decades and has been used by commercial data processing, aviation and aerospace industries, and many others, for example, for examining the Hubble Space Telescope [45].
The systematic approach, starting from functional requirements, with understandable, reliable (intermediate) results, supported by an efficient, automatic test case generation, ensures that there are no gaps in the testing process and the resulting specifications.
Future work is twofold, first to ease the creation of classification trees, e.g. by importing logs [46] and second the transformation of test specification from the CTE to executable test cases.

Fig. 3 .
Fig. 3. Optimizing test suites according to weighted coverage in CTE

Fig. 5 .
Fig. 5. Requirements matrix view in CTE showing linked CTE objects

Fig. 6 .
Fig. 6.CTE view of the HP ALM synchronization status after adding new values

Fig. 7 .
Fig. 7. Test coverage of a set of existing test cases to the coverage rule pairwise combination

Fig. 8 .
Fig. 8. Test evaluation report in CTE showing the considered test items and their failure rate