Testing with Büchi Automata: Transition Coverage Metrics, Performance Analysis, and Property Refinement

Abstract

Büchi automaton is one of the most commonly used formalisms for specifying and reasoning linear temporal properties. It is instrumental for developing model-checking algorithms for linear temporal logics. Until now Büchi automaton-based specification is primarily used in linear temporal logic model checking. In this paper, we develop test criteria and techniques essential for testing upon specifications in Büchi automata. These criteria measure the semantic relevancy of test cases to a requirement in Büchi automaton. We define the “weak” and “strong” variants of the criteria based on a test suite’s coverage on the transitions of a Büchi automaton. These criteria may be used to measure the quality of existing test cases with respect to requirements in Büchi automaton, and to drive test-case generation. We develop two algorithms that use an off-the-shelf model checker to automate test-case generation for the proposed test criteria. In our extended computational study we deploy two methodologies to measure the effectiveness of our approach. First, we measure the cross coverage of the transition coverage criteria against other existing test criteria. Second, we use a fault-injection technique to measure the sensitivity of our approach. In both cases, our approach shows a better performance compared with existing test criteria, and a good sensitivity in detecting errors systematically injected to a system. Furthermore, the proposed criteria uncover not only the deficiency of a test suite with respect to a linear temporal requirement, but also that of the requirement itself. We propose an algorithm to refine the requirement using the feedback from test-case generation.