Quality assuring the quality assurance tool: applying safety-critical concepts to test framework development

The quality of embedded systems is demonstrated by the performed tests. The quality of such tests is often dependent on the quality of one or more testing tools, especially in automated testing. Test automation is also central to the success of agile development. It is thus critical to ensure the quality of testing tools. This work explores how industries with agile processes can learn from safety-critical system development with regards to the quality assurance of the test framework development. Safety-critical systems typically need adherence to safety standards that often suggests substantial upfront documentation, plans and a long-term perspective on several development aspects. In contrast, agile approaches focus on quick adaptation, evolving software and incremental deliveries. This article identifies several approaches of quality assurance of software development tools in functional safety development and agile development. The extracted approaches are further analyzed and processed into candidate solutions, i.e., principles and practices for the test framework quality assurance applicable in an industrial context. An industrial focus group with experienced practitioners further validated the candidate solutions through moderated group discussions. The two main contributions from this study are: (i) 48 approaches and 25 derived candidate solutions for test framework quality assurance in four categories (development, analysis, run-time measures, and validation and verification) with related insights, e.g., a test framework should be perceived as a tool-chain and not a single tool, (ii) the perceived value of the candidate solutions in industry as collected from the focus group.


A.1 Suggested Supporting Documentation
The focus group identified a need for supporting documentation. Thus, identified activities listed in the DoDs may refer to one or more of the following supporting documents: (1) Guidelines for branching in source code version-control systems, (2) Guidelines for writing and documenting requirements, (3) Development guidelines, (4) Coding style, (5) Development checklist, (6) Guidelines for documentation, as well as (7) Guidelines for conducting peer-reviews.

A.2 Planning Phase Definition of Done
The suggested activities to be completed during the planning phase, before transitioning to the development phase, are: P1 Branch(es) created. Branching is done at an early stage to enable documentation of requirements during planning, and to isolate it from the stable framework branch.
P2 Proposed change(s) clarified as "top-level" requirements. The proposed changes may come from the Westermo Operating System (WeOS) teams during their planning phase, identified as new or altered functionalities in the framework, or come from within the test team. Here a requirement means a statement describing a functionality that is expected by the system based on the proposed change.
P3 Important framework interaction sequences identified. The purpose of this activity is to describe expected functionality at lower levels, enabling requirement decomposition and allocation, and to ease identification of possible errors and abnormal operating conditions in later risk analysis. The interactions should be based on the expected functionality.
P4 Third party functionalities identified and suitable libraries and tools selected. To avoid the use of too many tools (e.g. packet generators), using a tool already in successful use, could be tried before adding a new tool. If a new tool/library is needed, its history should be reviewed and the basis for selection documented. P5 Preliminary risk and impact analysis performed and documented. Utilize the interaction sequences of P3 to identify possible errors and their effects, including internal errors in the framework as well as errors caused by abnormal operating conditions. P6 Lower level requirements elicited and allocated to framework components. Break down requirements into smaller workable and testable units. Allocate these to framework components (e.g. modules/tools/tool-chains), according to identified interaction sequences. Also, requirements derived from the risk analysis should be allocated to suitable components.
P7 Development impediments identified and mitigated. Identify factors that may block or delay the development, and find mitigation strategies.

A.3 Development Phase Definition of Done
The suggested activities to be completed during the development phase, before transitioning to the test phase are defined in the provided list below.
D1 Complete implementation according to requirements and development guidelines. Avoidance of faults being introduced by misconceptions, defining e.g. conventions, error handling, and other practises. These could be combined with development checklists to reduce the effort for later reviews.
D2 Static code analysis only giving "low level" remarks. Linting 2 and/or other static analysis tools, e.g. Coverity 3 , should be set up to the development branch to enable continuous correction during development.
D3 Unit tests written. To test fine-grain logic, unit tests of developed components should be written and refined before, during and after the implementation is performed.
D4 Tests written to verify compliance with requirements. In parallel to the implementation, tests to verify compliance with requirements should be developed.
D5 Behaviour, instructions and constraints defined in documentation. Proper documentation of system behaviour, usage instructions and system constraints to be ensured.
D6 Peer-reviews completed and documented. Definition of methods for peer review, which should include examination of the implementation, tests and documentation.
D7 Issues found by peer-reviews corrected. After correction, this should be verified with the reviewer.

A.4 Test Phase Definition of Done
The suggested activities to be completed during the test phase, before transitioning to the tool validation test-suite and subsequent merge of the new functionality with the maintained stable solution, are defined in the provided list below. T1 Unit tests performed. Verify low level behavior by running the newly developed unit tests (this may be an iterative process, see DoD D3).
T2 Unit-integration tested. Test the integration of units as a group, as well as the data transfer between components.
T3 Requirement-based tests performed. These tests verify the expected functionality of the framework as described by the requirements.
T4 Fault injection tests performed. Fault-injection can be used in two ways, for two purposes. (i) Forced errors in components/sub-tools of the tool-chain can verify the error detection or prevention measures in other parts of the system (e.g. monitoring services, sanity checks, etc.). (ii) Faults could also be introduced in the WeOS code, and when running test cases for WeOS, we expect the framework to detect the problems (test cases should fail).
T5 All detected issues managed. After correction, applicable tests should be repeated for verification of sufficient correction.
T6 Risk and impact analysis documentation completed. The documentation should be revisited and completed. If necessary, a new analysis can be conducted to validate sufficiency of implemented measures.