Self-Adaptive Testing in the Field

1 INTRODUCTION

Traditionally, the software testing task has always been thought of as a fault-finding process performed during the development cycle [2]. Since the activities in this process are commonly performed in the development environment, the work in Ref. [8] refers to such activities as in-house or in-vitro software testing. However, in recent years, academic researchers and industry professionals are becoming more and more aware of the need to continue testing even after software release, when the system is in use, e.g., Refs. [4, 7, 22, 44]. In fact, many of the failures reported in production correspond to problems that would be difficult, if not impossible, to uncover by in-house testing [27] due to the tremendous complexity, evolvability, and interconnection of current software-intensive systems. The work in Ref. [8] defines this tendency of moving testing activities from the development to the production environment with different nuances. Testing in the field [8] can be subdivided into online testing, offline testing, and ex-vivo testing. Test cases are executed directly in the field, on the same instance of the system used in production, which is called online testing, or on a separated instance that still runs in production, called offline testing. Finally, ex-vivo testing refers to the case in which test cases are executed in-house but using data collected from the field.

The above cited work [8] reviews the literature on field-based testing techniques. It selected 80 primary studies published from 1989 to 2017 and classified them according to different dimensions, including, among others, how, when, and where the generation and activation of field tests take place. The authors of this review noticed that many of the collected works adopt some adaptation strategy in order to deal with uncertainty or confront emergent behaviors of the System Under Test (SUT). Furthermore, they conclude their summary of open challenges in field-based testing by stating that, concerning test cases generation, test cases shall adapt to the production environment [8], and that, concerning the oracles for field testing, these need to adapt to the unknown execution conditions that can emerge in the field [8].

The definition of adaptive testing is also investigated in the recent area of “software cybernetics” [13], which looks at how software and control processes interact. In this scenario, adaptive testing means that a software testing strategy is improved by taking advantage of the information collected during the testing process, which deepens our comprehension of the tested system [12]. However, this definition goes in a different direction since it does not refer to testing in the production environment, as we are concerned here.

As a matter of fact, we believe that the same reasons for moving testing activities from in-house to the field, such as tackling dynamism, context-dependence, and uncertainty, also support the need for self-adaptive testing approaches. Field-based testing is in general important to reveal those faults that escape in-house testing [8], but it becomes even more important for systems that change over time. In other words, self-adaptability represents an important feature when carrying out field testing activities.

Self-adaptive systems (SAS) have been actively researched in the recent years [16, 36, 50]. They can be described as software-intensive systems that modify themselves while in operation in response to internal activities or changes in the environment (the SUT environment, that is, the system under which the SUT is running, which may include software and hardware) or context (a broader concept that involves other variables, such as changes in the rules of the company to which the SUT belongs) in which they operate, and that may also be based on predetermined properties or policies [50]. This description, in our opinion, is equally appropriate for a software framework whose objective is to carry out testing in the field: the test approach may need to be modified in response to events in the field or due to shifting needs or expectations. In this work, we refer to such category of testing approaches as “self-adaptive testing in the field” (SATF for short). Generally speaking, SATF might be a useful option to supplement traditional testing rather than replace it.

In fact, several of the strategies proposed to test SAS are inherently designed to be used at runtime, generally when an adaptation takes place, and are themselves conceived as adaptive (we analyse many of such works in Section 4). Nonetheless, we do not believe, and this is confirmed by some of the works we collected (e.g., Refs. [15, 19, 29, 41]), that the SATF scope is restricted to SAS. It is unquestionably a wise technique for testing new service choreographies, as was first anticipated in the Choreos European project [9] where policies were established to adapt the runtime testing approach. More generally, in a recent opinion article that explores challenges and promising directions in testing “changing software in a changing world” [10], the authors state that the testing strategy should support the adaptive generation of test plans. From a more pragmatic perspective, based on the primary studies covered in this work we can notice that there are three types of systems for which the use of self-adaptive testing in the field approaches would play an important role: (i) SAS (e.g., Ref. [23]), (ii) systems that could be updated over time, due to the origination of new needs, among other reasons (e.g., Ref. [41]), and (iii) systems for which the configuration may be changed by the user at runtime (e.g., Ref. [15]). Of course, we could even have systems that present more than one of the above-mentioned dynamic characteristics.

In the aforementioned review on field-based testing [8], even though self-adaptation aspects do appear at various points, they are not examined as a stand-alone dimension. With respect to that work, this study puts the spotlight on field testing approaches that can, in a way or another, self-adapt themselves. We do not restrict a priori what aspect of the approach is adaptive or when the adaptation occurs, or what is the trigger that launches the adaptation. In fact, our research aims at properly defining such approaches, explaining what are their characteristics, and proposing a reference model.

This work extends a previous work published at SEAMS in 2022 [53], where we surveyed the literature of SATF, highlighted the main characteristics of the topic, and discussed some research challenges. In this work, we extend the previous one by (i) expanding literature coverage to papers published recently and to papers identified via snowballing (not performed in the SEAMS paper), (ii) providing a taxonomy for SATF according to the consolidated and refined list of characteristics of SATF and their relationships (also specified via a feature model), (iii) providing a reference architecture for self-adaptive approaches to field testing, and (iv) providing an updated list of research gaps and challenges in self-adaptive field-based testing.

More precisely, we aim to answer the following research questions:

RQ1 looks at existing definitions of SATF and at the characteristics of existing approaches and proposes a taxonomy for SATF. RQ2 builds on the data collected to answer RQ1, as well as on the taxonomy built in RQ1, and proposes a reference architecture for SATF systems. RQ3 discusses the knowledge gaps and challenges encountered when performing SATF.

On the methodological side, we constructed the taxonomy and the reference architecture by following a design science research methodology organized in three iterations. The first iteration consists of the work described in our previous article [53]. The other two iterations, are the new content described in this work. More details about the research methodology we followed are provided in Section 3.

Paper outline. Section 2 discusses related work. Section 3 describes the methodology followed for the study. Section 4 reports the results of the study in outlining a taxonomy for SATF. Section 5 presents the reference architecture for SATF approaches. Section 6 discusses the identified open challenges. Section 7 describes the activities we performed to validate the taxonomy and the reference architecture, and presents the outcome of the validation. Section 8 characterizes limitations and threats to validity of this study. Finally, in Section 9, we discuss the findings and suggest promising future research avenues.

2 RELATED WORK

As detailed in Section 3, to build awareness of the problem, we surveyed the literature on approaches for self-adaptive testing in the field. In this section, we prepare to this view of the state of the art on self-adaptive testing in the field with an analysis of existing secondary studies that are related to our work and with some additional works that we identified by performing specific search. For example, we searched for works discussing challenges of testing SAS since this is relevant for RQ3. Indeed, these works are not part of the set of primary studies we analysed in the survey we performed, but they are worth attention in this section.

Although there are a few secondary studies that partially relate to our subject, we are not aware of any previous literature reviews focusing on the same objective. A systematic literature review (SLR) has been carried out in 2016 aiming at recognizing and categorizing the challenges encountered when testing SAS [54]. This article gathered 25 primary studies (published from 2003 to 2015) and identified 12 types of challenges to be faced. The challenges mentioned by their work are described as follows:

We believe that many of these challenges could be mitigated when taking into account adaptive testing in the field, for example, the high number of configurations to be tested (Challenge 1), or context-dependency (Challenges 6 and 7), or test cases maintenance (Challenge 8), among others. In Section 6, we will come back to these challenges to check whether there is an overlap with the challenges we identified for SATF or whether SATF helps in mitigating some of these challenges.

The work in Ref. [48] focuses on a specific class of SAS, i.e., those utilizing health states for computing repair actions. It identifies the following challenges:

We believe that SATF approaches could help mitigating some of these challenges, and will discuss this more in depth in Section 6.

In 2021, two SLRs that concentrate on methods for testing a software system at runtime in its actual execution environment have been published. The already cited work in Ref. [8] distinguishes between testing methods that are undertaken in production (in-vivo) and others that use data from production. Their search query is more extensive than ours since it includes non-adaptive testing techniques in addition to adaptive ones, rather than only focusing on adaptive ones as we do. Therefore, in principle, our set of primary studies would be a subset of theirs. On the other hand, they cover the literature by looking at papers published until 2017, while we considered the literature until 2022, and 8 out of the 19 (i.e., 42%) primary studies we selected are, indeed, published after 2017. Furthermore, despite they include some of the existing works in SATF, in their work, the notion of test adaptability is not a central concern as for us, and there is no discussion on the specifics of test adaptation, such as what is adapted, how, and when.

Another SLR on runtime testing [37] limits the scope to approaches to test systems that are dynamically adaptive and distributed, and in contrast to the SLR in Ref. [8], which did not limit the domain of the tested application, the authors actually chose fewer papers (precisely, 43 from 2006 to 2020 in Ref. [37], against 80 from 1989 to 2017 in Ref. [8]). In Ref. [37], the authors pose eight research questions about the properties of runtime test approaches, and among them, RQ5 - which is phrased as: “What kind of dynamic adaptations can these approaches support?”—seems to be the one most closely related to our study. Nonetheless, considering the response they provide to the question, we comprehend that the adaptation they focus on concerns again the SUT, whereas their study does not address adaptation of the testing approach itself as we do here.

Lastly, a recent SLR [55] surveys faults that are experienced when testing adaptive systems and context-aware systems. As stated by the authors, such a study can be helpful for comprehending first the nature of flaws specific to those kinds of systems and then coming up with suitable testing approaches to find those types of faults. Thus, they concentrate more on fault-based testing methods. While still pertinent, their SLR entirely diverges from the focus of our current research because, unlike us, they do not characterize self-adaptive testing in the field as we aim to do.

Table 1 shows an overview of the above-mentioned secondary studies. It is important to notice that the work in Ref. [48] does not establish a specific period for its research. Also, we consider that a work focuses on self-adaptive testing even if this is not explicitly said, as in Ref. [37], a work that focuses only on testing dynamically adaptive and distributed systems, without considering the self-adaptive testing of other types of systems, as we do.

3 RESEARCH METHODOLOGY

In this work, we aim at defining a taxonomy for SATF, at contributing a reference architecture for SATF approaches, and at identifying and discussing the main challenges. To achieve these objectives, we followed the design research method [17], which is mainly used for creating new knowledge and artifacts that are required to address a particular phenomenon or problem.

As shown in Figure 1, we performed three iterations (first, second, and third), and each iteration consisted of three steps (awareness of the problem, solution development, and evaluation); overall, this leads to nine phases. Each phase in the research methodology is identified by a code <x>.<y>, where <x> represents the step and <y> represents the iteration.

Fig. 1.

View Figure

The first iteration has been reported in our previous article [53] (phases 1.1, 2.1, and 3.1, i.e., the three steps of the first iteration). The other two iterations represent the extension of [53] that we present in this work. The three steps permit us to work incrementally to answer the three research questions. Precisely, the answer to the three RQs is provided at the end of the third step, by providing a taxonomy (RQ1), a reference architecture (RQ2), and challenges (RQ3), which have been also benchmarked with challenges available in the related works. In the following, we describe details of the six phases, organized and presented by the three steps:

1. Awareness of the problem:

Phase 1.1 - Awareness, First Iteration: In the first iteration, the awareness of the problem was mainly developed through data collection from the literature. The collected data has been then analysed to extract information useful to define SATF and identify its characteristics and challenges.

Phase 1.2 - Awareness, Second Iteration: As described above, the second iteration has been performed to extend our previous work, so there was the need to update the previously performed review of the literature to papers published after [53]. In this iteration, we also complemented the review with snowballing to achieve a higher level of completeness. The identified primary studies have been then analysed to extract information useful to provide a first version of the taxonomy, reference architecture, and challenges.

Phase 1.3 - Awareness, Third Iteration: In this iteration, we exploited the feedback coming from experts of the domain to contribute a final and validated version of the solutions (see Phase 2.3) and we analysed the challenges highlighted in related secondary studies (see Phase 3.2).

2. Solution development:

Phase 2.1 - Solution, First Iteration: we contributed definitions, characteristics, and challenges of SATF. This step has been reported in our previous article [53].

Phase 2.2 - Solution, Second Iteration: we contributed a first version of the taxonomy of SATF, reference architecture, and challenges.

Phase 2.3 - Solution, Third Iteration: we contributed a final and validated version of the taxonomy of SATF, reference architecture, and challenges.

3. Evaluation:

Phase 3.1 - Evaluation, First Iteration: We validated the definition and characteristics of SATF thanks to the participants of SEAMS 2022 that attended the presentation of the paper at the conference. Moreover, we presented the findings in other events, like an event about robotic software engineering.¹ Specifically, we gave a presentation (disseminated the work), collected lively feedback in the form of questions/answers after the presentation, and had a follow-up with 3 attendees (overall) that showed interest in the work.

Phase 3.2 - Evaluation, Second Iteration: We validated the first version of the taxonomy and reference architecture with experts in the domain. Moreover, we compared the identified challenges with those present in related secondary studies.

Phase 3.3 - Evaluation, Third Iteration: We validated the final version of the taxonomy and reference architecture with a subset of experts involved in phase 3.2. Specifically, in this phase, we involved only those experts that suggested or triggered changes in the previous iteration of the validation (phase 3.2). The validation, in fact, focused on the performed changes, to check whether we correctly understood and implemented the recommended changes. We did not involve those experts that were already satisfied during phase 3.2.

In the remainder of this section, and specifically in Section 3.1, we describe the scoping review made in step 1.1 and its subsequent update made in step 1.2. Then, in Section 3.2, we describe the validation we made with experts of the domain, in steps 3.2 and 3.3. We believe that the other steps summarized above do not need further explanation. We provide a replication package of the activities done in this research at https://tinyurl.com/2kdb5jef.

3.1 Scoping Review

In this work, we employed the scoping review research methodology [18, 43, 46] to address the research questions mentioned in the introduction. The scoping review research methodology is popular in the medical field [18, 43, 46], but it is also gaining popularity in the field of software engineering [3]. Scoping reviews, according to [43], are analogous to systematic literature reviews and they adhere as well to a rigid structured process. On the other hand, they highlight some distinct essential methodological principles and have different objectives. In cases where systematic reviews are unable to achieve the essential goals or satisfy the demands of knowledge users, scoping reviews are increasingly recognized as a viable strategy [43].

Since SATF is not yet developed enough to effectively conduct a SLR, we think that scoping reviews are the most appropriate study methodology. Given that it identifies and analyzes knowledge gaps in the field of self-adaptive testing, this work may potentially serve as a useful precursor to upcoming systematic reviews. According to the traditional goals for performing a scoping review [43], this study will aid in (i) identifying knowledge gaps and important traits or elements associated with a concept, (ii) scoping a body of literature, and (iii) clarifying terminology and definitions used in the literature. The review protocol is presented in detail below to facilitate validation and replication studies. However, readers not interested in the methodology can skip to the brief summary in Section 3.1.7.

3.1.1 Search Strategy.

In light of the research questions mentioned in the introduction, it was decided that the search string to be used for searching the pertinent publications should comprise the term “software”, co-joint with the terms identifying testing in operation, such as: “online testing”, “runtime testing”, or “field-based testing”, and one other set of terms referring to the adaptation aspect, such as: “self-adaptive”, “self-organizing”, “autonomous”, “self-managing”, or “adaptive”. These keywords were then connected appropriately using the Boolean operations AND and OR, obtaining the following search string:

software AND (“online test*” OR “run-time test*” OR “runtime test*” OR “field-based test*”) AND (self-adaptive OR “self adaptive” OR self-organizing OR “self organizing” OR autonomous OR self-managing OR “self managing” OR “self organising” OR self-organising OR adaptive)

In comparison with the often cited previous review in Ref. [8], we have added here one more co-joint term relative to adaptiveness, which in the previous string was included by means of one only adjective “adaptive” included with an OR operator; i.e., here we focus specifically on self-adaptive approaches.

In order to be as comprehensive as possible, we applied our search across the “full text” of the articles under consideration. The databases that the aforementioned query was applied to are shown in Table 2. Three of the most popular databases, IEEExplore, ACM Digital Library, and Scopus, were employed in our scoping review.

Table 2.

#No	Database	URL
1	IEEExplore	https://ieeexplore.ieee.org/
2	ACM Digital Library	https://dl.acm.org/
3	Scopus	https://www.scopus.com/

View Table

Table 2. Search Engines for Scientific Papers

The number of records extracted for Scopus, ACM Digital Library, and IEEExplore, utilizing the aforementioned databases and the suggested search query, and only taking papers published from 2012 to 2022 into consideration, is equivalent to 637, 486, and 21, respectively².

3.1.2 Screening and Duplicate Removal.

We screened the papers by simply reading their title and abstract as there were a variety of papers that matched the query but were not pertinent. Most of them were related to other scientific fields or were clearly related to other meanings for the terms searched. For instance, the term “online test*” is also used in papers tackling e-learning. All of these papers considered as unrelated are then eliminated. Next, after removing the duplicates and combining the results from screening for several databases into a single set, we obtained 121 publications.

3.1.3 Selection Criteria.

The selection criteria are summarized in Table 3. The inclusion criteria are based on the research questions, while the exclusion ones are standard quality criteria (similar to those from Ref. [8]). Aiming at comprehensiveness, we considered publications written in English within the last 10 years that could be downloaded. It is important to highlight that our search returned no publications written in languages other than English or that could not be downloaded.

Table 3.

Inclusion criteria

Exclusion criteria

I1 - Papers that provide a definition on self-adaptive testing in the field. I2 - Papers that describe characteristics of approaches to perform self-adaptive testing in the field. I3 - Papers that describe gaps/challenges in self-adaptive testing in the field. I4 - Papers that describe components of a model or an architecture to self-adaptive testing in the field. I5 - Related papers published from 2012 up to 2022.

E1 - Papers that cannot be downloaded. E2 - Studies in languages other than English. E3 - Papers published before 2012. E4 - Unpublished papers. E5 - Secondary studies papers. E6 - Overlapping, duplicate and out-of-scope papers (i.e., not fulfilling I1, I2, I3 or I4).

View Table

Table 3. Inclusion and Exclusion Selection Criteria

Ultimately, 39 papers were obtained as a result of the application of inclusion and exclusion criteria.

3.1.4 Full-Text Checking.

The selected papers were merged and given to two of the authors for review. Each reviewer separately read the full text of every paper before choosing whether or not it should be included. In plenary meetings, all papers that generated conflicting perspectives were discussed. Finally, after a final agreement, 16 papers made it to the final list.

3.1.5 Snowballing.

The term “snowballing” describes the process of finding more publications by leveraging a paper’s reference list, backward snowballing, or its citations, forward snowballing [57]. We performed both backward and forward snowballing, using as the initial list the one obtained in the previous stage that contains 16 papers. In the snowballing phase, the screening of papers and application of inclusion and exclusion criteria are also performed. A first application of snowballing resulted in 3 papers accepted out of 753 obtained. Then, with the 3 papers accepted in the previous iteration, we performed a second snowballing search that resulted in 1 accepted papers among 141 obtained. Finally, by using the paper accepted in the preceding search, we finally executed the snowballing a third time, obtaining 78 papers but did not accept any of them. Then, the total the number of papers collected by the snowballing phases³ is equal to 4, resulting in 20 papers in total. Among these, we noticed an extended work of a paper already collected during the regular process. We then removed the older overlapping paper and kept the most recent one. Thus, the final list of papers collected in this work contains 19 papers.

3.1.6 Data Extraction.

Each author read the entire work and then individually extracted the information needed to respond to the three research questions (RQs) outlined in the Introduction (Section 1). Thus, we had two independent readings and classifications for each paper (in particular, for answering RQ2, we used a spreadsheet currently available from the replication package). These classifications were then discussed in a series of meetings that the authors attended in order to clarify any unclear categories and resolve any potential disagreements. It is important to notice that we performed the data extraction process incrementally in order to take into account the incremental building of the feature model and the reference architecture.

3.1.7 Review Protocol Summary.

This study has mainly focused on papers outlining strategies for self-adaptive testing in the field. The entire workflow employed in this scoping review is shown in Figure 2. 137 records in total were found across 3 separate databases. 121 records were later determined to be unrelated to our topic and eliminated. As a result, just 16 records were determined to be pertinent at the end of the process. Then, 4 papers were added through snowballing, and one overlapping paper of the previous 16-papers list was removed, resulting in a total of 19 papers. The final list of primary studies included in this paper is shown in Table 4. Comparing this final list against the list of selected primary studies in Ref. [8], we notice that: (i) only four studies are common to the two surveys, namely [19, 23, 24, 38], while the remaining 13 papers of this review were not included in [8]; and (ii) 8 papers of this review appeared after the search conducted in the previous survey.

Table 4.

Final List of Papers
Ref.	Year	Venue	Title
[42]	2012	International Workshop on European Software Services and Systems Research - Results and Challenges (S-Cube)	Verification and testing at run-time for online quality prediction
[25]	2013	International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)	Towards run-time testing of dynamic adaptive systems
[19]	2014	IEEE Transactions on Services Computing (TSC)	Dynamic test reconfiguration for composite web services
[24]	2014	International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)	Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty
[23]	2015	International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)	Automated generation of adaptive test plans for self-adaptive systems
[41]	2016	International Workshop on Automating Test Case Design, Selection, and Evaluation (A-TEST)	Automated workflow regression testing for multi-tenant saas: integrated support in self-service configuration dashboard
[51]	2016	International Middleware Conference (Middleware)	Bifrost - Supporting Continuous Deployment with Automated Enactment of Multi-Phase Live Testing Strategies
[38]	2016	Elsevier Science of Computer Programming (SCP)	Safe and efficient runtime testing framework applied in dynamic and distributed systems
[32]	2016	International Workshops on Foundations and Applications of Self* Systems (FAS* W)	Towards autonomous self-tests at runtime
[49]	2017	International Conference of Electronics, Communication and Aerospace Technology (ICECA)	Self-test framework for self-adaptive software architecture
[31]	2017	International Workshop on Variability and Complexity in Software Design (VACE)	Towards collective online and offline testing for dynamic software product line systems
[29]	2019	International Symposium on Software Reliability Engineering Workshops (ISSREW)	A hybrid framework for web services reliability and performance assessment
[39]	2019	Computer Software and Applications Conference (COMPSAC)	The SAMBA approach for Self-Adaptive Model-Based online testing of services orchestrations
[15]	2020	International Conference on Software Testing, Validation and Verification (ICST)	A Framework for In-Vivo Testing of Mobile Applications
[47]	2020	Wiley Software Testing, Verification and Reliability (STVR)	Testing microservice architectures for operational reliability
[30]	2021	International Conference on Web Research (ICWR)	On-demand Test as a Web Service Process (OTaaWS Process)
[21]	2021	Elsevier Information and Software Technology (IST)	Runtime testing of context-aware variability in adaptive systems
[14]	2022	International Conference on Automation of Software Test (AST)	Microservices Integrated Performance and Reliability Testing
[26]	2022	International Conference on Software Testing, Verification and Validation (ICST)	Testing Software in Production Environments with Data from the Field

View Table

Table 4. SATF Approaches

Fig. 2.

View Figure

4 TAXONOMY FOR SATF

In this section, we aim at providing an answer to RQ1, i.e., What are the main dimensions of a taxonomy for self-adaptive field-based testing?. We start by giving a definition to SATF through the analysis of the current literature on this subject (Section 4.1). Then, we present the dimensions of the taxonomy obtained to SATF (Section 4.2) and briefly discuss their relations (Section 4.3).

4.3 Relations Among the Dimensions

In this section, we provide an overall view of the various dimensions described in Section 4.2. Figure 3 puts together the various dimensions in a feature model that summarizes the components of a SATF approach and their relationships. It is important to notice that, in the figure, the mandatory components are marked with \(\bullet\) and the optional ones with \(\circ\). In the feature model, the Alternative symbol indicates that only one of the children can be defined. On the other hand, the Or symbol determines that more than of the children can be defined. When there is no symbol in the connection between components, it would be equivalent to a Boolean And symbol, that is, if the father is defined, so all the children must be. It is important to highlight that, both Alternative and Or symbols also mean that once the father has been defined, at least one child also must be.

Fig. 3.

View Figure

As described above, and visible in Figure 3, some of the features are alternatives, some mandatory, and some optional. Indeed, not all the possible configurations are valid. In the following, we describe the constraints that should be valid⁴:

—	SATF.Adaptation.“Object to adapt”.“Oracle Adapt.”.“Independent Oracle Adapt.” \(\Rightarrow\) SATF.Oracle.“Independent Oracle”: since an oracle is optional, Independent Oracle can be an object to be adapted only when an Independent Oracle is present.
—	SATF.Adaptation.“Object to adapt”.“Oracle Adapt.”.“Dependent Oracle Adapt.” \(\Rightarrow\) SATF.Oracle.“Dependent Oracle”: since an oracle is optional, Dependent Oracle can be an object to be adapted only when a Dependent Oracle is present. As highlighted by one of the interviewees, in practice, the dependent oracle is adapted when the associated test case is adapted. Since in principle, it is admissible to adapt a dependent oracle without updating the associated test case, we add no further constraints for this concern.
—	SATF.Adaptation.“Object to adapt”.Monitor \(\Rightarrow\) SATF.Monitor: since a monitor is optional, Monitor can be an object to be adapted only when a Monitor is present.
—	SATF.Adaptation.Trigger.Activation.On-demand \(\Rightarrow\) SATF.“Human Involvement”: since the human involvement component is optional, the activation of adaptations could only be performed on-demand in the presence of a human.
—	SATF.Monitor.“Monitor Scope”.“Configuration Change by User or Maintainer” \(\Rightarrow\) SATF.“Human Involvement”: since the human involvement component is optional, the monitor scope could be defined by configuration change by user or maintainer with the presence of a human.
—	Each test case should embed a (dependent) oracle or should be covered by an independent oracle or by a human acting as an oracle. Consequently, in this case, SATF.Oracle.“Independent Oracle” OR SATF.“Human Involvement” = TRUE, i.e., one of the two should be present.

5 REFERENCE ARCHITECTURE FOR SATF APPROACHES

In this section, we aim at providing an answer to RQ2, i.e., What are the main components of a reference architecture for self-adaptive field-based testing approaches? A reference architecture is a general architecture that is used as a foundation for the design of concrete architectures within a given context or application domain [11]; then, a concrete architecture can be considered as a specific instance of a software reference architecture. For instance, the work in Ref. [45] presents a reference architecture for connected vehicles. In this article, we contribute a reference architecture for SATF approaches. It has been built by taking into account the primary studies that have been identified and analysed in the literature review. Since we made an effort to generalize and interpret the data collected by analysing the primary studies, we also validated the reference architecture with our interviewees. Then, we improved the architecture according to the received feedback. Further details about the followed research methodology can be found in Section 3 and the results of the validation are described in Section 7.

An architecture can be described in various ways and it can be organized in various views and viewpoints [1]. For instance, software architectures can be described as “boxes and lines” when focusing mostly on the component and connector view, or they can be represented by referring to architectural patterns and tactics. We will first describe the main architectural decisions and then provide a component and connector view, which permits us to identify the main components and connections among them. The component and connector view is described in Figure 4. When describing the main architectural decisions, we will also describe how they have been defined, relate to the feature model described in the previous section, and have been influenced by the validation with experts. The main architectural decisions are described in the following:

Fig. 4.

View Figure

It is important to highlight that the figure shows a reference architecture, including the components that are conceptually required by SATF approaches. In specific cases, some of the components in the reference architecture might be integrated into components of the SUT. For instance, when the SUT is a self-adaptive system, it will probably implement a MAPE-K loop or a similar loop. Then, in the case in which the tester has available the code of the SUT or has available APIs to interact with the various components responsible for the adaptions of the SUT, e.g., monitor, analyser, planner, and executor, the tester might implement the self-adaptive testing approach by profitably exploiting or reusing parts of these components. In these cases, it might be interesting to refine the architecture in Figure 4 by identifying a better integration among the two MAPE-K loops, i.e., the loop of the SUT and the loop of the testing approaches. For instance, the monitoring component of the testing approach might reuse the monitoring APIs of the SUT to sense the SUT environment, exploit the knowledge of the SUT, or even profitably interact with its analyse component. However, since there can be too much variability and uncertainty in the architecture (e.g., a self-adaptive SUT might even implement an architecture not compliant with the MAPE-K loop style), the development aspects, as well as the accessibility of the SUT or its APIs, we decided to provide a general architecture assuming that the SUT is a black box component. Then, the reference architecture might be considered as a conceptual model and we delegate to the developers specific testing approaches improvements and optimizations that might become possible when targeting a specific class of SUTs.

6 CHALLENGES

In this section, we attempt to respond to RQ3, i.e., What are the known gaps/challenges in self-adaptive field-based testing?, and highlight the major research challenges and gaps in SATF. We here provide a list of these challenges/gaps and then compare them with the ones we found in the related literature (previously discussed in Section 2).

7 VALIDATION

In this section, we discuss the validation of the proposed taxonomy and reference architecture conducted with experts through a questionnaire and further discussions through online meetings. We first sent an e-mail to all of the 56 authors of the primary studies we collected. It is important to notice that one same person may be the author of more than one work. For 2 of them, the e-mail bounced, and we were not able to find an alternative valid address. Among the 54 authors that, in fact, received the e-mail, 19 expressed interest in participating in the questionnaire survey. These are the people we refer to here as the experts. The questionnaire contained five main questions and a last question asking whether the respondent was available for an interview during an online session. 5 experts (E2, E4, E7, E10, and E18) answered positively to the latter point. The idea behind the questionnaire was validating the structures in the feature model that were not solidly grounded on the data, but that we considered reasonable, whereas the interviews aimed at getting further feedback and additional comments.

In the first question, we asked the experts whether they think that a monitor should be optional. In other words, we wanted to validate with experts if they believe that a SATF approach may contain or not the monitor component. Also, for all the questions, we asked them to comment their opinion. 9 people agreed (E1, E3, E7, E10, E11, E13, E14, E16, and E18) with this statement. Among the comments, the most interesting ones reported that they agreed with this statement since “the need for a monitor component depends on both the objective of the testing and the system itself”(E10) and that “the outcomes of one test set execution can be used to gradually improve/tune the testing strategy for subsequent executions”(E18). We believe that this is enough to give the possibility to have the monitor component as optional, since the respondents that agreed had in mind scenarios in which the monitor can be optional. 7 people disagreed with the statement (E2, E4, E5, E8, E12, E15, E17). Among these respondents, the main comments are that in a SATF approach “there should be a monitor (even if hidden or called differently)”(E2), and that in a self-adaptive approach, adaptations should be based on something that is observed, so there is a need for a monitor (E4). Also, it was suggested that if the monitor is not present, the approach should contain any other mechanism for detecting changes that it has to adapt to. In fact, we have these other mechanisms in our taxonomy and reference architecture. Finally, a last comment said that if an approach does not have a monitor, it is not fully self-adaptive. However, we have a literature of human involvement (e.g., human-in-the-loop or even human-on-the-loop) in SAS that accepts the human acting as monitor, e.g., Ref. [40]. Three people neither agreed nor disagreed (E6, E9, and E19). We noticed that, for several respondents, this question was not completely clear. So, through the interview, we had the opportunity to better explain the question and guarantee that they properly understood it. Then, after further discussions during online sessions, all the interviewed respondents agreed with the statement that a monitor should be optional, even those that did not agree before.

In question two, we provided a list of possible events a monitor could expect in order to activate the testing (i.e., SUT adaptation, SUT evolution and/or configuration change made by user) and asked them if they agree that more than one of these events could be monitored by the same SATF approach. 15 people agreed (E1, E2, E3, E5, E7, E9, E10, E11, E12, E13, E15, E16, E17, E18, and E19) with this statement. Among the agreeing respondents, the most interesting comments made suggestions on what else could be monitored in a SATF approach. Their suggestions were valuable and very appreciated, resulting in the incorporation of new features into the feature model (e.g., Configuration change by maintainer (E4), External Component Change (E7), Environment Change or Evolution (E4 and E2), Test Results (E18) and Model Activities (E10)) and also into the reference architecture, as we explain later in this section. 2 people disagreed (E6, E8) with the statement. One of them (E6) seemed not to understand the question properly since his/her comment said that a single component adaptation should also be considered, what we already do here. The other respondent (E8) made a suggestion for including “requirements changes” to the list of possible events in the statement. We did not add any new item to the feature model based on this comment since the monitor is already capturing when these changes in the requirements result in SUT changes. Similarly, a person that neither agreed nor disagreed (E4) with the statement also wanted to make a suggestion regarding the inclusion of “changes operated by maintainers and/or evolution of the environment”. The other person (E14) stated that “any of them can in principle active testing as they can provide complementary information”. The discussion regarding this question during the interviews was extremely fruitful, since it gave us the opportunity to include their feedback and then refine both the taxonomy and the reference architecture.

In question three, we listed the possible objects to be adapted (i.e., test cases, oracle, monitor, and testing strategy) in a SATF approach and asked them whether they agree with the assertion that more than one of these objects could be adapted by the same SATF approach. 18 people (E1, E2, E3, E4, E5, E7, E8, E9, E10, E11, E12, E13, E14, E15, E16, E17, E18, and E19) responded it positively. We had a comment (E5) saying that these adaptations have different natures and are all relevant. Another (E14) said that even though their combination is possible, in real cases it may not be feasible. We also received the suggestion (E16) of including “test plans” as a possible object to be adapted, but this object was already included in the “testing strategy” object. Another interesting comment (E17) said that “the more objects changed at once, the riskier the adaptation itself”. One person (E6) disagreed with the statement, but we did not have the opportunity to discuss the reasons for that. A suggestion coming from the interviews (expert E4) that we incorporated into the taxonomy and reference architecture was the idea of having two different oracles: the dependent and the independent one.

In question four, we mentioned the possible strategies to trigger adaptation (i.e., periodically, by changes in the SUT, through a policy, and on-demand) and questioned if they agree that more than one of these strategies could be employed by the same SATF approach. 15 people agreed (E1, E2, E3, E4, E5, E6, E7, E9, E11, E12, E15, E16, E17, E18, E19) with the statement. Among these respondents, an interesting comment (E18) brought the idea that these strategies “are not fully orthogonal”, and that they could be combined, for instance, by using “a policy to periodically perform adaptation of the test suite”. 4 people neither disagreed nor agreed (E8, E10, E13, E14). One person (E10) among those suggested the inclusion of “changes in the users’ behavior”, meaning that “the way users interact with the SUT can impact its performance and reliability” and for this reason, could activate adaptation. The interviews were very useful to make clearer this question for the experts, since it may not be completely self explanatory. Once we could clarify it, all the interviewees agreed with the statement.

Finally, in question five, we listed the possible techniques (i.e., changing test parameters, the test structure and the test context) that could be employed in order to perform adaptation in SATF approaches. Then, we asked the experts if they agreed that more than one of these techniques could be applied by the same SATF approach. 13 people agreed (E1, E3, E5, E6, E7, E11, E12, E13, E15, E16, E17, E18, E19) with the statement, five people neither agreed (E2, E4, E9, E10, E14) nor disagreed, and one person disagreed (E8). The comments coming from this question were more related to the fact that they either did not understand the question or did not understand the difference between this question and question three. An interesting comment was that the respondent was “not sure if test structure is fundamentally different from test parameters”. All of these points could be better discussed during the interviews. With respect to the difference between this question and question three, we explained that they present different perspectives regarding the adaptation. In question three we simply look at what is the object adapted by the approach. On the other hand, question five focuses on the technique that is used to perform adaptation, that is, according to what is observed, what kinds of adaptation are necessary. Concerning the comment that connects test structure and test parameters, we had also the opportunity to explain the differences between them since this seemed to be unclear by only reading the question. Table 6 presents the proportion of responses for each question in the questionnaire.

During the interviews with experts, we were also able to present the reference architecture that is based on the data we gathered in this research. We presented them with an initial version of the architecture, generated based on the components found in the collected works. Then, we provided a brief explanation of each of the components of this initial architecture. Finally, we allowed them to make comments, suggestions or clarify their doubts about what we presented. The interviewees provided several useful comments with respect to the architecture. Besides, many of the suggestions regarding the feature model were also incorporated into the architecture. The changes in the reference architecture suggested by the interviews are summarized as follows:

Based on the conducted questionnaire and interviews, we concluded that the taxonomy (including the feature model) and reference architecture we built are good representatives for SATF approaches and that we are heading in the right direction in order to establish common definitions and terminologies to this still immature topic.

8 LIMITATIONS AND THREATS TO VALIDITY

This research has been performed by following well-established guidelines for empirical studies in software engineering including systematic studies [35], surveys [33], and interviews [52]. We identified the main threats to validity and mitigation strategies according to Ref. [58], as described in the following subsections.

9 CONCLUSIONS AND FUTURE WORK

In this article, we presented a new category of testing called self-adaptive testing in the field. The need of this new category of testing comes from the observation that modern systems are more and more required to evolve after they are released and are in operation. This is the case of SAS but also of systems that are continuously updated to new versions as required to fix bugs as well to offer new functionalities to their customers. Therefore, testing should be performed also when the systems are in operation in the field: this calls for field-based testing. Moreover, the test itself should adapt over time in an autonomous way to meet the changing and evolving systems over time. In this article, we offered a taxonomy of self-adaptive testing in the field together with a reference architecture of SAFT approaches. The taxonomy and reference architecture have been defined by surveying the literature and by collecting feedback and contributions from experts in the field through questionnaires and interviews. We also provided a wealth of challenges to be solved in SATF approaches, which, when compared to the amount of papers collected (i.e., 19 papers), endorse how this topic is immature and not yet recognized by software testing researchers as a self-standing emerging discipline.

We believe that our contributions of this article can help shaping the research in the future of this important category of testing. This work might also help practitioners to anticipate a need they will have in the near future or to select among the available techniques that are better matching their needs. Also, the reference architecture and the feature model may be employed as a guide to the development of novel SAFT approaches. When used in research, to the proposal of new approaches, the reference architecture leads the researcher to understand what are the main components and their sub-components in SATF approaches, and how they connect. On the other hand, the feature model lists the different possible characteristics a SATF approach may present, how they are connected and the existing restrictions associating them. In this way, the researcher can make wise decisions when the development is still in the initial phase. As future work, we plan to further explain how engineers and researchers can use the reference architecture and the feature model to build their own SAFT approach. We plan to build our SAFT approach and we will leverage this opportunity to dig also into this aspect.

ACKNOWLEDGMENTS

The authors express their gratitude to the experts who offered their time to answer the questionnaires and provide valuable feedback during interview sessions. The authors acknowledge the support of the PNRR MUR project VITALITY (ECS00000041), Spoke 2 ASTRA - “Advanced Space Technologies and Research Alliance”, of the PNRR MUR project CHANGES (PE0000020), Spoke 5 “Science and Technologies for Sustainable Diagnostics of Cultural Heritage”, and of the MUR (Italy) Department of Excellence 2023 - 2027 for GSSI.