Predicting Nonfunctional Requirement Violations in Autonomous Systems

2.1 Parametric Model Checking

A discrete-time Markov chain (DTMC) is a stochastic process comprising a set of states S associated with relevant configurations of the system under analysis, together with probabilities \(P(s,s^{\prime })\) that model the transitions between the system configurations associated with every pair of states \(s, s^{\prime } \in S\), where \(\sum _{s^{\prime } \in S} P(s,s^{\prime })=1\) for any \(s \in S\). To extend the range of nonfunctional properties that can be analysed using this modelling paradigm, a DTMC can be annotated with reward structures \(\mathit {rwd}:S\rightarrow \mathbb {R}_{\gt 0}\) that specify the values of attributes such as execution time, cost and resource use for the system configurations corresponding to different DTMC states. A parametric discrete-time Markov chain (pDTMC) is a discrete-time Markov chain in which some or all of the transition probabilities and/or rewards are unknown. These unknowns correspond to parameters of the modelled system and its environment.

Key to the run-time application of PRESTO, parametric model checking (PMC) [11, 22, 31, 40, 44] is a mathematical technique for the analysis of pDTMCs with properties specified as probabilistic computation tree logic (PCTL) [6, 20, 41] formulae, potentially extended with rewards [2]. For instance, the probabilistic reachability formula \(\mathcal {P}_{\ge 0.8}[\mathrm{F}\;\mathsf {success}]\) and the reward reachability formula \(\mathcal {R}_{\le 13}^{\mathsf {time}}[\mathrm{F}\;\mathsf {done}]\) can be used to formalise the sample requirements from the previous section, i.e., that a robot completes its task successfully with probability of at least 0.8 and that the mission is performed within at most 13s, respectively. For detailed descriptions of the PCTL semantics, see [2, 6, 41].

Probabilistic model checkers, such as PRISM [45] and Storm [26], allow the verification of PCTL-encoded requirements over DTMCs. These tools also support the analysis of quantitative PCTL formulae in which the bounds from the probabilistic operator \(\mathcal {P}\) and the reward operator \(\mathcal {R}\) are replaced by “=?”, e.g., \(\mathcal {P}_{=?}[\mathrm{F}\;\mathsf {success}]\) and \(\mathcal {R}_{=?}^{\mathsf {time}}[\mathrm{F}\;\mathsf {done}]\). These formulae can be analysed over a DTMC, yielding a numerical value, or over a pDTMC to provide a PMC expression, i.e., a rational function² over the pDTMC parameters. In this way, PMC allows the formalisation of the nonfunctional properties of a system as functions that be evaluated at run-time, when the system and environment parameters associated with the pDTMC unknowns are observed or predicted.

2.2 Piece-wise Linear Regression

Linear regression is a modelling technique that fits a linear equation of the form of \(value = at + b\), relating a response or dependent variable (value) with an independent variable (t) [50]. The linear equation for a set of n data points \(\lbrace (value_1,t_1),(value_2,t_2),\ldots ,(value_n,t_n)\rbrace\) can be obtained by minimising the sum of squared residuals between the predicted and actual responses: (1) \(\begin{align} \mathit {SSR} = \sum \limits _{i=1}^{n} (value_i - (a \cdot t_i + b))^2. \end{align}\) Given the slope a and the intercept b, predictions for the response value can be obtained for new values of t with the assumption of continued linearity. Piece-wise linear modelling does not assume the same slope and intercept throughout the time series but allows different linear trends to be fitted to different sections. An appropriate linear trend should result in both negative and positive residuals, so the fact that residuals have the same sign over a predetermined time period indicates a poor fit to the data. Thus a threshold on the number of consecutive data points with residuals having the same sign can be used to trigger an update of the model. Predictions are then made from the current linear equation.

3.1 Problem Definition

PRESTO aims at predicting the occurrence of nonfunctional requirement violations for an autonomous system in order to trigger early adaptation and avoid the actual violations. To produce its predictions, PRESTO assumes that the following inputs are available:

PRESTO further assumes that the pDTMC model and K requirements are available prior to the deployment of the autonomous system, and that the run-time observations \(x_{ij}\) are available one at a time, as soon as a new parameter measurement is obtained by the autonomous system through monitoring. Under these assumptions, PRESTO does the following after each new observation \(x_{ij}\):

Any actual adaptations/repairs triggered by PRESTO prediction of requirement violations are outside the scope of this article. For a wide range of such adaptations techniques, see [18, 23, 25]. Furthermore, PRESTO is not dealing with the possibility that interfering triggers may lead to conflicts in adaptation goals, which represents an open challenge according to recent research into the “uncertainty interaction problem” that affects self-adaptive systems [14, 16].

The two stages of PRESTO are depicted in Figure 1 and detailed in the following sections.

Fig. 1.

View Figure

3.2 Design-time Stage

In the first stage, PRESTO applies parametric model checking to the pDTMC model of the autonomous system and the PCTL formulae \(\Phi _k\), \(1\le k\le K\), from the nonfunctional requirements under verification. This parametric model checking is performed using a probabilistic model checker, such as PRISM [45] or Storm [26], and yields a set of rational functions (2) \(\begin{equation} \mathit {prop}_k (x_1,x_2,\ldots ,x_n) = \frac{P_k(x_1,x_2,\ldots ,x_n)}{Q_k(x_1,x_2,\ldots ,x_n)},\; 1\le k\le K, \end{equation}\) that formalise the relationship between the system properties from the K requirements and the n pDTMC parameters, where \(P_k(\cdot)\) and \(Q_k(\cdot)\) are polynomials.

The left-hand side of Figure 1 illustrates this PRESTO stage for a toy example in which the workflow performed by a simple system is modelled by a five-state pDTMC. In the initial pDTMC state (\(s_1\)), the system performs an operation \(\mathsf {op}_1\) that succeeds with probability \(p_1\), taking the system to state \(s_2\), where a second operation, \(\mathsf {op}_2\), is attempted. This operation succeeds with probability \(p_2\), bringing the system to a \(\mathsf {success}\) final state \(s_4\). If operation \(\mathsf {op}_1\) fails (which happens with probability \(1-p_1\)) the workflow is abandoned and the system ends in the final \(\mathsf {fail}\) state \(s_5\). In contrast, if operation \(\mathsf {op}_2\) fails (which happens with probability \(1-p_2\)), the system moves to state \(s_3\) from which it can \(\mathsf {retry}\) the entire workflow with probability \(p_3\) or needs to move to the \(\mathsf {fail}\) state \(s_5\) with probability \(1-p_3\). Two reward structures are defined over this pDTMC, specifying the mean times \(T_1\), \(T_2\), and \(T_3\) and energy consumptions \(e_1\), \(e_2\), \(e_3\) required to perform operations \(\mathsf {op}_1\) and \(\mathsf {op}_2\) and to initiate a \(\mathsf {retry}\), respectively. The example PCTL formula shown in Figure 1 corresponds to the requirement “The system shall complete its workflow successfully with probability of at least 0.8”, and the corresponding rational function \(\mathit {prop}(p_1,p_2,p_3)\) was obtained using the model checker Storm. Additional requirements specifying upper bounds for time and energy consumption used to complete the workflow (and referring to the two reward structures, e.g., \(\mathcal {R}^\mathsf {time}_{=?}[\mathrm{F}\; \mathsf {success\vee fail}]\le 90\)s and \(\mathcal {R}^\mathsf {energy}_{=?}[\mathrm{F}\; \mathsf {success\vee fail}]\le 1.2\)KJ) can be defined for the system; for readability, these are not shown in Figure 1.

3.3 Runtime Stage

In this PRESTO stage, the PMC expressions (2) are provided to the disruption prediction engine along with the time-series of the model parameters, monitored up to the current time point, as shown on the right-hand side of Figure 1. Three hyperparameters are used to configure the engine: the prediction window \(\tau\), which specifies the time period within which any disruptions are to be predicted, the update threshold N, which triggers an update of the linear regression model and a tailored tolerance \(\epsilon \ge 0\), used to reduce the number of updates required as described later in this section. The selection of these values and their impact on the results is discussed in Section 4.5.

The PRESTO disruption prediction engine evaluates the system’s compliance with its K nonfunctional requirements by executing the function \(PrestoEval(i,\mathit {value},\mathit {now})\) given in Algorithm 1 each time a new \(\mathit {value}\) for parameter \(x_i\) is obtained at the current time now. This function uses and, when necessary, updates the global variables described below.

The function PrestoEval (Algorithm 1) uses these global variables as follows. The new observation \((\mathit {value},\mathit {now})\) is added to the circular buffer storing the most recent N samples of parameter \(x_i\) (line 2) and, if the new \(\mathit {value}\) is above or below the estimate \(a_i\mathit {now}+b_i\) by more than the tolerance \(\epsilon _i\), the counter \(\mathit {cntr}^{-}_i\) or \(\mathit {cntr}^{+}_i\) is incremented respectively (lines 3–12). A counter that is not incremented is reset to zero. Lines 13–22) are only executed if the residuals for the last N consecutive measurements of parameter \(x_i\) were all greater than \(epsilon_i\). When this is the case, the auxiliary function UpdateFitting is called to apply linear regression to the N data points from \(\mathit {buffer}_i\), and thus obtain a new slope \(a_i\) and intercept \(b_i\) for parameter \(x_i\) (line 14). The two counters are reset to zero (lines 15 and 22) and, in the for loop (lines 17–21), the function ComputeViolationTime recalculates the violation times for any of the K requirements that depend on \(x_i\) (cf. the check in line 18). For each requirement k, \(1\le k\le K\), this function first assembles a univariate polynomial \(\mathit {pol}_k(t)\) by replacing the parameters \(x_i\), \(1\le i\le n\), from the rational function (2) corresponding to the kth requirement with the linear estimates \(a_it+b_i\) (line 26). Next, ComputeViolation calls the auxiliary function ComputeNextRoot to obtain the root of \(\mathit {pol}_k(t)\) that is closest to and larger than \(\mathit {now}\) (line 27), as (assuming that the kth requirement is not violated at time \(\mathit {now}\)) this root represents the time when the requirement will next be violated. The function PrestoEval returns the predicted violation times for each of the K requirements in line 23.

The auxiliary functions UpdateFitting and ComputeNextRoot are not presented in Algorithm 1. We assume that UpdateFitting implements standard linear regression [50, 66], and that ComputeNextRoot uses Newton’s or Horner’s method [55] to approximate the roots of \(\mathit {pol}_k(t)\) and returns the root closest to and larger than \(\mathit {now}\) if such a root exists, or returns \(\infty\) otherwise.

We note that most PrestoEval invocations will be highly efficient, as lines 13–22 will be skipped most of the time. In the worst-case scenario, these lines will only be executed once every N invocations for each of the K requirements, and \(N\gg K\) for the envisaged applications of PRESTO. The following theorems show the correctness and provide a formal complexity analysis of the PRESTO algorithm, respectively.

4.1 Case Studies

The first case study involves a fruit-picking robot, inspired by recent developments in the autonomous farming domain [65, 68], and the second considers a robot-assisted dressing scenario [17] to aid independent living. The models used in both case studies feature multiple transition probabilities and/or reward parameters that are specified as PMC expressions and changes in the model parameters reflect gradual degradation of internal or environmental conditions. The key characteristics of the two case study models are described in Table 1. The nonfunctional requirements for the case studies and the properties they represent are detailed in Tables 2 and 3. Our approach is applicable for any nonfunctional requirements that can be expressed in PCTL, including safety, reliability, performance and maintainability requirements [5].

Case study 1: Fruit-picking robot The robot is expected to harvest fruit on a farm by autonomously performing the following three tasks in a reasonable time:

The behaviour of the robot is referred to as a picking session. To ensure its reliability and performance, the robot is expected to comply with the three system-level requirements summarised in Table 2.

Case study 2: Robot Assisted Dressing System (RAD) The robot is designed to support a user with restricted mobility to live independently by allowing them to dress without the need for additional healthcare support. For each dressing session, the robot needs to perform five tasks in sequence with the user:

The RAD also needs to comply with the non-functional requirements presented in Table 3.

4.2 Illustration of the PRESTO Application

Before presenting the results obtained for experiments carried out across a wider range of scenarios, we illustrate the application of PRESTO for a single such experiment taken from our first case study. We first describe the pDTMC model for a fruit-picking session and then show the execution of the two stages of our approach. The pDTMC model and other intermediate results for the second case study are available on our project website [59].

Model. Figure 2(a) shows the pDTMC for a fruit picking session, which starts with the robot attempting to correctly position itself next to a piece of fruit for picking (state \(s_{0})\). This operation may succeed with probability \(\alpha p_1\), \(\alpha \in (0,1]\) is a degradation coefficient (discussed in more detail later in this section) and \(p_1\) represents the nominal probability of success for this operation. When this operation succeeds, the robot transitions to state \(s_{1}\), where picking is attempted, or may fail (with probability \(1-\alpha p_1\)), in which case the robot transitions to state \(s_{3}\), where picking is abandoned. Picking (in state \(s_1\)) succeeds with probability \(\beta p_2\) (where \(\beta \in (0,1]\) is a degradation coefficient and \(p_2\) the success probability under ideal conditions) and the system moves to state \(s_{4}\), followed immediately with a transition to \(s_{5}\) where the picking session ends. If picking is unsuccessful, the process moves to the decision state \(s_{2}\), where the robot decides whether to move back to state \(s_{0}\) and retry positioning, or to give up (\(s_{3}\)) and end the session (\(s_{5}\)).

Fig. 2.

View Figure

Figure 2(b) shows the encoding of this pDTMC in the high-level modelling language of the model checker PRISM [45]. Lines 3–8 define the model parameters associated with (a) the operational profile, (b) the degradation coefficients, and (c) the mean execution times and the mean energy consumption of the three tasks. Here the operational profile parameters, \(p_{j\in \lbrace 1,2,3\rbrace }\), are the probabilities when the robot is in perfect working order and we assume that they are fixed domain-specific values (e.g., provided by the farmer). As mentioned earlier, the degradation coefficients, \(\alpha\) and \(\beta\), reflect changes in these probabilities as the system gradually degrades, with the assumption that their values can be obtained by either monitoring [46] or self-testing [56]. For example, a decrease of \(\alpha\) and \(\beta\) could be associated with mud slowly building up on the wheels of the robot (leading to reduced success of positioning) and with lighting conditions (resulting in picking being more difficult), respectively. Finally, \(T_{j \in \lbrace 1,2,3\rbrace }\) and \(e_{j \in \lbrace 1,2,3\rbrace }\) represent the mean operation execution times and mean energy consumption for the three tasks. The picking session is modelled by the module FruitPicking (Lines 10–18 in Figure 2(b)), and the mean execution times and mean energy consumption are used to define reward structures over the pDTMC (lines 20–25). We note that this pDTMC has \(n=8\) parameters: \(\alpha\), \(\beta\), and \(T_i\) and \(e_i\), \(1\le i \le 3\).

Stage 1. In its design-time stage, PRESTO applies parametric model checking to the pDTMC model from Figure 2(b) in order to obtain PMC expressions for the system properties associated with the three requirements from Table 2. Using the model checker Storm to perform this PMC analysis yields the following property expressions (2) for these requirements: (3) \(\begin{equation} \begin{aligned}prop_1 (\alpha , \beta) &= \frac{855\beta \alpha }{513\alpha \beta -540\alpha +1000} \\ prop_2 (\alpha , \beta , t1, t2, t3) &= \frac{-342\alpha \beta t3 -540\alpha t3 +1000(t1+t3)+900\alpha t2}{513\alpha \beta -540\alpha +1000} \\ prop_3 (\alpha , \beta , e1, e2, e3) &= \frac{-342\alpha \beta e3-540\alpha e3 +1000(e1+e3)+900\alpha e2}{513\alpha \beta -540\alpha +1000} \end{aligned} . \end{equation}\) Stage 2. In the run-time stage, the property expressions (3) are used to predict requirement violation times during invocations of the function PrestoEval from Algorithm 1. Prior to the first invocation, the circular buffers for the eight pDTMC parameters are empty, and the linear estimates of these parameters are determined by their nominal values \(\alpha ^0\), \(\beta ^0\), and \(T_i^0\), \(e_i^0\), \(1\le i\le 3\). For this illustration, we consider one of the experiments from our evaluation, in which the PRESTO hyperparameters were \(N=400\), \(\tau =400\) and \(\epsilon =0\) determined based on domain knowledge.³ Points at which line 13 of Algorithm 1 is satisfied are highlighted in blue in Figure 3(a) to (h). As \(\alpha\) appears in all three properties (3), any updating of its linear approximation triggers the recalculation of new violation-time predictions for all three requirements as depicted in Figure 3(i), (j) and (k). However, as the earliest predicted violation time, Prop1, is more than \(\tau\) hours into the future, no adaptation is triggered.

Fig. 3.

View Figure

4.3 Experimental Setup

The PRESTO algorithm and the simulations were implemented using Python 3.7 with Stormpy 1.6.4 [26] to produce algebraic expressions from the Storm parametric model checker. All experiments were carried out using a MacBook Pro with a 2.3 GHz Quad-Core Intel Core i7 CPU and 32 GB RAM. The evaluation process, and simulator behaviour, are described below.

For each case study, we compared PRESTO to a baseline approach in which regular maintenance was carried out at fixed intervals to replicate the effects of a routine maintenance schedule. The cost and frequency of maintenance was varied throughout the experiments to evaluate the value of PRESTO against a range of possible maintenance plans. We simulated data for PRESTO and the baseline system for both case studies over a period long enough to ensure multiple violations, this being 6-months for the fruit-picking application and a year for the RAD application.

The current version of PRESTO predicts violations which arise due to disruptions caused by the slow degradation of internal, or environmental conditions. These changes are observed through the monitoring of system parameters which can be susceptible to noise. To simulate such data pattern, we generated time-series for the n parameters of each system from our case studies using the five steps detailed below and summarised in Algorithm 2:

These five steps ensure that the generated time series have a non-monotonic trend that reflects the gradual degradation of internal and environmental conditions and that will lead to requirement violations. In addition, two different levels of noise are added to simulate different environmental interference [29, 30] as: low-noise where \(\gamma = 0.15\) and high-noise where \(\gamma = 0.3\).

Process simulator: We simulated both case studies over a period of time, during which multiple disruptions are expected. For each case study, we considered two service strategies as follows:

In the baseline scenario, the parameters are re-set to pre-degradation values after each maintenance operation or violation and new time series are generated. This simulates the effect of a successful maintenance activity or repair and the process continues until the predefined simulation period ends.

For the PRESTO scenario, we not only use the regular maintenance as in the baseline scenario but also predict potential disruptions. If the predicted disruption is \(\tau\) away from “now”, a preventative service will be called, which re-sets all parameters to pre-degradation level. This simulates the effect of proactive adaptation.

We evaluate the performance of PRESTO through a cost formula (detailed in Subsection 4.5) that considers true positives (TP), false positives (FP), and false negatives (FN). A TP occurs when a disruption is predicted to occur less than \(\tau\) before the actual disruption. A disruption that occurs before it is predicted is classed as a false negative and a disruption predicted to occur more than \(\tau\) before the actual disruption is classed as a false positive case.

4.4 Research Questions

The purpose of our approach is to enable autonomous systems to predict requirement violations and avoid them through adaptations triggered before the violations occur. As such, the evaluation presented in this section focuses on comparing the number of requirement violations and the cost of dealing with such violations, preventatively or reactively. In both cases, the adaptation may consist of the system initiating a self-correction or repair process or, for autonomous systems with human in the loop [8, 21, 39], calling human operator support. To ensure that the evaluation is fair, we considered scenarios in which such self-correction/self-calibration is also applied regularly, as part of a maintenance process that the autonomous system undertakes automatically or that involves maintenance by a support engineer. In line with the fact that prevention is typically better than cure, we further assumed that the preventative adaptation cost was lower than that of reactive adaptation.

With these considerations in mind, we carried out experiments aimed at answering the following research questions:

RQ1 (Effectiveness): Does PRESTO reduce adaptation cost? We assessed whether PRESTO could reduce the overall adaptation costs associated with both regular maintenance and undetected violations, by predicting, and responding to, disruptions without the need for frequent, fixed-interval servicing.

RQ2 (Robustness): Is the performance of PRESTO sensitive to noise in the measurements of the system parameters? We assessed whether the effectiveness of PRESTO is affected by noise in the measurements, commonly observed in real data. We tested PRESTO’s ability to predict violations in the presence of measurement noise at varying levels.

RQ3 (Configurability): Can the PRESTO hyperparameters be tuned to obtain different tradeoffs between the overheads and effectiveness of the approach? We evaluated the impact of different combinations of hyperparameter values on the computational overheads and adaptation-cost savings delivered by our approach.

4.5 Results and Discussion

We used the simulator detailed in Section 4.3 to obtain results for both case studies using the baseline approach and PRESTO, and recorded the number of disruptions that occurred and those which were successfully predicted as well as those for which a repair was required (i.e., the disruption was missed). To address RQ1 and RQ2, we set the following parameters based on our best estimation: N=400, \(\epsilon\)=0, and \(\tau\)=10 hours for fruit-picking, and \(\tau\)=40 hours for RAD application. To facilitate a more informed selection of these parameters, we will investigate the impact of different parameter values in RQ3.

A total service cost was then recorded for each approach as follows. Let m be the number of undetected disruptions which repairs are required and \(c_1\) be the cost of repair. Let n be the number of maintenance operations and \(c_{0}\) be the cost of a service. For the “baseline” approach, the overall cost is calculated as (4) \(\begin{equation} cost_{baseline}=m \cdot c_{1}+n \cdot c_{0} , \end{equation}\) and, for PRESTO, the overall cost is (5) \(\begin{equation} cost_{PRESTO}=(TP + FP) \cdot c_{0}+FN \cdot c_{1} , \end{equation}\) assuming both TP and FP will trigger a preventative maintenance and FN will require a repair. RQ1 (Effectiveness): Figure 4(a) and (d) show how the number of disruptions is reduced as the frequency of regular services increases for both applications. The figures indicate that the total number of disruptions occurring with PRESTO is lower than the baseline approach unless preventative maintenance is carried out very frequently. For example, when there are no fixed interval services, the baseline approach reports on average 48 and 17 disruptions for the fruit picking and RAD applications respectively. In contrast, PRESTO suffers only 3 unpredicted disruptions for fruit-picking and 6 for RAD in the same period, giving a reduction in disruptions of 93.8% and 64.7% respectively.

Fig. 4.

View Figure

Although the number of disruptions decreases for both approaches as the number of regular services increases, the baseline approach may report disruptions after 100 regular services in the fruit-picking application in comparison to PRESTO which requires just 50 regular services to avoid all disruptions in the time period. Similarly, while 35 regular services are sufficient for PRESTO in the RAD application, the baseline approach requires an additional 5 services to void the disruption completely. This result suggests that PRESTO is able to reduce the frequency of services required.

Figure 4(b) to (f) shows the total service costs for each application with the ratio of repair to preventative maintenance (\(\frac{c_1}{c_0}\)) set to 2 and 3. It can be seen that the cost of the baseline approach was at least twice as high as PRESTO with no regular service. As the frequency of regular service increases, the gap in costs reduces as expected since disruptions are less likely with frequent routine maintenance. However, we note that such a maintenance strategy is highly wasteful.

RQ2 (Robustness): Measurement noise is commonly observed in sensory data and has a negative impact on system performance. Using data with different noise levels as discussed in Section 4.3, we found that PRESTO is largely insensitive to noise with the high noise level having a marginal effect on the number of disruptions and the cost of services (Figure 5). Indeed, in all cases, PRESTO continues to outperform the noise-free baseline approach. The ability of PRESTO to reject normally distributed noise with zero mean is due to the use of the linear regression algorithm which approximates an expected noise value of zero when enough samples are obtained.

RQ3 (Configurability): We evaluated the impact of varying two hyperparameters (i.e., N and \(\tau\)) on (1) the number of disruptions that PRESTO is able to predict and mitigate, (2) the total service cost incurred during the operation period and (3) computation overheads. The impact of \(\epsilon\) seen during these experiments is minor when compared to N, which is unsurprising since the experiments presented consider disruptions caused by the gradual degradation of model parameters. The hyperparameter \(\epsilon\) plays a less important role in such settings. For alternate modes of change, which will be considered in future development, \(\epsilon\) is believed to be more important for PRESTO. Given the limited impact caused by varying \(\epsilon\) we do not include the results in this article, instead, they are available on our supporting website [59]. We carefully chose a wide range of parameter values and tested the impact of their combinations. For example, the update threshold (N) varied from 50 to 1,200, and \(\tau\) varied from 5 hours to the maximal of 120 hours for the fruit-picking application and 10 hours to 680 hours for the RAD application. The results of these experiments are presented in Figures 6 and 7 for the fruit-picking application and RAD application respectively. In addition, we examined computation overheads for operating PRESTO in terms of mean time to calculate violation time (i.e., ComputeViolationTime(), line 14 in Algorithm 1), new linear estimates (i.e., UpdateFitting(), line 10 in Algorithm 1) and the average frequency that new linear estimates are required before a disruption is predicted.

Fig. 5.

View Figure

Fig. 6.

View Figure

Fig. 7.

View Figure

Figure 6 shows the evaluation results for the fruit-picking application. For a small \(\tau\) (\(\tau \le\) 40h), the results are more sensitive to varying N in comparison to a larger \(\tau\) (\(\tau \gt\) 40h). However, the impact of varying N on the results is less significant than the impact of varying \(\tau\). Small \(\tau\) (5h) leads to the greatest number of unpredicted disruptions, which then results in a high service cost. However, as \(\tau\) is increased, more disruptions can be predicted and mitigated by PRESTO until the point where all disruptions are mitigated.

The evaluation results from the RAD application are shown in Figure 7. In comparison to the results for the fruit-picking application, we found that a larger \(\tau\) is needed to completely avoid all disruptions in the RAD application. The results suggest that, despite a similar pattern, the actual value of this hyperparameter will depend on the application and should be carefully chosen according to requirements.

PRESTO is expected to work with a wide range of values for the hyperparameters N and \(\tau\). The selection of these values by exploiting domain knowledge is expected to be effective, as is the case for obtaining prior knowledge in Bayesian learning methods. The results presented in Figures 6 and 7 highlight the impact of these hyperparameters, and provide useful insight for their determination. In general, the results are less sensitive to different values of N in contrast to \(\tau\). We observe that a smaller \(\tau\), appropriate to its application context (such as 5 and 10 hours in fruit-picking and RAD applications, respectively), leads to a larger number of unpredicted disruptions. More disruptions can be avoided by gradually increasing \(\tau\) up to a point where no further improvement can be observed. Identifying the optimal values for N and \(\tau\) in PRESTO resembles determining optimal values for the hyperparameters of other machine learning techniques.

The impact of N and \(\tau\) is also reflected in the computation time as shown in Table 4. Although the linear equations are updated more frequently when both parameters are small (\(\tau =50\) and N =50), less time is required for computing linear estimates as fewer data points (N) are used in the regression. When both parameters are large (\(\tau =1,200\) and N =1,200), the opposite trend is observed as the equations are updated less frequently but more time is required for computation. Most computation overheads are due to calculation of the roots of polynomials derived from PMC expressions, but this only takes milliseconds to complete on the computer detailed in Section 4.3.

Discussion: The results presented above show that PRESTO is able to effectively predict and mitigate disruptions before they occur. Our results suggest that the smaller the number of regular services, the more significant the performance gain is from using PRESTO over the baseline approach. In fact, when the PRESTO is running in between regular services, the cost comparison between PRESTO and the baseline approach only relies on the ratio of true positives to false positives in relation to the cost difference between preventative maintenance and urgent repairs. To see this, consider a period P between scheduled maintenance arrangements and assume first that at least one violation occurs. Then equation 4 becomes (6) \(\begin{equation} cost_{baseline}=(TP + FN) \cdot c_{1} , \end{equation}\) as \(n = 0\) and \(m = TP + FN\); and PRESTO therefore reduces costs in the period P if: (7) \(\begin{equation} (TP + FP) \cdot c_{0} \lt TP \cdot c_{1} . \end{equation}\) Let the difference in cost in term of ratio be (\(\frac{c_{1}}{c_{0}}\)), then PRESTO reduces costs in the period P if: (8) \(\begin{equation} \frac{c_{1}}{c_{0}} \gt 1 + \frac{FP}{TP}. \end{equation}\) where \(TP \ne 0\).

PRESTO predicts disruptions by continually assessing the gradual change observed in system/environmental parameters, a common phenomenon in real-world applications [63, 64] that is underexplored by current research into self-adaptive systems. The first stage is design-time and we use the fact that computational resources are less of an issue by calculating PMC expressions that can be updated at run-time. This stage could be exploited further as prior knowledge of the order of importance of the parameters in the system equations and how they interact could reduce the level of monitoring required. In the second stage, at run-time, system disruptions are predicted by linear equations that are used to provide future parameter values for the PMC algebraic expressions. We use piece-wise linear modelling to simplify prediction whilst allowing the trend to change over time. This also allows information about changes in slope to be stored so that any underlying periodicity or seasonal component could potentially be recognised and accounted for. We summarise these and other opportunities for enhancing PRESTO when we propose future work directions for our project in Section 6.

Threats to Validity: Construct validity threats may arise from over-simplifications and unrealistic assumptions made when establishing experiments for evaluation. To mitigate these threats, we used two case studies inspired by real autonomous systems described in the recent research literature, i.e., a fruit-picking robot [65, 68] and a robotic assistive dressing system [17]. We also avoided the use of explicit costings and instead used cost ratios where we assumed that the cost of repair exceeded the cost of maintenance. In addition, the evaluation of PRESTO relies on simulated patterns of parameter change, which mimic a slow degradation in the system. Additional experiments are necessary to assess the performance of PRESTO for data patterns derived from real systems, and presenting more complex patterns of change.

Internal validity threats can originate from bias in interpretation of the experimental results. To mitigate this threat we evaluated PRESTO by answering three independent research questions. To further reduce the likelihood of having a biased result, the evaluations were compared against a baseline approach that is widely used in deployed systems, and we considered a broad range of parameter values for costs and service frequency. Finally, to enable the independent verification of our results, the source code of our simulator and the data from all the experiments to produce the tables and figures in the article are available online at [59].

External threats to validity may arise if the systems under consideration or the assumptions made in the experimental framework are not indicative of the characteristics of other systems to which PRESTO may be applied. To mitigate these threats, we evaluated PRESTO using two case studies from different application domains. Each of the models arising from these case studies makes use of a pDTMC with a different model structures. External validity threats may also arise due to restrictions in the types of non-functional requirements supported by PRESTO. To mitigate these threats, we chose a set of properties that covered a broad range of requirements (reliability, performance, resource utilisation, etc.). Additional experiments are still required to confirm that PRESTO can predict a wide range of disruption types in other application domains.