Dealing with observational data in control

There is growing interest in the use of control theory for interdisciplinary applications, where data may be sparse or missing, be non-uniformly sampled, have greater uncertainty, and where there is no opportunity to collect repeat measurements. In such applications, problems posed by observational data and the issue of missing or irregular data need to be considered. We present a review on dealing with observational, missing and irregular data for control applications. This considers the following issues: (i) how to identify a system model from observational data subject to missing measurements, (ii) how to determine control inputs when output data includes missing measurements, and (iii) how to ensure stability when future update times may be missed. Dealing with observational data and missing measurements is a key problem within the statistics literature, so we introduce statistical methods for dealing with this type of data. We aim to enable the integration of well-developed statistical methods of dealing with missing data into control theory. An example problem of using anticoagulants to control the blood clotting speed of patients is used throughout the paper.

• Epidemic analysis, where optimal control has been applied with the aim of increasing recovery rates and decreasing infection rates with finite resources (Behncke, 2000;Hansen & Day, 2011;Nowzari, Preciado, & Pappas, 2016).
• Resource management and operations research, where control theory has been applied to planning, ecology and resource management, including MPC to determine policies for oceanic fisheries (Cliff & Vincent, 1973), optimal control for water resource management (Nicklow, 2000), and MPC to control the flow of traffic (Van den Berg, Hegyi, De Schutter, & Hellendoorn, 2003).
of designing interventions to achieve a specified performance in the presence of noise, uncertainty and external disturbances.However, in many of these applications the controller needs to be built from observational data which may be sparse or missing, be non-uniformly sampled and have large uncertainties.Further, often there is no opportunity to collect repeat measurements and there are questions as to causality of effects.This contrasts with the standard control approach where multiple measurements can be made on the system, the experimental protocol can be designed to enable 'ideal' data to be collected, retesting is straightforward and generally a single system is considered over a long time period.There is a risk, therefore, that when attempting control in some interdisciplinary applications there may be lack of consideration of the nature of the data that are available, especially when the data are observational rather than experimental.
If control system techniques are to be adopted for interdisciplinary applications where data collection is problematic, an awareness of the complexities of the data is required to avoid introducing bias.Our aim in this article is to review and discuss some of these issues, with most attention on causation and problems arising through missing and perhaps irregular data.We consider the challenges of: identifying a system model from observational data when measurements are missing, determining control inputs when output data are missing, and ensuring stability when future update times may be missed.
To fix ideas we will root our discussion on a medical example, namely the long term anticoagulation of patients with chronic conditions, using the records of 152 patients from Newcastle upon Tyne, UK.Anticoagulants are prescribed to control the blood clotting speed of patients at risk of thrombosis (blood too easily clots) or severe bleeding (blood does not clot sufficiently quickly).Clotting speed is measured through the International Normalised Ratio (INR), which is a standardised measure with high values indicating long clotting times (Baglin, Keeling, Watson, & the British Committee for Standards in Haematology, 2006).Usually INR is deemed under control if it is in the range 2-3 units.To achieve control the most common anticoagulant is warfarin, which patients can take in various dosages.Raising the dose increases the clotting time and lowering the dose the opposite.Hence, with warfarin dose prescribed at clinic visit k considered as the input u (k), and INR considered as the output y(k), we have a classic control problem.Unfortunately INR is rarely stable, the effect of warfarin differs between patients, and in addition it interacts with many lifestyle factors, including diet, alcohol, exercise and co-medication, meaning that control is not always straightforward.
Fig. 1 exemplifies, using the results from three patients.The upper part of each plot shows the dose of warfarin prescribed, and the lower part shows the measured INR at clinic visits.Patient A is typical of one class of patient.There is considerable noise in the INR values, even when the patient is essentially under control.There are periods where the INR increases, requiring a corresponding decrease in dose, and other periods where the opposite occurs.The need for changing dose frequently to react to changing INR is evident.On the other hand, the second plot shows results for a second class of patient, where there is some instability when the patient is first prescribed warfarin, but then a suitable dose is found and the patient remains stable for a considerable time.Unfortunately, once stability is achieved it is not necessarily maintained, as illustrated in the third plot.Here a patient was well controlled for several years until, for unknown reasons, the INR became highly volatile.Note also that the dose level for stability differs considerably between the three patients: optimum treatment is patientspecific.
This example will be referenced throughout the paper.In Section 2 we will consider briefly the issue of assigning causation in observational data.In Section 3 we will discuss system identification from observational data subject to missingness, explaining some of the ideas and methods that have been developed in the statistical research literature, where there is considerable experience in dealing with these problems.
In Section 4 we turn to control in the presence of missing data.We assume that the intention is to sample periodically in time, but that occasional planned sample times are omitted.In Section 5 we turn attention to more irregular and variable sampling rates and concentrate on event-based control.Discussion in Section 6 concludes the paper.A list of abbreviations used throughout the paper is given in Table 1.

Causation
Simple causal reasoning about feedback control systems is difficult as the inputs are influenced by the outputs (Åström & Murray, 2010).Therefore, in conventional control system design, the problem is usually split into two steps: i) system identification from open loop input-output experiments and ii) control design and implementation.To identify the system, chosen inputs are applied in open-loop experiments.The term open-loop implies that the response has no influence on the input and so the issue of causation is not considered.By splitting the control design into two steps, the issue of causation in identifying a suitable plant model can effectively be ignored.However, experimental data can sometimes only be obtained in a closed-loop situation, for example when a feedback controller is always required due to safety or stability issues.Hence, methods to identify system models from closedloop data have been developed within the control systems literature (e.g.Gilson and Van Den Hof, 2005;Van den Hof, 1998;Ljung, 1999;Söderström and Stoica, 1989;Taylor, Young, and Chotai, 2013;Verhaegen, 1993;Young, 1970;2012, Section 8.7).
A schematic of an open-loop plant used for system identification is E.D. Wilson et al. Annual Reviews in Control 46 (2018) 94-106 presented in Fig. 2a.The experimenter is free to set the input at any of the feasible levels and can identify changes in output caused by changes in input.However, when a plant is to be identified from observational rather than laboratory or experimental data, the problem of interference in input determination can rarely be ignored.The most obvious of these is that in observational data the input is set with the intention of maintaining control, not for the benefit of the observer.Consequently there may be rather little spread in the range of inputs selected, and bad decisions should be rare, which makes system identification more difficult.Further, often the input will be at least partially informed by the previous outputs, as in Fig. 2b.
In the warfarin example a large proportion of high INR values are followed by decreases in dose, and a large proportion of low INR values are followed by an increase in dose.We will model this in the next section and show that, although dose is associated with previous INR, the relationship is not deterministic and there are many occasions where the dose is unchanged despite high or low INR, and a rather smaller number of times where the dose moved in the opposite direction to expectations.Association between output and selected input can be allowed for by careful consideration of direct and indirect effects.Thus an output at time k can influence outputs at time + k 1 directly, through serial correlation in the response, and indirectly by its effect on the input u(k).Path analysis techniques can be used to disentangle these (Borgan, Fiaccone, Henderson, & Barreto, 2007;Fosen, Ferkingstad, Borgan, & Aalen, 2006).
Exogenous variables may also drive input selection.Continuing with the anticoagulation application, in the hospital notes accompanying the data there are regular explanatory comments on unusual INR possibly being caused by co-medication, increased or decreased alcohol, missed doses, preparation for surgical procedures, and so on.This often meant the change in dose was different from that which might be expected.One example alongside some unhealthily high INR values is the comment "Patient has bad sight and may have been taking 3 mg instead of 2 mg.Request large print".Another is a comment alongside a sharp decrease then later recovery of INR for one elderly patient: "Nursing home has been giving patient 1-2 glasses of cranberry juice every day!!.Advised of effects of cranberry on warfarin.Home will stop".The nursing home, like the current authors, was evidently unaware that cranberry can react with warfarin.In both cases the action was to fix the underlying problem and not to change dose to accommodate unusual INR.In these cases, if a reason for a dose change is recorded then it can be allowed for in modelling.
Most often there is no record of exogenous factors or confounders that could affect either the output or, importantly, decisions on input level.This is the scenario of Fig. 2c, where both input and output are associated in some way with a disturbance.Since the disturbance is not measured, at least directly, then disentangling the effect of the input from the effect of the disturbance confounder becomes challenging.This leads naturally to issues that have been discussed in various literatures, especially clinical epidemiology and social science, under the name of causal inference.Causation in such literature does not mean determining the true physical process, at a molecular level for instance when a drug is administered, but instead reliably attributing an effect to a cause at a more macro level.Thus changing warfarin by this amount will lead, on average, to a change in INR of that amount.
There are many approaches to causal inference, some of which are contentious (e.g.Dawid, 2000;Ding & Li, 2018;Pearl, 2009;Rubin, 2005).The most common is based on the potential outcomes or counterfactual framework introduced by Rubin (1974), though he acknowledged antecedents as far back as Neyman (1923).Associated with each potential input U(k) is the potential output + Y k U k ( 1, ( )) that would be observed if the actual input u(k) took that value.Causal effects are then defined in terms of comparisons of Unfortunately the fundamental problem of causal inference (Holland, 1986) is that only one of the potential outcomes can ever be observed, that corresponding to the selected u(k).Consequently, as Pearl (2009) states, behind every causal conclusion there must lie some causal assumption that is not testable in observational studies.
The most important of many technical assumptions are on the input selection mechanism: how, in the observed data, was the input selected?As stated, if the reasons are known then they can be adjusted for.The problem is when the reasons are not known.For instance, if a clinician prescribes a low dose because they are aware that the patient likes to drink cranberry juice, then an analyst unaware of the additional information will misinterpret the effect of cranberry as the effect of warfarin.A fundamental assumption is then of no unmeasured confounders: anything that can influence both the input and the output is recorded in the available data and can be adjusted for in the analysis, at least in principle (e.g.Ding & Li, 2018;Streeter et al., 2017).How realistic such an assumption is will be context-specific.Returning to the

System identification
Much of the control literature on missing-data, or non-uniform sampling, is concerned with designing stable control systems in the presence of timing uncertainty and missing information (see Section 5).
A key concern is the performance of the system in the presence of bounded uncertainties.The approach to this problem tends to be theoretical and system models are either pre-specified (Arzén, 1999;Åström, 2008;Heemels, Sandee, & Van Den Bosch, 2008), or a general state-space system model is considered (Cloosterman et al., 2010;Heemels & Van De Wouw, 2010;Lunze & Lehmann, 2010;Sala, 2005).It is generally assumed that the plant is stable and there are no model uncertainties (Lunze & Lehmann, 2010).The question of how to identify the system model from data with missing measurements is often not considered.It is nonetheless a key issue in designing controllers from observational data.
There are some examples in the control domain where identification from data with missing measurements is considered (see Shi &Fang, 2010 andDing &Ding, 2010 and the references within).The estimation of autoregressive with exogenous terms (ARX) models (Isaksson, 1993;Wallin, Isaksson, & Ljung, 2000), autoregressive (AR) models (Larsson & Söderström, 2002;Mirsaidi, Fleury, & Oksman, 1997), and autoregressive moving average (ARMA) models (Jones, 1980;Rosen & Porat, 1989) from data with missing, or incomplete measurements has been considered.A general frequency domain approach to system identification with missing data is presented in Pintelon and Schoukens (2000).This approach works for any model structure (e.g.ARX, ARMAX, output-error, etc.) and is based on treating missing input, or output data as unknown parameters.This can potentially lead to a large number of parameters to be estimated.The particular problem of missing output data is considered by Sanchis and Albertos (2002), where the input (control action) is assumed to be updated at a fixed rate, with the output measured synchronously with the input, but with an irregular availability pattern.This is similar to the situation that could be encountered in our anticoagulation example, where the patient continues to take a regular fixed dose (input), but appointments may be missed, so output data are sometimes unavailable.
In the control literature, when modelling missing data, the reason or mechanism for the data being missing is rarely considered.Data are either assumed to be missing periodically (Markovsky, 2013;Rosen & Porat, 1989;Sanchis, Sala, & Albertos, 1997), or a random Bernoulli pattern of missing data is assumed where each measurement has a fixed probability of being missing and misses are independent (Rosen & Porat, 1989;Shi & Fang, 2010).

Missing data mechanisms
The question of how to develop a model from data that contains missing measurements is a key problem in the statistics literature.Rubin (1976) introduced three important classifications for missing data.These are: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR).
Suppose that a vector of responses Y is intended to be measured.Let R be a vector of observation indicators, with the convention that the elements of R are one when the corresponding elements of Y are observed, and zero if they are missing.Divide Y into sub-vectors of observed and missing components, Y obs and Y miss respectively.Use f(.) as generic notation for probability distributions.Data are said to be: • Missing completely at random (MCAR) if the missingness mechanism is independent of the responses: obs miss • Missing at random (MAR) if the missingness mechanism is con- ditionally independent of the missing data, given the observed data: ).

obs miss obs
These concepts, and variants thereof (e.g.Mealli & Rubin, 2015;Seaman, Jackson, & J., 2013), underpin much of the statistical research into methods for dealing with missing data.They are not considered in the control literature, where data are generally assumed to be MCAR.
Assuming that the parameters that determine R are distinct from those that determine Y, it is accepted that simply ignoring the missing data reduces efficiency under MCAR (as there is less data to use) but otherwise leads to no difficulty.Some statistical methods, notably maximum likelihood (ML) estimation, remain valid under MAR if the missing data are ignored (Little, Rubin, & Zangeneh, 2017).To see this note that we need to maximise the observed data likelihood.This is obtained by integrating out Y miss from the full data distribution: So all parameters in f(R|Y obs ) and f(Y obs ) can be estimated consistently.Åström and Torsten (1965) introduced the ML framework to the control community, and Isaksson (1993) implements ML with a Kalman filter to estimate ARX model parameters from missing data.
Data are said to be ignorable in the context of likelihood inference when the parameters describing the measurement process are functionally independent of the parameters describing the missingness process and the data are MCAR or MAR.When these conditions are not met the data are non-ignorable.If data are ignorable, inferences can proceed by analysing the observed data only.Ignoring missing data under MNAR will usually lead to biased and inconsistent estimation.Correct inference needs to take into account the missing data and mechanism, and then depends upon the modelling assumptions all being correct.This is of course a common requirement across all areas of modelling and estimation but a difference in the missing data sphere is that it is impossible to verify MNAR assumptions no matter how much data are available: see, for example, the informatively-titled article "Every missingness not at random model has a missingness at random counterpart with equal fit" by Molenberghs, Beunckens, Sotto, and Kenward (2009).

Longitudinal data
Observational data for control design are likely to be longitudinal.In healthcare applications, longitudinal data (often termed time-series data in control) contain measurements from individuals that are taken repeatedly through time.In longitudinal data, partially observed sequences due to missed appointments, or especially dropout (patient leaves study after a time and there are no subsequent measurements taken) are very common (Diggle, 2002).There is a huge literature on missing data methods in longitudinal studies; see, for example: Diggle (2002), Molenberghs and Kenward (2007), Ibrahim and Molenberghs (2009), Molenberghs et al. (2004), Ibrahim, Chen, Lipsitz, and Herring (2005), Pigott (2001), andFarewell, Huang, andDidelez (2017) and the references therein.
Simple methods for analysing longitudinal data subject to missingness include complete case (CC) and last observation carried forward (LOCF) techniques.In CC analysis only subjects without missing data are included in the estimation procedure, while under LOCF missing values are simply imputed from the last observed value.Both of these methods have been justifiably criticised but seem still to be in use (National Research Council, 2010).
More principled methods rely on models to at least some extent, with early approaches falling into one of three general classes.Recalling that Y represents the responses of interest and R indicates whether values are observed or missing, writing ) leads to the selection model class, in which the marginal distribution of Y is modelled, by a multivariate Gaussian distribution for example, and then the observation indicator is allowed to depend upon Y through an appropriate binary data model.On the other hand writing ) leads to the pattern mixture class of models based on the marginal distribution of R and then separate models for Y given each possible R. Mathematically the two factorisations are equivalent but they lead naturally to different modelling assumptions.A third general class is also in widespread use, the shared parameter approach.
Here it is assumed that association between Y and R arises because of shared dependence upon an unobserved random effect (Fitzmaurice, Davidian, Verbeke, and Molenberghs, 2009;Henderson, Diggle, and Dobson, 2000, Chapter 19).
Given the assumed model, estimation techniques include direct likelihood, Bayesian analysis, the expectation-maximisation (EM) procedure, single and multiple imputation and estimating equations (Fitzmaurice et al., 2009, Chapters 15, 17-21).Doubly robust procedures have been developed which attempt to reduce the dependence upon untestable assumptions by creating valid inference either if assumptions about R are correct or if assumptions about Y are correct, but not necessarily both (Bang & Robins, 2005;Carpenter, Kenward, & Vansteelandt, 2006;Ding & Li, 2018).

Anticoagulation example
A simulated model based on the warfarin data is developed and used here, and later in the article, to demonstrate various issues with missing data.
The output y(k) is the log(INR) and the input u(k) the dose (mg) of warfarin.In the data (9851 records from 152 patients), 98% of dose changes are from the set − − { 0.5, 0.25, 0, 0.25, 0.5}.We fitted a propor- tional odds model (McCullagh, 1980) to the observed dose changes and used this to generate changes in inputs in the simulations.The most important factors are the most recent INR (if high tend to reduce dose) and the previous change in dose (tendency to reverse).Table 2 compares the actual changes with those predicted by the model (as the most likely).
For the outputs we used the simple model, ( 1) ɛ.
1 1 (1) Based on a typical patient, we take = a 0.4 1 and = b 0.25, 1 together with ε ∼ N(0, σ 2 ) where = σ 0.25.Example input-output data generated using this model are given in Fig. 3: the patterns look very similar to those in the real data.To generate different mechanisms for missingness in the observed data, we used a logistic model for the probability p(k) that y(k) is observed: 1 1 (2) (3) where = + x x x expit( ) exp( )/{1 exp( )}.
To illustrate we chose the parameter values in Table 3 to give 50% missingness under each of the three mechanisms.The non-zero coefficients are large as z(k) is usually small.Parameters were estimated by ML with allowance for correlation between successive y(k).When there were missing data, we estimated using all values k such that y(k), − y k ( 1) and − u k ( 1) were observed.Table 4 summarises our results.If there are no missing data, the parameters are estimated well.All means are within the simulated noise of the true values.Under MCAR there is no bias but there is more variabilitythis is simply because there are less data available for estimation.Under MAR there is no bias because we used ML estimation, which remains consistent if MAR missing data are ignored.Under MNAR there is severe bias in the estimated parameters.

Periodic control
This section considers control design and implementation with periodic missing data, i.e. data are sampled periodically but for certain samples the output data are missing.For model-based control, the problem can be split into independent issues of model identification and model-based design.The issue of model identification was summarised in the previous Section 3.3, where an ARX model was fitted to simulated data with missing outputs.In this section we study control design and performance.We first evaluate the performance of controllers designed using models developed from missing data, but where all control data are available, before considering the case when not all the control data are available.We end this section with discussion on optimal dynamic treatment (ODT) techniques and explore how these link to a general control problem.

Control design using incomplete observations
Controllers developed for non-standard applications tend to be based on PID, PI, or MPC designs.PID controllers are one of the most ubiquitous control solutions, and as expected PI and PID controllers have been applied to the control of non-traditional systems.Examples include: in the control of anaesthesia (Dumont, 2012;Dumont, Martinez, & Ansermino, 2009); for the control of blood glucose using insulin (Hoekstra et al., 2009;Rattan & Nasraway, 2013); and for micro-climate control (El Ghoumari, Tantau, & Serrano, 2005).Due to the ability to handle constraints, which can be numerous in real world problems, MPC methods are also popular, probably more so than PID and PI methods.Examples include: again the control of blood glucose using insulin (Chakrabarty et al., 2017;Hovorka et al., 2004;Plank et al., 2006); the design of adaptive interventions (Rivera et al., 2007); for the solution of decision problems in economics (Grüne, Semmler, & Stieler, 2015); and to control the heart rate (Aerts et al., 2008;Leor-Librach et al., 1999) and growth (Aerts et al., 2003) of animals.
In applying control to non-traditional areas it may be necessary to use observational data for control design.As described in the previous section, this can lead to a biased estimated model when data are missing.To illustrate the effect of this model bias on control system performance, we use the warfarin model of the previous section and

Table 2
Frequencies of observed and predicted dose changes.), as obtained under MNAR.For the PI control design we used LQ optimisation, as described in Section 5.4 of Taylor et al. (2013), with weights of . For the MPC controller, we used a receding horizon control law with a prediction horizon of three, that was based on minimising the sum of the squares of differences between a reference trajectory and predicted future outputs (based on a discrete state space model).No constraints on the inputs were used, so a simple analytical solution for the optimal inputs could be computed (Camacho & Alba, 2013).In this simple example, offset-free MPC was not used i.e. the MPC control law does not include the equivalent of integral action (Maeder, Borrelli, & Morari, 2009).
Sample sizes of 100 were used and the desired INR output (reference trajectory) defined to be 2.5 for the first 50 samples, then 3.5 for the remaining 50 samples.These are the normal targets for patients with occasional and recurrent deep vein thrombosis respectively (Baglin et al., 2006), with the increase mimicking a change in diagnosis.Performance is evaluated by considering the proportion of INR outputs that are within ± 0.5 of the desired INR, and performance metrics are the averages from 1000 different trials.Example control responses from one trial are summarised in Fig. 4 and performance metrics given in Table 5.In all cases the noise to signal ratio is high, so none of the methods yield results that are within the desired INR range for more than 50% of samples.Performance is improved when MPC is used with the correct model, however using MPC with an incorrect model results in steady-state errors.This is expected as the performance of MPC is strongly associated with the accuracy of the model.Steady state errors could be removed by, for example, introducing an integral of error state into the MPC rule (Akçakaya & Sümer, 2009;Di Ruscio, 2013;Exadaktylos & Taylor, 2010).MPC techniques that are robust to model mismatch have also been developed (Kothare, Balakrishnan, & Morari, 1996).In practice these methods have rarely been used when MPC has been applied to non-conventional control examples.However, they could increase control performance in cases when the model is estimated from observational data and may be biased.The performance of the PI controller is less affected by model mismatch.Again this is as expected since, although the PI weights were tuned using a specific model, the resultant response is relatively slow and reasonably robust; and inclusion of integral action ensures there are no steady state errors when the model is incorrect.

Control implementation with missing data
During control implementation there are two distinct approaches to dealing with missing data, either the missing values can be estimated (reconstructed/ imputed) (Phillips & Tomizuka, 1995;Shi & Fang, 2010), or the control decision can be delayed until data are available (Astrom & Bernhardsson, 2002;Heemels et al., 2008).The second of these is an event based control strategy (Åström, 2008) which will be discussed in Section 5, while the first of these is considered here.
The theory for time-driven, periodic systems is well developed.Therefore, when data are missing it is attractive to provide estimates of the outputs at a constant sampling rate, so as to have access to the systems theory for time driven systems (Heemels et al., 2008).A Kalman filter can be used to estimate outputs with uniform sampling period when measurements are missing (Heemels et al., 2008;López-Orozco, de la Cruz, Besada, & Ruipérez, 2000).Similarly, a Leuenberger-type observer has also been used to estimate the state of the plant at uniform times (Heemels et al., 2008;Phillips & Tomizuka, 1995).
Several Kalman filter based methods can be used to estimate the state of a system when data are missing (Khan & Gu, 2009;Sinopoli et al., 2004a).Of these, the simplest method is zero-correction (KF zero), where if no output data are available (e.g.data are missing) the predicted state estimate is used as the true state without updating (Khan & Gu, 2009).This method yields simple, fast estimation without any need to store observations.However, drawbacks include reliance on the system model and spike phenomena in the estimation error.An alternative approach is to use past observations (KF past), where the Table 3 Missingness simulation parameter values.previous observations are used to correct the predicted state (Khan & Gu, 2009).This reduces some of the shortcomings of the zero-correction method, but it requires the most recent observation to be stored.Khan and Gu (2009) have also considered the development of more complex Kalman filter based optimisation algorithms.Missing data can be even more simply estimated by replacing the missing measurement with the last measured value.This removes the need for Kalman filtering, but requires the previous observation to be stored.This last observed carried forward (LOCF) technique is an ad hoc method for inference that assumes data are MAR (Molenberghs et al., 2004).More principled statistical approaches to imputation include direct likelihood, Markov chain Monte Carlo, the EM algorithm and multiple imputation (Ibrahim & Molenberghs, 2009;Molenberghs & Kenward, 2007).
To illustrate, PI control of the warfarin model system is evaluated when approximately 50% of output data are missing and the missingness can be described as MCAR, MAR or MNAR using the method described in Section 3.3.Different methods are used to impute the missing data: KF zero, KF past and LOCF (also, an event based method, as discussed in Section 5.1.2). Results averaged over 1000 trails are given in Table 6.Values for the percentage in range are only given for the measured data, as in practice unmeasured values would be unavailable.
A key issue during control is ensuring that the unmeasured values also have reasonable values.Results show that the KF zero imputation method, which is most reliant on the model, provides best performance when the model is correct and worst when it is biased.

Optimal dynamic treatment
There is a link, so far perhaps not fully exploited, between control methodology and research into optimal dynamic treatment (ODT) techniques developed for personalised medicine.In ODT, also called adaptive strategy or adaptive intervention, medical interventions adjust to changing individual circumstances.For overviews and a range of techniques see, for example: Chakraborty and Moodie (2013), Luedtke and van der Laan (2016), Orellana, Rotnitzky, and Robins (2010), Schulte, Tsiatis, Laber, and Davidian (2014), Zhao, Zeng, Rush, and Kosorok (2012), Zhang, Tsiatis, Laber, and Davidian (2012), and Zhou, Mayer-Hamblett, Khan, and Kosorok (2017).An archetypal situation is the anticoagulation scenario of Section 1, where dosage of warfarin is adjusted to reflect change in INR as a result of new diets, co-medication and so on.Thinking of treatment as input u(k) and an appropriate health measure such as INR as outcome y(k), the generic control and ODT scenarios are one and the same.Despite this, there seems to have been little input from the control community into developments in ODT.This may be because many early applications of ODT, starting with Murphy (2003), concentrated on cases where there were few potential treatment options available, classically simply a choice between two or three possible treatments, and few (again, just two or three) repeat visits.There has been more recent interest in applications in which the treatment can take a large number of values, such as the dose of a drug, and can be considered as effectively continuous, and for chronic conditions in which follow-up is long, again as in the warfarin application (Barrett, Henderson, & Rosthøj, 2014;Henderson, Ansell, & Alsibani, 2010;2011;Rich, Moodie, & Stephens, 2014;Rosthøj, Keiding, & Schmiegelow, 2012).
The ODT literature has concentrated primarily on modelling, estimation and inference, i.e. system identification as discussed in Section 3.There has been less attention given to subsequent use of the   modelled systems for control, which means large and well established areas of control expertise could and should be brought to bear in personalised medicine; robust control for instance.In the other direction some of the techniques developed in ODT could be taken into control applications, including careful handling of sparse and noisy observational data, and perhaps the use of A-learning techniques borrowed from machine learning and adapted to allow causal inference (Chakraborty & Moodie, 2013;Murphy, 2003;Robins, 2004).
Missing data causes two distinct issues in ODTs.These are introduced by Rosthøj, Henderson, and Barrett (2014).The first of these is the issue of missing data in the past.This refers to the situation where the data set used for estimation of the optimal regime contains missing values.This issue is comparable to the identification of models from missing data (see Section 3) and standard techniques of parameter estimation in the presence of missing covariates or responses in longitudinal data can be used.The second issue is missing data in the future; when calculating the optimal treatment it is assumed that the patient will be seen again at all future time points, so the treatment can be changed if necessary.This second issue is potentially more of a problem and has been given much less consideration within the literature.
To our knowledge the issue of missing data in the future has been considered in the ODT literature only by Rosthøj et al. (2014), who proposed the use of a fixed-dose allocation rule.In their optimal dynamic fixed-dose strategy it is assumed that each patient has visits scheduled for regular times = … k 1, 2, but the possibility that some visits may be missed is allowed for, at least under the assumption that missingness is completely at random.At each time point k a fixed dose strategy assigns the current dose u(k) to the patient for all times within a future observation window.The optimal fixed dose is the dose the patient should stay on to optimise the potential outcome if it is assumed that the dose will not change throughout the window, hence providing protection against the patient missing future visits.It is proposed to recalculate the optimal fixed dose at each visit that does take place.Rosthøj et al. (2014) showed that when there are no missed visits the optimal dynamic fixed dose strategy does not differ considerably from the optimal dynamic strategy.When there are missed visits the optimal dynamic fixed dose strategy outperforms the optimal dynamic strategy.A similar strategy, without missing data, is proposed by van der Laan, Petersen, and Joffe (2005) and Petersen, Deeks, Martin, and van der Laan (2007).At each visit (or time point) a future treatment regime is fully specified, this is updated at subsequent visits.Such a strategy is a form of MPC, something that we return to in the next section.

Irregular control
The usual approach to digital control is to sample periodically in time, which is known as periodic, Reimann (Astrom & Bernhardsson, 2002), or single-rate (Hu & Michel, 2000) sampling.The previous sections have considered periodic data with missing outputs.However, in many applications such as medical treatment control, or resource management, scheduling may be irregular and not synchronised with any underlying sampling period.Irregular data are not uniformly sampled and occur when scheduled visits (or measurements) are not uniformly distributed.Data could be constrained to treat the problem as one of periodic missing data, but if a fast underlying time period is used then data may be very sparse; alternatively using a slower time period may give poor resolution of when the actual event occurred.In such cases it may be better to treat the data as irregular.
In our anticoagulation example, Patient A (Fig. 1) has relatively regular observations, with 68% of intervals being within 10% of 7, 14, 21 or 28 days.There are nonetheless some 13% of intervals in excess of one month.Table 7 shows the distribution of gaps between visits.Many patients have more irregular sampling times.
As in the case of missing data, irregular visiting schedules can be categorised into: visiting completely at random (VCAR), visiting at random (VAR) and visiting not at random (VNAR).The term VCAR is used when the visit time is independent of the outcomes, VAR when the visit time is independent of the outcome given previously observed data, and VNAR when the visit time is not independent of the outcome given previously observed data (Pullenayegum & Lim, 2016).

Event based control
In an event-based system, the sampling is event-triggered, as opposed to time-triggered.It is the occurrence of an event, rather than the passing of time, that determines when samples should be taken (Arzén, 1999;Åström, 2008;Heemels et al., 2008).Within the literature different terminologies have been used to describe event-based control (Lunze & Lehmann, 2010).These include event-based sampling (Åström, 2008), event-driven sampling (Heemels et al., 2008), Lebesgue sampling (Astrom & Bernhardsson, 2002), aperiodic control (Arzén, 1999), and asynchronous control (Losada, Rubio, & Bencomo, 2015).All describe sampling that is not time triggered, but driven by the occurrence of an event.
In time-driven controllers the focus is on performance, while in event-driven controllers the aim is to balance performance with practical aspects (Heemels et al., 2008) such as the availability of data, and resource utilization.There are several reasons for using event-triggered sampling.These include: (i) the nature of the measurement e.g. the system may use event driven sensors such as encoders (Åström, 2008); (ii) the difficulty in sticking to a time-triggered paradigm, such as in modern distributed control systems (Zhang, Branicky, & Phillips, 2001), or applications like medical treatments; and (iii) resource utilisation, for which event-triggered sampling may reduce the number of control updates necessary (Liu, Wang, He, & Zhou, 2014).Also, interestingly in the context of developing optimal treatment rules in healthcare, event driven control is the dominating control principle in biological systems, including humans (Åström, 2008;Gawthrop, Loram, Lakie, & Gollee, 2011;Loram, Van De Kamp, Gollee, & Gawthrop, 2012).
In event-driven control, the nature of the event can vary, and examples of events could include a signal passing a threshold, or the arrival of a data packet to a node.The arrival of new data, or lack of data, is an example of unintentional event driven control and the passing of a threshold an example of intentional event driven control.Most examples of intentional event driven control consider the event as the signal, or a state, passing a threshold.Research into unintentional event driven control has focused on the area of Networked Control Systems (NCS).
A block diagram of a simplified event based control loop is shown in Fig. 5.The controller operates in open-loop between the events, with feedback actions occurring at events.Periodic systems also operate in open loop between sampling instants.In periodically sampled systems, the standard procedure is to keep the control constant between the sampling instants using a zero-order-hold scheme.Sensor signals are periodically sampled, and a zero order hold used on actuators.In event based systems, update times are not scheduled or known.The way the open-loop signal is generated is key and properties of the system depend on how this signal is generated (Åström, 2008;Cervin & Åström, 2007;Lunze & Lehmann, 2010).In event based control the sampler is replaced by an event generator, and the hold by the control input generator.

Table 7
Sampling intervals for Patient A. We take intervals to be ≃ 7 etc. if within 10%.The maximum interval was 84 days.The balance of intervals are within one month but not close to multiples of 7 days.The concept of a control input generator (Fig. 5) was introduced by Åström (2008).It could represent a generalised hold, but is introduced due to the importance of the control signal generator in determining behaviour in event based control.Åström (2008) shows that it is desirable to have holds that give control signals that are initially large, and then decay fast to ensure the system is robust to changes in when the event occurs.
Gawthrop and colleagues (Gawthrop et al., 2011;Gawthrop & Wang, 2007;2009) use basis functions to specify the controller evolution in the open-loop between events, where the open-loop control signal is constrained to be the linear sum of pre-specified basis functions.The use of basis functions to generate the control signal between events is equivalent to using a generalised hold and, therefore, similar to using the control input generator described by Åström (Åström, 2008;Cervin & Åström, 2007).

Intentional event driven control
In event driven control, the nature of the event can vary.The most commonly studied type of intentional event is the case when a signal, or state, deviates by a set amount.This can be cast, for example, as the measurement signal crossing a specified level, or as the absolute error exceeding a certain amount.The logic for event detection is discretionary, however, having a very complex event detector may needlessly complicate the control algorithm (Arzén, 1999).When the behaviour is event driven, it is not until the event detection criteria has been met (e.g. until the measurement signal has deviated sufficiently from the desired set point) that a new control action is taken.
The difference between an event-based, and a sampled-data controller is considered by Åström and Bernhardsson (Åström, 2008;Åström & Bernhardsson, 1999;Astrom & Bernhardsson, 2002) for a simple system.In this key work, the authors derive analytical results for a simple single-order system, to compare the performance of eventbased and sampled-data control loops, where an event occurs when the measurement is out of range.In certain situations the event based strategy performs better in terms of minimising the output variance for similar mean sample times.In the event based strategy communications are increased in time intervals with large disturbances, and decreased when disturbances are small.
Another key early paper on intentional event based control considers an event based PID controller (Arzén, 1999).The event detection logic is kept simple i.e. an event occurs if the absolute value of the difference between the current value of the error e(t k ) and the value of the error the last time a control signal was calculated e(t s ) exceeds a limit, or if a set amount of time has elapsed since the last sample.It is shown through simulations that large reductions in resource utilisation, with only minor control performance degradation, are possible using an event based PID controller.The difficulty is guaranteeing stability and the lack of systems theory for event based control.
Event-based MPC algorithms have also been developed (Bernardini & Bemporad, 2012;Eqtami, Dimarogonas, & Kyriakopoulos, 2011;Henriksson, Quevedo, Peters, Sandberg, & Johansson, 2015).MPC is an optimal control strategy.At each sampling instant (k) a control strategy that minimises an objective function (cost) is computed over a given time horizon ( + k p).It is standard practice to use only the first com- puted control input and then resample the plant state and repeat the optimisation starting from this next state.In event triggered MPC, rather than just use the first computed control inputs, the control inputs until the next event are applied.The computed input sequence is applied to the system from time k until time instant + k i.At this time instant a new sample is taken and the optimisation is repeated from this state.Chakrabarty et al. (2017) apply event driven MPC to control blood glucose using insulin.Control actions in the optimal MPC input sequence are applied until the event condition is met.Event conditions are: the norm of the output estimation error exceeds a given threshold, or a given number of control actions in the optimal input sequence have been implemented, or safety conditions have been violated.This requires the system to be continually monitored and when the event condition is met, control is triggered (Heemels, Johansson, & Tabuada, 2012).The nature of the event is dependent on the system and method of data collection: for many applications if signals cannot be continually monitored and measured, an event may just be the arrival of new data.

Unintentional event driven control
Aperiodic sampling is not necessarily implemented through choice, instead it can be imposed due to the nature of the system being controlled.We term the case where the limitation is the information exchange and availability of data 'unintentional' event driven control.In our anticoagulation control example, the outputs are only measured when the patient attends an appointment, so it is not possible to continually monitor the system until an event defined by the signals occurs.Instead, we could treat the arrival of new information in itself as an event.This means that the system will not be able to react to increases in the error (e.g.due to set point changes, or disturbances).Decisions on changes to medications can take place only when the patient attends an appointment.
To illustrate, we modified the PI controller described in the simulation study in Section 4 into an event based PI controller using the approach described by Arzén (1999).In divergence from continually monitoring the error, as in Arzén (1999), the event condition is the arrival of data, e.g.Y obs (k) is available.The integral control gain is recalculated at each event by using the elapsed time since the last event to determine the sampling period, k I T s → hk I T s , where k I is the integral control gain, T s the sample period and set to one in simulations, and h is the number of elapsed samples since the last update.When data are missing the inputs are held.Control simulation results with 50% missing data are given in Table 6, Section 4. The method is not quite as accurate as imputing data, but requires approximately half as many updates, as decisions are only made when data are available.
Event based control shows promise for applications where data are not available at every periodic sampling instant.The key issues are the analysis of the control system and guarantees of worst-case performance.The absence of a systems theory for event driven controllers is a major reason why time driven control still dominates (Heemels et al.,Fig. 5. Block diagram of an event-based control loop (Lunze & Lehmann, 2010).Solid lines denote continuous signal transmission and the dashed lines event based signal transmission.The control input generator may also make use of an observer.

2008
).However, due to the huge increased interest in network control systems, the event driven control literature is rapidly expanding and staring to address these issues, and this includes steps towards developing an event-driven systems theory (Heemels et al., 2012;Heemels et al., 2008;Hetel et al., 2017;Lunze & Lehmann, 2010).
Network control systems provide an example of a system where aperiodic sampling and timing issues are imposed, and a vast amount of literature on aperiodic sampling is concerned with the NCS control problem.An overview of some of the main research themes is provided here, however, it is by no means exhaustive and it should be noted that several reviews of research within the area of NCSs exist (Gupta & Chow, 2010;Heemels & Van De Wouw, 2010;Hespanha, Naghshtabrizi, & Xu, 2007;Hetel et al., 2017;Liu et al., 2014;Tipsuwan & Chow, 2003;Yan, Yan, Zhang, & Zhao, 2014;Yang, 2006;Zhang et al., 2001).The main limitations of research into NCS are that most of the literature only considers linear systems, and most results do not consider plant uncertainty (Yang, 2006).
Unlike sampled data systems, there is no generalised framework for analysing and designing NCSs.Research into NCSs can be grouped according to the type of network-induced uncertainty (e.g.time varying sampling, time varying delay, packet loss), according to the approach used to model and analyse the system under these network induced uncertainties, and how the network induced uncertainty is modelled (Hespanha et al., 2007).Hetel et al. (2017) provide a thorough recent overview on the stability of systems with aperiodic sampling.
The input delay approach models the NCS as a continuous-time delayed differential equation (DDE).The stability is then studied using Lyaponov-Krasovskii methods (Hespanha et al., 2007), and linear matrix inequalities (LMIs).Mirkin (2007) showed that this approach is conservative as it does not take into account the piecewise constant nature of the control signal that occurs due to the zero-order hold.Alternative DDE approaches (Naghshtabrizi & Hespanha, 2005;Naghshtabrizi, Hespanha, & Teel, 2006;2008) have been proposed that are less conservative, and do take the piecewise constant nature of the control signal into account.In the discrete-time approach, discrete-time representations of the sampled data system are constructed based on the exact discretisation of the continuous-time plant over a sample interval.These models are then used in a robust stability analysis based on Lyaponov-functions (Cloosterman et al., 2010;Fujioka, 2009a;2009b) or LMIs (Cloosterman et al., 2006;Dritsas & Tzes, 2009;García-Rivera & Barreiro, 2007).In the constrained case, where the delay is smaller than the sampling period, lifted state vectors (Cloosterman et al., 2006;García-Rivera & Barreiro, 2007;Yamamoto, 1996), or Lyaponov-Krasonoskii functions (Xie & Wang, 2004) have been used to address the analysis and design problem.
Model based control of NCSs was introduced by Montestruque and Antsaklis (2004, 2002, 2003).It uses an explicit model of the plant to estimate the plant state between transmission times and provide approximate control signals when sensor data are not available.The model is updated using the true values of the plant state when available.The emulation approach considers continuous time controllers using a continuous-time (sampled-data) NCS model in the form of a hybrid system.This is used to quantify allowable levels (to ensure stability) of network uncertainties (maximum transmission intervals, maximum delay).The problem is considered as a linear-quadratic-Gaussian (LQG) problem by the stochastic optimal control approach, where the LQG matrix is based on the network delay statistics.This method generally assumes that time delays are less than the sampling period.

Discussion
Control theory is increasingly being applied to nonstandard applications.The use of control theory is appealing as it provides a systematic way of determining control inputs in the presence of noise and uncertainty.In such applications, data may be observational, sparse, missing or irregular, with no opportunity to collect repeat measurements.This poses the following challenges: i) system identification from observational data with missing measurements; ii) determining control inputs when output data are missing; and iii) ensuring stability when future update times are unknown.In this paper we reviewed methods in the literature that deal with these issues.The example of anticoagulation control using warfarin was used throughout to illustrate some of the challenges and potential solutions.
Identifying models from observational data is a fundamental issue within the statistics community.To analyse such data, the assumption of no unmeasured confounders is key, which requires that there is knowledge of all predictors of treatment decisions (for medical applications) that are also independent risk factors for the outcome of interest.A significant amount of literature exists on estimating model parameters from longitudinal, observational data with missing covariates; key to this are the assumptions that are made regarding the nature of the data and scheduling, or missing, mechanisms.Traditionally, data collection and analysis assumptions are rarely considered within the context of control.Generally, multiple measurements can be made on the system and the experimental protocol can be designed to enable 'ideal' data to be collected.Where data are missing, it is assumed that the data are either missing periodically, or MCAR.If control systems techniques are to be adopted for nonstandard applications, an awareness of the complexity of the data and any missingness mechanisms is required to avoid introducing bias.Here techniques from statistics could be used to ensure careful handling of sparse and noisy observational data.When estimated models are biased, robust control techniques provide a way of ensuring controllers behave well in the presence of modelling uncertainty, external disturbances and sensor noise.
There are two distinct approaches to handling missing data during control.The first of these is to impute missing data.Several methods for imputing missing data exist.If the imputation method is more reliant on a model of the system, then control performance is more reliant on the accuracy of this model.The second general approach is to use an event based control scheme.Using an 'unintentional' event based scheme would mean control inputs were only updated when data were available, resulting in a system that is unable to react to increases in error.Such a scheme still gives reasonable performance when applied to our anticoagulation simulation example, and required ≈ 50% fewer updates than the imputation method.Alternatively, if the system could be continually monitored, an 'intentional' event based control scheme could be used, where control updates are implemented when a signal, or state, deviates by a set amount.Event based methods lead to asychronous control, and the rapidly expanding control literature on NCS means a systems theory for asynchronous control is starting to develop.Such theory could usefully be applied to ensure stability when applying control to nonstandard applications.
The dominating control principal within biological systems is event based control.Event based control may, therefore, provide a natural solution for controlling the health of biological systems using medical interventions (as well as in the control of other nonstandard systems).Event-driven control requires the system to be continually monitored and when the event condition is met, control is triggered.The development of sensor technology means certain health variables, such as blood glucose using a continuous glucose monitor, can now be measured online.Where online monitoring is possible, event based control strategies may improve resource utilisation and reduce the number of dose (or treatment) changes necessary.However, in certain contexts, continual health monitoring may be infeasible or impractical.In this case the event triggering condition may just be the arrival of data.
If the system cannot be continually monitored, self driven control, an alternative asynchronous, proactive control strategy could be implemented.In self driven control the time of triggering the next control update is precomputed at the current control update time, based on predictions using the history of the data and knowledge of the plant dynamics (Heemels et al., 2012).A model of the plant dynamics is required, which relates back to the system identification problem.In general, self-triggering results in asynchronous sampling that is dependent on the predicted state of the system.The self driven control problem is closely related to event driven control in terms of ensuring stability given bounds on variations in the sample rate.In the context of medical treatment and ODT, self driven control would provide a method of scheduling future appointments (or measurements), but alone would not provide protection against missed visits.
In protecting against future missing measurements, within the control literature it is generally assumed that there are some bounds on the timing, and methods of designing control systems that are able to deal with given variations have been developed.This contrasts with the statistics approach, which considers that a visit may never occur again (Rosthøj et al., 2014).Again, these contrasting approaches are likely to be due to the nature of the system being controlled.Much of the work on missing and non-uniform sampling in control is within the context of NCS, where it may be reasonable to assume bounds on packet dropouts, or system delays.In the statistics literature, within the context of ODTs, patient scheduling is likely to be less predicable and so within this context it is harder to give upper bounds on the timing between visits.However, in certain contexts, assuming some bounds on the missingness (e.g.patients miss no more than two consecutive appointments, or patient must make at least one visit a month) may help to simplify the control problem and could help to develop stricter rules on the scheduling of the next appointments, or measurements, to ensure constraints are not violated.
An awareness of the context and assumptions imposed by applying control to non-traditional areas is essential.Many methods exist for dealing with missing, or non-uniformly sampled data and the suitability of these methods depends on the actual study being considered.For example, in the case where a patient's health is continually monitored (e.g.blood glucose), event-based control theory may be key for determining treatments.However, it is of no use if we are unable to monitor the event triggering condition.Similarly, an awareness of the limitations of the data is essential in order to understand the limitations and bias that can occur in the estimated models.

Fig. 1 .
Fig. 1.Anticoagulation of three patients.The three subplots have the same vertical scales but different horizontal scales.
of PI and MPC controllers designed using both an accurate system model (

Fig. 3 .
Fig. 3. Example of anticoagulation simulation.The blue lines represent dose in mg of warfarin and the red crosses INR.(For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 4 .
Fig. 4. Example control results.The desired INR is defined as the region between the dashed lines.

Table 1
Abbreviations used throughout document. .

Table 4
Effect of missingness on parameter estimates.Results of 20,000 simulations, each of sequence length 100.Approximately 50% of the observations are missing under each of MCAR, MAR and MNAR.

Table 5
Control performance averaged across 1000 trials for different control designs.

Table 6
Control performance when data are missing, averaged across 1000 trails for different control designs.