USING NONLINEAR MODEL PREDICTIVE CONTROL TO FIND OPTIMAL THERAPEUTIC STRATEGIES TO MODULATE INFLAMMATION

Modulation of the inflammatory response has become a key focal point in the treatment of critically ill patients. Much of the computational work in this emerging field has been carried out with the goal of unraveling the primary drivers, interconnections, and dynamics of systemic inflammation. To translate these theoretical efforts into clinical approaches, the proper biological targets and specific manipulations must be identified. In this work, we pursue this goal by implementing a nonlinear model predictive control (NMPC) algorithm in the context of a reduced computational model of the acute inflammatory response to severe infection. In our simulations, NMPC successfully identifies patient-specific therapeutic strategies, based on simulated observations of clinically accessible inflammatory mediators, which outperform standardized therapies, even when the latter are derived using a general optimization routine. These results imply that a combination of computational modeling and NMPC may be of practical use in suggesting novel immuno-modulatory strategies for the treatment of intensive care patients.


Introduction
Although the inflammatory response is crucial to restoring health following a wide range of biological stresses, uncontrolled systemic inflammation is also the primary cause of organ failure and death in victims of severe trauma, infections, and many other conditions leading to admission to an intensive care unit.Therefore, control of the inflammatory response has become a key focal point in the treatment of critically ill patients.Much of the theoretical work regarding severe inflammation has aimed to elucidate the mechanisms underlying systemic inflammation and to explain how the mediators of inflammation interact with one another across their respective time scales [1][2][3] .Significant insight has been acquired from these approaches, including an enhanced understanding of ways in which manipulations of inflammatory components may combat persistent inflammation.A main result coming from this research confirms that successful therapeutic interventions require appropriate timing relative to the evolution of the inflammatory response [4;5].
An early clinical approach to controlling inflammation was to target a sole inflammatory mediator [6].However, there is no one mediator that stands as the source for persistent inflammation [7][8][9].Instead, inflammation involves a cascade of processes, which is, perhaps, initiated by a few key factors but persists as a result of a complicated feedback process involving effectors that are produced later than the instigator.In addition, anti-inflammatory mediators may be present in elevated levels during prolonged inflammation, but their effect on pro-inflammatory mediators may be small or negligible due to the relative amounts of inflammation present in the system [10].Inversely, immunoparalysis, an overwhelmingly under responsive state resulting from a relative anti-inflammatory excess, may play a large role in predisposing patients to secondary infections that compromise organ recovery [11].Because the inflammatory response is such a complex process, involving multiple positive and negative feedback loops, it is extremely difficult to predict the response of the various mediators to perturbations (i.e. to therapy) made to one or more of the system's components.
As a result of this complexity, there is still much to be done to identify appropriate therapeutic targets to combat excessive and pervasive inflammation and to develop strategies for delivering appropriate interventions in the correct amounts and at the right times [12][13][14].One of the tools that can help optimize complex dose regimens is nonlinear model predictive control (NMPC).NMPC algorithms have mainly been developed and applied for industrial operations involving system processes that can be well described with mathematical models, usually systems of ordinary differential equations.NMPC is advantageous relative to other control algorithms because it combines predictions of the real system state at a future time, based on a mathematical model, with measurements derived from the system to calculate a control move that will help to optimize the desired outcome for a specific process variable.Recently, NMPC has been applied to a variety of biomedical processes including the regulation of glucose supply in diabetic patients and an exploration of optimal dosing of anticancer agents, among others [15][16][17].
Motivated by the recent success of NMPC in biomedical settings, we aimed to explore the utility of NMPC for the derivation of optimal therapeutic interventions for the control of inflammation triggered by a pathogenic infection, simulating the ominous clinical problem of severe sepsis.[18] There are two essential components of an NMPC scheme: the process to be predicted and the model predicting the process.In the clinical setting, in the case of a severe infection, the process to be predicted would be the time course of the inflammatory response in a patient.For this work, as an initially, fully controlled investigation of the feasibility of applying NMPC, we chose to emulate the patient's immune response with a model developed by Reynolds et al. (2006), such that the current exposition is completely simulation based.Initially, we used a predictive model and a patient model with identical equations, parameter values, and initial conditions, a scenario referred to as no mismatch.On each time step, the NMPC algorithm was applied to the predictive model to generate an intervention strategy, and this was directly applied to the patient model.Subsequently, we introduced patient-model mismatch in some parameter values and initial conditions, to achieve a more realistic representation of a clinical scenario.
Although the model developed by Reynolds et al. is more abstract than those used to simulate insulin or cancer chemotherapeutic agents in past biological applications of NMPC, its biological relevance is supported by previous analysis showing that it reproduces several important observations related to severe systemic inflammation in biological organisms.[19] Our implementation of NMPC in a reduced ODE model for inflammation will pave the road for future applications involving more quantitatively detailed models that are currently under development.[20] While this work is only based on in silico simulation studies, it is a significant effort toward bringing model-based immunomodulation strategies closer to the bedside of the critically ill.
The paper is organized as follows.In Section 2, we give a brief overview of the NMPC framework.In Section 3, we present the equation-based model of inflammation, along with additional methodological details, including the way in which we generate our simulated patient population and the specifics of the therapeutic strategies that we implement.The results of our simulations are presented in Section 4, while we conclude with a discussion in Section 5.

NMPC Overview
The NMPC algorithm must incorporate certain essential elements (Ogunnaike, 1994), which we now briefly describe.

I. The Specification of a Reference Trajectory
The reference trajectory defines a target path that we would like our process outputs to follow.Specifically, our model includes a variable representing tissue damage/dysfunction that we would like to cause to decrease to zero as quickly as possible.So, our reference trajectory for damage is the constant function, R D (t)ª0.

II. The Prediction of Process Output
The reference trajectory will be compared to a prediction of process behavior over a specified duration of time, h, called the prediction horizon.Simulation of the patient model (equations ( 1)-(4) in Section 3.1) describing the acute inflammatory response to pathogen yields this prediction.

III. The Definition of an Objective Function
The objective function quantifies the difference between the reference trajectory and the predicted process output.The objective function we use has the typical weighted sum of squares form.

IV. The Computation of a Sequence of Control Moves
Using the predictive model to simulate the system's response to changes in input over time h, we seek a sequence of control moves that will minimize the objective function.
The control moves are simply changes in the level of an input into the model, calculated in such a way as to achieve the desired goal.For example, in our model, control moves consist of steps in control terms in equations ( 2) and ( 4), which represent pro-and anti-inflammatory therapy, respectively.
The number of control moves, m, that are allowed during time h can also be specified.If m<h, which is typically the case, then the control terms are held constant after the m th control move when the system's response to the input over time h is determined.Only the first control move is actually implemented in the patient and predictive models as the dose for the current time step, after which the algorithm repeats, eventually determining the dose for the next time step.The MPC schematic in Figure 1, adapted from Ogunnaike et al., offers an excellent summary of this process [21].

V. Error Prediction Update
An important element of the algorithm lies in the error prediction step.After a control action is implemented in both the predictive and patient models, a measurement M(k) is taken from the patient model and compared to the corresponding quantity p(k) computed from the predictive model, where k is the present time step in the algorithm.
In a standard NMPC implementation, is minimized as a part of the objective function.In our NMPC scheme, when a mismatch exists, updating is done differently.This is due to the fact that in these instances, the variables that can be realistically measured are not those that we specifically wish to minimize, and hence are not appropriate for inclusion in the objective function.The strategies we implement for handling this situation are discussed in Section 3.2.

Designing an NMPC algorithm for the therapeutic control of inflammation
Having described the general components of NMPC schemes, we now present the acute inflammation model we use and the specific details of our customization of an NMPC algorithm.The algorithm we use is a modified version of that developed by Florian et al [16] (software acquired through personal communication).

The model for the acute inflammatory response to pathogenic infection
The model of acute inflammation was previously described [5;19] and consists of the ODE system: where R and f in (1) -( 4) are given by ( ) In the model, equation (1) represents the evolution of the bacterial pathogen population (P) that instigates the cascade of inflammation.Equation (2) governs the dynamics of the concentration of a collection of early pro-inflammatory mediators such as activated phagocytes and the pro-inflammatory cytokines they produce (N*).Equation (3) corresponds to a marker of tissue damage/dysfunction (D), which helps to verify response outcomes.Finally, equation (4) describes the evolution of the concentration of a collection of anti-inflammatory mediators (C A ) that inhibit many of the interactions within the system.Table I gives the parameter values used in Reynolds et al, which we refer to as the reference parameter set [5].
Equations ( 1) -(4), with AIDose=PIDose=0, admit three stable critical points (outcomes) under certain choices of parameter values [5]: 1. Healthy: (P,N*, D, C A ) = (0,0,0, Ĉ A ), for a small value Ĉ A > 0.  We label our finite time simulation outcomes based on these three states, with simulations that end with negligible P classified as healthy or aseptic depending on which state (N*, D, C A ) are approaching, as discussed further in subsection 3.5.
Figures 2 (a) and (b) show typical aseptic and septic scenarios, respectively.It is assumed that basic therapy, including the administration of antibiotics, resuscitation of fluids, and so forth, are implicitly modeled in system (1) -( 4).This means that the various outcomes mentioned above can occur despite administration of basic treatment.
Input to the NMPC algorithm consists of an anti-inflammatory therapy, present as a source term (+AIDose) in equation ( 4), and a pro-inflammatory therapy, incorporated as a source term (+PIDose) in equation (2).Constraints are defined that prevent dosing from going negative, meaning that therapy can be infused into the system but not extracted.
In all of the simulations that we discuss, the total simulation time is 168 hours (1 week).In addition, k is an hourly step, so doses are adjusted on an hourly basis.The goal of the NMPC control algorithm is to identify patient-specific therapy dosing profiles that can correct inflammatory responses that, without intervention, would result in either aseptic or septic scenarios.The therapy found by the control algorithm is referred to as targeted therapy.

The objective function, constraints, and error prediction under mismatch
The objective function J that we use contains terms to minimize damage levels (D), pathogen levels (P), and total therapy AIDose and PIDose given over the prediction horizon h and takes the form Minimization is done over piecewise constant time courses of AIDose and PIDose, achieved by a sequence of control moves, as discussed in Section 2. The Γparameters are the weighting constants and the zero function is used as the reference trajectory for both tissue damage/dysfunction (D) and pathogen (P).From computational experimentation, it became apparent that striving to minimize both damage and pathogen was essential.This, however, introduced the difficult challenge of maintaining a balance between these two objectives.An emphasis on minimizing damage might lead to unrestricted pathogen growth.On the other hand, an emphasis on minimizing pathogen might lead to an overzealous immune response determined to eliminate pathogen as soon as possible whatever the costs, after which it might be too late to control the excessive inflammation.For all simulations, we chose as not to emphasize one factor over the other.In addition, we also penalized the actual dose amounts, AIDose and PIDose, over the prediction horizon.The rationale for penalizing dosing is that drugs have financial costs as well as the potential for harmful side effects.The values for and AI Γ PI Γ are 1 if the corresponding form of therapy is used and 0 otherwise.
When patient-model mismatch is introduced, we need to specify which variables from the patient model will be measured to update the predictive model.We propose that it is fairly realistic to acquire accurate hourly measurements for N* and C A and hence these are designated as the variables to be measured.We assume that it is not feasible, however, to measure the variables D and P since damage is impossible to quantify in real time in clinical settings, and it is unlikely that a measurement of the pathogen population could be made at all, much less at every hour.Thus, our NMPC implementation is non-standard, in that the states measured from the process (patient) This updating scheme does not address possible discrepancies between the model and patient with respect to levels of damage and pathogen, both of which are the primary forces capable of pushing the system toward an unhealthy endpoint.Hence, when pathogen levels in the patient and model are vastly different due to differences between patient and model parameter values, an additional updating strategy not based on direct measurements is necessary.We added an update that kicks in when either (A) pathogen levels are low in the model but sufficiently high in the patient, or (B) pathogen levels are high in the model but sufficiently low in the patient.
The levels of pathogen are checked every four hours.If (A) occurs, then the patient's pathogen level for the next time step is reset to P=0.5, which is a relatively high value.If (B) occurs instead, then the model's pathogen levels are reset to zero.This can be looked at as a type of re-initialization of the pathogen value in the model.This use of the patient's pathogen values reflects the fact that in a clinical setting, a physician can potentially identify persistent infection based on high fever, positive blood cultures, toxic granulations in neutrophils, or rising procalcitonin levels.In other cases, a physician may be able to judge, based on the absence/resolution of symptoms, that it is highly unlikely that a patient has a high pathogen load.This rationale, together with the fact that we are not directly setting the model's pathogen state to the exact value present in the patient, implies that this updating strategy is a reasonable and clinically relevant way to alert the algorithm of significant discrepancies between the model and the patient, corresponding to either case (A) or (B) above.
Constraints on dosing levels must also be specified.This might be construed as an ad hoc regularization procedure, where one wishes to reduce the likelihood of large changes in therapy.The maximum dose amount of anti-inflammatory therapy allowed at a given step is calculated as the difference between the current level of C A (the antiinflammatory mediator) and a maximum allowable level of C A , given by C A,Max :=0.6264.[5] When pro-inflammatory therapy is used, the maximum dose amount allowed at a given step is calculated as the difference between the current level of N* and N* Max :=0.5.This maximum was selected as a value of N* that is high enough to consistently have an impact on pathogen but not so high that anti-inflammatory feedback cannot rein it in.

Building a patient population
Several of the reference parameters in equations ( 1) -( 4) and the initial conditions for P and C A , which we denote P 0 and C A0 , respectively, were allowed to vary to reflect patient variability (Table II).Virtual patients were generated with individualized parameter profiles as described below.In some preliminary simulations, in the absence of patient-model mismatch, the patient-specific parameter profile was also used in the underlying predictive model.When patient-model mismatch was introduced, the reference parameter set (Table I) was used for the predictive model and patient model parameters were chosen in the following way.For reference parameters with estimated values as given in Reynolds et al. [5] rather than ranges, we constrained them to lie within +/-25% of their respective reference value for all patients.Values for these parameters were drawn from a bounded uniform distribution.For the parameter k pg , an experimentally determined range is available.However, the range used here for k pg is a restricted subset of that given in Reynolds et al., selected such that placebo outcomes in the patient population are not driven primarily by k pg and P 0 , but by the parameter profile as a whole (Figure 3).
In addition, some parameters were chosen to co-vary since many molecules are produced by the same biological machinery or actuate their effects through similar pathways.That is, if parameters p 1 and p 2 co-vary and the value chosen for p 1 is +n% of its reference value, then the generated value for p 2 should also be close to +n% of its reference value.For example, we specified that the variability in k cnd , the rate of production of the anti-inflammatory mediator (C A ) by damaged tissue (D), was to vary by the same percentage as k cn , the rate of production of C A by activated phagocytes (N*), so that the rates at which these sources were producing C A were balanced.

Alternative Therapies
A significant advantage of in silico simulations is the ability to apply different therapies to the same virtual patient and compare the outcomes.For each patient receiving treatment, the algorithm generates a dynamic therapeutic profile specific to that patient's particular evolution, governed by the patient's unique parameter profile.
To gauge how well the targeted therapy achieves our objectives, we compare its performance to the results from the administration of three alternative therapies.
The simplest alternative therapy is Placebo Therapy, where no treatment is given.Static Therapy is designed to represent the therapy regimen currently given to critically ill patients with severe inflammatory disorders in the intensive care unit: a consistent dosing regimen of an anti-inflammatory therapy known as Activated Protein C. In practice, we implemented this therapy by creating a dosing profile that gives a small dose (0.005) of the anti-inflammatory therapy (via instantaneous injections, as usual) each hour over a period of 72 hours, after which therapy terminates.The control algorithm does not play a role in static therapy.Standard Therapy is generated by applying the control algorithm to the model ( 1)-( 4), with the fixed set of reference parameter values and the initial conditions (P 0 , N* 0 , D 0 , C A0 )=(0.5,0,0,0.125),as specified in Reynolds et al. (2006), to obtain dosing profiles for pro-and antiinflammatory therapy.This one set of dosing profiles is then administered to all patients.We simulated standard therapies with each of three different values for k pg (growth rate of the pathogen) in the underlying model: 0.52, 0.6, and 0.8.Targeted therapy in the mismatch case was also determined under each of these k pg values.The model's k pg value is a parameter that we are free to tune to get the best possible therapeutic outcomes.We initially chose k pg =0.52 since it is above the bifurcation point where healthy, aseptic, and septic states co-exist (k pg =0.514) and within the range of patient k pg values.A later choice of k pg =0.8, even though above the upper bound of the range for k pg patient values we used, allowed for a more aggressive therapeutic approach.Although results with this value were favorable for targeted therapy, the outcomes for standard therapy suffered.Thus, we subsequently explored the k pg value of 0.6, which is the upper bound of the patient k pg range.When we display our results for standard therapy and targeted therapy, we show outcomes for all three k pg values.
Figure 4 displays the three sets of dosing profiles that were calculated for the standard therapies explored.

Additional practical issues
In our simulations, we based the intervention time, or time of onset of therapy, on the level of N*, which denotes the early pro-inflammatory signals.During the course of an infection, if a virtual patient's N* level rises above a certain threshold, the patient is considered to show clinical manifestations warranting treatment.This implies a biomarker driven approach to initiating therapeutic intervention.This method, however, is not without its complications.For instance, how should the N* threshold be chosen?
In the current exploration, an N* threshold of 0.05 was selected, based on the finding that N* levels generally do not exceed this threshold in simulations yielding healthy outcomes.Of the 1000 virtual patients we generated and simulated, 620 exhibited values of N* that exceeded this threshold and thus received treatment.
As we worked through the process of customizing the algorithm, one of the main goals was to make the setup and assumptions more realistic.For instance, we noticed that the prescribed amount of anti-inflammatory therapy given to patients would sometimes cause the levels of the anti-inflammatory mediator to stay elevated for very long periods of time.This problem usually happened in scenarios when inflammation was very high after the eradication of pathogen.Consequently, to bring inflammation down, the anti-inflammatory therapy would be given continually, in an attempt to essentially saturate the system with as much C A as possible for as long as possible.
This situation would be avoided clinically by fear of secondary infections, and hence, we put a mechanism into place so that if the level of C A remained consistently elevated for more than 48 hours, the maximum allowable amount of C A was reduced by half.
It was also necessary to choose an algorithm for determining the outcome of an individual simulation once an entire therapy dosing profile was administered to a patient.
Initially, these results were based on the values of the variables at the end of the simulation time of 168 hours; however, sometimes the outcome was numerically ambiguous.For example, the tissue damage/dysfunction variable might be elevated beyond a threshold that we designate as a "healthy" level, but might also be decreasing toward equilibrium.In such a case, the system could potentially evolve to the healthy steady state, given more time, without the delivery of any additional therapy.Thus, for each patient, we took the values of the variables observed in the patient at 168 hours, corresponding to the end of the dosing regimen, and integrated the system with AIDose=PIDose=0 for an additional 300 (simulated) hours, by which time the solution invariably settled to a steady state.From such simulations, we systematically determined patient outcomes and tallied the results.A patient outcome was labeled septic if pathogen levels were above a threshold of 1.0 and damage and activated phagocytes were also above their designated thresholds of 1.0 and 0.05, respectively.
If pathogen levels were not above threshold, yet damage and activated phagocyte levels were, then the patient outcome was labeled aseptic, in accordance with the definitions of these physiologic states mentioned earlier.Otherwise, a patient was labeled healthy.(An additional check for inconclusive results was also made; however, there were no outcomes that fell into this category.)The individual outcomes (healthy, aseptic, and septic) from each simulation were totaled and reported as percentages of the total number of patients given treatment.
Finally, after experimenting with a number of values for the prediction horizon, h, and the move horizon, m, we chose h=12 hours, since this was long enough to capture essential model dynamics, and m=2, which provided moderately aggressive dosing.
Table III summarizes the setup details for the multi-patient simulations with and without the presence of patient-model mismatch.

Predictors of the response to a control intervention
Since it is of interest to identify initial conditions and model parameters predictive of the response to control in cases that would otherwise evolve to either a septic or aseptic outcome in the absence of therapy, we designed several classifiers to identify such predictors.Classifiers included logistic regression (LR), naïve Bayes, a support vector machine (linear kernel), a neural network, and a variety of decision trees.
Independent predictors included all initial conditions and model parameters that varied among individual cases (see section 3.3).The classification variable was restoration of health as a binary outcome.All models were run using 10-fold cross validation.LR performed consistently better than other classifiers as judged by receiver operating characteristic (ROC) performance and we report those results.

Results
The administration of standard therapy yielded better outcomes than the placebo case for each k pg considered, even though it was not individually tailored.Within standard therapies, only a k pg value of 0.52 did not harm patients, whereas values of 0.6 and 0.8 harmed approximately 4% and 14% of the patients, respectively.This means that without treatment those patients would have normally achieved healthy outcomes, but with treatment they did not.Although static therapy restored a healthy state in 30 (5.2%) patients that would not have reached this state with placebo, the overall outcome was not very different from placebo.This weak improvement resembles results attained in current clinical practice, unfortunately.[22] In addition, static therapy also harmed 5 (0.8%) of the patients receiving treatment.The outcomes of these therapies and those generated by the NMPC algorithm are shown in Tables IV and V.We next present results from the use of targeted therapy, first without patient model mismatch and second in the more realistic case when mismatch is present.

No Mismatch
In the absence of mismatch, the underlying model will predict patient dynamics perfectly, which is useful for algorithmic development and for determining the best possible performance of our control algorithm.Since the patient state and the model state are the same at every step in this case, no updating of model states is needed.In the first set of simulations under the no mismatch setup, we experimented with antiinflammatory therapy alone (Case 1).The results obtained demonstrate the need for an additional therapeutic option (Case 2) to correctly modulate the immune response, even in the absence of patient-model mismatch.

No Mismatch Case 1: Anti-inflammatory therapy only
In this initial no mismatch case, motivated by the clinical practice of targeting a single inflammatory mediator [23;24], we set the source term PIDose to zero for each time step and allowed only AIDose to take on nonzero values according to the output of the NMPC algorithm.This strategy achieved a healthy resolution in 99 patients that would have had unhealthy outcomes with placebo (Table IV).However, 9 patients that would have resolved to healthy in the absence of therapy were harmed, with 5 becoming septic and 4 aseptic under the targeted treatment regimen.In septic scenarios, the control algorithm deems the anti-inflammatory therapy irrelevant, since curbing inflammation will not help eliminate pathogen; thus, only aseptic patients could be helped under this treatment.Figure 5 shows the outcome of targeted therapy applied to the aseptic patient #26, where anti-inflammatory therapy was successfully able to redirect the response to a healthy outcome.On the other hand, Figure 6 shows results for another aseptic patient (#91) who was not helped with therapy, most likely due to the increased infection severity compared to patient #26. Figure 7 shows an unfortunate example of a therapy-driven septic outcome demonstrated with virtual patient #473.The five therapy-driven septic outcomes and four therapy-driven aseptic outcomes were due to the suppression of necessary inflammation by the antiinflammatory treatment early in the response.While therapy helped to minimize tissue damage, the pathogen gained the upper hand during the resulting immuno-suppressed state.Once the pathogen levels became sufficiently elevated, the algorithm discontinued the anti-inflammatory therapy and the inflammatory mediators were freed to respond fully to the infection.However, by this time, either the pathogen could not be reined in (septic outcomes) or the amount of inflammation needed to successfully eradicate the pathogen was excessive and could not be controlled even by continuing the anti-inflammatory treatment (aseptic outcomes).

No Mismatch Case 2: Anti-and pro-inflammatory therapy
Since anti-inflammatory therapy alone could not prevent septic outcomes and was also unsuccessful for many aseptic patients, we added a second, pro-inflammatory therapy, which could boost the immune response.Specifically, we allowed PIDose to take on nonzero values in equation ( 2) and reran the algorithm on each patient.The seventh column of Table IV summarizes the results for targeted therapy generated in the No Mismatch Case 2 with both anti-and pro-inflammatory therapy.
The use of two therapies did not harm any of the patients and 337 patients (~54%) were helped.Only 32 patients who had been aseptic in the placebo case remained aseptic with therapy and all 141 of the septic patients were rescued.Figure 9 shows a successful outcome for patient #2 under the dual therapy regime.In such successful interventions, initial pro-inflammatory dosing knocks out the pathogen, while subsequent anti-inflammatory dosing reins in the inflammation.If a clinician had full information about each patient, this would be the obvious therapy of choice, tailored specifically for each patient.Overall, 95% of the 620 patients receiving dual therapy resolved to healthy outcomes, making this approach much more favorable than using anti-inflammatory therapy alone as in the previous results.In addition, this strategy was superior to the standard and static therapies.However, the fact that no patient-model mismatch was included makes these results less impressive, since the presence of mismatch is inevitable and potentially severe in a real world setting.

Patient-Model Mismatch
We next considered the more clinically relevant scenario of patient-model mismatch, reflecting the reality that only highly incomplete observations of a patient's inflammatory response are available.Mismatch was introduced by using the reference parameters listed in Table I in the underlying predictive model, instead of patientspecific parameters.To correct for the mismatch introduced between patient and model, the algorithm had to rely on updates from hourly virtual patient measurements of N* and C A and indirect pathogen updates every four hours, as described in Section 3.
Only dual therapy was considered under the mismatch construct, permitting both AIDose and PIDose to take on positive values.
Table V displays the results for targeted therapy in the mismatch case alongside the results for placebo, static, and standard therapies, as in Table IV.The last three columns show targeted therapy results using model k pg values of 0.52, 0.6, and 0.8, respectively.The results show that under targeted therapy with the higher k pg values of 0.6 and 0.8, more than twice as many patients (~82%) achieved healthy outcomes as in the placebo case.In addition, targeted therapy with these higher k pg values helped more septic patients, with only 10% resulting in the septic outcome with k pg =0.6 and no septic outcomes with k pg =0.8.With a model k pg value of 0.52, however, targeted results were nearly identical to standard therapy results with the same k pg value (third column, Table V).Under targeted therapy with k pg =0.6 or 0.52, further analysis shows that no patients were harmed by treatment; however, under a model k pg value of 0.8, even though the overall results were better, 16 patients (~2.6%) were harmed.

Comparing intervention strategies and predicting controllability
Although the targeted therapies generated by the NMPC algorithm were individualized, the resulting dosing profiles nonetheless exhibit the same general features (fig.9).Not surprisingly, the control strategy first yields pathogen destruction by enhancing pro-inflammation, then modulates anti-inflammation to mitigate excessive inflammation and restore health.This pattern of two slightly offset large dosage peaks is preserved even when the control strategy is unsuccessful.In unsuccessful cases, one also notices a later surge of both pro-and anti-inflammatory therapy, a feature absent in successful control strategies.Moreover, in aseptic cases, the anti-inflammatory therapy saturates at our preset maximum level.Indeed, this anti-inflammatory cut-off can explain the lack of controllability of several aseptic cases, where it is plausible that allowing a more incisive anti-inflammatory therapy could thwart excessive proinflammatory activity.The cumulative dose of pro-inflammatory therapy varied by 31.6% across outcomes, with higher mean values for cases yielding septic outcomes, while the cumulative anti-inflammatory dose varied by 142%, with cases in which aseptic death occurred in spite of treatment receiving the highest doses.
To identify initial conditions and parameter values predictive of controllability, an LR classifier identified P 0 and k pg as the strongest predictors of non-controllability for cases that would proceed to both septic and aseptic outcomes in the absence of intervention.Although P 0 and k pg did not determine a patient's outcome in the absence of therapy (Figure 3), they nonetheless were dominant factors in determining responsiveness to therapy.The LR classifier had an ROC of 0.951 and misclassification rates of 7.9%.When we omitted P 0 and k pg from the classifiers, C A0 and the parameters k nn and k cn were also discriminatory of outcome, but the ROC dropped to 0.676, with 20% misclassification.

Discussion
We have shown that incorporation of an NMPC algorithm into a model of the acute inflammatory response to pathogenic infection allows for the derivation of therapeutic interventions that can produce healthy resolutions for patients that would have otherwise faced septic or aseptic outcomes.If we supply the model with complete information about each patient, with parameters selected from anywhere within a physiological range, then the NMPC routine almost always produces a healthy outcome, as long as both pro-and anti-inflammatory dosing are included.If we consider a more realistic scenario in which only clinically available patient information is supplied to the predictive model, then, although the results are less perfect than in the no-mismatch case, the NMPC algorithm again significantly enhances the likelihood of healthy outcomes across the patient population.The success rate of the algorithm depends on the pathogen growth rate, k pg , used in the predictive model, with use of a high growth rate leading to a therapy with a strong pro-inflammatory component that prevents septic outcomes, and use of a more moderate growth rate minimizing the rate of aseptic outcomes.
Therapeutic profiles appear very similar across outcomes.Differences in dynamic dosing are subtle indeed, while outcomes themselves are strikingly influenced by treatment.This finding argues strongly for individually targetted therapy, especially given the modest effects observed from constant, untitrated anti-inflammatory therapy and the diverse outcomes obtained from even a standardized dynamic dosing profile.
Although clinicians practice targetted patient care in modern intensive care units to support organ systems and combat infections, existing guidelines as to how to treat patients with potentially lethal infections are generic, with very limited room for individualized titration [12].In particular, targeted immunomodulation remains an elusive goal [13;25].Even when appropriate drugs are available, there is no unifying concept as to how these should be used and combined to improve patient outcome.Model-based immunomodulation of complex inflammatory diseases thus represents an extraordinary opportunity for very significant advances in the care of the critically ill.Indeed, the idea of using a control-based algorithm to generate appropriate therapeutic regimens makes particular sense in light of the fact that the acute inflammatory response features multiple interconnected nonlinear feedback loops that would vastly complicate the design of successful therapies under clinical conditions.
In this work, we implement control algorithms in a four-equation reduced ordinary differential equation model for the acute inflammatory response [5].Reynolds et al.
briefly explored the possibility of modulating the outcome of a simulated infection, in the context of this model, by altering the anti-inflammatory mediator levels at a particular time point.It was found that most of the alterations perturbed the system from an otherwise healthy resolution to an unfavorable state.The NMPC algorithm that we implemented yields a much more refined approach to the design of therapeutic strategies.In particular, our results illustrate the need for multimodal therapeutic strategies to successfully modulate a complex inflammatory response.
Our approach has several limitations.Our representation of the inflammatory response is extremely simplified and our lumped model variables and parameters cannot be mapped directly to quantities accessible in clinical or laboratory settings.It would be advantageous to incorporate the algorithm into a more detailed model that can give quantitative predictions about specific mediators of the acute inflammatory response (e.g.[20]).In addition, the pharmacokinetics of existing or potential therapies could be incorporated to generate specific predictions with respect to particular dosing profiles.The addition of noise to the measurements would also be helpful in producing more realistic simulations.The customizations of our NMPC algorithm and features of the controller that we used were based on pragmatic clinical considerations, yet some decisions were admittedly arbitrary.For example, although we limited the intensity of anti-inflammatory therapy to cap the risk of secondary infections associated with states of elevated anti-inflammation, it appears clear that allowing more intensive antiinflammatory therapy would rescue more cases otherwise destined for aseptic death.
Ideally, such limitations on therapy would be implemented much more precisely, based on clinical or experimental data.Another issue for future consideration is the construction of an optimal objective function, which is nontrivial, even in this very simplified context.We experimented with a wide combination of weights and penalty terms, arriving at our objective function empirically.We believe that, for clinical applications, biological input will be essential in constructing suitable objective functions, and that there could be a good argument made for the inclusion of time-varying weights in the objective functions used.
The translation of biomarker-targeted methods, such as we have implemented in this work, to clinical practice will require empirically validated predictions.In a realistic experimental protocol, a mathematical model that forecasts several immune mediators fairly well from a mouse/rat model of endotoxin challenge would be used as the underlying model, predicting the immune response of each animal.A small set of initial observations of individual animals subjected to an infectious challenge would be taken to generate an ensemble of candidate models for each subject.The NMPC algorithm, applied to the predictive ensemble of models, would suggest a therapeutic intervention that would be implemented in each animal, after which measurements of various analytes would be taken to update the ensemble (adaptive control).After repeated iterations of this procedure, comparisons of outcomes in animals receiving such targeted therapy would be made to those observed in animals either receiving a standardized therapy or no therapy at all.
In conclusion, as a result of this study and in light of other biomedical applications of feedback control, we maintain that the utilization of feedback control methods to assist in treating the critically ill is a strategy worth further exploration.In practice, clinicians currently implement a sort of dosing strategy algorithm, in which they iteratively determine the next treatment step based on all of the information available to them.To utilize their knowledge and expertise most effectively, it is logical to propose the incorporation of model-based control algorithms as a tool to guide them in this process of selecting patient-specific treatment strategies.Figure 3 Scatterplot showing the distribution of patient outcomes with respect to pathogen growth rate (k pg ) and the initial level of pathogen (P 0 ).For the given ranges of k pg and P 0 the various outcomes (healthy, aseptic, and septic) of the patient profiles in the placebo case are well mixed in the k pg -P 0 plane.Thus, k pg and P 0 are not the primary drivers of patient outcome in the absence of treatment.

Figure 4
Three different standard therapy dosing profiles, with pro-and anti-inflammatory dosing schedules.Standard therapy was calculated using a model k pg value of 0.52 (top panel), 0.6 (middle panel), or 0.8 (bottom panel).The same pair of doses, both pro-and anti-inflammatory, was administered to all patients receiving treatment.

Figure 5
The NMPC algorithm is applied to an aseptic virtual patient (#26), using antiinflammatory therapy alone and no patient-model mismatch.Solid curves: Targeted therapy; Dashed curves: Placebo.An anti-inflammatory therapy regimen generated by the algorithm successfully redirects an otherwise aseptic response to the healthy state.

Figure 6
The NMPC algorithm is applied to another aseptic virtual patient (#91), using antiinflammatory therapy alone and no patient-model mismatch.Solid curves: Targeted therapy; Dashed curves: Placebo.Unlike the results shown in Figure 5, antiinflammatory therapy fails to redirect this aseptic response to the healthy state, due to the higher levels of pathogen growth seen in this virtual patient.Figure 7 The NMPC algorithm is applied to a virtual patient (#473) who would have survived without treatment.Anti-inflammatory treatment alone is used and there is no patientmodel mismatch.Solid curves: Targeted therapy; Dashed curves: Placebo.The use of anti-inflammatory therapy suppresses the immune system in an attempt to minimize damage; however, this prevents a successful response, causing the pathogen to grow such that the immune system is unable to control it.By the time the immune system begins to respond, it is too late, showing that the therapy can have adverse effects.
The NMPC algorithm is applied to patient #2, who was septic under placebo, with both pro-and anti-inflammatory components and with no patient-model mismatch.Solid curves: Targeted therapy; Dashed curves: Placebo.The therapy found by the algorithm successfully changes the outcome from septic to healthy.This figure is representative of the No Mismatch Case 2 results for all virtual patients who were septic under placebo.

Figure 9
Mean pro-inflammatory and anti-inflammatory dosing profiles.The doses for successful (left panels) and unsuccessful (right panels) control strategies are similar, with an early pro-inflammatory peak that helps control pathogen (bottom row) and a later antiinflammatory peak that reins in inflammation (top row).The grey bands indicate the 2.5%-97.5% dosing range.Unsuccessful strategies feature late surges in both antiinflammatory and pro-inflammatory components.The profiles for aseptic outcomes are qualitatively similar to those for septic outcomes (data not shown).Patient initial conditions (P 0 , N*, D, C A ) = (P 0 -random, 0, 0, C A0 -random) (See Table II for ranges for P 0 -random and C A0random)

Tables
The following are applicable in the mismatch simulations only: Model initial conditions (P 0 , N*, D, C A ) = (0.5, 0, 0, 0.125) Pathogen Update Every 4 hours k pg value in underlying model various explored: 0.52, 0.6, or 0.8
the variables appearing in the objective function are disjoint sets.To harness the measurements of N* and C A taken from the patient model at the end of the each time step, we use these values as our initial conditions for N* and C A in the predictive model at the start of the next one-hour time step, with AIDose and PIDose set to the values prescribed by the NMPC algorithm and with P and D in the predictive model evolving continuously across time steps.

Figure 2 (
Figure 2 (a) -(b) Representative placebo simulations of two virtual patients.(a) Simulation of a typical aseptic outcome in the absence of treatment.(b) Simulation of a typical septic outcome in the absence of treatment.
production of anti-inflammatory mediator (C A ) k nd 0.015 -0.025 Activation of phagocytes (N*) by tissue damage (D) k np 0.075 -0.125 (Co-varies w/ k nd ) Activation of phagocytes (N*) by pathogen (P) k cnd 36.0 -60.0 (Co-varies w/ k cn ) Controls the effectiveness of activated phagocytes (N*) versus damage (D) in the production of the anti-inflammatory mediator (C A ) k nn 0.0075 -0.0125 (Co-varies w/ k nd ) Activation of phagocytes (N*) by already activated phagocytes (N*) (or the cytokines that they produce)

Figure
Figure 3

Table I
Model parameter reference values for the system (1) -(4).Table IIModel parameters in which variability was assumed in the patient-model mismatch case.Patient parameters were generated by choosing a random value from a uniform distribution oneach given range

Table III
Summary for NPMC Simulation Setup: