Building confidence in quantitative systems pharmacology models: An engineer's guide to exploring the rationale in model design and development

This tutorial promotes good practice for exploring the rationale of systems pharmacology models. A safety systems engineering inspired notation approach provides much needed rigor and transparency in development and application of models for therapeutic discovery and design of intervention strategies. Structured arguments over a model's development, underpinning biological knowledge, and analyses of model behaviors are constructed to determine the confidence that a model is fit for the purpose for which it will be applied.

For the full design, implementation, and analysis detail that underlies this model we re-139 fer the reader to the models accompanying publication and supporting materials [3]. To In this section we outline a process using structured argumentation that assists the record-161 ing of justifications and rationale for both the biological detail and engineering processes 162 that underlie the development of a computational model. The process and associated 163 tools to support that process take inspiration from the field of safety-critical systems, 164 where it must be demonstrated that a software system is as safe as reasonably practicable 165 [17]. Acceptable safety can be established and presented using arguments over evidence. 166 For increased accessibility and ease of communication, Goal Structuring Notation (GSN) 167 [30, 2] was developed as a visual notation for the presentation of arguments detailing to allow an alteration of focus from safety cases to providing a rationale for fitness for 177 purpose. The argument is presented as a tree of connected argument components, of 178 specific shapes (Figure 1). The semantics are detailed in Figure S3. These components 179 start from a top-level claim (a GSN goal). At the beginning of the process a set of 180 fitness-for-purpose requirements (referred to as goals or claims, that the argument seeks 181 to substantiate) should be established, with an accompanying set of strategies that can be 182 used to assess whether the requirement has been met. The strategies typically break down 183 goals into sub-goals, and eventually link to evidence supporting the claim, alongside the 184 source of the evidence where appropriate. If a requirement cannot be fully supported by areas of biological study that have been overlooked or require further laboratory work.

189
The process of constructing a claim using the semantics in Artoo is described in Figure   190 S5.  evidence that is acceptable (as defined by the creator of the argument structure). In the 205 case of arguing fitness for purpose, the claim can be left as undeveloped, that is the claim 206 can remain in the argument structure, but highlights a clear gap in the evidence base, 207 thus providing informative transparency of the lack of evidence to support the claim.

208
Such a modification is vital in QSP modelling applications, where expert opinion and 209 assumptions have to be used to mitigate the fact that the understanding of the biological 210 system may be incomplete.

211
Taking the description in Figure S4 as a template of how to develop a claim, we turn 212 attention to developing claims that encompass all stages encountered in model develop-213 ment. In Figure S5 we have split the process into seven distinct phases, all of which, 214 we believe, greatly benefit from the adoption of a structured argumentation approach in 215 revealing the rationale employed at that stage. To exemplify creation of argumentation 216 at each phase, we now go through each in turn, providing case study examples in the 217 context of leishmaniasis.

218
Step 1 -Define Purpose of the Model 219 As can be seen in Figure S5, understanding and defining the intended purpose of a model 220 is a key part of the process, as the rationale for the other key phases of model development  Figure S3, this top level claim is usually associated with context nodes that define the 229 key terms used to specify that purpose. From here, strategies are then set that will be 230 used to argue that the top level claim is met: that the tool is fit for its specified purpose. purpose. Attached to this claim are six strategies that will be used to support the claim.

237
It is hopefully easily noticed that these six claims correspond to the six rounded rectan-  ical evidence is being used should be produced: without a specification of the data used or 247 any assumptions employed, it is difficult for researchers using model-derived predictions 248 to relate this prediction to their own experimental study.
Step 2 of our process supported 249 by argumentation is used to assess (i) the scope of any supplied biological data; (ii) the   identified, well informed, justified, assumptions will need to be introduced into the model. 280 It is critical that the justification for any such assumptions are documented alongside the 281 predictions generated by the model, as their introduction may have an influence on the 282 validity of that prediction. If, for example, the purpose of the model is to produce pre-283 dictions that inform laboratory research, it is vital that confidence in the assumptions are 284 a fair reflection of the experimental system on which they will be testing this prediction: 285 key when financial and technical resources have to be considered within a study.

286
In Figure 4 we expand on two examples from the cytokines that were being considered 287 in Step 2. We demonstrate two common simplifying assumptions. For IL-1, the proposed 288 mechanism of action on parasite load is killing of parasites indirectly via heat shock. It 289 can be argued that heat shock is neither necessary nor sufficient for parasite clearance, 290 as evidenced by the lack of impact of IL-1 receptor blockade on acquired resistance or 291 granuloma formation [33]. Considering the purpose of the model, it is reasonable to assume 292 that IL-1 can be excluded despite the fact that there is some evidence that it could impact 293 parasite load. This exclusion of IL-1 is one type of simplifying assumption. Figure 6 294 also shows a partially developed argument for merging IFN and TNF which ends in the 295 undeveloped claim that they perform the same function and can be merged into a single  on what is observed [14]. Therefore, it is important that the rationale for the choice of 308 modelling system be exposed. As an example, a claim could be made that an agent-based 309 modelling paradigm is most suitable for addressing the question of concern. Strategies 310 would then be employed to determine whether this is indeed the case, or whether other 311 approaches such as Ordinary Differential Equation (ODE) modelling would be more ap-312 propriate. By using argumentation at this stage, the developer has a record of the im-313 plementation decisions that were taken, with a fully evidenced justification of why these 314 decisions were taken.  Figure 2, the strategy is to argue the appropriateness of the adopted approach, in this 318 case stochastic Petri nets. From here, our claim is that the adopted paradigm provides 319 the means to represent the required aspects of the biological system. To support this 320 claim, one would be required to compare the available approaches, and as such the stated 321 strategies involve examining implementing the model as a Petri net, agent-based model, 322 or ODEs. For the scope of this tutorial, Figure 5 expands on the Petri net suitability 323 claim, arguing that we can capture the required stochasticity, capture granuloma hetero-324 geneity, handle small integer number calculation, and produce an implementation that is 325 computationally tractable. In this case we are able to evidence all four claims, suggesting 326 we have a suitable approach for capturing the key aspects specified in the claim.   The first, supported by collaborators opinion, would suggest that these cells could po-357 tentially be abstracted out of the model altogether, as they do not influence the models 358 purpose. However this is contradicted by the second, which makes the claim that these 359 cells contribute to the cytokine environment of interest. As such, we argue that these are 360 required, but can be abstracted to a single proxy cell type that expresses the cytokines 361 identified in Figure 3. Step 6 -Engineering the Implementation 363 When going through this process alongside the development of a simulation, the developer 364 will now have justified the modelling approaches they are going to use (step 4) and the 365 abstractions they will make in implementation (step 5). The next step is to implement   Figure 7, the strategy to argue that the Leishmaniasis simulation was ade-380 quately tested has been to ensure adequate structural coverage of the code by tests. In 381 this case, as is typically the case in high integrity software engineering, this strategy is 382 split into three phases: requirements testing [36]; unit testing [23]; and manual review. 383 Requirements testing ensures that the system has a collection of requirements describ-384 ing the tasks that the system should perform, and it ensures that each requirement has an 385 associated test (or collection of tests) that demonstrates the system fulfilling the require-   Even using unit tests, it may not be possible to achieve full code coverage for some 398 types of code. For example: robustness checks, system libraries, or code that only executes 399 when running the system in a different mode. For these cases, the code is reviewed 400 manually to either determine that it will not execute in the situations we are providing, 401 or to argue why it does not need to be tested (for example, a commonly used system 402 library). Given the criticality of models we consider adequate testing to mean achieving 403 90% statement coverage and 90% branch coverage through requirements tests and unit 404 tests, with the remaining code reviewed manually.   However, for computational modelling studies to achieve that potential, it is critical that 506 the relationship between the model and the biological system being captured is fully 507 understood. Any researcher would need to have a high level of confidence in a model-508 derived prediction before seeking to invest time, expertise, and financial resources into 509 investigating that prediction further in the real system. imply that the biological system has been captured appropriately [5].

517
In this tutorial we have detailed a process through which the rationale underlying 518 the design, implementation, and analysis of a model of a biological system is generated. 519 We see this process being applied either within a process of model construction or as 520 a tool through which an assessment of a previously developed model can be performed.  convince researchers that a model is appropriately constructed and analysed to meet its 547 intended purpose. 548 We believe that arguing over the rationale for each of the model development phases          An Assumption node provides a means of specifying any information that is assumed to be true when arguing over a claim or designing a strategy to examine a particular claim. Explicitly stating the inherent assumptions eases the process by which others can assess the extent to which the argument over a particular claim holds.
J Justification A Justification node should contain the reasoning for the application of a particular strategy in order to substantiate a claim. Justifying the approach used explicitly can reveal the extent to which alternative strategies have been considered, and why this strategy was selected over those alternatives.

Unsubstantiated Claim
Unlike the application of structured notation in formal safetycase arguments, a claim can be shown to be unsubstantiated in the approach described in this paper. Biological systems are not fully understood, and it may not be possible to generate evidence to substantiate the complete set of fitness for purpose requirements. Where this is the case, it is critical that the lack of evidence is explicitly stated in the argument, and the limitations of the model are shown. In our approach, a lack of substantiating evidence is shown by attaching a white diamond to that claim As the argument becomes more complex, it may become difficult to follow. As such we have introduced a black diamond notation, representing the continuation of the argument surrounding this claim on a different diagram Node containing the Evidence that is used to substantiate an attached claim. In Artoo, it is possible to hyperlink to this evidence, which could include publications, experimental results, statistical analyses, etc.

Claim Continued
Evidence Definition Notation Description

In Context Of Supported By
Connected To Figure S3: Semantics of diagram language used in Artoo Identify a claim that the argument seeks to support. This is represented within a rectangle. The objective is to detail how this claim can be supported with available evidence, if possible.
Each claim is usually examined within a given context. This may involve defining the meaning of key terms stated in the claim. For example, if the purpose of the model was to adequately capture a biological system, adequate must be defined. Any context definitions are given in rounded rectangles A B C D E

Box 1: Developing a Claim
Each claim is accompanied by one or more strategies that will be used to determine if that claim can be supported. This could, as examples, be a particular experimental strategy or systematic literature review. A strategy is always stated in a parallelogram.
A claim or strategy can also be accompanied by a justification or assumption node to provide more detail on the choice of the claim being made or the strategy that was followed. Semantics in Supplementary Figure 1 Strategies, unless leading to evidence (see part E), are then broken down into sub-claims, and the process repeated. In this case, the strategy is divided into four sub-claims, each of which examines a key part of the model development process.
If evidence can be provided that supports a claim, this is stated in an evidence node: a circle. In Artoo, electronic links to this evidence can be provided. Diamonds on the diagram indicate either: (i) a claim cannot be supported. If no evidence can be provided, a white diamond can be used to show this is the case.
(ii) the argument is detailed on another diagram. A black diamond is used to show the claim is fully described elsewhere. Figure S4: Process of developing a specific claim, using the diagrammatic notation used in Artoo.  Figure S5: Process through which assessing the rationale for model design, implementation, and analysis should be conducted. Each stage of the process is grounded in the purpose for which the model was developed. Arrows linking to Purpose are bidirectional as the purpose shapes what assumptions and abstractions are appropriate, and conversely, decisions about assumptions and abstractions that are made can de facto alter the purpose for which the model is fit. Note the lack of defined end point: arguing fitness for purpose has potential to inform later iterations of model and study development.