Real-time dynamic modelling for the design of a cluster-randomized phase 3 Ebola vaccine trial in Sierra Leone

Background: Declining incidence and spatial heterogeneity complicated the design of phase 3 Ebola vac- cine trials during the tail of the 2013–16 Ebola virus disease (EVD) epidemic in West Africa. Mathematical models can provide forecasts of expected incidence through time and can account for both vaccine efﬁ- cacy in participants and effectiveness in populations. Determining expected disease incidence was critical to calculating power and determining trial sample size. Methods: In real-time, we ﬁtted, forecasted, and simulated a proposed phase 3 cluster-randomized vaccine trial for a prime-boost EVD vaccine in three candidate regions in Sierra Leone. The aim was to fore- cast trial feasibility in these areas through time and guide study design planning. Results: EVD incidence was highly variable during the epidemic, especially in the declining phase. Delays in trial start date were expected to greatly reduce the ability to discern an effect, particularly as a trial with an effective vaccine would cause the epidemic to go extinct more quickly in the vaccine arm. Real-time updates of the model allowed decision-makers to determine how trial feasibility changed with time. Conclusions: This analysis was useful for vaccine trial planning because we simulated effectiveness as well as efﬁcacy, which is possible with a dynamic transmission model. It contributed to decisions on choice of trial location and feasibility of the trial. Transmission models should be utilised as early as possible in the design process to provide mechanistic estimates of expected incidence, with which decisions about sample size, location, timing, and feasibility can be determined. (cid:1) 2016 The Authors. Published by Elsevier Ltd. ThisisanopenaccessarticleundertheCCBYlicense(http:// creativecommons.org/licenses/by/4.0/).


Introduction
West Africa experienced the largest outbreak of Ebola virus disease (EVD) to date during 2013-16. This epidemic resulted in more than 25,000 cases and 10,000 deaths. As the epidemic unfolded in 2014, development of candidate vaccines was accelerated, including evaluation in phase 1-2 studies and phase 3 planning. However, the rapidly changing incidence both geographically and in time posed major challenges to the design and planning of phase 3 trials. Typical study design calculations do not allow for varying infection rates within and between communities over time, which is especially problematic during the tail of an epidemic, when few cases occur. Computer simulations employing empirical statistical models can mitigate some of these concerns however they require accurate assumptions on incidence, heterogeneity and, in addition, do not capture the mechanism of an outbreak. Moreover, an effective vaccine used widely in a given area (as would be the case in large-scale, population-based vaccine trials) could in itself further reduce the incidence.
Dynamic models of EVD transmission were developed during the epidemic to understand the patterns of spread of the virus and predict the course of the outbreak [1][2][3][4]. If these models are appropriately parameterised and updated, then they can be used to predict incidence and how it may change in space and time [5]. In addition, dynamic models can account for both the direct and indirect effect of vaccine-induced immunity and its impact on the transmission dynamics. That is, they can be used to assess the extent that the trial itself may affect the transmission dynamics.
Collaboration between the Centre for Mathematical Modelling of Infectious Disease (CMMID) at the London School of Hygiene & Tropical Medicine and Janssen Research & Development (Janssen R&D) was established to rapidly extend a mathematical model of EVD [3] to simulate a cluster-randomized phase 3 vaccine trial in Sierra Leone. The dynamic model and trial simulations were updated in real-time to match the latest incidence data available. This collaboration thus enabled a real-time, dynamic assessment of the feasibility of a potential phase 3 trial, which ultimately was implemented as a safety and immunogenicity study: EBOVAC-Salone (NCT02509494). This paper describes how the model was used to inform the planning of the trial as well as the decision-making to abandon the effectiveness part of the protocol.

Collaboration
Collaboration was initiated between CMMID and Janssen R&D in February 2015. Janssen R&D was seeking a partner to guide study design and feasibility planning of a phase 3 effectiveness trial for their heterologous prime-boost vaccine regimen (Ad26.ZEBOV as prime and MVA-BN Ò -Filo 28 days later as boost), for which phase 1 trials were on-going.
CMMID had previously developed mathematical models of EVD transmission to assess the potential for large outbreaks [6], impact of community care centres on the evolving epidemic [7], and bed capacity in Sierra Leone [3]. In addition, CMMID members liaised with WHO on the design and analysis of the WHO EVD vaccine trial [8].
Collaboration offered a unique opportunity to explore the use of a dynamic transmission model to evaluate study feasibility. In this paper, we present the model-based incidence projections and trial simulations from 15th February 2015, similar to those sent from LSHTM to the team at Janssen on a weekly basis from February 2015 to May 2015. These were in turn employed by the clinical study team to evaluate and guide power calculations, study design as well as trial feasibility. To illustrate the impact of the evolving epidemic, an update of the projections and simulations at the end of April is provided as supplementary materials.

Vaccine trial design
A large-scale cluster-randomized phase 3 trial was designed to evaluate the effectiveness of prime-boost vaccine regimen against laboratory-confirmed EVD in an outbreak setting in Sierra Leone [9]. Sierra Leone is administratively divided into districts, districts into chiefdoms, and chiefdoms into sections. A trial cluster would be a section. With vaccine availability at time of study design of up to four hundred thousand doses of both prime and boost vaccine, approximately 160 clusters of 5000 participants (800,000 in total) were to be assigned in a 1:1 ratio to immediate vaccination versus no vaccination (control), whereby vaccination would be offered to the control group after effectiveness was established.
Initially, feasibility, statistical power, and type I error of the trial were evaluated using simulations which assumed constant incidence through time [9]. Control incidence assumptions of 3, 5, 10, 20 and 40 EVD cases per arm per month (400,000 personmonths) were evaluated, with allowance for heterogeneity between clusters based on CMMID projections and simulations. However, the rapidly changing epidemic dynamics in early 2015 meant that these static predictions were unlikely to capture the epidemiological picture.

Transmission model for trial
The transmission model extended a previously published model for transmission of EVD [3]. It was a stochastic compartmental model, where the population was divided into classes ( Fig. 1): Sus-ceptible (S), Exposed (E), Infectious not yet notified (I), Infectious and notified (J) and Removed (R, for recovered and immune, or dead). The infectious compartment was split in two subcompartments I and J in order to account for a delay of (on average) 4.8 days to notification of new cases [4]. The model was extended to mimic the trial design closely, but modelling cluster-level randomization was not possible because there was insufficient data available at this spatial scale for fitting. It is often difficult to predict the tails of epidemics, which are characterized by small, local outbreaks, and stochastic variation. Instead, we assumed a 1:1 randomization at the district level and treated the clusters as independent units.
Susceptible people were assumed to be recruited to the trial for the length of the accrual time, T r , by entering either the vaccine (V S ) or control (C 1 ) arms. An average of 2 weeks after receiving the prime, vaccinated participants entered the compartment, V P , where they were assumed to have a reduced risk of infection, r p . On receipt of the boost vaccine, they were assumed to enter V B , and immediately gain the target vaccine efficacy, r b ( Fig. 1). Control participants were assumed to proceed from C 1 to C 2 at the same rate as V S to V P to maintain comparability. Parameters that govern rates of transition are given in Table 1.
To account for external influences on transmission -such as variation in human behavior and introduction of control measures -we assumed that the transmission rate could change over time; the extent and direction of change was estimated during the model fitting process [3]. Hypothetical vaccine efficacy values were defined in February 2015 for the power calculation of the effectiveness trial. These values were conservative estimates, chosen to ensure that the planned trial would have sufficient power in the event of unpredicted changes in incidence, and to decrease the risk of the study. These hypothetical assumptions are only working hypotheses and do not necessarily reflect the potential effect of this candidate vaccine, and these hypothetical values need to be assessed in the future.

Incidence data
The model was fitted to weekly confirmed and probable EVD incidence data from three districts in Sierra Leone (Kambia, Port Loko, and Western Area) that had on-going epidemics in February 2015 and were therefore candidate areas for a potential vaccine trial. Data were drawn from the WHO and Sierra Leone situation reports and ran from 25th May 2014 until the date of fitting and forecast [10,11]. We used Bayesian methods to fit the model to the data, namely particle Markov Chain Monte Carlo, which allows parameter estimation in a stochastic framework.

Forecasting
We sampled the reproduction number (R t ) 5000 times at the last fitted data point, and forecasted the epidemic until extinction under the assumption that the reproduction number did not change from that time. We retained only forecasts that went extinct by 1st January 2016 because all regions showed waning epidemics, and although persistence for a further year was possible, it was deemed unlikely (Fig. S2). Sampled reproduction numbers therefore usually lie below 1 (Fig. S1). Updated estimates of the reproduction number distribution made in April 2015 have very little density above 1, which suggests this was a reasonable assumption.
The forecasted persistence probability at each point of time t was defined as the probability that at least one infectious individual remains in the arm at that time, and was computed empirically by summing over the N forecast trajectories that went extinct by 1st January 2016: with i the index of the sample from the posterior.
We evaluated the impact of trial start date by comparing the simulated number of cases in the vaccine or control arms. We used a one-sided Wilcoxon signed rank test to test for the pairwise difference in the total number of cases between the vaccine and control arm. In addition, we assessed the effect of vaccine efficacy on trial success. The persistence probability is calculated as the proportion of 5000 epidemic simulations that are non-extinct at each time point.

Model fits and projections
Using data as of 15th February 2015, we fitted the model to weekly confirmed and probable EVD cases (Fig. 2). At that time, the epidemic was in the tail phase, which is clear from the rapidly decreasing persistence probability. Visually, incidence data that has since been observed show excellent agreement with the forecasted epidemics (Fig. 2), where 65%, 59%, and 65% of weekly values lie within the 50% credible interval (CI) in Kambia, Port Loko, and Western Area respectively. 92%, 92%, and 94% of points lie within the 95% CI. Short-term model projections were contrasted with the static model assumptions from the power analysis of [9] (Fig. 2). In [9] the control incidence of 5 per 400,000 person-months was identified as a threshold of sufficient statistical power to initiate the trial. The model indicates that the incidence could drop below this threshold between June and August for all three districts, though the estimates are subject to great uncertainty as can be seen from the wide 95% credible intervals. Already in April-May, the incidence was possibly too low to initiate the trial.
We now describe the results of the vaccine trial simulations. The baseline scenario presented in this paper, unless otherwise specified, is that the trial began on 1st May 2015 using a conservative working hypothesis of 50% reduction in susceptibility following prime vaccine, rising to hypothetical 80% after the boost vaccine. The populations of Kambia (population 340,000) and Port Loko (population 558,000) were smaller than the target of 400,000 participants in each arm, so we show simulations for those districts with each arm of the trial encompassing both -170,000 in Kambia, and 230,000 in Port Loko. Western Area had a large enough population that the trial could be conducted solely in that district.

Effect of the start date of the trial
For forecasts made on 15th February 2015 with trial start dates on 1st May, 1st June and 1st July 2015, there were fewer cases in both arms when the start date was later (Fig. 3), due to the continued decline of the epidemic. The later the start date, the lower the probability that the epidemic was still on-going by the start of the trial. Although the trial begins on the first of each month, delays involved in the recruitment of participants, onset of protective immunity, and time to boost vaccination mean that cases occurring in the population are not necessarily trial endpoints, and therefore do not accrue in the cumulative cases shown in Fig. 3. While the difference between vaccine and control arms could have been large by December 2015, the probability of the epidemic persisting until December was very low. We found the overall number of cases was larger if the trial was in Kambia and Port Loko combined, compared to Western Area solely.
The occurrence of a vaccine trial in a candidate region would affect the persistence probability of the epidemic in that region if the vaccine were efficacious. In a declining epidemic, this would cause the epidemic to go extinct faster, reducing the persistence Fig. 1. Model diagram. Susceptible people were recruited to the trial by entering either the vaccine (V S ) or control (C 1 ) arms. Two weeks after receiving the prime, vaccinated participants developed protective immunity (r p ), and entered the compartment V P . On receipt of the boost vaccine, they enter V B , and immediately gain the target vaccine efficacy (r b ). Control participants proceed from C 1 to C 2 at the same rate as V S to V P to maintain comparability. Transitions have Erlangdistributed waiting periods with shape equal to 2, apart from S to E, I to J, and J to R, which are exponentially distributed as well as S to V S or C 1 , which are step-wise processes. probability. This can be seen from Fig. 3, as starting the trial earlier increased the probability of earlier elimination in the vaccine arms. Starting the trial later would result in a reduced probability of detecting a difference in the number of cases between the two arms and an increased probability of having no cases in either arm (Fig. 4). In some simulations more cases would be observed in the control group in comparison to the vaccinated group (a ''negative" effect). Fig. 4 indicates that over 5000 replicates of the trial, the distribution of the total number of cases is always significantly different between the vaccine and control arms at all starting time (p < 0.05, Wilcoxon signed rank test). However, when considering the proportion of trials with a positive or negative effect (i.e. below or above the diagonal in Fig 4), these proportions decrease and become more similar as the trial start later, with the proportion of trials with no cases increasing in the same time. While a split in favour of the vaccinated group would be expected under the assumed prime/boost effect, the simulations indicate that for a trial starting on 1st May, a negative (versus positive) effect would be observed in respectively 18% vs. 55%, 12% vs. 32% and 11% vs. 38% of simulations in Kambia, Port Loko and Western Area. For a trial starting 1st July 2015 the difference became smaller with 13% vs 15%, 4% vs 5% and 3% vs 4% of simulations for the three districts, respectively. The probability of observing no cases in either arm increased when the trial started later due to increasing stochastic extinctions (supplementary information).

Effect of vaccine efficacy
In simulations of this large trial, which is started in the declining phase of the epidemic, any effective vaccine causes a decrease in persistence probability of the epidemic (Fig. 5). Higher hypothetical vaccine efficacy leads to quicker extinction of the epidemic, although the differences are very small. The simulations gave very comparable expected number of cases in the vaccine arm between the hypothetical vaccine efficacy values of 60%, 80% and 90%, which is shown by the grey, red, and yellow boxplots (Fig. 5). This finding was consistent for Kambia, Port Loko and Western Area.

Real-time updates
The epidemic progressed from February to May 2015, and we updated the model fits and projections in real time. Here we display updated results as of 26th April 2015 (Fig. 6). The persistence probability of the epidemic in the potential study areas changed as incidence in those areas decreased and the model was fitted to more available data. There was a sharp decrease in persistence probability for Port Loko, due to the decline in the epidemic occurring there. In contrast, for Kambia and Western Area, the projections from February changed very little by April.

Discussion
This close collaboration between CMMID and Janssen R&D in forecasting and planning a phase 3 Ebola vaccine trial had many key benefits: firstly, the production of up-to-date epidemic projections gave better situational awareness to the clinical study team and key decision makers at Janssen. These forecasts were based on fitting a mechanistic transmission model to the current epidemiological data, thereby providing rigorous and realistic predictions. Secondly, the mechanistic model provided a means by which to assess the feasibility of the phase 3 effectiveness trial, and how this changed through time. This is critical to trial planning, determining whether to proceed, and to understanding the effect of logistical delays or constraints on feasibility. And thirdly, by fitting mechanistic models to potential study regions individually, the forecasts provided a better understanding of the variability between candidate sites and the impact that a trial might have had on the epidemic. This allowed study team and decision makers to assess the relative probability of trial success based on geographically specific information.
Other trials planned for Ebola vaccines in various parts of West Africa faced challenges to feasibility as a result of the declining incidence [5,12]. In this study, modelling was used to help gauge the feasibility of the cluster-randomized design, by forecasting incidence in potential regions, which was then used in power calculations [9]. The dynamic transmission model could account for both vaccine efficacy in those vaccinated as well as vaccine effectiveness in the population. Trial simulations indicated how many cases to expect in the vaccine and control arms for various trial locations, start dates and hypotheses of vaccine efficacy and how this changed over time. The trial simulations thus guided decisions of trial location and feasibility. For example, the simulations indicated that Kambia was more likely to have sustained transmission compared to Western Area and Port Loko. Further, the rapid decrease of the persistence probability over time urged the vaccine development team at Janssen to explore alternative trial designs, and, partly as a result of this work, it was decided to abandon the effectiveness trial protocol as planned. Instead, a safety and immunogenicity study was initiated in Kambia in October 2015.
The trial was intended to start during the declining phase of the outbreak and the assumption on potential start dates reflects realistic assumptions about operational timing. Additional work could explore the feasibility of the trial starting at earlier points of the outbreak. Importantly, this work was performed using conservative estimates of vaccine efficacy after prime and boost vaccination, which were used to calculate power of the effectiveness trial. The effect of different assumptions of potential effect of prime and boost vaccinations could be reassessed, also taking into account durability of protection as suggested by recent immunogenicity data in humans highlighting a strong and sustained immune response [13]. Future work could also integrate formal testing of vaccine efficacy [9] in the dynamic transmission modelling framework, in a similar way as for an individually randomized trial [5].
Although the model was able to accurately forecast the incidence by district, a key limitation was that we could not model cluster-level randomization due to insufficient data at this scale. Also, although the model could suggest which was likely to be the most favourable setting for the trial from an epidemiological point of view, in reality, there may be logistical constraints such as local customs, staff availability, manufacturing capacity and storage, trials running in parallel, and other factors that would affect trial feasibility. The scenarios that we explored were considered realistic at the start of the collaboration.
The model structure used here does not explicitly include different transmission mechanisms such as during unsafe burials [14,15]. Instead, we used a flexible, stochastic, transmission rate to capture the combined effect of these different transmission components. The precise contribution of different factors was likely to have changed over the course of the epidemic. For example, by November 2014, trained burial teams and a safe burial command centre were established in Kambia and Port Loko (having been previously established in Western Area) [16], so the risk of Fig. 4. Distribution of total cases observed in each arm of the trial, stratified by start date, for the baseline scenario. Note that the color scale (which indicates the number of simulations) is logarithmic. In simulations above the diagonal, more cases occurred in the vaccine arm. The p-values are from a one-sided Wilcoxon signed rank test for fewer cases in the vaccine arm. There is one simulated point not shown, where the number of cases in the control arm is 100, and in the vaccine arm is 93, which occurred in Port Loko. transmission due to unsafe practices was likely decreased by these interventions. However, the model was able to capture overall patterns of disease transmissions that occur as a result of changes in transmission routes.
While the use of mechanistic transmission models in evaluating vaccination programs is well established, their use in trial design, planning, and analysis, is a relatively new and growing area of research [17]. Designing interventions to reduce influenza transmission gives different preferred trial designs whether the goal is achieving power or taking account of economic constraints [18,19]. Modelling has been used to propose new trials for HIV antiviral treatment in serodiscordant couples [20], and has been used specifically for vaccine trials for malaria [21,22], intestinal helminths [23], wildlife vaccines [24], and nasopharyngeal bacteria [25]. For Ebola vaccine trials, a semi-mechanistic model developed during the epidemic addressed the feasibility of a proposed phase 3 trial in high risk individuals [12].
Our collaboration represents a novel example of close collaboration between modellers and trial planners to guide the design of a phase 3 trial during an epidemic. We delivered up-to-theminute projections for both the epidemic and trial feasibility from academic researchers to industry partners. This type of information is critical to trial planning and clinical development, and mathematical models of disease transmission should be integrated into trial design at the earliest possible stage.

Funding source
The authors acknowledge funding from the Innovative Medicines Initiative 2 (IMI2) Joint Undertaking, under grant agreement  115854 (RME, WJE), the Research for Health in Humanitarian Crises (R2HC) Programme, managed by Research for Humanitarian Assistance (Grant 13165) (AC, SF, AJK), and the Norwegian Institute of Public Health (CHW, WJE). IMI2 receives support from the European Union's Horizon 2020 research and innovation programme and the European Federation of Pharmaceutical Industries and Associations (EFPIA). The funders had no influence on the content of this analysis or decision to publish.

Author contribution
AC, SF, WJE, AJK developed the model. AC programmed the model and ran all simulations. AC, RME made the figures. AC, RME, NG, AV, RM, SF, AJK, CHW, TV, WJE analysed the outputs and interpreted the model. RME, AC, NG wrote the first draft of the paper. All authors contributed to the draft and have approved the final version.

Conflict of interest
AC, WJE and CHW have acted as unpaid advisors to the WHO on Ebola vaccination and report travel and accommodation paid for by the WHO to attend meetings. WJE is a co-investigator on, and RME is funded by, the European Commission Innovative Medicines Initiative-funded EBOVAC trial of the Johnson & Johnson primeboost Ebola vaccine candidate. WJE's partner is an epidemiologist at GlaxoSmithKline, in a role unrelated to the company's development of an Ebola vaccine. AC and CHW have acted as unpaid advisors to the EBOVAC trial, for which CHW reports travel and accommodation paid for by the EBOVAC consortium to attend a meeting. NG, AV and TV are employees of Janssen R&D. RM was employee of Janssen R&D during the time this work was done and is now employee of Merck.