Pipeline analysis of a vaccine candidate portfolio for diseases of poverty using the Portfolio-To-Impact modelling tool

Background: The Portfolio-To-Impact (P2I) P2I model is a recently developed product portfolio tool that enables users to estimate the funding needs to move a portfolio of candidate health products, such as vaccines and drugs, along the product development path from late stage preclinical to phase III clinical trials, as well as potential product launches over time. In this study we describe the use of this tool for analysing the vaccine portfolio of the European Vaccine Initiative (EVI). This portfolio includes vaccine candidates for various diseases of poverty and emerging infectious diseases at different stages of development. Methods: Portfolio analyses were conducted using the existing assumptions integrated in the P2I tool, as well as modified assumptions for costs, cycle times, and probabilities of success based on EVI’s own internal data related to vaccine development. Results: According to the P2I tool, the total estimated cost to move the 18 candidates currently in the EVI portfolio along the pipeline to launch would be about US $470 million, and there would be 0.69 expected launches across all six diseases in EVI’s portfolio combined during the period 2019-2031. Running of the model using EVI-internal parameters resulted in a significant increase in the expected product launches. Conclusions: Not all the assumptions underlying the P2I tool could be tested in our study due to limited amount of data available. Nevertheless, we expect that the accelerated clinical testing of vaccines (and drugs) based on the use of controlled human infection models that are increasingly available, as well as the accelerated approval by regulatory authorities that exists for example for serious conditions, will speed up product development and result in significant cost reduction. Project findings as well as potential future modifications of the P2I tool are discussed with the aim to improve the underlying methodology of the P2I model.


Introduction
The Special Programme for Research and Training in Tropical Diseases (TDR) recently developed a new modelling tool, called Portfolio-to-Impact (P2I), that allows users to model the impact of different research portfolios 1 . The P2I tool can be used to estimate the costs of moving a portfolio of candidate products for poverty-related and neglected diseases (PRNDs) through the pipeline and the likely launches that would result. It can also help to identify potential funding bottlenecks and operational challenges. The modelling tool, which is deterministic, uses Excel based software to estimate the "minimum funding needs to accelerate health product development from late stage preclinical study to phase III clinical trials" and to "visualize potential product launches over time 1 ." As a financial forecasting tool that estimates the funding needs for pharmaceutical product development, the tool and its outputs are of value to funders of product development, product development partnerships, and other stakeholders involved in research and development (R&D) policy. Terry and colleagues, who developed the P2I model, note that "its real utility lies in its predictive value for modelling the impact of different funding strategies at the portfolio level" 1 .
To the best of our knowledge, the first published use of the tool was by Young and colleagues 2 . These researchers first conducted a pipeline portfolio review to identify current candidates in the pipeline for 35 PRNDs. They then used the P2I tool to estimate (a) the costs to move these candidates through the pipeline, (b) the likely launches, and (c) the highly needed products that would still be "missing" at the end. As of August 31, 2017, they found 685 PRND product candidates, of which 538 candidates met inclusion criteria for input into the model. Their modelling estimated that it would cost about US$ 16.3 billion (range $13.4-19.8B) to move these candidates along the pipeline, resulting in about 128 (89-160) expected product launches. The study found that "there would be few launches of complex new chemical entities; launches of highly efficacious HIV, tuberculosis, or malaria vaccines would be unlikely 2 ." The European Vaccine Initiative (EVI), established in 1998 as the European Malaria Vaccine Initiative (EMVI), is a not-for-profit organization that supports the development of effective, affordable and accessible vaccines against diseases of poverty and emerging infectious diseases. To achieve this goal, EVI supports translational vaccine research and development (R&D) spanning from preclinical development through to the establishment of a clinical proof of concept. To date, EVI has supported the development of about 40 vaccine candidates through to earlyand mid-stages of clinical development. Initially focussing only on malaria vaccines, in 2009 in the context of a strategic revision EVI broadened its scope and since has built a vaccine portfolio that addresses critical challenges and opportunities in vaccine R&D for a variety of diseases of poverty and emerging infectious diseases.
Currently EVI´s vaccine portfolio comprises around 20 vaccine candidates at different stages of development between late preclinical and mid-stage clinical development. In order to estimate future financing needs required for delivering the EVI portfolio and the potential public health impact of product launches, we conducted an analysis of EVI´s vaccine candidate portfolio using the P2I tool.
Together with similar pipeline portfolio reviews using the P2I tool that are currently being conducted by other product development partnerships (PDPs), the results will inform product developers as well as funders and policy makers regarding future funding needs. The results may also guide future investment priorities to maximise the chances of developing products for diseases that are missing urgently needed health products.
We used the P2I tool because, to the best of our knowledge, it is the first publicly available comprehensive portfolio model that includes data on cost, success rate, and cycle time per phase for various product types based on data from a very large number of previous product development candidates (over 25,000) 1 . The P2I tool is thus complementary to other available tools that can help guide prioritization in vaccine development, such as the multi-stakeholder Vaccine Innovation Prioritization Strategy 1 and Total Systems Effectiveness Framework 2 .

Methods
In this section, we begin by summarizing how the Microsoft Excel-based P2I tool was developed, which phases of product development are included in the tool, and which costs are excluded. After this summary, we then describe the four key steps in our analysis of EVI's vaccine portfolio, which are summarized in Figure 1.

Amendments from Version 1
Unclear wording highlighted by the referees in certain parts of the manuscript has been addressed by revising the corresponding sections. The same was done for some paragraphs that had been misunderstood by the referees, the language in these sections has been revised and additional wording been included. Additional information that was requested regarding EVI, including the institutional vaccine portfolio management process and the potential value for portfolio management and decision making of the results from the portfolio analysis using the P2I tool, has been included. The revised manuscript includes additional wording that provides further information on the development and underlying methodology of the P2I tool, including its potential limitations. This also includes further details and information regarding the definition of certain terms used for the development of the P2I tool. Moreover, figures considered to be redundant by the referees have been taken out of the manuscript.

Figure 1. Four key phases in the analysis.
A summary of how the P2I tool was developed, and which phases and costs are included A detailed research paper describing the development of the first version of the P2I tool itself (P2I version 1, or P2I v1) has been previously published 1 . The development of a second version of the P2I tool, called P2I version 2 (P2I v.2), has also been previously described 2 . As summarized below, the model is based on assumptions for costs per phase, attrition rates (probability of success) per phase, and cycle times per phase for four development phases (preclinical to phase III, see Figure 2) for a number of different kinds of medical products, called archetypes.
The P2I v.1 model has 11 archetypes: simple or complex vaccines; simple, innovative, or complex new chemical entities (NCEs); simple or complex repurposed drugs; simple or complex diagnostics; simple or complex biologics; and two diagnostic archetypes, assay development or simple technical platform development. Descriptions and examples of each are described elsewhere 1 . For each of these 11 different archetypes, the model has in-built assumptions on costs, attrition rates, and cycle time per phase.
One valuable feature of the model is that it is highly adaptable, users can input additional archetypes into the Excel tool. In the P2I v.2 model, additional archetypes include "unprecedented vaccines" (for candidate vaccines for HIV, TB, and malaria to, which are "considered as unprecedented as current platforms have not led to suitable vaccines" 2 ) and vector control products.
As described elsewhere 1 , assumptions on development costs at each phase of product development for the 11 archetypes included in the P2I.v1 model were initially based on an analysis of clinical trial costs from Parexel's R&D cost sourcebook. The assumptions on attrition rates and cycle times at each phase were initially based on the historical attrition rates and cycle times of more than 25,000 development candidates. All of these assumptions were further refined and validated based on (i) academic literature, (ii) industry publications and databases, and (iii) 133 stakeholder interviews with a wide variety of product development partnerships (PDPs), pharmaceutical companies, and major funders of global health R&D. As described elsewhere 2 , additional sources of assumptions for the new archetypes in P2I.v2 were the McKinsey Risk-Adjusted Portfolio (RAP) Model and clinical trial data shared by the Bill & Melinda Gates Foundation.
As described below, the three archetypes of relevance for our analysis of the EVI portfolio were simple, complex, and unprecedented vaccines (reference 2 describes these three different archetypes in more detail). After classifying each EVI candidate into its archetype and phase, we then ran the model. There are two main model outputs. The first is "launches"; in this paper, the term launch refers to a candidate making it through phase III and thus being ready for the next steps, e.g. the regulatory and manufacturing steps. The second is the total costs to move all candidates through the pipeline from their current phase from now to 2031 (the model also gives a breakdown of these total costs into annual costs by year, from 2019 to 2031).
The model includes only advanced preclinical to phase III, and thus the cost estimates are an under-estimate of the full costs of product development. In particular, the model excludes all costs related to basic research through lead optimization; chemistry, manufacturing, and controls; good manufacturing practice; manufacturing build up and scale-up costs; regulatory or registration fees (post-phase III); and all post-market commitments (e.g., phase IV pharmacovigilance studies).
Step 1: review of EVI's portfolio and classification of vaccine candidates into archetypes The first step was to review, organize, and classify the vaccine candidates to allow them to be entered into the P2I model. To enter candidates into the model, we needed to include (a) the target disease, (b) the current phase of development (the model assumes that the candidate is at the start of that phase), and (c) the archetype. Table 1 summarizes the vaccine candidates included in the analysis. Since all product archetypes were vaccines, we used the archetype classification from the P2I v.2 model (Table 2), in which candidates are classified as simple, complex, or unprecedented.
Step 2: modelling of costs to move candidates through pipeline and likely launches, using P2I v.2 model with its existing assumptions Once EVI's portfolio of candidates was classified into archetypes (see Table 1), we then entered them into the P2I v.2 model and ran the model. In this first run of the model, we used exactly the same assumptions on cycle time, cost, and attrition rate per phase as in the P2I.v2 model 2 . These assumptions are shown in Table 3. The assumptions were derived from three sources: the P2I model (shown in orange in Table 3), the McKinsey Risk-Adjusted Portfolio (RAP) Model (shown in yellow), and the Bill & Melinda Gates Foundation (shown in blue) 2 . The model outputs were the estimated costs to move the current candidates through the pipeline and the estimated number of launches.
Step 3: modelling of costs to move candidates through pipeline and likely launches, using the P2I v.2 model with modified assumptions As a third step, we did a second run of the model after making selected modifications to some of the assumptions used in P2I v.2.
3 See: https://www.who.int/tdr/publications/r_d_report/en/ EVI has collected data on its own parameters for cycle times, attrition rates, and costs per phase, which are shown in Table 4. We wanted to assess what effect the use of EVI's parameters would have upon the outputs of the P2I model, but we recognized that many of these parameters were based on only two or three data points and were thus unreliable. For the second run of the model, we therefore made a pragmatic decision to use only those EVI parameters that were based on 10 data points, i.e. EVI's parameters on success rate in phase 1 and the duration of phase 1. Thus, in the second run of the model, we made two modifications: • Instead of using a success rate of 50% for phase 1 for unprecedented vaccines (the assumption in P2I v.2, as shown in Table 3), we used EVI's success rate of 70% (see Table 4).
• Instead of using a phase length of 2 years for phase 1 for unprecedented vaccines (the assumption in P2I v.2, as shown in Table 3), we used EVI's phase length of 1.45 years or 17.4 months (see Table 4).
For the EVI parameters that were included in the second run of the model, we used the following definitions: • Success rate: EVI uses two different measures: (i) technical success: a clinical trial is considered successful if it concluded without being terminated prematurely for whatever reason, and (ii) phase transition success: a clinical trial is considered successful if after completion of the trial a decision is made to move to a subsequent clinical trial (even if no funding is available to do so; successful in this sense means also if you conduct, Any vaccine platform that requires a novel approach that has not been successfully approved for use in the past. Conferral of immunity against disease-causing microorganism is difficult to induce and maintain and natural immunity is not protective against reinfection. Platform may elicit incomplete/insufficient immunity and require boosting over time.

Unprecedented vaccine
Was not included as an archetype in P2I v.1

HIV, TB, malaria vaccines
All vaccine candidates for HIV, TB, and malaria are classified as "unprecedented" due to much higher attrition rates at phases II and III than other complex vaccines for example, another phase I clinical trial of the same antigen with a different formulation, dose, or age group).
• Clinical trial duration: EVI excludes clinical trial preparation, including dossier preparation and waiting for approval, in the duration. The duration is the period between the start and completion times. The start time is the time point when the clinical study opened for recruitment of participants, or the actual date on which the first participant was enrolled. The completion time is the time point when the final participant was examined or received an intervention for the purposes of final collection of data for the primary outcome.
There were no candidates in phase III. Among the 18 candidates, 15 were classified as unprecedented vaccines, 2 as simple vaccines, and one as a complex vaccine.
As described in the Methods section above, we first ran the model using the assumptions included in the P2I v.2 model. The results of this first run are described in section (ii) below. We then re-ran the model with some modifications to the assumptions that were made by EVI, based on EVI's own historical data; section (iii) describes the results of this second run. Finally, we conducted a sensitivity analysis around the results of both runs of the model; the results of these sensitivity analyses are in section (iv).
(ii) First run of the P2I financing modelling tool, using assumptions from P2I v.2 Estimated costs to move candidates through the pipeline. The total estimated cost to move the 18 candidates for six diseases along the pipeline to launch would be about US $470 million (Table 5). Just over one third (35%) of the expected costs would be incurred by development of the 8  Step 4: sensitivity analysis As a final step, we conducted a sensitivity analysis, using an approach developed by Mestre-Ferrandiz et al. at the United Kingdom Office of Health Economics 3 . We examined the impact of changing all probabilities of success per phase to 10% higher and 10% lower, and all costs per phase to 10% higher and lower. We also examined the impact of all possible combinations of these changes (e.g., 10% higher probability of success per phase and a 10% higher cost per phase, 10% higher probability of success per phase and a 10% lower cost per phase, etc.). We conducted this sensitivity analysis for both runs of the model.

Expected launches (expressed as launch probabilities).
Overall, for all 18 candidates under development, the P2I model estimates that there would be 0.69 expected launches across all six diseases combined, as shown in Table 5 (we have left all results unrounded). Table 5 summarizes the expected launches by disease based on the current candidates for six diseases in EVI's portfolio (the "expected launches" throughout this paper are expressed as launch probabilities, where 1.0 is 100% probability of a launch). Table 6 summarize the estimated launches for all six diseases combined (as launch probabilities) by year from 2019-2031 alongside cost estimates by year to move candidates for all six diseases through the pipeline from their current phase.
(iii) Second run of the P2I financing modelling tool, with modifications of selected assumptions The re-run (second run) of the model using the modified assumptions increased the projected portfolio costs by US $46 million, bringing the total cost estimate to US$ 517 million to move all 18 candidates through the pipeline. The changes in estimated cost are driven by the increase in expected costs for the unprecedented vaccine candidates, i.e., the 8 malaria vaccine candidates, 5 placental malaria vaccine candidates and 2 leishmaniasis vaccine candidates. The remaining candidates for Nipah virus, Zika virus, and shigellosis were not affected by the change in parameters.
With regards to the expected number of launches, in the re-run of the model, the launch probability increased for malaria, placental malaria, and leishmaniasis vaccines: • For malaria, the estimate of expected launches increased from 0.098 to 0.11 • For placental malaria, the estimate of expected launches increased from 0.05 to 0.07 • For leishmaniasis, the estimate of expected launches increased from 0.024 to 0.03. Table 7 provides a comparative summary of the costs and launch probabilities for each run of the model.
(iv) Sensitivity analyses for both runs of the model We conducted a sensitivity analysis for the first and second runs of the model.

Sensitivity analysis for first run of the model.
In the sensitivity analysis for the first run of the model (which used assumptions from P2I v.2), we found that the total estimated costs to move all candidates in EVI's portfolio for all six diseases through the pipeline from their current phase ranged from US$ 417.08 million to US$ 528.11 million ( Table 8). The combined launch probability for launching candidates across all six disease types ranged from 0.51 to 0.91.

Sensitivity analysis for second run of the model.
In the second sensitivity analysis for the second run of the model (which used modified assumptions), we found that the total estimated cost to move all candidates in EVI's portfolio for all six diseases through the pipeline from their current phase ranged from US$ 482.48 million to US$ 581.9 million ( Table 9). The combined launch probability for launching candidates across all six disease types ranged from 0.53 to 0.96.

Conclusions and discussion
The mission of EVI is to accelerate the development of vaccines for diseases of poverty and emerging infectious diseases. Compared to other PDPs with a narrower focus, for example on a single disease, EVI has a broader scope and consequently a more heterogeneous portfolio, currently comprising 18 active vaccine candidates covering five different diseases/pathogens (malaria, including placental malaria, leishmaniasis, shigella/ ETEC, Nipah and Zika viruses). An analysis of EVI´s current vaccine portfolio, providing estimations for future vaccine development costs and expected product launches, was considered important to inform future decision-making and priority setting at EVI, as well as to provide valuable information to global health funders and policy makers. At EVI, decision regarding which vaccine candidates to include into the organisation's portfolio, and which ones to move forward, are based on a rigorous selection of candidates made by the EVI Board, based on input from an independent scientific advisory committee. For selecting vaccine candidates and advancing their development, EVI employs a portfolio management approach that has defined gating criteria (Go/No-Go criteria), ensuring that only the best leads are fed in and candidates that do not meet the criteria set are weeded out early on, thereby balancing the number of projects supported with available resources 4 . Results delivered by the portfolio analysis using the P2I tool, in particular the total estimated costs and expected success rates, therefore provide valuable information that informs this selection and decision-making process.
Overall, using the pre-defined assumptions established in the P2I tool, our modelling resulted in a total estimated cost of The costs that we estimated using the P2I tool are likely to be an underestimate of the true costs. Vaccine development is a reiterative process, meaning that many steps, such as clinical trial phases, will be conducted several times. This process is in contrast to the rationale of the P2I tool, which assumes a straightforward development of product candidates without the reiteration of any particular development steps. For example, very often several phase I trials for a particular antigen are conducted, i.e. multiple trials in which different formulations, different routes of administration or different technology platforms for antigen presentation are being tested and compared. Also, very often several phase I clinical trials are conducted in which a vaccine´s safety is assessed in various age groups in an age-deescalating manner (i.e. the vaccine is tested consecutively in volunteer groups with decreasing age). It is therefore rather unlikely that any vaccine candidate would immediately advance to a phase II clinical trial after the conduct of a single phase I clinical trial only. Consequently, the total costs for the development of the different vaccine candidates in EVI´s portfolio are likely to be significantly higher than those estimated by the P2I tool. In addition, as several phase I or phase II clinical trials are likely to be conducted for the same vaccine (as explained above), due to such "reiterated" phases the overall timelines to reach the market are expected to be longer.
With regards to the estimated future launches, for all 18 candidates under development, the P2I model estimates that there would be 0.69 expected launches across all six diseases combined. In the re-run of the model using EVI's own internal parameters, thanks to higher success rates at EVI as compared to the P2I tool´s predefined parameters, the launch probability increased for malaria, placental malaria, and leishmaniasis vaccines (from 0.098 to 0.11, from 0.05 to 0.07, and from 0.024 to 0.03, respectively). Directly related to this increased launch probability, the total estimated costs for moving these vaccine candidates through the pipeline increased accordingly. This difference between the estimated future product launches emphasizes that, in order to increase the chances of ultimate success with this kind of product development, it is important to make the product development process technically as efficient as possible, i.e. reducing the attrition rates as much as possible. If attrition rates during product development cannot be reduced, the only other chance of achieving ultimate product launches is to boost the overall number of product candidates in the pipeline, obviously in the end resulting in higher total costs linked to the launch of a single product due to the high costs linked to failed product candidates.
However, rather than looking at the isolated results on likely launches from the analysis of a single organization's portfolio, as has been done in this particular study, more meaningful results will be obtained from such simulations using the P2I tool by conducting a much wider portfolio analysis in which the launch estimation results of the entire global vaccine candidate portfolio are estimated in an integrated, combined manner. Only this kind of "full global portfolio" study can provide a reliable prediction of the product success rates on a global level for the next few decades.
The P2I tool has a number of other limitations, which were described in detail by Young et al. 2 We highlight six specific limitations. First, as a static, deterministic model, it does not take into account possible improvements in product development techniques over time (e.g. R&D efficiencies that reduce costs). Similarly, as a static model, it does not take into account the possibility that candidates may sometimes have to go "backwards" to an earlier phase. For example, once a candidate enters into phase I there may be bottlenecks that require that the candidate return to a preclinical evaluation stage (e.g., if different formulations need to be re-evaluated or alternate adjuvants need to be tested).
Second, the model's assumptions for costs, attrition rates, and cycle times for phase are based on product development data from across multiple diseases (including non-communicable diseases-the model does not reflect possible differences in R&D parameters between different diseases). Although the assumptions were based on a very large number of data points (from 25,000 development candidates) and validated with experts, it is unclear how many of these data points came from vaccine development for neglected and emerging infectious diseases.
Thus there is some uncertainty as to how accurate the assumptions are for the costs, attrition rates, and cycle times per phase for the three vaccine archetypes used in our study.
Third, the model does not include all phases of development (e.g. it excludes drug discovery, basic research, and regulatory review). The exclusion of phase IV studies, also known as post-marketing surveillance, is a major limitation-determining long-term safety and effectiveness is critical, yet it can be a lengthy, expensive process. We acknowledge that using the P2I tool, which only includes advanced pre-clinical to phase III, will always lead to an under-estimate of the total costs, since the costs of early pre-clinical research and post-phase III research can both be substantial. For example, based on data from a sample of 106 NCEs, DiMasi et al. 5 estimate that developing an NCE to the point of marketing approval costs $2.6 billion; this includes $1.2 billion in "time costs" (the expected returns that private investors forgo while a drug is in development) 5 . Of this $2.6 billion, $1.1 billion is in pre-clinical development costs and $1.5 billion in clinical development costs. Previous research has suggested that the cost of regulatory approval stage may represent up to 5.7% of the total R&D cost 6 .
By only including advanced pre-clinical to phase III, the model also provides no insight into the costs and complexity of the array of activities that need to happen after phase III for a new product to have a public health impact. The phase after phase III is often considered as a "valley of death" for product development-a product may pass successfully through phase III but then there may be no concerted, strategic plan for large-scale manufacturing or scale-up. Demand forecasting, developing a long-term business case, understanding the public health value of new products, and analysing the delivery system and scale-up approach are all critical components in determining the ultimate public health utility of a new health technology. We have previously noted that the P2I model "is "agnostic" when it comes to the public health value of the estimated launches-it cannot judge their clinical utility" 2 .
Fourth, accurate classification of candidates into their archetypes can be challenging. As we previously noted, "the P2I v.2 model requires users to classify every candidate into an archetype, but categorizing candidates based on the archetype definitions was challenging-especially determining a candidate's complexity. It will be helpful for future iterations of the model to include more fine-grained, detailed descriptions" 2 .
Fifth, the model assumes that the costs, attrition rates, and cycle times per phase for vaccine development would be the same regardless of the setting where the study is done. Yet in reality, it is likely that these parameters would be different if References a study were conducted in a high-, middle-, or low-income country. It would be helpful for future iterations of the P2I tool to incorporate these differences across study settings.
Sixth, the model also assumes that the costs of vaccine development per phase do not vary and are predictable. Yet there can be substantial variation and unpredictability in items such as the cost of manufacturing the product candidates and adjuvants, or the maintenance and quality control of clinical trial sites.
In this study, several adaptations to the P2I tool initially were considered with the aim of improving the tool´s usefulness and reliability. First, we considered making adaptations of the assumptions for success rates, costs and cycle times based on EVI's longstanding experience with conducting studies in resource-limited, low-income settings. Second, we considered making adaptations for the same parameters based on (a) the fast-track clinical development strategy often used by EVI (consisting of a strategy in which the first-in-human evaluation involves a staggered multi-centre phase Ia/b clinical trial resulting in shorter timelines 7,8 ), and (b) the inclusion of accelerated clinical testing based on controlled human infection models available, for example, for malaria 9,10 . Third, in the original proposal we considered including adaptations of the assumptions for costs and cycle times for vaccines that might be eligible for accelerated approval by regulatory authorities, for example the "Expedited Programs for Serious Conditions--Drugs and Biologics" from the US Food and Drug Administration 11 .
In the end, we were able to do one re-run of the P2I model using EVI's own parameters for success rates and cycle times for phase I clinical trials for unprecedented vaccines (70% and 17.4 months, respectively, compared to 50% and 24 months defined in the original P2I tool for unprecedented vaccines). However, although to date EVI has been involved in the conduct of over 30 clinical trials, data from only a limited number of studies could be used in the analyses conducted. When it came to estimating clinical trial costs, for example, for several trials it was not possible to extract the specific costs for preclinical or clinical trial activities out of the overall study costs.
Concerning the proposed modification of P2I parameters based on accelerated clinical testing using controlled human challenge models and on accelerated approval by regulators, we realized that although these two issues are likely to speed up vaccine development, currently there is not enough evidence/ data based on which the P2I parameters could be adapted and analyses be run to assess their impact.
In conclusion, we found that despite the limitations discussed above, the P2I tool was flexible and adaptable enough to be used to study EVI's portfolio. We believe that the P2I model represents a useful tool to analyze the portfolio of global health products under development. Findings from the analysis of the overall EVI vaccine portfolio using the P2I tool will be taken into consideration in the next revision of the EVI Strategic Plan, and estimations for individual vaccine candidates will inform decisions regarding whether or not to continue with the development of individual vaccine candidates once they reach major milestone or stage gating criteria. We expect that studies like ours will inform future updates of the model that will further increase its value for product developers, R&D funders, and decision makers.

Data availability
All data underlying the results are available as part of the article and no additional source data are required.
The particular vaccine candidates included in this study have been anonymized for intellectual property reasons.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
11 February 2020 Reviewer Report https://doi.org/10.5256/f1000research.24213.r59534 © 2020 Excler J. This is an open access peer review report distributed under the terms of the Creative Commons , which permits unrestricted use, distribution, and reproduction in any medium, provided the original Attribution License work is properly cited. Regarding clinical trial duration, there may still be a misunderstanding. I was not referring to Phase IV but to Phase II (prior to Phase III). Your manuscript refers to the duration from first enrolled to last visit. It does not include the time of analysis of basic safety data and of immunogenicity data, the latter being sometimes long for complex vaccines (e.g., HIV vaccines). At best, the immuno analysis may take 6-12 mo, cleaning of the data and statistical analysis may well take another 6-12mo. This is not taken into account in your duration.

© 2019 Bottazzi M. This is an open access peer review report distributed under the terms of the Creative Commons
, which permits unrestricted use, distribution, and reproduction in any medium, provided the original Attribution License work is properly cited.

Maria Elena Bottazzi
Texas Children's Hospital Center for Vaccine Development, National School of Tropical Medicine, Baylor College of Medicine, Houston, TX, USA This manuscript attempts to describe the applicability and adaptability (with assumption modifications) of a recently developed computer (Excel)-based modelling tool called Portfolio-to-Impact (P21 -version 2) that has as objective to estimate the minimum funding needs (costs) to move of target a portfolio candidates (drugs or vaccines) from late stage preclinical up to phase 3 trials and estimate the number of targets with the potential to reach the launch stage. Of note, the authors mention that prior publications describe the P21 tools in detail for both Versions 1 and 2 and that a prior publication had already analyzed the application of the P21 tool using a broad portfolio of 538 PRND candidates.
For this manuscript the authors use as a case study a selection of targets from the of the vaccine portfolio European Vaccine Initiative (EVI). They focus on 18 candidates (3 in preclinical, 11 in phase 1 vaccine and 4 in phase 2) and with three different categories: 15 as unprecedented vaccines (13 malaria and 2 leishmania), 2 as simple vaccines (Zika and nipah) and 1 as a complex vaccine (shigella/ETEC). As methodology, the authors compare the outputs obtained after using the original P21V2 tool assumptions (cycle time, cost and attrition rate) with a modified set of EVI assumptions.
As a reviewer, I believe the paper has some potential usability and interest, especially for other PRND vaccine developers and PDPs and those with small portfolios. There are several major observations, however that require attention and/or clarification to strengthen the paper before it is to be indexed: Even though the paper highlights that the tool was developed specifically for PRNDs, it lacks more detailed information, discussion and evaluation of the bottlenecks or considerations especially when the development is being done outside HICs and the likely the challenges that would be applicable when the vaccines could be developed partly in and with LMICs organizations. The examples using the EVI portfolio still leaves unclear where the different phases are done. What if preclinical is in Europe but the clinical in Africa, would it be different the cost or probability?
The model and paper don't seem to take into consideration nor discuss the variable and unpredictable costs during vaccine development. For example, cost of goods, manufacture, proprietary components such as adjuvants, stability, regulatory, quality control costs, all needed to maintain and even replenish the clinical lots during the transition into the advanced clinical stages.
It also makes no mention on the complexity of the costs, time and probability needed to continuously mature the production processes and its QC testing and reach suitability for use throughout the different clinical stages and pre-launch or post-launch. For instance, when measuring the probability and time to launch, does this take into consideration where and who would be the large-scale manufacturer? Would there also be the need for manufacturing infrastructure?
Even though the manuscript speaks about using "time to launch" and "probability to launch", this is very vague and offers no value to the reader since there is a big valley after phase 3 trials that is not discussed. The public health value proposition, business case and demand forecasting for 4.
not discussed. The public health value proposition, business case and demand forecasting for PRNDs is a very complex system that require the involvement of multiple WHO offices and committees, GAVI and others. This should be better elaborate in the manuscript.
The manuscript lacks to elaborate on the reality that even once a product enters into Phase 1 there are bottlenecks, which may require that a product returns to a preclinical evaluation stage, for example if different formulations need to be re-evaluated or alternate adjuvants need to be tested. It also is not clear about costs and time when the clinical development is in resource-poor areas where regulatory hurdles may hamper the timelines.
Step 2 method description and table 3: if in the first run, the authors are using "exactly the same assumptions as P21V2 model", it is unclear why there is also reference to two additional set of assumptions: the RAP and the BMGF assumptions. Are these readily available? If not, this would make the applicability of P21V2 obsolete especially for other PDPs and PRND vaccine developers with smaller or non-gates supported portfolios.
Step 3 method description and table 4: the authors mention that the modified EVI assumptions may be unreliable because they are based only on 2-3 data points and decided to make a pragmatic decision to use only those with 10 data points. This in itself is flawed. If the objective of the authors is to showcase how the tool assumptions or a modification of tool assumptions could support vaccine producers in their forecasting exercise, the model should be based on close to reality, especially since most PRND vaccine portfolios do not contain large number of vaccine candidates or data points. The authors should address and comment on this especially since it seems the probability rates are too generous in the EVI assumptions.
The disease focus of the selected portfolio is quite varied and for some (ie malaria) with multiple targets versus for some only one target. A more detail discussion on the applicability of the tool for the evaluation of 1-2 targets versus >2 targets would be very useful.

If applicable, is the statistical analysis and its interpretation appropriate? Not applicable
Are all the source data underlying the results available to ensure full reproducibility? Partly

Are the conclusions drawn adequately supported by the results? Partly
No competing interests were disclosed.

Competing Interests:
Reviewer Expertise: Neglected tropical diseases, vaccine development, product development, global Reviewer Expertise: Neglected tropical diseases, vaccine development, product development, global health

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.
Author Response 03 Jan 2020 , European Vaccine Initiative (EVI), Heidelberg, Germany Stefan Jungbluth

Comment:
Even though the paper highlights that the tool was developed specifically for PRNDs, it lacks more detailed information, discussion and evaluation of the bottlenecks or considerations especially when the development is being done outside HICs and the likely the challenges that would be applicable when the vaccines could be developed partly in and with LMICs organizations. The examples using the EVI portfolio still leaves unclear where the different phases are done. What if preclinical is in Europe but the clinical in Africa, would it be different the cost or probability? Response from authors: This is an excellent point. The model is "agnostic" as to where the development is being done, which is a limitation. In the revision, we now acknowledge this limitation in the Discussion section (in the discussion of limitations). We write: "Fifth, the model assumes that the costs, attrition rates, and cycle times per phase for vaccine development would be the same regardless of the setting where the study is done. Yet in reality, it is likely that these parameters would be different if a study was conducted in a high-, middle-, or low-income country. It would be helpful for future iterations of the P2I tool to incorporate these differences across study settings."

Comment:
The model and paper don't seem to take into consideration nor discuss the variable and unpredictable costs during vaccine development. For example, cost of goods, manufacture, proprietary components such as adjuvants, stability, regulatory, quality control costs, all needed to maintain and even replenish the clinical lots during the transition into the advanced clinical stages. It also makes no mention on the complexity of the costs, time and probability needed to continuously mature the production processes and its QC testing and reach suitability for use throughout the different clinical stages and pre-launch or post-launch. For instance, when measuring the probability and time to launch, does this take into consideration where and who would be the large-scale manufacturer? Would there also be the need for manufacturing infrastructure? Response from authors: This is another excellent point. We now discuss this in the limitations section of the discussion. We write: "Sixth, the model also assumes that the costs of vaccine development per phase do not vary and are predictable. Yet there can be substantial variation and unpredictability in items such as the cost of manufacturing the product candidates and adjuvants, or the maintenance and quality control of clinical trial sites." As discussed in detail in the revision, there are many costs that are excluded from the P2I model, which is indeed a limitation. The model only includes advanced clinical to phase III (it does not include post-phase III large-scale manufacturing, regulatory approval, marketing, etc.). We state: "Third, the model does not include all phases of development (e.g. it excludes discovery, basic research, and regulatory review). The exclusion of phase IV studies, also known as post-marketing "Third, the model does not include all phases of development (e.g. it excludes discovery, basic research, and regulatory review). The exclusion of phase IV studies, also known as post-marketing surveillance, is a major limitation-determining long-term safety and effectiveness is critical, yet it can be a lengthy, expensive process. We acknowledge that using the P2I tool, which only includes advanced pre-clinical to phase III, will always lead to an under-estimate of the total costs, since the costs of early pre-clinical research and post-phase III research can both be substantial. For example, based on data from a sample of 106 NCEs, DiMasi estimate that developing an et al NCE to the point of marketing approval costs $2.6 billion to, which includes $1.2 billion in "time costs" (the expected returns that private investors forgo while a drug is in development). Of this $2.6 billion, $1.1 billion is in pre-clinical development costs and $1.5 billion in clinical development costs. Previous research has suggested that the cost of regulatory approval stage may represent up to 5.7% of the total R&D cost. "

Comment:
Even though the manuscript speaks about using "time to launch" and "probability to launch", this is very vague and offers no value to the reader since there is a big valley after phase 3 trials that is not discussed. The public health value proposition, business case and demand forecasting for PRNDs is a very complex system that require the involvement of multiple WHO offices and committees, GAVI and others. This should be better elaborate in the manuscript.

Response from authors:
In the revision, we now make it much clearer what we mean by launch (i.e. a candidate makes it through phase III and is ready for post-phase III steps, such as regulatory review). We also now, in the discussion, explore the valley after phase III. In the revision, we now state: "By only including advanced pre-clinical to phase III, the model also provides no insight into the costs and complexity of the array of activities that need to happen after phase III for a new product to have a public health impact. The phase after phase III is often considered as a "valley of death" for product development-a product may pass successfully through phase III but then there is no concerted, strategic plan for large-scale manufacturing or scale-up. Demand forecasting, developing a long-term business case, understanding the public health value of new products, and analysing the delivery system and scale-up approach are all critical components in determining the ultimate public health utility of a new health technology. We have previously noted that the P2I model "is "agnostic" when it comes to the public health value of the estimated launches-it cannot judge their clinical utility."

Comment:
The manuscript lacks to elaborate on the reality that even once a product enters into Phase 1 there are bottlenecks, which may require that a product returns to a preclinical evaluation stage, for example if different formulations need to be re-evaluated or alternate adjuvants need to be tested. It also is not clear about costs and time when the clinical development is in resource-poor areas where regulatory hurdles may hamper the timelines Response from authors: This is an excellent point, which we now address in the revised discussion section: "The P2I tool has a number of other limitations, which were described in detail by Young . We et al highlight X specific limitations. First, as a static, deterministic model, it does not take into account possible improvements in product development techniques over time (e.g. R&D efficiencies that reduce costs). Similarly, as a static model, it does not take into account the possibility that candidates may sometimes have to go "backwards" to an earlier phase. For example, once a candidate enters into Phase I there may be bottlenecks that require that the candidate return to a preclinical evaluation stage (e.g., if different formulations need to be re-evaluated or alternate adjuvants need to be tested)." 10 11 2" 2 Comment: Step 2 method description and table 3: if in the first run, the authors are using "exactly the same assumptions as P21V2 model", it is unclear why there is also reference to two additional set of assumptions: the RAP and the BMGF assumptions. Are these readily available? If not, this would make the applicability of P21V2 obsolete especially for other PDPs and PRND vaccine developers with smaller or non-gates supported portfolios.

Response from authors:
We now discuss the RAP and the Gates Foundation as data sources for P2I.v2; see the revised Methods section. The RAP is not public; the Gates Foundation data are shown in reference 2.

Comment:
Authors mention that the modified EVI assumptions may be unreliable because they are based only on 2-3 data points and decided to make a pragmatic decision to use only those with 10 data points. This in itself is flawed etc Response from authors: We agree that model should be based on real data, however we do not consider it as meaningful to modify the P2I tool assumptions if such alternative parameters only include 2-3 data points. A better way forward will be to combine these low numbers of alternative data points with additional data points coming from other users of the tool, thereby increasing the overall total number of data points available. Such a combined, increased number of data points can then be used as alternative parameters for the analysis of product portfolios

Comment:
Probability rates are too generous in the EVI assumptions. Response from authors: Probability rates included are the real success rates obtained at EVI. These high success rates are also not surprising, given that they relate to phase I clinical trials that test for vaccine safety where success rates usually are rather high.

Comment:
Disease focus of the selected portfolio is quite varied and for some (ie malaria) with multiple targets versus for some only one target. A more detail discussion on the applicability of the tool for the evaluation of 1-2 targets versus >2 targets would be very useful.

Response from authors:
Wording that had already been included in the article has been slightly modified for clarification. As mentioned in the article, especially for the estimation of expected product launches the P2I tool should be used for an overall analysis of global product candidate portfolio, compared to the analysis of individual organisation´s portfolio only. Such combined global studies will provide far more meaningful data. This article describes the application of the P2I model to the vaccine development portfolio of EVI. The intent of the exercise is to provide a portfolio funding estimate from late preclinical through completion of phase III. The authors proceed in four steps by classifying their vaccine candidates into archetypes, running the standard P2I v2 model, re-running the model using modified assumptions based on EVI's own data, and conducting a sensitivity analysis for the model assumptions. The authors conclude that the P2I model, despite several limitations, is suitable and sufficiently flexible to study EVI's vaccine portfolio.
There are a number of methodological questions that merit further consideration or clarification.
First, the notion of product launch is vague. The model calculates costs from late preclinical up to completion of phase III, so this approach seems to provide numbers for 'launch readiness' rather than product launch and should be clearly stated.
Second, significant other costs for vaccine development do not seem to be included, for example process development, GMP lots and investment into manufacturing. Due to this limitation, the calculated funding needs provide only a partial view of the overall required investment.
Third, clinical phase duration does not take into account the time to prepare for clinical trials, perform data analysis and generate a study report. Together with the author's comment that each clinical trial phase is likely to comprise several iterative studies, the time (and cost) assumptions appear very optimistic.
Fourth, the classification into archetypes is somewhat arbitrary. One could argue that the RTS,S and M72 (out of scope of this exercise) vaccines provide precedents for malaria and TB, respectively, and should increase the phase II and III probability of success assumptions.
Overall, this study seems to demonstrate that the usefulness of any model depends on the quality of the assumptions. Unfortunately, as the authors indicate, there is a scarcity of data in type II and III disease vaccine development, thereby rendering the generation of assumptions unreliable.
In the reviewer's opinion, the article would benefit from answering two questions: how has the model helped EVI to prioritise its portfolio and how can the model be improved.
Minor comments: Table 5 and figures 2 and 3 are redundant  Table 6 and figure 4 are redundant. It would also help to clarify what 'cost' and 'cumulative annual' means. Is that 'cumulative cost'? Table 7 'of' needs to be deleted in the title

Are sufficient details of methods and analysis provided to allow replication by others? Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Yes
No competing interests were disclosed.

Competing Interests:
Reviewer Expertise: Vaccine development, PRDs, global health I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Comment:
First, the notion of product launch is vague. The model calculates costs from late preclinical up to completion of phase III, so this approach seems to provide numbers for 'launch readiness' rather than product launch and should be clearly stated Response from authors: Reviewer 1 also raised this point. We now explain this more clearly. We state: "in this paper, the term launch refers to a candidate making it through phase III and thus being ready for the next steps, e.g. regulatory and manufacturing steps."

Comment:
Second, significant other costs for vaccine development do not seem to be included, for example process development, GMP lots and investment into manufacturing. Due to this limitation, the calculated funding needs provide only a partial view of the overall required investment.

Response from authors:
The reviewer is correct, and this is one of the limitations of the P2I model. We now state this limitation clearly in the methods, and discuss its implications in the Discussion section. In the revised Methods, we now state: "The model includes only advanced preclinical to phase III, and thus the cost estimates are an under-estimate of the full costs of product development. In particular, the model excludes all costs related to basic research through lead optimization; chemistry, manufacturing, and controls; good manufacturing practice; manufacturing build up and scale-up costs; regulatory or registration fees (post-phase III); and all post-market commitments (e.g., phase IV pharmacovigilance studies)."

Comment:
Third, clinical phase duration does not take into account the time to prepare for clinical trials, perform data analysis and generate a study report. Together with the author's comment that each clinical trial phase is likely to comprise several iterative studies, the time (and cost) assumptions appear very optimistic.

Response from authors:
In the revised paper, we now explain exactly how these assumptions were generated and validated (and also point readers to the two previous research articles that explain these assumptions in greater detail).

Comment:
Fourth, the classification into archetypes is somewhat arbitrary. One could argue that the RTS,S and M72 (out of scope of this exercise) vaccines provide precedents for malaria and TB, respectively, and should increase the phase II and III probability of success assumptions.

Response from authors:
We respectfully disagree that classification is arbitrary, and we now refer readers to our previous two studies showing more details on the classification process. But we certainly do agree that classification is an imperfect science, and we now discuss this limitation in the revised Discussion section.

Comment:
Overall, this study seems to demonstrate that the usefulness of any model depends on the quality of the assumptions. Unfortunately, as the authors indicate, there is a scarcity of data in type II and III disease vaccine development, thereby rendering the generation of assumptions unreliable.

Response from authors:
We agree that all models are dependent on the quality of the model assumptions. We now explain in greater detail how these were derived and validated. We respectfully disagree that the assumptions are unreliable; indeed, we know of no author assumptions that are more reliable than those in the P2I model. We now state, in the revision: "As described in detail in reference 1, assumptions on development costs at each phase of product development for the 11 archetypes included in the P2I.v1 model were initially based on an analysis of clinical trial costs from Parexel's R&D cost sourcebook. The assumptions on attrition rates and cycle times at each phase were initially based on the historical attrition rates and cycle times of more than 25,000 development candidates. All of these assumptions were further refined and validated based on academic literature, industry publications and databases, and 133 stakeholder interviews with a wide variety of product development partnerships (PDPs), pharmaceutical companies, and major funders of global health R&D. As described in detail in reference 2, additional sources of assumptions for the new archetypes in P2I.v2 were derived from the McKinsey Risk-Adjusted Portfolio (RAP) Model and from clinical trial data shared with us by the Bill & Melinda Gates Foundation." In the discussion section, when we discuss limitations, in the revision we now write: "the model's assumptions for costs, attrition rates, and cycle times for phase are based on product development data from across multiple diseases (including non-communicable diseases-the model does not reflect possible differences in R&D parameters different diseases). Although the between assumptions were based on a very large number of data points (from 25,000 development candidates) and validated with experts, it is unclear how many of these data points came from vaccine development for neglected and emerging infectious diseases. Thus there is some uncertainty as to how accurate the assumptions are for the costs, attrition rates, and cycle times per phase for the three vaccine archetypes used in our study." the next phases of development.
The term "launch" may deserve some clarification. Are the authors talking about registration for launch on the public market in LMICs, private market in LMICs, or per Gavi scheme for eligible countries? What about MICs which are no longer Gave-eligible and whose list is increasing? It is unclear whether the model considers vaccine manufacturing by traditional big western pharmas or developing country vaccine manufacturers (DCVM) and consider bringing the vaccine to WHO prequalification. This latter point is of importance as often this may require additional clinical studies adding to the overall development costs and duration before launch.
It is difficult to understand Tables 6-7 (Title of  Tables 8 and 9 would need ample explanation about what is considered in these tables.
Coming back to the estimated cost of vaccine development to launch, it is difficult to weigh the value of the findings compared to real life costs. For example, the cost of RTS,S malaria vaccine developed by GSK is estimated to be close to $500M. For Nipah and Zika, what is the development cost estimated by CEPI? How does it compare to EVI estimates? What the P2I model ever tested a posteriori for vaccines already developed and for which the development cost is know in order to measure the accuracy of the model prediction?

If applicable, is the statistical analysis and its interpretation appropriate? Not applicable
Are all the source data underlying the results available to ensure full reproducibility? No

Are the conclusions drawn adequately supported by the results? Partly
No competing interests were disclosed.

Competing Interests:
Reviewer Expertise: Vaccine Development, Global Health I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

Comment:
It is unclear to a reader unfamiliar with this tool who are the target stakeholders of the outcome of the analysis: manufacturers, non-for-profit developers, donors, public health policy makers? Although the authors provide references, this should be summarized and clarified Response from authors: We have added a sentence in the introduction to explain this. We now state: "As a financial forecasting tool that estimates the funding needs for pharmaceutical product development, the tool and its outputs are of value to funders of product development, product development partnerships, and other stakeholders involved in research and development (R&D) policy. Terry and colleagues, the developers of the P2I model, note that "its real utility lies in its predictive value for modelling the impact of different funding strategies at the portfolio level."

Comment:
It is also unclear how this tool articulates for a given vaccine with the development of the Business Case component of the Full Public Health Value Proposition and the Total Systems Effectiveness (TSE) and the Vaccine Innovation Prioritization Strategy (VIPS) developed by WHO.

Response from authors:
In the introduction, we have now added: "We used the P2I tool because, to the best of our knowledge, it is the first publicly available comprehensive portfolio model that includes data on cost, success rate, and cycle time per phase for various product types based on data from a very large number of previous product development candidates (over 25,000) . The P2I tool is thus complementary to other available tools that can help guide prioritization in vaccine development, such as the multi-stakeholder Vaccine Innovation Prioritization Strategy and Total Systems Effectiveness Framework."

Comment:
What is the added value of the P2I model compared to the other models referred in Table 3? This should also be clarified. Response from authors: Table 3 does not show other models. It shows the P2I model's underlying assumptions on cost, cycle time, and success rate per development phase for three different categories of vaccines (simple, complex, and unprecedented). It also shows the source of data for these assumptions. In the text, we state: "In this first run of the model, we used exactly the same assumptions on cycle time, cost, and attrition rate per phase as in the P2I.v2 model . These assumptions are shown in . The assumptions were derived from three sources: the P2I model (shown in orange in Table 3 ), the McKinsey Risk-Adjusted Portfolio (RAP) Model (shown in yellow), and the Bill & Table 3 Melinda Gates Foundation (shown in blue) ."

Comment:
It might also be useful to remind how EVI prioritized their vaccine candidate portfolio (no reference cited) and why they selected some of them for this P2I analysis. Response from authors: New wording describing the vaccine candidate selection and priorisation process, including a reference, have been included in the discussion. Reasons re the selection of vaccine candidates that were included in this study were already provided in the original article.

Comment:
1 1 2 2 Data one can obtain from the CEPI web page are the total maximum funding amounts awarded by CEPI for the different Nipah and Zika vaccines that currently are in the CEPI portfolio. However, we do not consider these figures as meaningful comparators for our estimations as the details of the funded projects (eg vaccine technology, other specific activities included in different projects etc) are not known to us and it is therefore unclear how the vaccines funded by CEPI compare to the one in EVI´s portfolio (NB: The Nipah vaccine candidate in EVI´s portfolio is funded by CEPI but vaccine development is only funded down to phase II clinical trial).

Comment:
What the P2I model ever tested for vaccines already developed and for which the a posteriori development cost is known in order to measure the accuracy of the model prediction? Response from authors: No vaccines that have finalised their development are available at EVI for such an a posteriori analysis, so for the time being this issue cannot be addressed.
No competing interests were disclosed.

Competing Interests:
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com