Developing new health technologies for neglected diseases: a pipeline portfolio review and cost model

Background: Funding for neglected disease product development fell from 2009-2015, other than a brief injection of Ebola funding. One impediment to mobilizing resources is a lack of information on product candidates, the estimated costs to move them through the pipeline, and the likelihood of specific launches. This study aimed to help fill these information gaps. Methods: We conducted a pipeline portfolio review to identify current candidates for 35 neglected diseases. Using an adapted version of the Portfolio to Impact financial modelling tool, we estimated the costs to move these candidates through the pipeline over the next decade and the likely launches. Since the current pipeline is unlikely to yield several critical products, we estimated the costs to develop a set of priority “missing” products. Results: We found 685 neglected disease product candidates as of August 31, 2017; 538 candidates met inclusion criteria for input into the model. It would cost about $16.3 billion (range $13.4-19.8B) to move these candidates through the pipeline, with three-quarters of the costs incurred in the first 5 years, resulting in about 128 (89-160) expected product launches. Based on the current pipeline, there would be few launches of complex new chemical entities; launches of highly efficacious HIV, tuberculosis, or malaria vaccines would be unlikely. Estimated additional costs to launch one of each of 18 key missing products are $13.6B assuming lowest product complexity or $21.8B assuming highest complexity ($8.1B-36.6B). Over the next 5 years, total estimated costs to move current candidates through the pipeline and develop these 18 missing products would be around $4.5B (low complexity missing products) or $5.8B/year (high complexity missing products). Conclusions: Since current annual global spending on product development is about $3B, this study suggests the annual funding gap over the next 5 years is at least $1.5-2.8B.


Introduction
In 2015, United Nations member states adopted the Sustainable Development Goals (SDGs), an expansive global agenda that includes ambitious health targets. These health targets include, by 2030, "end the epidemics of AIDS, tuberculosis, malaria and neglected tropical diseases," "end preventable deaths of newborns and children under 5 years of age," and reduce the global maternal mortality ratio to under 70 per 100,000 live births (see Sustainable Development Goals (SDGs)). Recent studies based on modelling the impacts of scaling up health tools and strengthening health systems show that it is highly unlikely that these targets will be achieved using today's health technologies alone-achievement will also require breakthrough innovations, such as high efficacy preventive vaccines for HIV, malaria, and tuberculosis [1][2][3] . The Commission on Investing in Health, in its Global Health 2035 report, also found that achieving "grand convergence"-a reduction in avertable infectious, maternal, and child deaths to universally low levels-will require accelerated health product development 1 .
Investing in the development and delivery of health technologies is one of the most effective ways to achieve rapid reductions in avertable mortality. For example, Jamison and colleagues recently showed that the diffusion of such technologies accounted for about 80% of the decline in child mortality from 1970 to 2000 4 . In addition, researchers have used a new economic evaluation tool termed "extended cost effective analysis" to show that many health technologies for diseases of poverty provide not only health but also financial protection, and are pro-poor 5 . A further way in which investing in health product development has economic benefits is that the returns to investment can be very large. For example, the March of Dimes invested about US$26 million (M) in developing the polio vaccine, and since routine vaccination was introduced, treatment cost savings have generated a net benefit of around US$180 billion (B) in the United States alone (unless otherwise stated, all dollar figures within this report have been adjusted to 2017 US dollars) 6 . A Copenhagen Consensus study estimated that every US$1 invested in HIV vaccine development would return US$2-$67, assuming a vaccine of 50% efficacy becomes available by 2030 and annual R&D costs are roughly US$0.9B 7 .
There is thus a strong case for investing in product development for neglected diseases (in this paper, we use the term "neglected diseases" to refer to the 35 infections or health priorities defined by Policy Cures Research as neglected, including HIV/AIDS, malaria, tuberculosis, pneumonia, diarrheal diseases, neglected tropical diseases and reproductive health needs of developing countries (see G-FINDER project)). Yet funding for neglected disease product development fell steadily from 2009 to 2015, with the exception of a short-term injection of Ebola funding (see G-FINDER report for 2016). While mobilizing additional finance for such R&D is needed, funders face several information gaps that are an impediment to resource mobilization. In particular, there is a lack of consolidated information on: • which candidates are currently in the pipeline and at what development phase; • the estimated costs to accelerate this portfolio of candidates to production; • the anticipated product launches that would result from such acceleration; and • the critical, highly needed products that would still be "missing" under the status quo.
There have been a small number of studies published on the estimated cost to develop a single drug. For example, DiMasi and colleagues estimate that it takes $2.6B (in 2015 US dollars) to develop a new chemical entity (NCE), an estimate based on surveying 10 pharmaceutical firms to obtain information on 106 randomly selected new drugs 8 . It is unclear how relevant this study is to developing products for neglected diseases, and the estimate can be criticized for, among other things, including $1.2B in "time costs" (the expected returns that private investors forgo while a drug is in development). However, while there are published estimates of the costs of developing individual products, to the best of our knowledge, there have been no estimates of the costs to move the full portfolio of current product candidates through the pipeline.
There has been an estimate by the WHO's Consultative Expert Working Group on R&D (the CEWG) on overall funding needs for neglected disease product development 9 . The CEWG

Amendments from Version 2
Recently, we conducted a new pipeline portfolio review and cost model using the P2I v.2 tool. In our process of conducting quality control, we discovered small errors when comparing the 2017 pipeline and P2I V.2 modelling results to the 2019 pipeline and P2I V.2 modelling results.
Errors and impact: Candidates in the pipeline entered into P2I model: under vaccines, one unprecedented vaccine had been mis-categorized as a simple vaccine, which has now been corrected. Also, the values for assay development and simple platform development were inversed, which was also corrected.
Impact: Correcting these errors did not affect the modeling results. The changes are reflected in the updated Figure 2 and Supplementary File 2.
The updated model contained an error in one of the intermediate calculation sheets that mis-categorized the input candidates as a different archetype. This error was found in the "Total port annual costs" sheet for portfolio input numbers 173 and 186. While this error existed in the intermediate calculation sheet it was not visible in either input or output sheets, which is why this was not discovered prior to this. The error was corrected by ensuring that the correct formula was applied to 173 and 186 input lines in the "Total Port Annual Costs" sheet.
Impact: Correcting the error in the model resulted in a small change in cost for tuberculosis (from $2556 million to 2569.48 million) and the total cost for all products (from $ 16,335.29 million to 16,348.78 million). This error was corrected in Figure 4 (costs of moving current product candidates through the pipeline from their current phase), Figure 5 (costs of moving candidates through the pipeline by disease), Table 7 (costs of moving current product candidates through the pipeline from their current phase), Table 9 (sensitivity analysis), and Supplementary File 3.
argued that $6B should be spent annually on such research. This estimate was not based on an empirical analysis of what is in the pipeline, what is missing, and what it would cost to develop the missing tools. Instead, it was derived by doubling the amount that the public sector in low-and middle-income countries invested in health R&D in 2005 (which was $3B, according to the 2008 Global Forum for Health Research report 10 ). There remains a dearth of cost estimates for the actual pipeline of neglected disease candidate technologies.
Our study aimed to help close these information gaps. We conducted a pipeline portfolio review, using public domain knowledge, to examine which candidate products (e.g., drugs, vaccines, diagnostics) are currently in the pipeline for neglected diseases. Based on the results of this review, we estimated the costs to move these candidates through the pipeline, the likely launches, and the "priority" health technologies that would be "missing" under the status quo. To do this, we used an adaptation of a financial tool-the Portfolio to Impact (P2I) modeldeveloped by TDR, the Special Programme for Research and Training in Tropical Diseases, for the World Health Organization 11, 12 . We made the adaptations of the P2I tool ourselves, resulting in P2I version 2 (P2I v.2); adaptations included adding more product types (e.g., vector control products) and modifying some of the tool's underlying assumptions based on additional data.

Methods
In this section, we describe the study's six steps: (i) pipeline portfolio review; (ii) adaptation of the original P2I model; (iii) classification of candidate health products into archetypes; (iv) inputting the pipeline of candidates into the adapted model; (v) estimating the costs of developing "missing" products; and (vi) sensitivity analysis.
i) Pipeline portfolio review Scope. A global review of candidate products for neglected diseases was conducted by Policy Cures Research (authors NC and VC) in 2017, and represents a snapshot of the pipeline as of August 31, 2017. Policy Cures Research, a non-profit global health research and advocacy organization, developed the scope of the R&D pipeline presented in this analysis based on the G-FINDER landscape reports on global funding of R&D for neglected diseases and the reproductive health needs of developing countries. The G-FINDER scope is based on three key principles: the disease or health issue disproportionately affects the developing world; there is a need for new products (i.e., there is no existing product, or improved or additional products are needed); and there is market failure (i.e., there is insufficient commercial market to attract private R&D investment). Accordingly, the R&D pipeline presented here encompasses the 33 neglected diseases included in the G-FINDER report, as well as developing country-specific reproductive health needs, and Ebola, giving a total of 35 neglected diseases or conditions.
Product types included were drugs, vaccines, diagnostics, vector control products, contraceptives, and multipurpose prevention technologies (MPTs); however, in line with the G-FINDER scope, not all product types were included for all areas. Microbicide candidates for HIV have been included under the drug category, while contraceptives and MPTs (which prevent pregnancy and sexually transmitted infections) have been grouped as reproductive health technologies. Medical devices (except for diagnostics and contraceptives), and general or supportive therapies (e.g., oral rehydration or nutritional supplements), were excluded.
We define the pipeline to include product candidates at all stages of development-from discovery through product registration. For drugs, vaccines, and reproductive health products, development stage was broken down into discovery; pre-clinical studies; and clinical trials (further divided into Phase I, Phase II and Phase III trials). Diagnostics and vector control products have different product development and regulatory pathways, and the development stage for these products was broken down into concept and research; feasibility and planning; design and development; and clinical validation and launch readiness. Early stage drug discovery projects that are not linked to a specific, discrete pipeline candidate were excluded. Candidates were no longer considered to be part of the R&D pipeline-and therefore were excluded from this analysis-once granted regulatory approval by a national regulatory authority, or if their development had been placed on hold indefinitely.
In line with the scope of the G-FINDER report, additional restrictions were applied to selected disease and product categories with potential commercial (high-income country market) overlap. For example, drug candidates for HIV/AIDS were only included if they were label-extensions or reformulations specifically intended for developing country use (e.g., paediatric or slow-release formulations; fixed dose combinations; or low-dose formulations for prophylaxis).
Further details on the specific diseases and related product areas within the scope of the pipeline analysis are given in the Policy Cures Research R&D matrix and R&D scope document. Additional details about the search are in the online methodology.  12 . The tool was developed by TDR. In brief, P2I v.1 is a financial portfolio model that estimates funding needs to move a portfolio of candidate health products through the pipeline from late stage pre-clinical to phase III clinical trials, as well as potential product launches over time ( Figure 1). The model, which is deterministic, is based on assumptions for costs, attrition rates (probability of success), and cycle times for four development phases (preclinical to phase III) for eleven different kinds of medical products, called archetypes (Table 1). A detailed description of how these assumptions were developed is given in the accompanying study on development of the P2I v.1 tool 12 .
In brief, the assumptions on development costs at each phase of product development were initially based on a bottom-up analysis of clinical trial costs from Parexel's R&D cost sourcebook 13 .
The assumptions on attrition rates and cycle times at each phase were initially based on a review of the attrition rates and cycle times of more than 25,000 development candidates. These assumptions were further refined and validated based on (i) academic literature 14 , (including literature on the development of antibody therapeutics) (ii) industry publications and databases (such as Pharmaprojects), and (iii) stakeholder interviews with a wide variety of PDPs, biopharmaceutical and diagnostic companies, and major funders of global health R&D. For the stakeholder interviews, a total of 228 stakeholders representing a cross-section of the global R&D landscape were contacted to request an interview and 133 agreed to be interviewed, a response rate of 58% (see reference 11 for further details). Overall, the stakeholder interviews largely confirmed the validity of the assumptions derived from the R&D cost sourcebook, and did not lead to any significant changes. As a final validation step, the P2I model and its assumptions were reviewed by TDR's Science and Technical Advisory Committee, who provided an additional round of expert inputs.
The P2I v.1 tool allows users to make multiple types of adaptations. For example, (i) users can add archetypes; (ii) users can input disease-specific assumptions (e.g., the attrition rates in developing TB biologics); and (iii) if users have their own additional data on costs, attrition rates, and cycle times across the portfolio, they can enter the tool and modify the assumptions. For this study, we made all three types of adaptations.
First, we wanted to add archetypes that were not in version 1, such as vector control products and unprecedented vaccines (discussed further below). Second, we wanted to refine the model as it applied to a specific set of TB candidate archetypes. Third, we took the opportunity to modify a small number of the assumptions. These three adaptations were made based on data shared by the Bill & Melinda Gates Foundation (Per Liljenberg, personal communication). The Foundation supported the development of the P2I v.1 tool, and has used the tool to estimate its own portfolio development costs, but it has made a number of adaptations derived from three Foundation costing exercises: • In 2010, the Foundation identified, reviewed, and costed out all projects in its product development portfolio, building a portfolio model (including costs, time, and risk) called the Risk-Adjusted Portfolio (RAP) Model, based on publicly available data as well as from a proprietary database. The probability of technical success was based on an attrition database, which contains detailed attrition rates across more than 3,000 products over a 10-20 year period. Additionally, PDPs and industry experts were engaged to identify appropriate ranges for specific programs.
• This costing exercise was repeated in 2012-2013.
• In 2016, the costing exercise was repeated once more, using data compiled from various sources, including from the Foundation itself, from the 2010 RAP Model, and from TDR (the data used in P2I v.1).
Below we summarize the adaptations in the P2I v.2 model, which are based on outputs of the 2016 exercise.  The P2I v.2 model has two additional archetypes, "unprecedented vaccines" and "other products": • Unprecedented vaccines. In addition to sub-classifying vaccine candidates into "simple" and "complex," we added "unprecedented vaccines" as a third vaccine candidate category. We assigned candidate vaccines for HIV, TB, and malaria to this third sub-category, which we considered as unprecedented as current platforms have not led to suitable vaccines. We assumed that vaccines targeting each of these diseases would require the development of innovative platforms and a better understanding of the basic biology and of immune protection. The assumptions for unprecedented vaccines were provided by the Bill & Melinda Gates Foundation, which used the P2I v.1 assumptions for complex vaccines as a starting point. As shown in Table 2, the assumptions for unprecedented vaccines are the same as for complex vaccines, with two exceptions. In the P2I v.2 model, unprecedented vaccines have a lower probability of success in phases II and III than complex vaccines (in phase II, 5% for unprecedented vaccines versus 22% for complex vaccines; in phase III, 40% probability of success for unprecedented vaccines versus 64% for complex vaccines). These probability rates for unprecedented vaccines (5% in phase II, 40% in phase III), provided by the Bill & Melinda Gates Foundation, were based on around 10-25 data points per estimated value (these data were from both the Foundation portfolio and publicly available sources). These lower probabilities of success compared with complex vaccines reflect the Foundation's real world experience of trying to develop highly effective vaccines against HIV, TB, and malaria.
• Other products: We used the "other products" archetype for vector control products (which were not included in P2I v.1). The assumptions on costs, attrition rates, and cycle times per phase for the "other products" archetype were derived from product development data shared by the Bill & Melinda Gates Foundation and from the RAP Model.
In addition to including these two archetypes, we also modified several assumptions, as follows: • The costs of phase III for simple and complex vaccines.
We used an assumption of $201M for simple vaccines and $223M for complex vaccines. These phase III cost assumptions, which are higher than those in the P2I v. ). An assessment of benchmarks for biologics was made by Foundation experts, and could be classified as cautious expert judgement based on early industry trends. The Foundation judged that since most early successful monoclonal antibody projects were related to the tumor necrosis factor alpha pathway but biologics are now finding a much broader range of biological targets, the success rates are likely to fall. This was a prediction based on early signals from industry reporting challenges to time, cost and success rates in many ongoing programs at the time, rather than an assessment based on actual program failures.
• Development phases for diagnostics. For diagnostics, the selection and validation phase was further sub-divided into two sub-phases: "concept and research" followed by "feasibility and planning" (see section (iii) for further details). Table 2 shows the assumptions that we used in P2I v.2.
Finally, P2I v.1 allows users to input a portfolio of up to 150 product development projects (the model was developed in the context of potentially launching a pooled fund for R&D that would have had capacity constraints). However, the pipeline of candidates for neglected diseases has far more than 150 projects, so we adapted the tool to be able to input all of these projects.
iii) Classification of product candidates into archetypes As a starting point, we used the initial archetypes and the descriptions of each archetype from P2I.v1 (Table 1). In order to further "operationalize" each archetype-i.e., to make it easier for the research team to make decisions about how to classify each candidate-we worked with technical experts to further define each archetype. Table 3 shows the original archetype descriptions from P2I v.1, our additional definitions, and examples of candidate classifications.
As shown in Table 3, product candidates were classified into six broad archetypes-repurposed drugs, NCEs, vaccines, biologics, diagnostics, and "other products" (which refers to vector control products). Repurposed drugs, NCEs, and biologics were further sub-classified into simple versus complex; vaccines into simple, complex, or unprecedented; and diagnostics into assay development versus simple technical platform development. For candidates in the pipeline that were contraceptives, microbicides, or MPTs, these were classified according

New chemical entity (NCE)
Simple Validated target/mechanism of action primaquine Any chemically synthesized drug that is part of a well-established class of drugs or has a mechanism of action that has already been approved for market use in humans. It is not the first of its class to be approved.
Complex Novel target/mechanism of action without understanding of disease pathogenesis imatinib Any chemically synthesized drug that is first in its class as determined by having a novel target/mechanism of action regardless of the drug's indication.

Biologics
Simple Validated target/mechanism of action human monoclonal antibody m102.4 Any drug that is synthesized from a living organism that is part of a wellestablished class of drugs or has a mechanism of action that has already been approved for market use in humans. It is not the first of its class to be approved.

Complex
Novel target/mechanism of action polyclonal IgG antibodies Any drug that is synthesized from a living organism that is first in its class as determined by having a novel target/mechanism of action regardless of the drug's indication.

Vaccines
Simple Platform has been used to develop other vaccines.
Hep A, Hep B, polio, killed or live attenuated vaccines Any vaccine platform that has been extensively researched and approved for use in the past. Pathogen has readily-identifiable vaccine targets that lack complexity. Conferral of immunity against disease-causing microorganism is expected as natural immunity to the pathogen is protective. Platform is likely to elicit robust protective response. to the constituent drug (e.g., microbicides in the pipeline were classified as repurposed drugs, NCEs, or biologics). If there was more than one active drug ingredient in the MPT, the candidate was classified according to the most complicated component. We did not consider if the polymer or technology itself was innovative in itself as this went beyond the scope of our costing framework.

Complex
The classification of candidates was made by different members of the research team, based on their expertise (repurposed drugs, NCEs, and biologics: KS, KC; diagnostics: BR; vaccines: SP, LD, TS; other products: VC). The classification was based on a combination of (a) technical expertise of the researchers, (b) academic literature, (c) relevant publicly available product databases (e.g., for classifying drugs, ChemBL and chemspider), (d) information from international clinical trials registries, including the WHO International Clinical Trials Registry Platform, (e) websites of PDPs, e.g. the Medicines for Malaria Venture website, (e) patent databases, and (f) relevant reports and news releases from bilateral and multilateral funding agencies, companies, PDPs, other product developers, and non-government organizations. In assigning each candidate product to an archetype, we documented any relevant source material that guided the classification (e.g., a published research article on the candidate's mechanism of action).

Classification of candidate diagnostics into archetypes was conducted by a technical expert in diagnostics R&D (BR).
Using the Policy Cures Research list of candidates, these were further classified into six more specific development phases, as required by P2I v.2: concept; feasibility; early development; late development; validation; and commercialization. Table 4 summarizes what these phases mean and how they compare with two other classification systems for technology readiness (the Technology Readiness Level (TRL) and Manufacturing Readiness Level (MRL), developed by the United States Department of Defense). When it came to inputting candidates into the adapted P2I cost model (as shown in Table 2), diagnostic candidates at the concept phase were placed in the category "concept and research"; those in the feasibility stage were placed in "feasibility and planning"; those in either early or late development were placed in "design and development"; those in validation were placed in "clinical validation and launch readiness"; and those in the commercialization phase were excluded from the cost modelling.
iv) Inputting the pipeline of candidates into the adapted P2I model As described previously, we adapted P2I v.1 into P2I v.2, adding two archetypes (unprecedented vaccines, and other products) and modifying a number of the assumptions. We then used the model "prospectively"-that is, once we had determined which candidates were in the pipeline, and their target disease/condition, archetype and phase, we then inputted these candidates into P2I v.2.
For each disease and archetype, we inputted the number of candidates that were in each phase of development (for repurposed drugs, NCEs, biologics, vaccines, and other products, the phases were preclinical, phase I, phase II, or phase III; for diagnostics, the phases were concept and research, feasibility and planning, design and development, or clinical validation and launch readiness). The analysis was undertaken in 2017, and hence we chose a start date of 2017 (consistent with P2I v.1).
Based on the assumed costs, attrition rates, and time per phase for each archetype, the model estimates the costs and outcomes of moving product candidates through the pipeline from their current phase. When a candidate is put into in a specific phase, the model assumes that it is at the start of that phase and so it includes the costs of moving that candidate through to the end of its current phase.
For each archetype for each disease, the final outputs are (a) the costs of moving the archetype candidates through the pipeline from their current phase, and (b) the estimated product launches at the end of this process. For launches, we rounded only at the very end of the model. For example, for disease X, if there were 3 simple vaccine candidates at Phase II that led to 1.3 expected launches and 3 simple vaccine candidates at Phase III that led to 1.4 expected launches, we rounded the cumulative total-in this case, the total was 2.7. For this paper, we have chosen a conservative approach to presenting the launches-we have considered a launch to be a binary event, i.e., we have always rounded down (in this case, 2.7 rounds down to 2 launches). However, in Supplementary File 1, we also present the results without any rounding (e.g., in this example, 2.7 launches) and with rounding to the nearest integer (2.7 would round to 3). Both of these other approaches give less conservative estimates of the number of launches 15 .
We did not apply a discount rate to our cost estimates. Probability of success, time of phase, cost variables, archetype and complexity classification were assumed to remain constant throughout the lifecycle of the model. We modeled only the current pipeline (i.e., we assumed that no new candidates would enter the pipeline).
(v) Estimating the costs of priority "missing" products As described in the Results section below, for several diseases and product types (e.g., highly efficacious vaccines for HIV, TB, and malaria), the model suggested that there would be no product launches based on the current status quo (i.e. based on the pipeline of candidates that are in the public domain). In order to estimate the costs to develop those products that are likely to be "missing" but are highly needed, we reviewed the suggested list of "important" or "game changing" diagnostics, drugs, and vaccines prioritized by the Commission on Investing in Health (Table 5). The Commission's list was developed through expert consensus. We examined the overlap between the Commission's proposed products and those that our modelling suggested would still be missing (the 18 missing products are noted in Table 5). For each missing product, we used the P2I v.2 model in a retrospective manner to estimate the number of additional candidates that would be needed at preclinical phase-over and above the existing candidates-to lead to one expected launch of that product, and the associated additional cost. For example, as shown in the Results section, we found 41 HIV vaccine candidates in the pipeline and the modelling suggested that these would result in 0.49 launches. Thus, to estimate the additional costs to reach one launch, we estimated the number of additional candidates needed at preclinical phase and the associated additional costs to achieve an additional 0.51 launches (in this case, an additional 125 candidates would be needed at preclinical phase to achieve 0.51 launches, at an additional cost of $2.8 billion).
(vi) Sensitivity analysis As a final step, we conducted a sensitivity analysis, adopting an approach proposed by Mestre-Ferrandiz et al. at the United Kingdom Office of Health Economics in their study, "The R&D Cost of a New Medicine." We examined the impact of changing all probabilities of success per phase to 10% higher and 10% lower, and all costs per phase to 10% higher and lower. We also examined the impact of all possible combinations of these changes (e.g., 10% higher probability of success per phase and a 10% higher cost per phase, 10% higher probability of success per phase and a 10% lower cost per phase, etc.). We conducted this sensitivity analysis both for moving current candidates through the pipeline and for the costs of priority "missing" products. We did not conduct a sensitivity analysis varying the length of time per phase, because in the P2I model the length of time is independent of the cost variables (the cost parameters are per phase, not per year). We also conducted a sensitivity analysis in which we used only the assumptions from the P2I v.1 model, which helps to show the impacts of the modifications that we made to P2I v.1 when we developed P2I v.2.
While the approach of changing product development assumptions by 10% higher and lower has been used in previous studies, we recognize that achieving these changes-particularly reducing the attrition rates-may not be realistic. As Paul et al. note: "There is little doubt that reducing the attrition rate of drug candidates in clinical development represents the greatest challenge and opportunity for pharmaceutical R&D, and arguably for sustaining the viability of the entire industry 16 ." Reducing costs may be more feasible, e.g. through simpler trials or improved trial management. Nevertheless, we included a sensitivity analysis to show the impact of changing the underlying assumptions on our estimates.

Results
The pipeline of candidates for neglected diseases In the pipeline portfolio review, we found 685 product candidates for neglected diseases as of August 31, 2017. We excluded 147 of these from the model because (a) there was insufficient information about their development phase, (b) they were already marketed, or in a development phase that is excluded from the P2I.v2 model, or (c) there was insufficient information about the candidate to be able to classify it into an archetype. After exclusion, 538 candidates were included in the model.
Supplementary File 2 gives detailed information on these 538 candidates, showing (a) the candidate's name, (b) the health area that it targets, (c) the development phase that it was in at the time this study was conducted, (d) the archetype to which we assigned the candidate, (e) relevant source material that guided the classification, and (f) the sponsors and collaborators conducting the study of this candidate 17 . Figure 2 shows the breakdown of candidates by the archetypes used in P2I v.2 (208 vaccines, 108 NCEs, 101 diagnostics, 90 repurposed drugs, 16 vector control products, and 15 biologics) and Figure 3 shows the breakdown by disease/condition. The pipeline is dominated by three diseases-malaria (109 candidates), HIV/AIDS (99 candidates), and tuberculosis (98 candidates)-which comprise nearly 6 out of 10 (57%) of all candidates. About 1 in 10 candidates (11%) are for reproductive health needs in developing countries. Table 6 shows the breakdown of candidates by disease and archetype for the 10 diseases/conditions that have the most candidates. For several health areas-particularly neglected tropical diseases-there were only one or two candidates (one candidate for cryptococcal meningitis, giardiasis, leptospirosis, multiple diarrheal diseases, multiple vector-borne diseases, trichuriasis, and two candidates for hookworm, leprosy, lymphatic filariasis, meningitis, rheumatic fever, and trachoma).
Costs to move candidates through the pipeline Based on inputting these 538 candidates into the P2I.v2 model, the modelling suggests that it would cost about $16.3B over about 10-12 years to move all of these candidates through the pipeline (Table 7). Supplementary File 3 gives detailed information on the breakdown of these estimated costs 18 . As shown in Table 7, in many cases the overall costs per phase are higher for the simpler archetype (e.g., the costs in all phases are higher for simple vaccines than complex vaccines); this finding simply reflects the fact that there are more candidates of simple complexity in the pipeline (see Figure 2 and Supplementary File 2 17 ).
Over half of these estimated costs (55%, $8.9B) would be for Phase III clinical trials, one third (33%, $5.3B) for Phase II trials, just under one tenth (9%, $1.4B) for preclinical development and the remainder (4%, $0.7B) for phase I trials (the percentages do not add up to 100%, since they have each been rounded). About three quarters of the costs (76%, $12.4B) would be incurred in the next 5 years assuming that all candidates are taken through all stages ( Figure 4); in reality, of course, there would be a "go/no go review" at each stage gate. The "front loading" of costs over the first five years reflects the large number of candidates at early phases and the subsequent attrition through each phase.

Expected product launches
Based on public domain information on the current portfolio of candidates for neglected diseases, and the assumptions of success at each phase for the different archetypes included in the P2I v.2 model, 128 product launches would be expected (this figure is based on rounding down the number of launches for each disease archetype; Supplementary File 1 shows the results without any rounding and also with rounding to the nearest integer) 15 . The dominant product type for the anticipated launches would be diagnostics, which would make up almost 6 in 10 expected launches (57%, expected 73 launches), followed by repurposed drugs (13%, 16 expected launches) and simple NCEs (13%, 16 expected launches).
As shown in Figure 6, the model estimates that just over one quarter (27%)   The modelling suggests that there would be very few launches of complex vaccines (only 2). It also suggests that launches of vaccines for HIV, TB, or malaria and launches of complex NCEs would be unlikely.
Supplementary File 3 18 gives detailed information on the portfolio of anticipated product launches, broken down by disease and archetype.
Sensitivity analysis: moving current candidates through pipeline Table 9 shows the results of the sensitivity analysis, in which we examined the impact on costs and expected product launches of changing all probabilities of success per phase to 10% higher and 10% lower, all costs per phase to 10% higher and lower, and all possible combinations of these changes. The sensitivity analysis found that the total costs to move the current pipeline through to production range from $13.4B-19.8B and the anticipated launches range from 89-160.
The second sensitivity analysis that we conducted in which we used only the assumptions of P2I v.1 found that the total costs to move the current pipeline through to production would be $13.4B, with 151 anticipated launches.   Estimates of the costs of "missing" products As shown in Supplementary File 3, based on the current known pipeline (i.e., candidates in the public domain), there are unlikely to be product launches for several diseases and product types 18 . A comparison of these "missing" products with the list of needed priority products proposed by the Commission on Investing in Health highlights 18 missing products. As shown in      Table 10 estimates the number of candidates needed at the preclinical phase, development cost, and development time to launch one product, for each different archetype, assuming an "empty" pipeline (i.e., starting product development from scratch). However, the pipeline for these 18 missing products is not currently empty; Table 11 shows the number of additional candidates that would be needed at preclinical phase-over and above the existing candidates-to lead to one expected launch of that product, and the associated additional cost. Table 11 includes the results of the sensitivity analysis (changing all probabilities of success per phase to 10% higher and 10% lower, all costs per phase to 10% higher and lower, and all possible combinations of these changes).
The total estimated additional costs (over and above the costs to move current candidates through the pipeline) to reach one estimated launch of each of these 18 missing products ranges from $13.6B to $21.8B over 10-12 years, depending on the complexity of the products (Table 11). Around three-quarters (75%) of the costs are likely to be incurred over the first 5 years, i.e., $10.3B-16.6B. The sensitivity analysis found that the total additional costs range from $8.1B-36.6B.

Discussion
Our study, based on data in the public domain, found 685 product candidates for neglected diseases as of August 31, 2017, of which 538 fitted archetype descriptions and could be entered into a portfolio costing model, P2I v.2. The pipeline is dominated by product candidates for HIV, TB, and malaria; of the candidates included in the model, almost 6 in 10 (57%) targeted these three diseases.
The dominance of these three diseases when it comes to product candidates is in alignment with the proportion of funding for neglected disease product development that is directed at HIV, TB, and malaria. In the 2017 G-FINDER report, which analyzes financing data for 2016, out of a total of $3.2B invested in neglected disease product development, 70% ($2.2B) was targeted at these three diseases (G-FINDER calls these "tier one diseases," as they are in the top tier of funding) (see G-FINDER report for 2017).
In contrast, for several diseases there were just one or two candidates, reflecting much lower levels of R&D funding. For example, G-FINDER notes that a number of diseases are in the bottom funding tier ("tier three"), meaning that they each receive less than 0.5% of global funding for neglected disease product development. We found very few candidates for these diseases, e.g. just one each for cryptococcal meningitis and leptospirosis and two each for leprosy, rheumatic fever, and trachoma. The proportion of total funding directed at each disease is poorly correlated with its overall disease burden 19 . Indeed, several high burden diseases receive very little R&D funding. For example, the Global Burden of Disease Study 2016 estimated that there were about 450 million people with hookworm in 2016 20 , yet it received just $3.87 million in funding for product development in that year (see G-FINDER report for 2017). In their analysis of the relationship between R&D innovation and disease burden, Barrenho et al. found that for neglected tropical diseases, "innovation is disproportionately concentrated in low burden diseases." The model suggests that moving all 538 candidates through the pipeline from late stage preclinical to launch (end of phase III clinical trials) would cost an estimated $16.3B, of which around $12.4B would be spent in the first 5 years (on average $2.5B per annum). Given this level of investment, we would expect about 128 product launches, two-thirds of which (66%) would be for HIV, TB, and malaria.
By far the largest number of launches would be for diagnostics, which are likely to make up almost 6 in 10 expected launches (57%, expected 73 launches). This high number of estimated diagnostic launches is at least partly explained by the underlying model assumptions. In particular, in the P2I v.2 model, the success rate in the design and development phase is 100% and 75%, respectively, for assay development and simple platform development, and it is 100% at the clinical validation and launch readiness phase for both assay and simple platform development. To the best of our knowledge, there have been no peer-reviewed, published studies on attrition rates through the pipeline for diagnostic development. Nevertheless, the success rates used in P2I v.2 may be overly optimistic.
Our study suggests that the current pipeline is unlikely to produce several critically needed technologies: highly efficacious vaccines against HIV, TB, malaria, and hepatitis C; a combined vaccine against multiple diarrheal diseases; a complex NCE for TB; and NCEs for 12 NTDs. This finding underscores the need for substantially scaled up resources and innovative development approaches to fill these gaps. Using the P2I v.2 model, our study estimates that the additional total cost to launch one of each of 18 prioritized "missing products" ranges from $13.6B to $21.8B, depending on the complexity of the products. Of these additional costs, about $10.3B-16.6B would be spent in the first 5 years (an annualized average of $2B-3.3B).
Thus, overall, in the first 5 years, total estimated costs to (a) move all current candidates through the pipeline and (b) develop these 18 priority missing products would be around $4.5-5.8B per year. We recognize that these two estimates were generated in somewhat different ways and so it may not be appropriate to combine them. The former was derived by "forward induction," taking the current pipeline of candidates and projecting forwards. The latter was derived by "backwards induction," estimating the cost of a scenario in which additional candidates are available at the pre-clinical stage-this scenario assumes additional funds will be targeted at these "missing products." Nevertheless, we believe it is helpful to provide an aggregate estimate as it gives an idea of the overall funding gap for neglected disease product development.
How do these aggregate estimated costs compare with current spending on product development for neglected diseases? The annual G-FINDER surveys have found that the annual spending since 2008 has been around $3B, suggesting that the funding gap is at least $1.5-2.8B. There are several reasons why this is likely to under estimate the total funding need, including (i) in all likelihood, there are additional candidates in the pipeline that we did not capture in our study (because there is no information about them in the public domain), (ii) we only estimated funding needs for 18 high priority "missing products," not all missing products, and (iii) as discussed below, not all costs are included.
Closing this large financing gap will require a major effort to mobilize new resources from across the public, philanthropic, and private sectors. High-income governments have been the most important source to date for financing product development for neglected diseases, but they are arguably under-investing in such research given the very large health, social, and economic returns to investment. Findings from the G-FINDER surveys suggest that middle-income countries are under-performing in terms of their overall contribution to R&D for neglected diseases, given their economic capacity and their burden of disease. The Brookings Institution's Private Sector Global Health R&D Project has proposed ways in which private sector investments could be stimulated, such as through advanced market commitments for hookworm and schistosomiasis vaccines. Yamey and colleagues recently proposed a number of other strategies that could potentially help to close the product development funding gap 21 . These include a health investors' platform "to inform public, private, and philanthropic investorsand attract new investors-to fund those candidate products likely to have the largest public health benefits" and a new type of matching fund that pairs global and national resources for shared R&D priorities.

Strengths of the study
Our study has several strengths, and we highlight two in particular. The first is its novelty. To the best of our knowledge, our study is the first to estimate the costs of global health product development from preclinical to the end of phase III based on the existing portfolio of candidates across multiple neglected diseases, and the first study to use and adapt the P2I v.1 model for this purpose. This approach complements earlier efforts to model a single therapeutic portfolio 22 . Our approach of costing a portfolio of candidates using the current pipeline as a basis adds a different dimension to the field of global health R&D costing, and one that aligns with the way in which funders pursue a diversified portfolio of product development projects.
The value of the model is in the ability to estimate costs and probable launches based on a portfolio of candidates. The model is much less reliable when it comes to predicting what will happen to any one specific candidate-in other words, it is prone to the ecological fallacy (making inferences about a single candidate based on data from across a portfolio of candidates).
Second, by moving beyond the costing of individual product types for specific diseases (e.g., costing only an HIV vaccine or a TB diagnostic), our study has shown more broadly-across the portfolio of neglected diseases-where the pipeline is most robust, where it is lacking, which product launches are most likely, and which products will probably still be missing based on existing candidates. For global health R&D advocates, this broad picture could potentially help to highlight critical funding and product development gaps.

Strengths of the P2I tool
We highlight two strengths of the tool itself. First, the P2I v.1 tool that we used and adapted is available online, as are our model assumptions, model inputs and outputs, and detailed information on the portfolio review, which means that readers can replicate, improve on, and further adapt our work 12 . The P2I v.1 tool was designed for flexibility, as shown in the way in which we adapted it, and we encourage others to refine it further. In particular, we hope that those who have access to updated, high quality data on costs, attrition rates, and cycle times will share and contribute these data to further iterations of the model. All R&D cost modelling exercises are, of course, inherently uncertain, but the steps we have taken will, we believe, allow others to "stress test" our work. In the accompanying study on the development of the P2I tool 12 , we describe how the tool can be used to estimate health impact. We note: "The P2I model allows users to estimate the impact of a launched product on both disability, measured in disability-adjusted life years (DALYs) averted, and mortality, measured in deaths averted." It would thus be possible to estimate the economic value of these health impacts, and therefore to estimate the rates of return on portfolio investment.
Second, most of the model assumptions were based on a large number of data points (e.g., assumptions on success rates and cycle times were based on data from of 25,000 development candidates), and were validated through examining peer-reviewed literature, industry reports/databases, and expert interviews. While no assumptions used in R&D cost models can ever be perfect, we believe that the process for developing the assumptions was "robust enough" to give realistic, real world benchmarks for costs, success rates, and cycle times per phase.
While a detailed discussion of the literature on clinical development success rates for investigational drugs is beyond the scope of this paper, our assumptions appear to be roughly in line with reported industry standards (the amount of variation differs by product types) 23

Limitations of the study
Our study has a number of limitations. First, the pipeline portfolio review only provided a snapshot at a single point in time.
The pipeline is constantly changing, and between the end of our pipeline review (August 31, 2017) and today, it has already changed.
Second, our review is probably incomplete, given the lack of publicly available information on some products under development. It is particularly challenging to find information about candidates that are at the pre-clinical research phase, since studies at this phase are not included in clinical trial registries. Unfortunately, proprietary interests and non-disclosure agreements mean that we have no knowledge at all of some candidates under development. While we did our best to gain as full a picture of the pipeline as possible by using a variety of methods-such as searching databases and interviewing product developers-it is likely that we missed some candidates. Using a different search strategy or searching further databases could have identified additional candidates, such as anti-bacterial products under development (though only certain anti-bacterial candidates would qualify for the strict inclusion criteria in our study). For all these reasons, and as mentioned above, our model probably under-estimates the total costs and number of product launches. We call on the global health research community to commit to making its product development research more transparent.
A third, related limitation is that for some candidates, there is very little information on their development phase. We found it particularly challenging to obtain public data, even in trial registries, on the development phase for candidate MPTs, diagnostics, and vector control products. In some cases, we had to make a judgment call based on whatever data we could find.
A fourth limitation relates to trial capacity. We modelled the costs to launch a number of critical "missing" products-but the model does not account for whether there is the actual trial capacity to conduct the additional studies that would be needed should the additional funding be mobilized.
A fifth limitation relates to the modifications that we made to the P2I v.1 tool to create P2I v.2. As described in the Methods section, these modifications were made using a variety of data sources (see Table 2), which introduced some variation in the strength of the evidence underlying the adaptations. For example, the assumptions on probabilities of success for unprecedented vaccines in phases II and III were based on a relatively small number of data points-around 10-25 data points per estimated value. As we note in the methods section, in P2I v.2, the adjustments made to the assumptions in P2I v.1 for biologics were derived from cautious expert judgement based on early industry trends. To assess the impact of the adaptations, we conducted a sensitivity analysis in which we ran the model using only the P2I v.1 assumptions. This resulted in a lower cost estimate for moving existing candidates through the pipeline ($13.4B using P2I v.1 assumptions versus $16.3B using P2I v.2 assumptions) and a higher estimate of expected launches (151 vs. 128).

Limitations of the P2I modelling tool
As with all modelling tools, the P2I tool that we used in this study has several limitations. First, the model is deterministic and static. It does not take into account possible improvements in product development techniques over time, such as reductions in cycle time or lowering of development costs. Historical evidence suggests that factors such as attrition rates and costs per phase do change over time, and the P2I tool does not capture such evolutions over time 23 . We tried to address this limitation through our sensitivity analysis, but this does not fully capture the uncertainties surrounding the model assumptions. The P2I v.1 and v.2 models also do not account for down selection, in which portfolio managers decide at various stages of the development process to drop certain candidates. By examining only the current, static portfolio of candidates-and costing just these-our study does not account for future candidates that will enter the pipeline. It also does not address the minimum specifications that a product may need to meet. For instance, although the current TB diagnostic pipeline is forecast to result in several new approved diagnostics, there is no guarantee that these products would be fit-for-purpose, or offer a significant improvement over current tools.
Second, the P2I v.2 model requires users to classify every candidate into an archetype, but categorizing candidates based on the archetype definitions was challenging-especially determining a candidate's complexity. It will be helpful for future iterations of the model to include more fine-grained, detailed descriptions.
Third, the assumptions on costs, success rates, and cycle times per phase for each archetype were based on taking averages from a large number of data points provided by industry and PDPs across a range of diseases. In the model, the averages for each archetype are the same for every disease. For example, the costs, success rates, and cycle times per phase for developing a simple vaccine for schistosomiasis are the same as those for developing a simple vaccine for hepatitis C. Thus the model does not reflect differences that there may be between diseases when it comes to product development cost structures, success rates, and cycle times. In addition, the data that we used to develop the assumptions, provided by industry and PDPs, were not just on neglected diseases-they were on development candidates for multiple infectious and noncommunicable diseases, including cancer therapies. Pooling from across a huge number of data points is likely to have led to more robust assumptions, but the inclusion of data from all different types of products (e.g., cancer drugs) may have led to estimates that do not reflect neglected disease product development alone. Furthermore, for some archetypes (e.g. biologics, diagnostics, vector control products) there were few available data on development costs and success rates to inform the model assumptions; greater availability of such data would be helpful for further refining the P2I model. However, we note that these are commercial estimates and many costs-such as the location of production-are strategic in nature and so it is much harder to derive an average cost.
In the area of neglected disease R&D, the PDPs challenge these commercial costs and their own cost estimates are much lower. For example, DNDi, the Drugs for Neglected Diseases initiative, a non-profit PDP, estimates that it has spent about $39-52 million per NCE, a figure that adjusts upwards to $130-195 million when risk of failure is taken into account.
Most of the supporting basic research and pre-clinical research for neglected disease R&D is publically funded as a public good and may serve many different products. Therefore, trying to develop a meaningful average cost between for-profit and not-for-profit in the later development stages was not feasible and such an average was purposefully excluded in the P2I tool. Given all of these excluded costs, the P2I model was never intended to provide a full price tag; instead, the aim was to offer the first tool to provide an evidence-based method for R&D cost comparisons where there are historical data to draw on. The intention is to inform prioritization and decision making at a global level and not to price individual product development.
It is worth repeating that the P2I v.1 and v.2 models examine the total pipeline of all candidates, with no judgements made on "go/no go" at each stage gate. In reality a portfolio would be managed and ranked with only the most promising candidates likely to be moved on to a subsequent phase. The type of decision required would be based on potential public health impact and feasibility of success. As these are products that are aimed at poor populations, where the market is largely absent, other considerations such as access and affordability will also influence the decisions on a go/no go basis 11 . Fifth, the model is "agnostic" when it comes to the public health value of the estimated launches-it cannot judge their clinical utility. For example, the model estimates that there could be 27 TB diagnostics (just over a fifth of all launches). This is the type of scenario that the model was intended to highlight, in order to stimulate debate about what is an appropriate approach moving forwards. Despite this high number of launches, the large unmet need for TB diagnostics may still not be addressed as almost all new developments are of an incremental rather than transformational nature. While some of the incremental developments in the pipeline might be of benefit to address different needs in different epidemiological and geographic settings, others might be redundancies. In addition, the quality and performance of the launched assays may vary widely, and therefore they may not meet public health needs. And, as mentioned, this large estimate may also reflect the underlying assumptions of high success rates in both the design and development phase and the clinical validation and launch readiness phase.

Conclusions
This study has shown that the P2I v.1 tool is flexible enough to be adapted and used to estimate the costs and probable launches associated with moving a portfolio of current candidates for neglected diseases through the pipeline. It has pointed to gaps in the pipeline, which can be valuable in directing and prioritizing future R&D financing. It has also given an indication of the size of the financing gap, which can be helpful for future resource mobilization. Our recommendation is that other interested parties explore the use of the P2I v.1 and see how they can adapt it to create their own scenarios and share their results so that at a global level we can improve the process of supporting R&D for neglected diseases.

Data availability
All data underlying the results are available as part of the article and no additional source data are required.

Mikel Berdud
Office of Health Economics, London, UK Jorge Mestre-Ferrandiz Independent Consultant, London, UK The research article by Young and colleagues, entitled Developing new health technologies for neglected , is a research estimating; (i) the funding requirements diseases: a pipeline portfolio review and cost model to move products -currently at development stages -for Neglected Tropical Diseases (NTDs) through the pipeline until market launch, and (ii) the funding required to increase the R&D effort in order to ensure at least one launch for each of 18 missing priority products ranges for NTDs of greatest need. Resulting from adding up these two estimates, authors present a headline figure of $4.5B-$5.8B/year of funding requirement over the next 5 years, which is around $1.5B-$2.8B larger than the actual funding ($3B/year).
It is an original piece of work which represents the first attempt to estimate the R&D cost required specifically for a global health strategy aimed at tackling NTDs and meet the health targets included in the UN Sustainable Development Goals (SDGs) adopted in 2015. Its originality resides in using the methodology to estimate the R&D cost of a new drug validated by the literature to estimate the cost of moving the current portfolio of candidate drugs for NTDs through the pipeline. To our knowledge, this is the first time that such empirical approach has been used to estimate the R&D investment required for global health, which typically has been estimated by comparing historical data of global R&D expenditure for NTDs with its achievements in terms of health burden and health gains.
Research is also relevant mainly because, in a context where the R&D expenditure for NTDs has been declining steadily, it helps understand governments, policy makers, philanthropic donors and industry, not only how much additional investment effort must be done in the following years to achieve UN global health objectives, but also in which diseases (and health technologies) the effort must be specifically increased and by how much -although this latter is only partially addressed as it requires analyses on the return rate to be performed.
In general, we feel methods and data are appropriate. The estimate is based on a final sample of 538 product candidates covering 35 NTDs which were selected from a wider sample of 685 candidates after applying inclusion criteria. Cost estimates are calculated by using an adapted version of Portfolio to Impact financial modelling tool (P2I v.2). Adaptations included more product types and changes over assumptions to tailor it to current portfolio for NTDs. This is also an outcome of this research which opens the door to further research although adaptations should be challenged, discussed and revised. All data assumptions to tailor it to current portfolio for NTDs. This is also an outcome of this research which opens the door to further research although adaptations should be challenged, discussed and revised. All data are publicly available, as well as the P2I model, so interested readers could, at least in theory, replicate results (we have not done it).
We have listed up to 11 comments. Our first general comment relates to references and placing this analysis relative to previous work. Whilst literature has been quite productive estimating the R&D cost of a single new drug, authors only cite the work by DiMasi et al. (2016) . We encourage authors to read in detail the work by Schuhmacher et al. (2016) which reviews all the relevant literature on the topic, including the literature estimating the R&D cost, development phases success rates and development times. We consider that, the reading of this work will give the authors a deeper knowledge of probabilities of success, time intervals and R&D cost of R&D phases including drug-discovery, pre-clinical and regulatory approval. The cost of the drug-discovery and the regulatory approval stages are missed in the analysis with the latter not even mentioned. This issue is addressed below. Exploring the latest estimates released in the literature, authors would be able to compare them with the ones used for the modelling in the present work for validation purposes and eventually for filling potential gaps still remaining in the analysis. Additionally, we would encourage authors too, to read Hay et al. Our second general comment is regarding the modelling adaptations and insufficiencies. The model estimates the R&D cost in a validated way, but the analysis missed the cost of basic research, drug-discovery and, most importantly, the cost of regulatory review submission and marketing authorization that is typically included as an additional phase in the relevant literature. Authors explain why they have not been able to estimate the cost of basic research and drug-discovery stages and justify their exclusion using the argument that they are underestimating the true cost. It has been shown (Di Masi et al (2016), Mestre-Ferrandiz et al. (2012)) that pre-clinical costs can be a sizeable share of total cost, so it would be nice to have some discussion about how big, or small, these are. The underestimation argument might be also used to exclude regulatory approval cost, in case that such costs were not possible to estimate, but authors omit any explicit mention to the reason why they have not been included in the analysis. As the cost of regulatory approval stage is "standard" for all archetypes of drugs and diseases, it should not be very complex to estimate. Consequently, authors should consider its possible inclusion -or at least its omission should be explained further -as it may represent up to a 5.7% of the total R&D cost (Schumacher et al., 2016).
Third, we feel the issue of "opportunity cost" / "cost of capital" is only discussed very briefly. This is an important issue, as raised by Mestre-Ferrandiz et al. (2012), as it can have a significant impact on the total cost of R&D. The authors, when commenting the estimate of the R&D cost of a new drug published by DiMasi et al (2016), argue that "[…] the estimate can be criticized for, among other things, including $1.2B in "time costs" (the expected returns that private investors forgo while a drug is in development)." We feel this assertion needs further discussion. We will not question whether $1.2B of opportunity cost is too much or not, but under our view, not incorporating the cost of capital into the present analysis is one of the weaknesses of this research. Mostly because the whole literature is aligned to include it to some extent. Is the measure of the cost of capital and not its inclusion which is subject to debate. The fourth general point we want to make is that we are also concerned by the lack of specificity of the information used to make assumptions. Assumed probabilities of success, development times and phase specific costs for the costing modelling are not always specific for new drugs developed for NTDs. They are based on mixed sources as Table 2. shows. The majority, highlighted in orange, come from P2I v.1 modeling and are not specific for candidates for NTDs. Other, highlighted in blue and yellow, come from Risk-Adjusted Portfolio Model and B&M Gates Foundation in-house data. When comparing assumptions coming from different sources between complex and unprecedented vaccines (a new archetype), only the probability of success for phase 2 and 3 are different, being the rest of assumptions exactly equal. Authors should explain better why the reminder assumptions imply the same values for the parameters, even though they come from different sources and are based on different samples. Some assessment of why phase 2 probability of success for unprecedented vaccines is as low as 5% should be also provided. It seems to us to be very low and although the number of datapoints used for its estimation is shown in the paper, such number of datapoints result unmeaningful as reader is not aware about the number of datapoints used for the same estimate in the case of complex vaccines. The same applies -although to a lower extent -to the phase 3 probability of success. In addition, all assumptions for simple and complex repurposed drugs are equal although many of them come from different data sources too. We wonder here how sensible is to assume that for pre-clinical and phase 1 stages both have the same costs when simple repurposed drugs are assumed to have enough safety data to start development in phase 2, while complex repurposed drugs require some phase 1 clinical trials (see table 3). The reader could be confused with assumptions in table 2 and, beyond the explained differences in data sources, the article does not make clear neither the differences/equalities in assumptions coming from different sources nor their impact on the results. Fifth, the authors perform a sensitivity analysis to assess impact of assumptions on results. They use the same approach as Mestre-Ferrandiz et al. (2012) which consists in decrease/increase by 10% both, probabilities of success and times of development phases. Such approach allows to perform four combinations. We will not question the appropriateness of performing the same sensitivities for the present work, although it would be useful to explain how realistic it is to achieve the 10% change for each of the variables (for example, achieving a 10% reduction in time might be very complicated, but could be achieved for phase 3 attrition rate). In addition, we want to point out that sensitivities are only presented for the cost of moving product candidates through the pipeline (at least this is what the paper seems to present in page 20). Sensitivities also affect the number of additional products required per phase to ensure one launch for the 18 priority products, as well as their cost per development phase. To our understanding, such sensitivity is not presented in the paper and its impact seems to be significant on the headline figure of the paper ($4.5B-$5.8B). Why authors do not present these sensitivities should be explained or they should be included. Another sensitivity missed in the paper is one assessing impact of time of development assumptions, which is due to neglecting the cost of capital from the analysis -as mentioned above. Authors should inevitably check how much weight these assumptions have on the cost estimate, but they neglect to perform such sensitivity analysis. Finally, less important but worth to point out is that authors may consider presenting the estimate of the cost resulting by applying only the assumptions in the P2I v.1 model. The resulting figure should be compared with the one obtained by introducing the other assumptions in the modeling (i.e. RAP, BMGF).
Sixth, the total resulting cost -the headline figure of the article -comes from adding figures from two sources: (i) the cost of moving actual current portfolio for NTDs through the pipeline and, (ii) the increased portfolio needed to ensure at least one launch for 18 priority products for NTDs. Approaches used to estimate each are opposed one to the other. While the former consists in projecting the reality (forecasting the true cost by forward induction), the latter consists in estimating the cost of a hypothetical case (forecasting by backward induction). The two estimates are presented separately. As readers we case (forecasting by backward induction). The two estimates are presented separately. As readers we have been able to separate them clearly along the paper. However, the headline figure, the one compared to actual funding (G-FINDER) to assess the funding gap, results from adding up the two separate estimates. Question is, how sensible is to add up these to figures? For the 18 priority products, the approach assumes that increased funding will increase the number of product candidates, but additional funds may be driven to other candidates with larger expected returns. The fact that reality is showing scarcity of investment in these priority areas is showing the necessity for other incentives rather than just funding. This should be discussed more in depth in the paper, as well as authors raising this caveat about the interpretation of the aggregated figure.
Seventh, as authors recognise, cost modelling is static. It means that assumptions over the drug development productivity (i.e. portfolio composition, costs of development phases, probabilities of success) remain unchanged along time. It is a strong assumption, as suggested by the literature (for instance, Schuhmacher et al., 2016). Attrition rates, time of development and costs per phase have been changing during the last decades due to technology changes, biomedical science advance, etc. New candidates are also expected to enter in the portfolio. Such constant evolution of the drug R&D landscape will affect the costs and funding required to tackle NTDs. One possible way to assess this impact is to analyse the historical evolution of attrition rates, time of evolution and cost of phase of development, estimate the trend and forecast to where they are going so the assumptions in the modelling could be calibrated taking into account such dynamic behavior of the pharmaceutical R&D process and technology.
Eight, if we take a simplistic approach to estimate the cost of one product, we understand that for the $16.3B there are 128 products launched (over the next 10-12 years). Dividing one by the other, it gives $127m. Applying the same algebra for the 18 "missing" products gives a figure of circa $1bn (total cost of $13.6bn and $21.8bn). Costs differences are very significant, so would be interesting to understand in more detail the reasons for such (big) differences.
Nine, it would be nice to see the expected product launches, year by year, to understand the evolution. Interestingly, for "missing" products, the authors say that around 75% of the costs are likely to be incurred over the first 5 years -why is that?
Ten, there is a very nice analysis done by Barrenho and colleagues in Imperial College Business School, on "Does global drug innovation correspond to burden of disease? The neglected diseases in developed and developing countries". The authors should see how their work differs. The paper can be found here: . https://spiral.imperial.ac.uk/handle/10044/1/51581 Finally, and as some further research, it would be nice to understand from the authors whether some modelling could be done, based on this analysis, to estimate the rate of return of the R&D funding, and whether it differs by archetype, for instance. This information could be very useful for policy makers to help them identify how best to spend the extra dollar on new drugs for NTDs. As an illustration, some analyses can be found in: Buxton et al., 2008, Glover et al, 2014.
Other minor points worth mention are: It would be nice to do further analysis on the products that have been put on hold indefinitely and explore reasons for that.
Sometimes authors justify assumptions n expert views and/or assessments, but these are not 5 6 of innovation and incentives to encourage medical research and development; the economics of antimicrobial resistance (AMR) We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.
lower cost estimate for moving existing candidates through the pipeline ($13.4B using P2I v.1 assumptions versus $16.3B using P2I v.2 assumptions) and a higher estimate of expected launches (151 vs. 128)." With respect to having the same assumptions for simple and complex repurposed drugs, this reflects an attempt to be as "conservative" as possible when it comes to costs. See also point 5 below-we have now conducted a sensitivity analysis in which we use only the assumptions from the P2I.v1 model, which helps to show the impacts of the P2I v.2 modifications based on RAP and BMGF data.

5.
: We have now conducted two new sensitivity Conducting additional sensitivity analyses analyses suggested by the referees, which are described in the revised Methods and presented in the revised Results: (a) A sensitivity analysis for the 18 "missing" products, including both the additional number of candidates needed at preclinical phase and the additional investment needed to achieve one launch. (b) A sensitivity analysis in which we use only the assumptions from the P2I.v1 model.
We did not conduct a sensitivity analysis varying the length of time per phase, because in the P2I model the length of time is independent of the cost variables (the cost parameters are per phase, not per year)-we now mention this in the Methods section (under sensitivity analysis). In response to the referees' suggestion to reflect how realistic it would be to change our assumptions by 10% higher or lower, we now briefly mention this in the section on sensitivity analysis and cite Paul et al's work. We would also recommend that this tool is intended to generate different scenarios by adjusting these variables. There is no single answer but a set of outcomes within a range. This is based on historical data and will change over time as new data are added.

The appropriateness of combining the two different estimates (current pipeline versus
we now discuss the pros and cons of aggregating these two estimates in the missing products): revised Discussion section.

7.
: we have expanded our discussion of this in the The static nature of the cost modeling revised section on the "Limitations of the P2I tool." 8.
: the referees suggest taking a "simplistic approach" to estimate the cost of Costs per candidate one product, by simply dividing the 'portfolio cost' by the anticipated launches (i.e., $16.3B divided by 128). As mentioned in our response to other reviewers, we do not feel this is appropriate since the P2I tool is a portfolio financial modeling tool-it is not intended to model individual product costs.

9.
we are unable to produce this illustration/figure Visualizing product launches year by year: because of the nature of the rounding. As we state in the paper: "For this paper, we have chosen a conservative approach to presenting the launches-we have considered a launch to be a binary event, i.e., we have always rounded down (in this case, 2.7 rounds down to 2 launches)." We did the rounding at the end of the modeling (rather than at each phase of the model), which precludes us from showing launches over time. The referees ask why 75% of costs are incurred in the first 5 years-we now address this in the revision (it reflects the large number of candidates at early phases and the subsequent whittling down through each phase). In the revised Results section, we now state: "The "front loading" of costs over the first five years reflects the large number of candidates at early phases and the subsequent attrition through each phase." candidates at early phases and the subsequent attrition through each phase."

10.
: we now mention and cite this study in our revised Citing the Barrenho study et al. Discussion. 11.
: while we are unable to do this Using the model to estimate the rate of return on R&D analysis for the current paper, the model could be used in this way. As mentioned in our response to Referee Report 1, we now discuss this in the revised paper.
The referees list 4 minor points at the end of their report: They suggest conducting an analysis on the products that were put on hold indefinitely, exploring reasons why they were put on hold-this future analysis would be interesting, but is outside the scope of the current project. with several questions: (1) How do these rows compare with the published literature? (2) What was the influence of the stakeholders in revising the parameters from the published literature reference point? (3) Did the stakeholders move the parameters in any particular direction (yielding greater or lesser expense and risk)? These parameters drive the results and how they were achieved is important. It appears that a similar model was used within the Bill & Melinda Gates Foundation to evaluate product development partnerships supported by the Foundation. If so, it would be good to know explicitly how the model was honed in response to actual experience.
Supplemental File 1 lists the neglected diseases studied and the expected R&D outcomes based on the model. I don't have the expertise to comment on most of these categories, but the numbers related to bacterial diseases seemed too low to me. For example, the model predicted no drugs, diagnostics or vaccines targeting multiple diarrhoeal diseases and just two vaccines (but no drugs or diagnostics) for Enterotoxigenic E.coli (ETEC). I suspected that the model input data might have missed existing clinical and preclinical projects that could conceivably target these diseases, from antibiotic R&D sources that have not traditionally been included in neglected disease databases. Examining Supplemental File 2 allowed me to spot check whether companies known to me to be pursuing such work were listed. (The CARB-X portfolio is listed at ). I suspect that the authors' data sources were heavily www.carb-x.org weighted towards the neglected diseases community and did not include more traditional antibacterial projects that are generally Type I or Type II diseases. If my hypothesis is correct, then some relatively modest changes in the results would probably occur: the researchers would find additional projects that required financial support in order to reach the market and therefore fewer entirely missing programs for the target of new classes of antibiotics and prevention against bacterial diseases.
In future articles, I hope for updates from these authors, comparing the model predictions with actual results with regard to the parameters in Table 2, and calling out specific updates in light of actual experience. This is valuable work.
Another portfolio-level effect might be added to a future model. A single project, progressing alone might have certain parameters on cost, time and attrition. How does that change when they are not alone, but part of a cohort of projects, often from the same group of funders, such as charitable foundations? I hypothesize that if the funders coordinate open sharing of data and information, the cohort should be more efficient, but it is also possible that groupthink could send entire research portfolios down unproductive pathways or that rapid scale up could constrain available human and technical resources, raising costs and delaying timelines. I also note the ongoing controversies over the cost of a new drug. Could the authors use these parameters to reverse engineer the Tufts study and arrive at a per drug cost estimate? Would be an interesting comparison to this model. A minor typo on Table 5: should it be "POC load for HIV?" viral Thank you for the opportunity to review this article.

Is the study design appropriate and is the work technically sound? Yes
Are sufficient details of methods and analysis provided to allow replication by others? Yes industry R&D cost sourcebook (e.g., assumptions on attrition rates and cycle times at each phase were initially based on a review of the attrition rates and cycle times of more than 25,000 development candidates in this source book). The stakeholder interviews were used for validation. The interviews confirmed the assumptions and did not have a significant impact in terms of changing them. As discussed above, the interviews did highlight many of the missing costs in the model. For example, stakeholders noted that the P2I model misses many industry costs, e.g. the costs of the regulatory expenses, depreciation of infrastructure, marketing costs, etc.

3.
we acknowledge that our pipeline portfolio review is likely to have missed Missing candidates: candidates under development for the 35 neglected diseases/conditions, and we discuss this in our paper. The referee notes in particular that we may have missed anti-bacterial projects; in the revision, in the Discussion section, we now make this point. Under Limitations, we now state: "Using a different search strategy or searching further databases could have identified additional candidates, such as anti-bacterial products under development (though only certain anti-bacterial candidates would qualify for the strict inclusion criteria in our study)." It is important to emphasize that the reason that the number of pipeline candidates for bacterial diseases is lower than the referee expected is primarily due to the strict inclusion and exclusion criteria used, rather than being an oversight. For example, the reason that no drug candidates targeting Enterotoxigenic E.coli (ETEC) ETEC were included in the portfolio is that this category is explicitly excluded from the scope of G-FINDER, and therefore of the portfolio inputs.

4.
The referee has a number of excellent suggestions for future studies and updates Future work: that we could provide (e.g. comparing model predictions with actual results, and examining "cohort effects" in which funders share a cohort of projects). We hope to be able to conduct such studies in the future.

5.
. Two referee reports Reverse engineering the cost of a new drug using our portfolio costs (Reports 2 and 3) suggested that we do this, i.e. they suggest we divide the 'portfolio cost' by the anticipated launches (i.e., $16.3B divided by 128). However, we do not feel this is appropriate since the P2I tool is a portfolio financial modeling tool-it is not intended to model individual product costs. Also, the estimate of $16.3B relates to moving the current portfolio of candidates through the pipeline, some of which are currently at Phase 3.

6.
We have fixed the typo in the revised version. Typo in Table 5

Lloyd Czaplewski
Chemical Biology Ventures Ltd, Abingdon, UK Developing new health technologies for neglected diseases: a pipeline portfolio review and cost model by Young et al., is a well written and comprehensive review of the pipeline and an attempt to estimate the cost of delivering Sustainable Development objectives set by the UN. So this is an important piece of work and the headlines from the Abstract will be widely used and reported.
Considerable skill and effort led to the identification of 538 products with sufficient available information to include in their cost and launch analysis. The process for project identification was clearly described although it would have been helpful to have provided the search terms used so that others could reproduce the search or update it in the future.Value is added by identifying gaps in the pipeline and estimating the costs required to support the preclinical and clinical development to launch of 18 high value products. No estimate of the cost of the fundamental research and drug discovery activity required to generate projects to feed this pipeline was provided.
All 538 products went into the analysis but portfolio management involves decisions about which projects to fund and a sense of prioritisation. It is not clear that the analysis provided is useful to inform policy makers about what level of funding is realistically needed to attempt to meet the UN objectives to reduce the impact of neglected diseases.
It is interesting that WHO's CEWG estimate of a $6 bn annual budget based on a doubling of current funding is fairly close the papers conclusion that $5 bn is required for the next 5 years to gain portfolio momentum.
The Portfolio to Impact (P2I) spreadsheet was updated to reflect neglected diseases and more product types were added to include vector control products. The P2I model does not use costs related to basic research through lead optimization; chemistry, manufacturing, and controls (CMC); good manufacturing practice (GMP); manufacturing build up and scale-up costs; regulatory or registration fees (post-phase III); and all post-market commitments (e.g., phase IV pharmacovigilance studies). Most of these costs are project specific and therefore difficult to model but they could represent significant costs leading to an under-estimate of costs particularly preclinical costs in the first 5 years.
Many of the references are papers in academic literature and it is not clear what level of diligence has gone into assessing the feasibility and potential of each project.
The diseases with most projects are HIV/AIDS 84, Malaria 88, Reproductive Health 59 and TB 62. Instead of funding all 293 projects in this group, if the top 20% of projects in each disease were selected by an expert funding panel, they could provide focus and reduce the number of therapeutic and vaccine projects from 421 to 186 which is a 56% reduction. This may be a more realistic start point for a portfolio assessment.
The world may not need 13 new HIV/AIDS or 14 new Malaria therapeutics or vaccines asap. It needs some now and some later. some now and some later. So the analysis is informative but does not use likely portfolio management to build a case for what funding would actually be needed to deliver.
Does it make sense to proposed funding to bring 27 TB diagnostics to market? How many are needed? This is commented on in the limitations -some common sense around how many diagnostics are need would be helpful.
This reviewer is slightly confused by Table 7 which seems to be at odds with Ref 12. The costs of a complex project at each stage of development are most likely more than the costs of a simple project yet the costs of a complex vaccine or NCE seem to be less than the costs of a simple version. The authors may wish to check this or at least help the reader understand why.
In several places the authors describe the costs associated with moving all 538 candidates through the pipeline. It would be helpful to make project attrition clear as not all 538 projects will progress through each stage using their estimated success rates. It is not obvious that the stated project success rates have been applied to projects.
Would it be possible to estimate what the return on investment is with respect to vaccines, therapeutics, diagnostic and others? Where is the best value spend? Preventative measures and fund those first?
The take home numbers in the abstract may not be helpful as the lack of portfolio management, recognition of attrition, everything started in year one rather than some sense of planning and phasing means its not overly useful.
If the suggested level of funding was available, is there adequate capacity and infrastructure in R&D and in clinical trial sites to be able to deliver? Where is the pinch point? What would the consequence of staggering the start of projects?
The authors have provided an excellent database of neglected disease projects but may find that additional attention to portfolio project management may yield a model for future funding that reflects a deliverable reality.

If applicable, is the statistical analysis and its interpretation appropriate? Partly
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results?
DiMasi 's estimate of the capitalized cost (i.e. cost including opportunity costs) of $2.6B to the et al. point of marketing approval to develop an NCE comprises $1.1B in pre-clinical and $1.5B in clinical costs. The cost of the regulatory approval stage may represent up to 5.7% of the total R&D cost. The P2I model does not include other types of critical research that are needed to develop new products for neglected diseases, such as developing appropriate animal models, or to bring new products to poor populations, such as policy and implementation research. In addition, our estimate of costs does not include opportunity costs. There has been a contentious debate about the merits of including these costs-for example, Angell argues that cost estimates should be not capitalized because drug companies "are not investment houses" and they "have no choice but to spend money on R&D if they wish to be in the pharmaceutical business." Nevertheless, the literature on R&D costs does tend to include opportunity costs, and they can be large (as previously mentioned, Di Masi 's $2.6B estimate includes $1.2B in opportunity costs). et al.
However, we note that these are estimates and many costs-such as the location of commercial production-are strategic in nature and so it is much harder to derive an average cost. In the area of neglected disease R&D, the PDPs challenge these commercial costs-their own cost estimates are much lower. For example, DNDi, the Drugs for Neglected Diseases initiative, a non-profit PDP, that it has spent about $39-52 million per NCE, a figure that adjusts upwards to estimates $130-195 million when risk of failure is taken into account. Most of the supporting basic research and pre-clinical research for neglected disease R&D is publically funded as a public good and may serve many different products. Therefore, trying to develop a meaningful average cost between for-profit and not-for-profit in the later development stages was not feasible and such an average was purposefully excluded in the P2I tool. Given all of these excluded estimated costs, the P2I model was never intended to provide a full price tag; instead, the aim was to offer the first tool to provide an evidence-based method for R&D cost comparisons where there are historical data to draw on. The intention is to inform prioritization and decision making at a global level and not to price individual product development.
It is worth repeating that the P2I v.1 and v.2 models examine the total pipeline of all candidates, with no judgements made on "go/no go" at each stage gate. In reality a portfolio would be managed and ranked with only the most promising candidates likely to be moved on to a subsequent phase. The type of decision required would be based on potential public health impact and feasibility of success. As these are products that are aimed at poor populations, where the market is largely absent, other considerations such as access and affordability will also influence the decisions on a go/no go basis ."

4.
: the referee notes that "many of the references are in academic literature. We would also reiterate that the basis for the model is those data reported in the PAREXEL Biopharmaceutical R&D Statistical Sourcebook -which is the industry standard for recording costs in the stages of R&D we focus on.

5.
The referee asks: "Does it make sense to proposed funding to bring 27 TB TB diagnostics: diagnostics to market? How many are needed?" We now address this question in more detail in 20 22 diagnostics to market? How many are needed?" We now address this question in more detail in the revised Discussion. We now state: "Fifth, the model is "agnostic" when it comes to the public health value of the estimated launches-it cannot judge their clinical utility. For example, the model estimates that there could be 27 TB diagnostics (just over a fifth of all launches). This is the type of scenario that the model was intended to highlight, in order to stimulate debate about what is an appropriate approach moving forwards. Despite this high number of launches, the large unmet need for TB diagnostics may still not be addressed as almost all new developments are of an incremental rather than transformational nature. While some of the incremental developments in the pipeline might be of benefit to address different needs in different epidemiological and geographic settings, others might be redundancies. In addition, the quality and performance of the launched assays may vary widely, and therefore they may not meet public health needs. And, as mentioned, this large estimate may also reflect the underlying assumptions of high success rates in both the design and development phase and the clinical validation and launch readiness phase."

6.
: These costs reflect the costs of moving through Confusion about Table 7 existing candidates the pipeline. Therefore, the reason why the costs in all phases are higher for simple vaccines than complex vaccines, and why the costs in phases 2 and 3 are higher for simple NCEs than complex NCEs, is that there are more simple vaccine candidates and simple NCEs in these phases in the pipeline (see Figure 2 and Supplementary File 2). We have now added a sentence in the results to clarify this point: "As shown in Table 7, in many cases the overall costs per phase are higher for the simpler archetype (e.g., the costs in all phases are higher for simple vaccines than complex vaccines); this finding simply reflects the fact that there are more candidates of simple complexity in the pipeline (see Figure 2 and Supplementary File 2)."

7.
: the referee requests information on project attrition-Supplementary File 3 Project attrition shows project attrition across the portfolio.

8.
: the referee asks if we can generate these Estimating the returns on investment (ROI) estimates (referee report 3 also asks about the possibility of generating an ROI). While doing so is outside the scope of this study, in the revised paper we now note that the P2I tool would potentially allow readers to do this. We now state: "In the accompanying study on the development of the P2I tool, we describe how the tool can be used to estimate health impact. We note: "The P2I model allows users to estimate the impact of a launched product on both disability, measured in disability-adjusted life years (DALYs) averted, and mortality, measured in deaths averted." It would thus be possible to estimate the economic value of these health impacts, and therefore to estimate the rates of return on portfolio investment." 9.
: the referee asks whether there would be adequate capacity to actually deliver on Trial capacity increased funding, should such funding become available. The P2I model adjusts for this by estimating the percentage uptake and capacity for development. This was set at 100% in this study i.e. an assumption that all products would be treated equally and moved through all stages of development. A setting of 50% would model for only half of available products being developed. However, the reality is that this model could only inform what shape a portfolio would take but would not replace the qualitative and strategic decisions needed about go/no-go decisions.