INTRODUCTION

In 2007, the Ontario Ministry of Health and Long-term Care in the province of Ontario, Canada, established the Drugs for Rare Diseases Working Group (DRDWG) to develop an evidence-based evaluation framework to inform funding decisions for drugs for rare diseases (DRDs). Drug funding decisions made by public payers usually take into consideration a drug’s effectiveness (derived from randomized controlled studies) and its cost-effectiveness. However, for truly rare diseases, it may be difficult to conduct clinical trials of adequate sample size and duration to rigorously assess a drug’s effectiveness. In addition, DRDs are often very expensive and, with little or no confirmatory data on their effectiveness, are rarely found to be cost-effective. However, a policy of not funding any of these drugs on the basis of insufficient evidence and/or inefficient use of public spending might be considered unreasonable by the public.1 On the other hand, routinely funding DRDs on a compassionate basis might also be considered unfair.2 In a recent environmental scan, the Canadian Agency for Drugs and Technologies in Health identified both highly variable definitions of what constitutes a rare disease and rapid expansion in regulatory mechanisms to approve and incentivize development of drugs for rare diseases internationally over the past 20 years, but virtually no specific reimbursement frameworks.3

In Canada, most health care is publicly funded and managed at the provincial government level. In Ontario (population 13 million), the Ontario Public Drug Programs (OPDP) within the Ontario Ministry of Health & Long-term Care oversees provincial drug expenditures. OPDP has an annual budget of 4.5 billion Canadian dollars (CAD) and provides drug funding for eligible program recipients. Manufacturers apply to OPDP for public reimbursement of their drugs under six different programs. The Committee to Evaluate Drugs, an independent expert panel, advises the OPDP on which drugs to fund based on clinical and cost-effectiveness assessments of each individual drug product. The OPDP Executive Officer, reporting to the Ontario Minister of Health, considers the recommendation of the Committee to Evaluate Drugs and makes the final funding decision. Typically data from randomized trials is necessary for the adequate evaluation and listing of new drug entities on the OPDP formulary.

The Drugs for Rare Diseases Working Group (DRDWG) was established in response to the 2007 recommendation by the Committee to Evaluate Drugs against the public funding of idursulfase for Hunter syndrome (mucopolysaccharidosis II) due to insufficient evidence of effectiveness.4 Recognizing the inherent challenges of gathering robust data for drugs for rare diseases, the Executive Officer established the DRDWG to investigate and develop a different policy mechanism for the evaluation of these drugs.

The DRDWG initially consisted of nine members selected by the Executive Officer: three representatives from the Ontario Ministry of Health and Long-term Care (the Executive Officer and two pharmacists), four current or previous members of the Committee to Evaluate Drugs (two physicians, a pharmacist, and a health economist), and two other physicians, one with formal pharmacoeconomic training and the other an expert in the treatment of inherited disorders of metabolism in children. Several principles formed the basis of the policy development process. First, the perspective would be that of a public payer addressing requests for funding for a specific drug. Other guiding precepts included respect for the principles of "accountability for reasonableness" of Daniels and Sabin,1 with the expectation that the ideal policy product would be transparent, consistent, address unique aspects of the treatment of a specific rare condition while being adaptable to other dissimilar conditions, and include an appeals process. This process has been described previously, and a seven-step policy framework based on the principles of evidence-based medicine and epidemiological causality was developed, implemented, and made publicly available (Table 1).5,6 The aim of this report is to describe the initial experience using this framework to develop funding recommendations for drug applications, and to examine the subsequent effect on drug funding decisions and costs.

Table 1 Evaluation Framework for Funding Drugs for Rare Diseases
Full size table

METHODS

Between March 2008 and February 2013, eight drugs were identified as candidates for review by the DRDWG. We reviewed the degree to which their evaluations conformed to the seven-step evaluation process: 1. confirm that the condition for treatment with the candidate drug is truly “rare”; 2. understand the disease (usual history based on best available evidence); 3. understand the potential value of the candidate drug (based on best available evidence); 4. model the potential clinical effectiveness of the candidate drug; 5. evaluate cost implications and generate a funding recommendation; 6. review the drug evaluation with disease experts and stakeholders; and 7. assessment for re-evaluation (Table 1). For the purposes of this report, two members of the Working Group (WC and CS) collected the data from evaluations to date, and these were assessed by all of the authors. We also recorded the completeness of the estimates of clinical effectiveness and cost for each drug, the development of funding recommendations, and the results of funding decisions by the Executive Officer. For drugs approved for funding, the results of individual patient funding decisions were also reviewed.

RESULTS

Of the eight drugs evaluated using the framework (Table 2), six were for rare inherited metabolic enzyme deficiencies, one for a malignancy, and two for uncommon idiopathic conditions. The estimated average annual drug cost per patient ranged from 28,000 to 1,200,000 CAD. Idursulfase for Hunter syndrome, the first drug evaluated, was used to iteratively develop the framework and is included in the results. The other drugs and disease indications were: alglucosidase for Pompe disease, miglustat for Niemann-Pick disease type C, laronidase for mucopolysaccharidosis I, galsulfase for mucopolysaccharidosis VI, vorinostat for cutaneous T-cell lymphoma, canakinumab for cryopyrin-associated periodic syndrome, and eculizumab for paroxysmal nocturnal hemoglobinuria. At each step of the assessment, the members of the DRDWG came to a consensus only after review of adequate data about both the disease and drug. If a disease or drug failed to meet criteria at any of the steps, it was considered unsuitable for further evaluation using the framework.

Table 2 Drugs Evaluated Using the Drugs for Rare Diseases Working Group (DRDWG) Framework
Full size table

Evaluation Process

  1. Step 1.

    Confirm that the condition for treatment with the candidate drug is truly “rare”

    For each drug evaluation, the “rarity” of the disease indication was carefully reviewed. Rarity limits not only the ability to conduct adequately powered randomized trials, but also understanding of the disease's natural history, which may be biased toward more severe variants of the disorder. In addition, variability in the clinical phenotype of a rare disease can make the selection of suitable clinical outcomes extremely difficult and force researchers to rely on surrogate outcomes of uncertain validity. Often, the DRDWG experienced some pressure to define an explicit threshold. However, it was recognized that review under this process, which potentially modeled clinical effectiveness, could be error prone and should be used exceptionally, not routinely. In addition, there was consensus that the existence of a process for evaluating drugs for rare diseases should not diminish the motivation to conduct randomized trials in conditions that were simply uncommon or difficult to study (i.e., “best possible evidence” should be required). The disease indications for six of the eight drugs reviewed were considered of sufficient rarity to warrant evaluation: Hunter syndrome (1.3:100,000 male live births), Pompe disease (1:40,000 live births), Niemann-Pick type C (incidence 1:150,000), mucopolysaccharidosis I (1:50,000 live births), mucopolysaccharidosis IV (1:700,000 live births), and cryopyrin-associated periodic syndrome (incidence 1:1,000,000). The disease indications for two drugs (cutaneous T-cell lymphoma and paroxysmal nocturnal hemoglobinuria) were considered ineligible for further consideration under this framework, because it was determined that the size of the available patient population and the nature of the disease were amenable to good-quality clinical trials to assess meaningful endpoints, and thus should not be evaluated through a revised policy framework based on lower levels of evidence. These two drugs were referred for regular review through the standard evaluation process of the Committee to Evaluate Drugs, where standard evidence from randomized trials is expected.

  2. Step 2.

    Understand the disease

    This step entails gaining an understanding of the basic pathophysiology, natural history, and health effects of the disease condition and the mechanism of action of the candidate drug. This then allows for a critical review of the actual or potential effects of the drug treatment on the disease trajectory. Five of the six eligible disease indications were considered to have sufficient natural history information available to understand the disease and to predict potential effects from drug therapy. Insufficient information about the natural history for mucopolysaccharidosis VI precluded modeling for this disease. As a result, the DRDWG was unable to predict the nature and extent of benefit that galsulfase could provide. In this case, the paucity of natural history information and clinical trial data on the drug’s efficacy made it impossible for the DRDWG to render an evidence-informed funding recommendation. The drug was rejected for funding on this basis, with feedback to the manufacturer to provide natural history data for a future resubmission.

  3. Step 3.

    Understand the potential value of the candidate drug

    The theoretical value of the drug for the disease indication is evaluated at this step. Although there was some debate within the Working Group about the specificity of canakinumab for cryopyrin-associated periodic syndrome, ultimately all five drugs eligible were considered to be of potential clinical value.

  4. Step 4.

    Model the potential clinical effectiveness of the candidate drug

    Using the data identified in steps 2 and 3, clinical effectiveness can be estimated using modeling techniques that include estimates of the magnitude and the variability of treatment effects, with explicit acknowledgment of the limitations of the data and techniques used. When possible, the DRDWG used Bayesian-style decision modeling to synthesize data of variable quality from different sources. Typically, the prior probabilities for the most clinically relevant outcomes were drawn from studies of the natural history of the disease in question, and the impact of the new drug on these outcomes over time was estimated based first on the best available evidence of effectiveness from clinical studies; modeling for time frames beyond that was informed by the clinical studies.

    The example of idursulfase for Hunter disease has been described previously.7 In brief, a Markov model of the disease course was developed with input from the DRDWG and validation by external clinical experts. Disease epidemiology was extracted from available case series data, with use of informed expert opinion when necessary. Cycle lengths were 6 months, and a lifetime horizon was used, with a maximum age at death set at 80 years. Disease progression was described from diagnosis to musculoskeletal symptoms to respiratory symptoms to cardiovascular symptoms to cardiorespiratory failure. It was assumed that patients would progress sequentially through these health states, and the rate of progression was estimated from natural history data. A set of transition probabilities was determined, allowing estimation of the average life expectancy of cohorts of Hunter disease patients based on their age and current health status. The potential benefits of idursulfase based on available clinical data were then also included, as were life years gained with this intervention. Based on this modeling of clinical effectiveness, the DRDWG concluded that idursulfase could reduce the likelihood of development of musculoskeletal and respiratory symptoms, but there was neither evidence nor biologic plausibility to suggest an impact on neurodegeneration or on the progression of cardiac disease and cardiorespiratory failure. The model informed the recommendations of the DRDWG, and the Executive Officer endorsed a policy of considering requests for reimbursement of idursulfase for patients with confirmed diagnosis of Hunter disease who are aged 6 years or older and who have no or minimal non-progressive neurocognitive impairment.

    All five drugs that were not excluded based on evaluation in steps 1–3 of the framework next underwent similar modeling of clinical effectiveness. The models were used to predict the potential benefit or lack of benefit of the drug treatments, thereby informing funding recommendations and clinical eligibility criteria.

  5. Step 5.

    Evaluate cost implications and generate a funding recommendation

    All five drugs eligible for step 4 were evaluated for cost implications, including budget impact. As it was obvious that the cost per quality-adjusted life year for all five drugs was beyond conventional funding thresholds, formal pharmacoeconomic analyses were not performed. In addition, specific reimbursement guidelines, including clinical criteria for initiation, continuation, and termination of funding, were developed for all five drugs. The Executive Officer agreed with all five funding recommendations, and these drugs are now publicly funded in Ontario according to clinical criteria developed by the DRDWG.5

  6. Step 6.

    Review the drug evaluation with disease experts and stakeholders

    Input from external disease experts was sought in steps 1 to 5 as required. Upon completion of the evaluation, outcomes of the review, including the data and assumptions used to inform the clinical and economic predictive modeling, were shared with physician and patient stakeholder groups to identify areas of disagreement or error. This also provided stakeholders with the opportunity to engage in the DRDWG process, which should facilitate acceptance of funding policy conclusions. Formal stakeholder consultations were conducted with physicians, patients, and industry representatives for idursulfase, the first drug reviewed through the framework. For miglustat, a face validity exercise was conducted with experts. For the other three recommendations, the reimbursement guidelines were circulated to physician and patient groups, and when necessary, reimbursement guidelines were revisited based on feedback.

  7. Step 7.

    Reassessment

    New information regarding disease incidence, natural history, and the effectiveness or cost of drug therapy may trigger re-evaluation at this step. Three of the drugs completing full evaluation have undergone reassessment (idursulfase, alglucosidase, and miglustat). In each case, the reassessment found that revisions of the initial funding guidelines were not required based on new information that had emerged since the initial review.

Drugs that Completed Evaluation

Five drugs underwent full framework evaluation, including Markov modeling of clinical effectiveness. When there were recommendations for drug cost reimbursement for individual patients, these included highly specific criteria for treatment initiation, mandatory requirements for continuation of funding, and criteria for discontinuing treatment based on futility. Based on the recommended eligibility and exclusion criteria for fundable conditions, all five drugs were subsequently approved for funding by the Executive Officer of the Ontario Public Drug Programs after risk-sharing agreements were made with the manufacturers.

Drugs that did not Complete Evaluation

Two disease indications were considered ineligible for review under the framework: cutaneous T-cell lymphoma and paroxysmal nocturnal hemoglobinuria. The manufacturer of vorinostat did not pursue reimbursement through the standard Committee to Evaluate Drugs evaluation process. Eculizumab was subsequently reviewed, and it was recommended that eculizumab not be funded. However, the Committee also indicated that there was likely a subset of patients for whom eculizumab might be of benefit and that this patient subgroup was not well defined in the data provided by the manufacturer. As a result, OPDP engaged an expert committee that included two members of the DRDWG to develop a predictive model to help estimate the potential benefit of eculizumab in modifying disease outcomes; funding guidelines were subsequently developed. For galsulfase, the OPDP Executive Officer decided to provide funding on a case-by-case basis, on compassionate grounds.

Funding Policy Results

For the five drugs recommended for funding based on these criteria, 32 patients met the eligibility criteria and received the drug for ongoing treatment. However, at least one of these patients met criteria determining futility of the treatment, and the drug was no longer funded for this patient. Five patients have been denied funding for these drugs based on predefined exclusion criteria (Table 2).

DISCUSSION

This evaluation framework has been a useful guide for the evaluation of drugs for rare diseases in Ontario, providing a more consistent and transparent process. The framework incorporates principles of decision making informed by the best possible evidence, epidemiological causality (Bradford-Hill’s criteria), and Bayesian modeling of clinical effectiveness. Modeling of effectiveness has been a very important component of the evaluation and often informs not only the specific clinical criteria for funding recommendations (i.e., eligibility criteria, futility rules, stopping criteria), but also the criteria for risk-sharing agreements with industry, in order to reinforce these criteria. We feel that by accepting a model of effectiveness based on a candidate drug’s potential effects on disease natural history, this evaluation framework implicitly reflects an ethical basis through principles analogous to “accountability for reasonableness.”

However, the evaluation framework has several limitations. Evaluations require dedicated and often protracted efforts by a group of experts and stakeholders. So far, full evaluations have been limited to diseases that might be considered “ultra-rare.” Specific definitions of “rare” have been deliberately avoided, and each disease of interest has been evaluated case-by-case, based on a consensus opinion on whether randomized trials with pragmatic endpoints could be completed within a reasonable time frame and with reasonable effort and cost. Such decisions must consider the prevalence, severity, and clinical heterogeneity of the disease, as well as the potential efficacy of the candidate drug. Care has been taken to avoid creating a perception that the DRDWG evaluation framework is an opportunity to avoid the difficulty and expense of pragmatic randomized clinical trials when these are possible. An expert in Markov modeling (DC) has been essential to the formulation of funding recommendations. However, such modeling requires credible information about the natural history of the candidate disease, and for “ultra-ultra rare” diseases, this may not be feasible and limits the scope of evaluation.

The evaluation framework described represents a compromise to address the dilemma of inadequate trial evidence upon which to make funding policy decisions about drugs for rare diseases. It is designed to retain fidelity to the evidence-informed evaluation process upon which drug reimbursement is based in Ontario, but with application to exceptional circumstances, without digression to a policy of universal compassionate funding without consideration of reasonable eligibility criteria, futility rules, and stopping rules that can be gleaned within reason from available evidence. The evaluation process allows for re-evaluation of funding recommendations as new data arises, some of which may be produced by comparing the results of funding decisions made with those predicted by the disease models. There should not be major barriers to adapting this type of evaluation framework outside Canada, among both public and private payers, provided that adequate committed expertise is available and transparency remains a high priority. The evaluation process has been—and continues to be—refined iteratively with each drug evaluation. Ultimately, further observation and analysis will be required to determine the value of this evaluation process. In particular, the passage of time will allow comparison of predicted clinical and economic outcomes from the modeled versus the real-world clinical and economic outcomes. While the results of real-world outcomes can theoretically be fed back into the model to improve future decisions, the reality of ultra-rare diseases means that it will take several years to obtain improved information from one province alone. Future collaborations toward a similar iterative policy framework might expedite knowledge creation across Canada, or even internationally.