Rethinking Statistical Approaches to Evaluating Drug Safety

After more than 100 deaths caused by the Elixir Sulfanilamide disaster in 1938, the US Congress passed the Federal Food, Drug, and Cosmetic Act (FD&C Act) which, for the first time in US history, required pharmaceutical companies to submit full reports of investigations regarding the safety of new drugs. However, it was not until 1962, after the passage of the Leaver-Harris Amendment of the FD&C Act, that the US Food and Drug Administration (US FDA) was authorized to require evidence of efficacy for approval of new drugs. Consequently, for the approval of a new drug, the US FDA requires adequate and controlled clinical trials be conducted on humans to demonstrate their effectiveness and safety. The safety of a drug ought to be the primary focus and should come before its efficacy.

However, there has recently been an alarmingly increasing trend of safety issues of drugs after their approval. For instance, the most notorious example is the withdrawal of Vioxx® (rofecoxib) in 2004 after its excessive cardiovascular risk was not only confirmed by a retrospective nested case-control study1 but also by prospective clinical trials.2, 3 The most recent safety saga is also the cardiovascular risk associated with Avandia® (rosiglitazone), one of the most widely used thiazolidinediones that are agonists for peroxisome-proliferator-activated receptor γ (PPAR-γ. Nissen and Wolski4 reported a meta-analysis that showed that the risk of myocardial infarction is 43% higher than that of the control group. These safety issues reflect that the current evaluation processes are not adequate for the assessment of the safety of drugs for approval. Consequently, an US congressional hearing and an US FDA joint advisory committee meeting were called on June 6 and July 30, 2007, respectively, to review the issues surrounding drug safety.5

As mentioned before, safety comes before efficacy. However, at least in design, conduct and analysis of the adequate and well-controlled clinical trials, this may not be totally true for pharmaceutical companies to develop new drugs and for regulatory agencies to approve them. The current paradigm for the approval of a new drug is two-fold. First, the drug must be proven to be efficacious. Second, it must be verified whether there is any excessive safety risk even though the objective is to prove that the drug is safe. As a result, the selection of study design, endpoints, statistical methods, and sample size are to maximize the probability of proving the effectiveness of the drug. On the other hand, most clinical trials conducted during the development of the drug do not select the optimal design with a sufficient number of patients and correct endpoints to prove that the drug is safe. In addition, most of the analyses of safety data are descriptive in nature and no inferences are made. Consequently, these trials are neither adequate nor well controlled for the evaluation of safety.

One of the most critical but often neglected components during the review process for drug approval is the statistical evaluation of the evidence of safety. To prove the effectiveness of a drug, the approach is to adopt the traditional hypothesis of equality. In other words, the effectiveness of a drug is proven by rejecting the null hypothesis of equal efficacy between the test drug and control to prove that the alternative hypothesis of a superior efficacy of the test drug is true. This approach is adequate for proof of efficacy of drug products. However, the same approach is not appropriate for the evaluation of safety because it is for proof of hazard or excessive risk of drugs.6 Therefore, failure to reject the null hypothesis of no excessive risk cannot prove that a drug is safe.7 To prove that a drug is safe, based on the concept of risk management, we suggest that the non-inferiority hypothesis with consideration of managing the magnitude of the safety risk is more appropriate than the traditional hypothesis.

Graham et al.1 reported the results of a nested case control study on cardiovascular risk of COX-2 inhibitors. This study is based on a claim database of a national integrated managed care organization in conjunction with the mortality status and cause of death from the California Department of Health and Center for Health Statistics. Table 1 shows the odds ratios of acute myocardial infarction with the use of selected NSAIDs compared to remote use of a NSAID. Table 1 reveals that the upper limits of the 95% CI for odds ratios of acute myocardial infarction for celecoxib and ibuprofen are less than 1.50. It follows that, with respect to the risk of acute myocardial infarction, celecoxib and ibuprofen pose no excessive risk compared to remote users of NSAIDs. For naproxen, even though the p-value for the hypothesis in Equation (1) is 0.05, it can be concluded that naproxen is of no excessive risk of acute myocardial infarction compared to remote users of NSAIDs, because the upper limit of 95% CI for odds ratios of acute myocardial infarction is less than the upper safety margin of 1.5. On the other hand, since their upper limits of all 95% CIs for odds ratios are larger than the safety margin of 1.5, it cannot be concluded that rofecoxib is of no excessive risk of acute myocardial infarction over the remote users of NSAIDs even though some p-values based on the hypothesis in equation (1) is greater than 0.05.

Table 1
Odds Ratios of Acute Myocardial Infarction with Use of Selected NSAIDs Compared with Remote Use of a NSAID

• Click for larger image

Bresalier et al.3 reported that the risk of cardiovascular event associated with rofecoxib in the APPROVe trial, which was a chemoprevention trial involving 2,586 subjects with colorectoral adenoma. Table 2 provides the relative risks of confirmed serious thrombotic events of rofecoxib compared to the placebo group. Table 2 shows that although the 95% confidence interval of relative risk includes 1 between month 0 and month 18, its upper limit is 2.15, which is greater than the safety margin of 1.5. Therefore, contrary to the claim made by the paper, rofecoxib cannot be concluded to be of no excessive risk with respect to serious thrombotic events.

Table 2
Relative Risks of Confirmed Serious Thrombotic Events of Rofecoxib Compared with Placebo in APPROVe Trial

• Click for larger image

Table 3 presents partial results by Nissen and Wolski4 who performed a meta-analysis on a total of 26,000 patients for the effects of Avandia® on the risk of myocardial infarction and death from cardiovascular causes. As seen in Table 3, the upper limits of all 95% CIs for odds ratios are greater than 1.5. Despite the fact that some of the 95% CIs for odds ratios include 1, it cannot be concluded that Avandia® possesses no excessive risk of myocardial infarction and death from cardiovascular causes. In response to the meta-analysis by Nissen and Wolski,4 Home et al.16 reported an interim analysis of the RECORD trial of a total of 4,000 patients with a planned median follow-up of 6 years for the effect of Avandia® on cardiovascular outcomes. The hazard ratios of death from cardiovascular causes, acute myocardial infarction, or congestive heart failure are given in Table 4. With respect to death from various causes, the upper limits of all 95% CIs for hazard ratios are less than 1.5. Therefore, from the data accumulated at the cut-off date of March 30, 2007, with a mean follow-up of 3.75 years, Avandia® can be concluded to be of no excessive risk of death. On the other hand, however, the upper limits of the 95% CIs of hazard ratios with respect to acute myocardial infarction, and congestive heart failure are greater than 1.5. Therefore, Avandia® cannot be claimed to be of no excessive risk of acute myocardial infarction and congestive heart failure.

Table 3
Risk of Myocardial Infarction and Death from Cardiovascular Causes of Avandia®

• Click for larger image

Table 4
Hazard Ratios of Death from Cardiovascular Causes of Avandia® in RECORD Trial

• Click for larger image

For approval of a new drug, sponsors and regulatory agencies must assure that the drug is safe. However, the continuing saga of emerging safety problems after approval warrants us to rethink and review the current approach to evaluation of the safety data before approval. As formulated by the traditional hypothesis in equation (1), the current paradigm for evaluation of safety is based on the concept of proof of hazard to detect existence of excessive safety risk of the new drug. Therefore, failure to reject the null hypothesis of no excessive risk does not necessarily prove that the new drug is of no excessive safety risk and hence cannot guarantee that it is safe. In addition, the magnitude of safety risk has not been considered in the traditional hypotheses in equation (1). As a result, the current statistical approach to assessing the drug safety is neither appropriate nor adequate.

The notion that a drug is safe implies that the drug is of no excessive safety risk in the targeted patient population. In other words, no excessive safety risk means that the safety risk of the drug can not exceed a clinically inconsequential margin with respect to the control group in the targeted patient population. Therefore, the non-inferiority hypothesis with consideration of the magnitude of the safety risk in equation (2) is more appropriate to prove that the drug is of no excessive risk. In addition, statistical methods for the non-inferiority hypothesis are available and can directly be applied to verify the safety of the drug.

The selection of safety margins is an extremely important issue in the implementation of the non-inferiority hypothesis for evaluation of safety of drug products. Safety margins depend on the seriousness and consequences of the adverse events, targeted patient population, duration of intended use, magnitude of effectiveness, desirable benefit-risk ratio, and many other factors. Therefore, safety margins should vary depending on different diseases, classes of drugs, and targeted patient population. They should be determined jointly by clinicians, epidemiologists, pharmacists, statisticians, and other personnel involved in drug development as well as approval processes from sponsors, academia, and regulatory agencies. During phase I and II trials, information on safety profiles of new drugs can be obtained. Therefore, for phase III studies, the protocol should specify the primary safety parameters and safety margins in addition to the primary efficacy endpoints. Furthermore, phase III trials should be powered to verify that the drug is of no excessive risk based on the non-inferiority hypothesis. Only in this way, the safety of drugs and patients can be guaranteed.