Intended for healthcare professionals

  1. Research
  2. Associations between...
  3. Associations between stopping prescriptions for opioids, length of opioid treatment, and overdose or suicide deaths in US veterans: observational evaluation
CCBYNC Open access
Research

Associations between stopping prescriptions for opioids, length of opioid treatment, and overdose or suicide deaths in US veterans: observational evaluation

BMJ 2020; 368 doi: https://doi.org/10.1136/bmj.m283 (Published 04 March 2020) Cite this as: BMJ 2020;368:m283

Linked Opinion

What an opioid safety initiative can teach us about using information to improve patient outcomes
  1. Elizabeth M Oliva, national coordinator and principal investigator1 2,
  2. Thomas Bowe, statistician1 2,
  3. Ajay Manhapra, physician and assistant professor3 4 5 6,
  4. Stefan Kertesz, research physician and professor7 8,
  5. Jennifer M Hah, assistant professor9,
  6. Patricia Henderson, data scientist1,
  7. Amy Robinson, pharmacy program manager10,
  8. Meenah Paik, data scientist1,
  9. Friedhelm Sandbrink, national director11 12 13,
  10. Adam J Gordon, director and professor14 15 16,
  11. Jodie A Trafton, director and principal investigator1 2 17
  1. 1Veterans Affairs Program Evaluation and Resource Center, Veterans Affairs Office of Mental Health and Suicide Prevention, Menlo Park, CA, USA
  2. 2Veterans Affairs Center for Innovation to Implementation, Veterans Affairs Palo Alto Health Care System, 795 Willow Road, Building 324, Menlo Park, CA 94025, USA
  3. 3Advanced PACT Pain Clinic, Veterans Affairs Hampton Medical Center, Hampton, VA, USA
  4. 4Department of Psychiatry, Yale School of Medicine, New Haven, CT, USA
  5. 5New England MIRECC, West Haven, CT, USA
  6. 6Departments of Physical Medicine and Rehabilitation and Psychiatry, Eastern Virginia Medical School, Norfolk, VA, USA
  7. 7Birmingham Veterans Affairs Medical Center, Birmingham, AL, USA
  8. 8University of Alabama at Birmingham School of Medicine, Birmingham, AL, USA
  9. 9Division of Pain Medicine, Department of Anesthesiology, Perioperative, and Pain Medicine, Stanford University, Stanford, CA, USA
  10. 10Pharmacy Services, Veterans Affairs Sierra Pacific Network, Palo Alto, CA, USA
  11. 11National Pain Management Program, Veterans Health Administration, Washington, DC, USA
  12. 12Department of Neurology, Washington DC Veterans Affairs Medical Center, Washington, DC, USA
  13. 13Uniformed Services University, Bethesda, MD; George Washington University, Washington, DC, USA
  14. 14Program for Addiction Research, Clinical Care, Knowledge and Advocacy (PARCKA), Salt Lake City, UT, USA
  15. 15Division of Epidemiology, Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, USA
  16. 16Informatics, Decision-Enhancement, and Analytic Sciences Center (IDEAS), Veterans Affairs Salt Lake City Health Care System, Salt Lake City, UT, USA
  17. 17Department of Psychiatry, Stanford University School of Medicine, Stanford, CA, USA
  1. Correspondence to: E M Oliva elizabeth.oliva{at}va.gov
  • Accepted 15 January 2020

Abstract

Objective To examine the associations between stopping treatment with opioids, length of treatment, and death from overdose or suicide in the Veterans Health Administration.

Design Observational evaluation.

Setting Veterans Health Administration.

Participants 1 394 102 patients in the Veterans Health Administration with an outpatient prescription for an opioid analgesic from fiscal year 2013 to the end of fiscal year 2014 (1 October 2012 to 30 September 2014).

Main outcome measures A multivariable Cox non-proportional hazards regression model examined death from overdose or suicide, with the interaction of time varying opioid cessation by length of treatment (≤30, 31-90, 91-400, and >400 days) as the main covariates. Stopping treatment with opioids was measured as the time when a patient was estimated to have no prescription for opioids, up to the end of the next fiscal year (2014) or the patient’s death.

Results 2887 deaths from overdose or suicide were found. The incidence of stopping opioid treatment was 57.4% (n=799 668) overall, and based on length of opioid treatment was 32.0% (≤30 days), 8.7% (31-90 days), 22.7% (91-400 days), and 36.6% (>400 days). The interaction between stopping treatment with opioids and length of treatment was significant (P<0.001); stopping treatment was associated with an increased risk of death from overdose or suicide regardless of the length of treatment, with the risk increasing the longer patients were treated. Hazard ratios for patients who stopped opioid treatment (with reference values for all other covariates) were 1.67 (≤30 days), 2.80 (31-90 days), 3.95 (91-400 days), and 6.77 (>400 days). Descriptive life table data suggested that death rates for overdose or suicide increased immediately after starting or stopping treatment with opioids, with the incidence decreasing over about three to 12 months.

Conclusions Patients were at greater risk of death from overdose or suicide after stopping opioid treatment, with an increase in the risk the longer patients had been treated before stopping. Descriptive data suggested that starting treatment with opioids was also a risk period. Strategies to mitigate the risk in these periods are not currently a focus of guidelines for long term use of opioids. The associations observed cannot be assumed to be causal; the context in which opioid prescriptions were started and stopped might contribute to risk and was not investigated. Safer prescribing of opioids should take a broader view on patient safety and mitigate the risk from the patient’s perspective. Factors to address are those that place patients at risk for overdose or suicide after beginning and stopping opioid treatment, especially in the first three months.

Introduction

In the United States, concerns about the opioid crisis—the rise in deaths from overdose1 and opioid use disorders234—has prompted a range of initiatives to mitigate the risks associated with prescribing opioids for chronic pain, and to encourage tapering or stopping when risks exceed benefits.5678910111213 These efforts have seen a fall in the prescribing of opioids by 35% between 2012 and 2017, per 100 000 people.14 Guidance on reducing opioids typically warns against sudden changes to care. For example, a Veterans Affairs/Department of Defense Guideline specifies that “[a]brupt discontinuation should be avoided unless required for immediate safety concerns.”13 Some studies suggest a rise in adverse events after stopping treatment with opioids, but national studies with large sample sizes are lacking.

In 2012, a retrospective study was conducted in the Veterans Health Administration of 509 veterans whose long term treatment with opioids was stopped. Patients with substance use disorders were matched to patients with no disorders. The researchers found evidence of suicidal thoughts and self-harm in patients with and without substance use disorders who ended treatment with opioids after long term use.15 A nested case-control study from Kaiser Permanente Colorado in patients on long term treatment with opioids between 2006 and 2017 (228 patients who took an overdose of opioids matched to 3547 control patients) found that variations in the dose of opioid increased the likelihood of overdose. Stopping treatment for at least three months (that is, 0 mg of morphine equivalence) mitigated the risk.16 In an analysis of Vermont Medicaid data between 2013 and 2017 of 494 patients treated with high doses of opioids for long periods of time who stopped taking opioids, researchers found that almost half had been admitted to hospital or visited the emergency department for an opioid or substance use related event, after they had stopped treatment.17 The impact of stopping treatment in 572 patients on long term treatment with opioids was examined in a safety net primary care clinic in Seattle with opioid registry data from 2010 to 2015. Stopping opioid treatment was associated with an increased risk of death from overdose18 but the analyses could not assess the characteristics of the patients that might have contributed to the deaths. These characteristics could be important because patients with substance use disorders and mental health diagnoses have historically been more likely to be treated with opioids for chronic pain,19 which might increase the risk for psychiatric or medical instability when treatment with opioids is stopped.152021 Given these uncertainties, Frank and colleagues called for research on the impact of reducing the prescribing of opioids.22

The Veterans Health Administration has taken a multifaceted approach to the safety of prescribing opioids. Sequential safety initiatives produced a 37% reduction in the prescribing of opioids between 2014 and 2016, largely by reducing the introduction of long term use of opioids, but also by reducing opioid treatment in some long term users.232425 The purpose of our evaluation was to examine the likelihood of death from overdose or suicide in veterans prescribed opioid analgesics in the early implementation period of the Veterans Health Administration’s opioid safety initiative.24 We studied the interaction between stopping opioid treatment and length of opioid use because opioid tolerance, dependence, and addictive behaviors might develop with longer exposure to opioids and might contribute to the risk of death from overdose or suicide after treatment with opioids is stopped.21,262728

Methods

Stanford University’s institutional review board determined that this quality improvement project was not research.

Patient population

The Veterans Health Administration’s Corporate Data Warehouse (extracts from electronic medical record data) identified patients with an outpatient prescription for an opioid analgesic in fiscal year 2013 (1 October 2012 to 30 September 2013). Opioids used to treat opioid use disorders were excluded (eg, sublingual buprenorphine formulations). Opioid prescriptions were for short acting or long acting drugs. Short acting opioids were butorphanol, codeine, dihydrocodeine, hydrocodone, hydromorphone, meperidine, morphine, opium/belladonna, oxycodone, oxymorphone, pentazocine, tapentadol, and tramadol. Long acting opioids were fentanyl transdermal patch, buprenorphine transdermal patch, methadone (excluding methadone from licensed opioid treatment programs), sustained release hydromorphone, sustained release morphine, sustained release oxycodone, sustained release oxymorphone, sustained release tapentadol, and sustained release tramadol.

Evaluation period

We examined the last opioid prescription episode in fiscal year 2013 and death related to overdose or suicide to the end of fiscal year 2014. Thus the evaluation period was 1 October 2012 to 30 September 2014 (fiscal year 2013 to fiscal year 2014).

Measures

Prescribing data for opioids and personal patient data for fiscal year 2013 (age, sex, current marital status, rural residence), number of Elixhauser medical diagnoses,29 and diagnoses for mental health and substance use disorders were from the Veterans Health Administration’s Corporate Data Warehouse.21 Prescribing data for opioids were used to identify a patient’s last opioid prescription episode in fiscal year 2013, maximum daily morphine milligram equivalents for an opioid prescription, type of opioid treatment, date of stopping treatment, and status of treatment (stopped or continued).

Last opioid prescription episode in fiscal year 2013

Opioid prescription episodes were identified by the drug name, quantity, days’ supply, and release dates for each patient with an opioid prescription from 1 October 2012 to 30 September 2014. We identified the last opioid prescription episode in fiscal year 2013 for every patient. To identify the start and end dates of an opioid prescription episode, prescription data were tracked for five years before the date of death or the end of the next fiscal year. An opioid prescription episode was defined as a period in which gaps in possession of opioids never exceeded twice the days’ supply for a newly written prescription, 180 days between releases of a refill of an existing prescription, or 300 days between any two prescription releases regardless of refill status.

Length of last opioid prescription episode in fiscal year 2013

The length of the last opioid prescription episode in fiscal year 2013 was calculated with the start and end dates of the last prescription for opioids (categorized as ≤30, 31-90, 91-400, and >400 days). This variable is described as length of treatment with opioids.

Maximum daily morphine milligram equivalents

Morphine equivalents for each patient’s opioid prescription(s) were calculated for fiscal year 2013 with the method described by the Centers for Disease Control and Prevention for calculating the daily amount of opioid prescriptions—that is, converting each opioid prescription into daily morphine milligram equivalents and adding them together.30 Daily morphine milligram equivalents for each patient’s Veterans Health Administration opioid prescription(s) were added together to give the maximum daily morphine milligram equivalents. The statistical analyses used log maximum daily morphine milligram equivalents.

Type of opioid treatment

Type of treatment was the opioid prescribed in fiscal year 2013 (hierarchical classification): long acting opioid, short acting opioid, or tramadol only.3132

Date of stopping treatment and opioid status

The date when treatment with opioids was stopped was estimated for each patient as the date of the last prescription for opioids in addition to the days’ supply. The date when opioid treatment was stoppedwas when patients were estimated to have no pills, if that occurred before the date of death or before the end of fiscal year 2014. For descriptive use only, the date when treatment with opioids was stopped was used to create a binary status variable: stopped or continued treatment with opioids. Patients were assigned to a stopped or continued treatment group. Patients in the continuing treatment group were estimated to have pills on hand at the time of death or at the end of fiscal year 2014.

Outcome measure

Our previous work with this population found similar risk factors for overdose and suicide,21 and thus the primary outcome in this evaluation was a binary variable of death caused by overdose (including opioid, sedative, acetaminophen, or other drug poisoning) or suicide, according to ICD-10 (international classification of diseases, 10th revision), 10th revision, from the National Death Index search results of the Veterans Affairs/Department of Defense Mortality Data Repository (appendix 1). Deaths from overdose and suicide were not limited to those involving opioids.

The Veterans Affairs and Department of Defense jointly search the National Death Index for data on numbers and causes of death as part of their surveillance on suicide. Searches involve matches with death certificates in the National Death Index, based on name, social security number, date of birth, sex, race, and marital status. The National Death Index matches records in a two step process. The first step is selection of potential matches based on seven matching criteria: social security number; exact month and within a year of birth, and first and last name; exact month and within a year of birth, first and middle initials, and last name; exact month and day of birth, and first and last name; exact month and day of birth, first and middle initials, and last name; exact month and year of birth, first name, and father’s surname; and if the patient is a woman, exact month and year of birth, first name, last name (on user’s record), and father’s surname (on National Death Index record). A scoring and classification procedure results in a probabilistic score and suggested final match by the National Death Index. A possible match is a record that matches one of the seven criteria. The Veterans Affairs uses algorithms to find the best of all possible matches. Possible matches are run through several sets of potential true match criteria. A match is counted if a possible match meets one of the sets of criteria. Matches are run through additional criteria to determine a best match for each social security number by ranking the matching variables of the National Death Index by order of importance. The record with the highest priority matching criteria is selected for each social security number.

Cox non-proportional hazards regression model

We used SAS PHREG (UNIX SAS version 9.2; Executive Information Systems, Bethesda, MD) to conduct a Cox non-proportional hazards regression model that analyzed the last opioid prescription episode started in fiscal year 2013, with death from overdose or suicide as the primary outcome. The main covariate of interest was the interaction of time varying opioid cessation (date of stopping treatment) by length of treatment with opioids (≤30, 31-90, 91-400, and >400 days). SAS PHREG can test and correct for violations of proportional hazards assumptions inherent in the proposed model that includes an interaction with a function of time (length of treatment with opioids).3334 Consistent with approaches that incorporate time related variables, length of treatment with opioids was only included in the interaction term with stopping opioid treatment. We also included stopping opioid treatment as a time varying covariate to help avoid immortal time bias.3536 Also, to account for potential bias introduced by opioid treatment before the start of the observation period (that is, the index fiscal year), entry into the model was left censored based on length of opioid use before fiscal year 2013. For example, patients with an opioid prescription episode 30 days before the start of fiscal year 2013 were not included in the risk pool for the first 30 days of their treatment because patients could not have died from suicide or an overdose before the start of fiscal year 2013 and still enter the study. Thus patients were not included in the model until they entered the outcome observation period, with the index date for the model being the start of fiscal year 2013 (1 October 2012) for patients treated with opioids before fiscal year 2013 or the start date of the last opioid prescription episode that occurred in fiscal year 2013 for patients who were not treated with opioids before fiscal year 2013. Also, all models were adjusted for baseline demographic characteristics; number of medical diagnoses; diagnosis of a mental health, substance use, or nicotine use disorder; type of opioid treatment; and the log of the maximum daily morphine milligram equivalents. The analyses excluded individuals with incomplete data. Patients who died from causes other than overdose or suicide were right censored at the time of death. Cox non-proportional hazards regression models were also conducted for overdose and suicide outcomes separately as sensitivity analyses.

Life table descriptive data

We used SAS LIFETEST to better understand the potential clinical impact of the interaction between stopping treatment with opioids and length of treatment on death from overdose or suicide over time. With SAS LIFETEST, we created life tables that characterized observed death rates for overdose and suicide calculated every 25 days. Two life tables were created, the first characterized death from overdose or suicide in patients while treated with opioids and after stopping treatment, and the second characterized death from overdose or suicide after stopping treatment with opioids by length of treatment. For the first life table, death rates for overdose or suicide for patients who started an opioid prescription in fiscal year 2013 (to address immortal time bias for patients who started treatment before fiscal year 2013) were calculated from the start date of their last opioid prescription, when treated with opioids. For patients in the group who stopped treatment with opioids, death rates were calculated from the date of stopping treatment because, by definition, patients had to have survived up to the date of stopping treatment to enter this group. The first life table essentially shows death rates for overdose and suicide in patients who continued to be treated with opioids by length of treatment. The second life table was created to show death rates in patients who stopped treatment with opioids by length of opioid treatment. Appendix 2 includes a heuristic summary of the data assembly method used in this evaluation.

Patient and public involvement

Although patients and the public were not directly involved in this evaluation, the questions were informed by the concerns of patients and the public, as expressed directly through congressional representatives or the media to the Veterans Health Administration, the experiences of authors involved in this evaluation, and the concerns about potential iatrogenic effects of stopping treatment with opioids. As this was a quality improvement project, the findings have been shared internally on numerous national calls of the Veterans Health Administration with leadership and providers. A national workgroup was convened to identify additional quality improvement efforts to address findings from this evaluation (eg, medication use evaluation, natural language processing approaches). The Veterans Health Administration computerized decision support systems have been updated to highlight risks and facilitate monitoring and risk management of patients after stopping treatment with opioids.

Results

The number of patients with an outpatient prescription for an opioid analgesic in fiscal year 2013 was 1 394 102. Table 1 describes the characteristics of the patients based on treatment with opioids for fiscal year 2013. The incidence of stopping treatment with opioids was 57.4%. Mean age of the patients was around 60 years, about 8% were women, around half were married, and most (61%) lived in urban areas. Just over half had three or more medical diagnoses. Alcohol use disorders were diagnosed in about 10% of patients and nicotine use disorders in about 22%. The most common mental health conditions were depressive disorders and post-traumatic stress disorder.

Table 1

Characteristics of patients by status of opioid treatment in fiscal year 2013

View this table:

In fiscal year 2013, 10% of patients were prescribed long acting opioids (n=134 647), 68% short acting opioids (n=947 107), and 22% tramadol only (n=312 348). Because we evaluated all patients with an outpatient prescription for an opioid analgesic (including those with short term prescriptions for acute postoperative or post-injury conditions), most were treated with short acting opioids, as expected. A higher proportion of patients who continued treatment with opioids were prescribed long acting opioids than those who stopped treatment with opioids. Maximum morphine milligram equivalents for an opioid prescription was nearly twice as high in patients who continued with their opioid treatment compared with those who stopped treatment.

We conducted a multivariable Cox non-proportional hazards regression model on 1 394 089 patients in fiscal year 2013 (13 patients were excluded because of missing data) with 2887 deaths from overdose or suicide (1851 overdose, 1249 suicide; deaths from overdose or suicide were not mutually exclusive). Thirty two per cent of patients were prescribed opioids for up to 30 days (n=445 918), 9% for 31-90 days (n=121 294), 23% for 91-400 days (n=316 520), and 36% for longer than 400 days (n=510 370). We found a significant interaction between stopping treatment with opioids and length of treatment that suggested a greater risk of death from overdose or suicide after stopping treatment the longer patients were treated with opioids (P<0.001;table 2). Specifically, the hazard ratios for patients who stopped opioid treatment (with reference values for all other covariates) were 1.67 (≤30 days), 2.80 (31-90 days), 3.95 (91-400 days), and 6.77 (>400 days). Other factors independently associated with a greater risk of death from overdose or suicide were prescriptions for long acting or short acting opioids compared with tramadol, log maximum daily morphine milligram equivalents, number of medical diagnoses, and having a diagnosis of a mental health or substance use disorder. Older age, female sex, and being married were independently associated with a lower risk of death from overdose or suicide.

Table 2

Multivariable Cox non-proportional hazard regression model estimates for death from overdose or suicide in patients with an outpatient prescription for an opioid in fiscal year 2013 at the Veterans Health Administration

View this table:

Sensitivity analyses examining multivariable Cox non-proportional hazards regression models for death from overdose and suicide separately found the same general pattern of results as the combined model (appendix 3). Stopping treatment with opioids was associated with a greater risk of death from overdose and suicide the longer patients were treated with opioids. Specifically, for death from overdose, the hazard ratios for patients who stopped treatment with opioids (with reference values for all other covariates) were 1.39 (≤30 days), 2.41 (31-90 days), 3.25 (91-400 days), and 5.65 (>400 days), and the corresponding hazard ratios for death from suicide were 2.02 (≤30 days), 3.43 (31-90 days), 4.78 (91-400 days), and 7.99 (>400 days). In the combined model, we found no association between rural residence and death from overdose or suicide, whereas rural residence was associated with a lower risk of death from overdose but a higher risk of death from suicide in the separate models. The pattern of significant results was similar in the combined and overdose models, respectively, for age (P<0.001; P<0.001), number of medical diagnoses (P<0.05; P<0.001), and short acting opioids (P<0.01; P<0.05), but in the suicide model, age and short acting opioids were not significant. The number of medical diagnoses was associated with a lower risk in the suicide model (P<0.05) compared with a higher risk in the combined and overdose models.

Figure 1 (top panel) shows life table descriptive data over 375 days, with the conditional probabilities of death from overdose or suicide for patients treated with opioids who started an opioid prescription in fiscal year 2013 (n=952 918) and for patients who stopped treatment (n=799 668). The risk of death from overdose or suicide was increased for three months after starting or stopping opioid treatment. The incidence of death was increased more, however, in the three months after stopping treatment with opioids with the rates not stabilizing for almost six to 12 months after stopping. The full life table descriptive data for figure 1 (top panel) are in appendix 4.

Fig 1
Fig 1

Conditional probability of death from overdose or suicide over 375 days (in 25 day intervals). Top panel=probability in patients treated with opioids (for patients who started treatment with opioids (n=952 918) in fiscal year 2013) and after stopping opioid treatment (n=799 668). Bottom panel=probability by length of treatment with opioids in those who stopped opioid treatment

Figure 1 (bottom panel) shows the life table descriptive data over 375 days, illustrating the interaction between stopping treatment with opioids and length of treatment. In the first 25 days after stopping treatment, deaths per 100 000 patients based on length of opioid treatment were 20 (≤30 days), 40 (31-90 days), 58 (91-400 days), and 104 (>400 days). The pattern of increased numbers of deaths in the first three months after stopping treatment with opioids for all time periods was similar to figure 1 (top panel), but stabilization across the longer times (31-90, 91-400, and >400 days) was not as clear, with variability observed from three to 12 months after stopping treatment with opioids. The full life table descriptive data for figure 1 (bottom panel) are in appendix 5.

Discussion

Principal findings

In this quality improvement evaluation, we examined the risk of death from overdose or suicide associated with stopping treatment with opioids and length of treatment, in patients in the Veterans Health Administration prescribed opioids in fiscal year 2013. We found that the interaction between stopping treatment and length of treatment with opioids was associated with an increased risk of death from overdose or suicide the longer patients had been treated with opioids before stopping. The risk was pronounced in patients treated the longest. Also independently associated with a greater risk of death from overdose or suicide were not being married, male sex, having a diagnosis of a mental health or substance use disorder, medical comorbidities, and higher doses of opioids, similar to previous reports.26272837 Our descriptive analyses showed the rise in the number of deaths from overdose or suicide in the months after stopping or starting opioid treatment, highlighting these vulnerable risk periods for patients. Our findings suggest that transitions (starting and stopping treatment) were associated with an increase in the risk for overdose and suicide for patients treated with opioids for any length of time, with increasing risk the longer patients were treated.

Comparisons with other studies

Previous studies focused mostly on smaller regional samples of patients treated with high doses of opioids or for long periods of time,1718 whereas our study used national data from the Veterans Health Administration and included all patients prescribed opioids. All patients exposed to opioids had an increased risk of death from overdose or suicide after starting or stopping treatment with opioids. Although patients treated with opioids for long periods (eg, >400 days in our evaluation) had the highest hazard ratios after stopping treatment, even those treated for up to 30 days had a rise in the risk of death after treatment was stopped (hazard ratio of 1.4 for death from overdose after stopping opioid treatment and 1.7 for death from overdose or suicide). Those treated with opioids for 31-90 days had a hazard ratio of 2.4 for death from overdose after stopping opioid treatment (2.8 for death from overdose or suicide). Our study reinforces the importance of the first three months after stopping opioid treatment, extending the high risk period from 77 days17 to at least 100 days.

Priorities for future research

Future research should examine factors that might affect the risk or explain these findings. Glanz et al16 found that high dose variability was a major predictor of increased risk for opioid overdose, and the rate of tapering has been hypothesized to moderate the risk after stopping treatment. This is concerning given that Mark et al17 found that over half of patients treated with high doses of opioids for long periods of time suddenly stopped treatment with no reductions in dose (in one day), with 86% stopping within 21 days. The reasons for stopping treatment are also important. Demidenko et al15 found evidence of suicidal thoughts and self-harm in patients with and without substance use disorders whose clinicians stopped their long term treatment with opioids. James et al18 observed an increased risk of death from overdose in patients whose provider initiated stopping treatment although the findings were not statistically significant. In 77% of patients, at least one reason for stopping treatment came from the patients’ providers whereas in only 18% had the patient decided to stop treatment. The risk of stopping treatment might reflect worsening pain, conflicts with or disengagement from providers, particularly during exacerbations of mental health disorders (eg, aberrant behaviors, substance misuse, or mania), symptoms related to withdrawal, or a combination of these factors.153839 Our findings suggest that future studies should include all patients prescribed opioids, not just those on long term treatment, and look at suicide behaviors in addition to outcomes of overdose.

Our evaluation did not focus solely on long term treatment with opioids or whether stopping treatment was voluntary, but our findings raise questions on whether evaluating stopping treatment with opioids in isolation is an effective population based strategy to reduce death from overdose or suicide. As we observed risks associated with starting and stopping treatment, our findings are consistent with: the recommendations in the Veterans Affairs/Department of Defense Clinical Practice Guideline against starting long term opioid treatment,13 the recently released guide for clinicians from the US Department of Health and Human Services on the appropriate dose reduction or discontinuation of long term opioid analgesics (HHS guide),4041 an article by some of the lead authors of the guidelines from the Center for Disease Control and Prevention on long term prescribing of opioids42 highlighting the need for close monitoring and mitigation of risk after reducing or stopping treatment with opioids to avoid adverse outcomes, and the guidelines from the Veterans Affairs/Department of Defense and Center for Disease Control and Prevention to reduce reliance on opioids for the treatment of acute pain conditions.43 More research is needed to understand outcomes after stopping treatment with opioids in practice, particularly when to stop and which patients might benefit or be harmed by stopping treatment. Death from overdose or suicide was not limited to those with a prescription for opioids in this evaluation; patients who stopped treatment might have had unmanaged pain, or symptoms of opioid withdrawal and mood instability, which could have driven other risky strategies to obtain relief (eg, illicit opioids, sedating medications, self-harm). A better understanding of the risk after stopping treatment with opioids might help inform and improve the clinical care and outcomes for these patients.

Implications for clinical care

For clinical care, our findings suggest that patient centered efforts to mitigate the risk of death from overdose or suicide in patients prescribed opioids need to look at a broad array of risk factors for patients rather than focus solely on long term prescription management. Our findings suggest close monitoring for at least three months for any patient stopping treatment with opioids but especially in those on long term treatment or those with other risk factors (eg, mental health and substance use disorders). Pain conditions that required prescription of opioids, along with comorbid conditions, should be continually managed and treated. Surveillance, engagement with patients, and efforts to mitigate the risks are important when treatment is started and stopped, and especially in patients who have been treated with opioids for a long period of time before stopping.

Preservation of relationships with healthcare providers is important because the reasons for their concerns about prescribing opioids (eg, mental health or substance use disorders, polypharmacy, medical comorbidities, or social risks) might be the same issues that place patients at risk for overdose or suicide after stopping or starting treatment with opioids. Thus clinicians should avoid dismissing patients from care, ensure that patients continue to receive coordinated care, and have policies that allow them to account for each patient’s unique circumstances when making clinical decisions about prescribing opioids.404142 Also, more detailed guidance and standard mitigation of risk after stopping or starting treatment with opioids (eg, guidance on prescription drug monitoring programs, urine drug screening, prescription of naloxone, frequency of follow-up appointments, evaluation of suicide risk) might be warranted. In general, implementing recommended safety strategies for starting patients on long term opioid treatment (eg, close monitoring, engagement with patients, pain management) might be relevant even for short term treatment, and is especially important after stopping treatment, particularly in those patients ending longer courses of opioids.

Limitations of the study

There are limitations to this evaluation. First, an observational evaluation cannot prove cause and effect and is subject to confounding. We cannot say from these data whether the increased risk was caused by a change in prescription; circumstances that triggered the decision to start or stop prescribing an opioid (eg, illness, injury, development of addiction, or other adverse events) might drive the increased risk of overdose or suicide. Second, our primary variable (stopping treatment with opioids) did not take into consideration the reasons or clinical intentions for stopping, or the speed of its execution.4144 We did not record whether patients stopped treatment voluntarily or because their pain resolved, or if they were concerned about adverse outcomes. Moreover, because we included all patients prescribed opioid analgesics (a large proportion of whom were treated with short acting opioids), we expected our findings would be biased towards a lower risk associated with stopping treatment, because many patients stop taking opioids after recovery from short term injuries whereas long term pain is a risk factor for suicide.454647 The interaction between stopping treatment with opioids and length of treatment suggested a lower risk for patients treated for shorter periods, but the risks were relatively high even in patients prescribed opioids for 90 days or less. Third, we lacked data on adherence to treatment, access to unused drugs after the estimated prescription period, and opioids obtained outside of the Veterans Health Administration. Fourth, to ensure that the risks were not related to restarting treatment with opioids after stopping, we examined the number of patients who had another opioid prescription in fiscal year 2014 after the date of stopping treatment in fiscal year 2013 to fiscal year 2014 (that is, after the date of stopping treatment for their last opioid prescription episode identified in fiscal year 2013 but within our observation period of fiscal year 2013 to fiscal year 2014). Of the 799 668 patients who stopped treatment with opioids in our evaluation, 4071 (0.5%) had another prescription for an opioid in fiscal year 2014 after their date of stopping treatment. None of these 4071 patients died of suicide or overdose during our evaluation period. Finally, our findings might not apply to other populations.

Conclusions

The patients in our evaluation were at greater risk of death from overdose or suicide after stopping treatment with opioids, with the risk increasing the longer patients had been treated before stopping. Our descriptive data suggested starting treatment with opioids is a risk period. In general, our evaluation identified the first three months after starting or stopping treatment with opioids as risk periods that offer clear opportunities for clinical intervention to improve the safety of prescribing opioids. These periods have not been a focus of guidelines for long term treatment with opioids.

Close monitoring of patients for up to one year after stopping treatment with opioids might be warranted. We suggest that efforts to reduce deaths from overdose and suicide take a broader view on patient safety, beyond opioid management, to mitigate the risk from the patient’s perspective. Efforts should look at risk factors that might place patients at risk for overdose or suicide whether they continue or stop treatment with opioid analgesics. Greater preventative measures and engagement with healthcare providers are needed in the first three months after starting or stopping treatment with opioids.

What is already known on this topic

  • In the US, a rise in deaths from overdose and opioid use disorders has prompted initiatives to reduce the risks associated with prescribing opioids, encouraging stopping treatment when risks exceed benefits

  • Studies have identified a rise in adverse events (death from overdose, suicidal thoughts, self-harm, and hospital admissions or emergency department visits for substance related events) after stopping treatment with opioids in patients treated long term

What this study adds

  • In patients prescribed opioids in the Veterans Health Administration, stopping treatment with opioids at any time had an increased risk of death from overdose or suicide, with the risk increasing the longer patients were treated

  • Efforts to mitigate the risk (patient monitoring, prevention of suicide and overdose) should be intensified for at least three months after starting or stopping treatment with opioids

Acknowledgments

We thank Eleanor Lewis for feedback on drafts of this paper and acknowledge the sacrifices of men and women who have served in the US military.

Footnotes

  • Contributors: EMO supervised the evaluation and is the guarantor. All authors contributed to the design and conduct of the evaluation, interpretation of the data, and preparation, review, and approval of the manuscript. TB, PH, AR, and MP assisted with data collection and management. TB and PH assisted with data analyses. EMO is the corresponding author and attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding: This project was supported by the Veterans Affairs Office of Mental Health and Suicide Prevention but the contents of this work do not represent the views of the Veterans Affairs or the US government.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the Veterans Affairs Office of Mental Health Suicide Prevention for the submitted work; JMH received support from grants from the Office of the Director, National Institutes of Health (grant R01DA045027); SK owns stock in CVS Health, Thermo Fisher Scientific, and Zimmer Biomet, not exceeding 5% of his assets; he also personally owned stock in two pharmaceutical companies (Abbott and Merck) before 31 December 2017, which amounted to less than 5% of his assets, and were sold before the end of 2017. SK’s spouse personally owns stock in Abbott, Merck, and Johnson & Johnson, not exceeding 10% of her assets. SK has offered opinions in the peer reviewed and lay literature on the ethics of non-consensual or suddenly stopping opioids in patients with pain. All other authors declare no financial relationships with any organizations that might have an interest in the submitted work in the previous three years, and no other relationships or activities that could appear to have influenced the submitted work.

  • Ethical approval: Not required. This quality improvement project was not considered research by the institutional review board of Stanford University.

  • Data sharing: No additional data available.

  • The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the evaluation being reported; that no important aspects of the evaluation have been omitted; and that any discrepancies from the evaluation as originally planned (and, if relevant, registered) have been explained.

  • Dissemination to participants and related patient and public communities: We shared preliminary findings from an early version of this evaluation in the Fall 2018 with the US Food and Drug Administration when the agency was looking into the risk associated with stopping treatment with opioids (note: the US Food and Drug Administration released a safety announcement on 9 April 2019, on the harms of sudden discontinuation of opioids (www.fda.gov/drugs/drug-safety-and-availability/fda-identifies-harm-reported-sudden-discontinuation-opioid-pain-medicines-and-requires-label-changes) which received substantial public attention). Findings and recommendations from this study will be incorporated into the Veterans Health Administration’s nationally standardized patient and provider educational materials and computerized decision support systems designed to support risk-benefit and opioid management discussions for patients on or considering opioid prescriptions. These standardized tools are used in required informed consent discussions for chronic opioid treatment and risk reviews before starting opioids and are provided to support ongoing management of patients treated with opioids. At the time of acceptance of the manuscript, this revision process is in progress and has been partially implemented. The authors will work with the Veterans Health Administration Communications to develop a press release on the findings to increase public awareness, subsequent to publication of the manuscript.

http://creativecommons.org/licenses/by-nc/4.0/

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

References

  1. ↵
    1. Rudd RA,
    2. Seth P,
    3. David F,
    4. Scholl L
    . Increases in drug and opioid-involved overdose deaths - United States, 2010-2015. MMWR Morb Mortal Wkly Rep2016;65:1445-52. doi:10.15585/mmwr.mm655051e1 pmid:28033313
    OpenUrlCrossRefPubMed
  2. ↵
    1. Saha TD,
    2. Kerridge BT,
    3. Goldstein RB,
    4. et al
    . Nonmedical prescription opioid use and DSM-5 nonmedical prescription opioid use disorder in the United States. J Clin Psychiatry2016;77:772-80. doi:10.4088/JCP.15m10386 pmid:27337416
    OpenUrlCrossRefPubMed
  3. ↵
    1. Mokdad AH,
    2. Ballestros K,
    3. Echko M,
    4. et al.,
    5. US Burden of Disease Collaborators
    . The state of US health, 1990-2016: burden of diseases, injuries, and risk factors among US states. JAMA2018;319:1444-72. doi:10.1001/jama.2018.0158 pmid:29634829
    OpenUrlCrossRefPubMed
  4. ↵
    1. Martins SS,
    2. Sarvet A,
    3. Santaella-Tenorio J,
    4. Saha T,
    5. Grant BF,
    6. Hasin DS
    . Changes in US lifetime heroin use and heroin use disorder: prevalence from the 2001-2002 to 2012-2013 National Epidemiologic Survey on Alcohol and Related Conditions. JAMA Psychiatry2017;74:445-55. doi:10.1001/jamapsychiatry.2017.0113 pmid:28355458
    OpenUrlCrossRefPubMed
  5. ↵
    St Amour M. Central Maine patients fear weaning off opioids as they struggle with chronic pain. 21 January 2017. https://www.centralmaine.com/2017/01/21/central-maine-patients-fear-medication-weaning-as-they-struggle-with-chronic-pain/
  6. ↵
  7. ↵
    1. García MC,
    2. Dodek AB,
    3. Kowalski T,
    4. et al
    . Declines in opioid prescribing after a private insurer policy change - Massachusetts, 2011-2015. MMWR Morb Mortal Wkly Rep2016;65:1125-31. doi:10.15585/mmwr.mm6541a1 pmid:27764082
    OpenUrlCrossRefPubMed
  8. ↵
    Centers for Medicare and Medicaid Services. Centers for Medicare and Medicaid Services (CMS) opioid misuse strategy 2016. United States Department of Health and Human Services; 5 January 2017. https://www.cms.gov/Outreach-and-Education/Outreach/Partnerships/Downloads/CMS-Opioid-Misuse-Strategy-2016.pdf
  9. ↵
    1. Murthy VH
    . Ending the opioid epidemic - a call to action. N Engl J Med2016;375:2413-5. doi:10.1056/NEJMp1612578 pmid:27959718
    OpenUrlCrossRefPubMed
  10. ↵
    1. Dowell D,
    2. Haegerich TM,
    3. Chou R
    . CDC guideline for prescribing opioids for chronic pain - United States, 2016. MMWR Recomm Rep2016;65:1-49. doi:10.15585/mmwr.rr6501e1 pmid:26987082
    OpenUrlCrossRefPubMed
  11. ↵
    1. Meara E,
    2. Horwitz JR,
    3. Powell W,
    4. et al
    . State legal restrictions and prescription-opioid use among disabled adults. N Engl J Med2016;375:44-53. doi:10.1056/NEJMsa1514387 pmid:27332619
    OpenUrlCrossRefPubMed
  12. ↵
    1. Kertesz SG
    . Turning the tide or riptide? The changing opioid epidemic. Subst Abus2017;38:3-8.
    OpenUrl
  13. ↵
    Opioid Therapy for Chronic Pain Work Group. VA/DoD clinical practice guideline for opioid therapy for chronic pain. Washington, DC: Department of Veterans Affairs Department of Defense; 1 March 2017.
  14. ↵
    1. Bohnert ASB,
    2. Guy GP Jr.,
    3. Losby JL
    . Opioid prescribing in the United States before and after the Centers for Disease Control and Prevention’s 2016 opioid guideline. Ann Intern Med2018;169:367-75. doi:10.7326/M18-1243 pmid:30167651
    OpenUrlCrossRefPubMed
  15. ↵
    1. Demidenko MI,
    2. Dobscha SK,
    3. Morasco BJ,
    4. Meath THA,
    5. Ilgen MA,
    6. Lovejoy TI
    . Suicidal ideation and suicidal self-directed violence following clinician-initiated prescription opioid discontinuation among long-term opioid users. Gen Hosp Psychiatry2017;47:29-35. doi:10.1016/j.genhosppsych.2017.04.011 pmid:28807135
    OpenUrlCrossRefPubMed
  16. ↵
    1. Glanz JM,
    2. Binswanger IA,
    3. Shetterly SM,
    4. Narwaney KJ,
    5. Xu S
    . Association between opioid dose variability and opioid overdose among adults prescribed long-term opioid therapy. JAMA Netw Open2019;2:e192613. doi:10.1001/jamanetworkopen.2019.2613 pmid:31002325
    OpenUrlCrossRefPubMed
  17. ↵
    1. Mark TL,
    2. Parish W
    . Opioid medication discontinuation and risk of adverse opioid-related health care events. J Subst Abuse Treat2019;103:58-63. doi:10.1016/j.jsat.2019.05.001 pmid:31079950
    OpenUrlCrossRefPubMed
  18. ↵
    1. James JR,
    2. Scott JM,
    3. Klein JW,
    4. et al
    . Mortality after discontinuation of primary care-based chronic opioid therapy for pain: a retrospective cohort study. J Gen Intern Med2019;34:2749-55. doi:10.1007/s11606-019-05301-2 pmid:31468341
    OpenUrlCrossRefPubMed
  19. ↵
    1. Breckenridge J,
    2. Clark JD
    . Patient characteristics associated with opioid versus nonsteroidal anti-inflammatory drug management of chronic low back pain. J Pain2003;4:344-50. doi:10.1016/S1526-5900(03)00638-2 pmid:14622692
    OpenUrlCrossRefPubMedWeb of Science
  20. ↵
    1. Sullivan MD,
    2. Edlund MJ,
    3. Zhang L,
    4. Unützer J,
    5. Wells KB
    . Association between mental health disorders, problem drug use, and regular prescription opioid use. Arch Intern Med2006;166:2087-93. doi:10.1001/archinte.166.19.2087 pmid:17060538
    OpenUrlCrossRefPubMedWeb of Science
  21. ↵
    1. Oliva EM,
    2. Bowe T,
    3. Tavakoli S,
    4. et al
    . Development and applications of the Veterans Health Administration’s Stratification Tool for Opioid Risk Mitigation (STORM) to improve opioid safety and prevent overdose and suicide. Psychol Serv2017;14:34-49. doi:10.1037/ser0000099 pmid:28134555
    OpenUrlCrossRefPubMed
  22. ↵
    1. Frank JW,
    2. Lovejoy TI,
    3. Becker WC,
    4. et al
    . Patient outcomes in dose reduction or discontinuation of long-term opioid therapy: a systematic review. Ann Intern Med2017;167:181-91. doi:10.7326/M17-0598 pmid:28715848
    OpenUrlCrossRefPubMed
  23. ↵
    1. Gellad WF,
    2. Good CB,
    3. Shulkin DJ
    . Addressing the opioid epidemic in the United States: lessons from the Department of Veterans Affairs. JAMA Intern Med2017;177:611-2. doi:10.1001/jamainternmed.2017.0147 pmid:28288245
    OpenUrlCrossRefPubMed
  24. ↵
    1. Lin LA,
    2. Bohnert ASB,
    3. Kerns RD,
    4. Clay MA,
    5. Ganoczy D,
    6. Ilgen MA
    . Impact of the opioid safety initiative on opioid-related prescribing in veterans. Pain2017;158:833-9. doi:10.1097/j.pain.0000000000000837 pmid:28240996
    OpenUrlCrossRefPubMed
  25. ↵
    1. Hadlandsmyth K,
    2. Mosher H,
    3. Vander Weg MW,
    4. Lund BC
    . Decline in prescription opioids attributable to decreases in long-term use: a retrospective study in the Veterans Health Administration 2010-2016. J Gen Intern Med2018;33:818-24. doi:10.1007/s11606-017-4283-8 pmid:29380212
    OpenUrlCrossRefPubMed
  26. ↵
    1. Bohnert ASB,
    2. Valenstein M,
    3. Bair MJ,
    4. et al
    . Association between opioid prescribing patterns and opioid overdose-related deaths. JAMA2011;305:1315-21. doi:10.1001/jama.2011.370 pmid:21467284
    OpenUrlCrossRefPubMedWeb of Science
  27. ↵
    1. Fulton-Kehoe D,
    2. Sullivan MD,
    3. Turner JA,
    4. et al
    . Opioid poisonings in Washington State Medicaid: trends, dosing, and guidelines. Med Care2015;53:679-85. doi:10.1097/MLR.0000000000000384 pmid:26172937
    OpenUrlCrossRefPubMed
  28. ↵
    1. Zedler B,
    2. Xie L,
    3. Wang L,
    4. et al
    . Risk factors for serious prescription opioid-related toxicity or overdose among Veterans Health Administration patients. Pain Med2014;15:1911-29. doi:10.1111/pme.12480 pmid:24931395
    OpenUrlCrossRefPubMed
  29. ↵
    1. Elixhauser A,
    2. Steiner C,
    3. Harris DR,
    4. Coffey RM
    . Comorbidity measures for use with administrative data. Med Care1998;36:8-27. doi:10.1097/00005650-199801000-00004 pmid:9431328
    OpenUrlCrossRefPubMedWeb of Science
  30. ↵
    Centers for Disease Control and Prevention. Calculating total daily dose of opioids for safer dosage. https://www.cdc.gov/drugoverdose/pdf/calculating_total_daily_dose-a.pdf. 2019.
  31. ↵
    1. Oliva EM,
    2. Bowe T,
    3. Tavakoli S,
    4. et al
    . Development and applications of the Veterans Health Administration’s Stratification Tool for Opioid Risk Mitigation (STORM) to improve opioid safety and prevent overdose and suicide. Psychol Serv2017;14:34-49. doi:10.1037/ser0000099 pmid:28134555
    OpenUrlCrossRefPubMed
  32. ↵
    1. Midboe AM,
    2. Lewis ET,
    3. Paik MC,
    4. et al
    . Measurement of adherence to clinical practice guidelines for opioid therapy for chronic pain. Transl Behav Med2012;2:57-64. doi:10.1007/s13142-011-0104-5 pmid:24073098
    OpenUrlCrossRefPubMed
  33. ↵
    1. Borucka J
    . Extensions of Cox model for non-proportional hazards purpose. Ekonometria2014;3:85-101doi:10.15611/ekt.2014.3.07.
    OpenUrlCrossRef
  34. ↵
    Wilson M. Using SAS to assess and model time-to-event data with non-proportional hazards. Paper presented at: Proceedings of the Midwest SAS User Group Conference 2010.
  35. ↵
    1. Crowley J,
    2. Hu M
    . Covariance analysis of heart transplant survival data. J Am Stat Assoc1977;72:27-36doi:10.1080/01621459.1977.10479903 .
    OpenUrlCrossRefWeb of Science
  36. ↵
    1. Suissa S
    . Immortal time bias in pharmaco-epidemiology. Am J Epidemiol2008;167:492-9. doi:10.1093/aje/kwm324 pmid:18056625
    OpenUrlCrossRefPubMedWeb of Science
  37. ↵
    1. Gomes T,
    2. Mamdani MM,
    3. Dhalla IA,
    4. Paterson JM,
    5. Juurlink DN
    . Opioid dose and drug-related mortality in patients with nonmalignant pain. Arch Intern Med2011;171:686-91. doi:10.1001/archinternmed.2011.117 pmid:21482846
    OpenUrlCrossRefPubMedWeb of Science
  38. ↵
    1. Lovejoy TI,
    2. Morasco BJ,
    3. Demidenko MI,
    4. Meath TH,
    5. Frank JW,
    6. Dobscha SK
    . Reasons for discontinuation of long-term opioid therapy in patients with and without substance use disorders. Pain2017;158:526-34. doi:10.1097/j.pain.0000000000000796 pmid:28192376
    OpenUrlCrossRefPubMed
  39. ↵
    1. Jawad I,
    2. Watson S,
    3. Haddad PM,
    4. Talbot PS,
    5. McAllister-Williams RH
    . Medication nonadherence in bipolar disorder: a narrative review. Ther Adv Psychopharmacol2018;8:349-63. doi:10.1177/2045125318804364 pmid:30524703
    OpenUrlCrossRefPubMed
  40. ↵
    US Department of Health and Human Services. HHS guide for clinicians on the appropriate dosage reduction or discontinuation of long-term opioid analgesics. 2019. https://www.hhs.gov/opioids/treatment/clinicians-guide-opioid-dosage-reduction/index.html.
  41. ↵
    1. Dowell D,
    2. Compton WM,
    3. Giroir BP
    . Patient-centered reduction or discontinuation of long-term opioid analgesics: the HHS guide for clinicians. JAMA2019;1-3. doi:10.1001/jama.2019.16409 pmid:31600366
    OpenUrlCrossRefPubMed
  42. ↵
    1. Dowell D,
    2. Haegerich T,
    3. Chou R
    . No shortcuts to safer opioid prescribing. N Engl J Med2019;380:2285-7. doi:10.1056/NEJMp1904190 pmid:31018066
    OpenUrlCrossRefPubMed
  43. ↵
    1. Bohnert ASB,
    2. Ilgen MA
    . Understanding links among opioid use, overdose, and suicide. N Engl J Med2019;380:71-9. doi:10.1056/NEJMra1802148 pmid:30601750
    OpenUrlCrossRefPubMed
  44. ↵
    1. Mark TL,
    2. Parish W
    . Opioid medication discontinuation and risk of adverse opioid-related health care events. J Subst Abuse Treat2019;103:58-63. doi:10.1016/j.jsat.2019.05.001 pmid:31079950
    OpenUrlCrossRefPubMed
  45. ↵
    1. de Heer EW,
    2. Ten Have M,
    3. van Marwijk HWJ,
    4. et al
    . Pain as a risk factor for suicidal ideation. A population-based longitudinal cohort study. Gen Hosp Psychiatry2018;S0163-8343(18)30027-6. doi:10.1016/j.genhosppsych.2018.11.005 pmid:30528078
    OpenUrlCrossRefPubMed
  46. ↵
    1. Racine M
    . Chronic pain and suicide risk: A comprehensive review. Prog Neuropsychopharmacol Biol Psychiatry2018;87(Pt B):269-80. doi:10.1016/j.pnpbp.2017.08.020 pmid:28847525
    OpenUrlCrossRefPubMed
  47. ↵
    1. Ilgen MA,
    2. Zivin K,
    3. McCammon RJ,
    4. Valenstein M
    . Pain and suicidal thoughts, plans and attempts in the United States. Gen Hosp Psychiatry2008;30:521-7. doi:10.1016/j.genhosppsych.2008.09.003 pmid:19061678
    OpenUrlCrossRefPubMedWeb of Science
Back to top