You are here

  • Home
  • Archive
  • Volume 13, Issue 10
  • Quantitative content analysis of Freedom of Information requests examining the extent and variations of tools and training for conducting suicide risk assessments in NHS Trusts across England

Article Text

Article menu
Mental health
Original research
Quantitative content analysis of Freedom of Information requests examining the extent and variations of tools and training for conducting suicide risk assessments in NHS Trusts across England
  1. Sophia Fedorowicz1,
  2. Robert C Dempsey2,
  3. Naomi J Ellis3,
  4. Olivia Mulvey2,
  5. Christopher J Gidlow4
  1. 1Centre for Health and Development, Staffordshire University, Stoke-on-Trent, UK
  2. 2Department of Psychology, Manchester Metropolitan University, Manchester, UK
  3. 3Faculty of Health Sciences, Staffordshire University, Stoke-on-Trent, UK
  4. 4Centre for Sport, Health and Exercise Research, Staffordshire University, Stoke on Trent, UK
  1. Correspondence to Sophia Fedorowicz; sophia.fedorowicz{at}student.staffs.ac.uk

Abstract

Objectives Determining the risk for suicide is a difficult endeavour. Clinical guidance in the UK explicitly advises against using risk assessment tools and scales to determine suicide risk. Based on Freedom of Information (FoI) requests made to NHS Trusts in England, this study provides an overview of suicide risk assessment tools in use, training provided in how to use such assessments, and explores implementation of suicide risk assessment guidance in practice in English NHS Trusts.

Design A cross-sectional survey of suicide risk assessment tools and training gathered via FoI requests and subjected to a content analysis.

Setting FoI requests were submitted to NHS Trusts across England.

Results A wide variety of suicide risk assessments tools were identified as being used in practice, with several trusts reported using more than one tool to determine suicide risk. Forty-one trusts reported using locally developed, unvalidated, tools to assess risk of suicide and 18 stated they do not use a tool. Ten trusts stated they do not train their staff in suicide risk assessment while 13 reported use of specific suicide risk assessment training. Sixty-two trusts stated they do not centrally record the number of assessments conducted or how many individuals are identified as at risk. Content analysis indicated the frequent wider assessment of risk not restricted to suicide risk.

Conclusions There is wide variation in suicide risk assessment tools being used in practice and some lack of specific training for healthcare staff in determining suicide risk. Few trusts routinely record the number of assessments being conducted or the number of individuals identified at high risk. Implementation of specific training is necessary for the suicide risk assessment process to identify patient needs and develop therapeutic engagement. Routinely recording how many assessments are conducted is a crucial step in improving suicide prevention.

  • mental health
  • protocols & guidelines
  • risk management

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2023-072004

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Freedom of Information requests are conducted by information governance staff based on publicly available written policies. This data provides an insight into the organisational practices and policies across NHS Trusts, including recommended clinical guidance provided to healthcare staff.

    • The current sample represents 39.6% of the 217 NHS Trusts in England (as of April 2020), assumptions about the generalisability of these findings to all trusts should be treated with caution.

    • The data presented here represents the organisation policies and not necessarily the practice of individual healthcare practitioners. This study’s findings need to be interpreted cautiously as they might not reflect actual clinical practice.

    • The data presented here do not provide an insight into healthcare practitioners’ personal experiences about using suicide risk assessments or of training in determining suicide risk.

    Introduction

    Every year there are approximately 700 000 recorded deaths by suicide across the world and many more attempt suicide.1 Target 3.4.2 of the United Nations sustainable development goals is to reduce suicide mortality by a third by 2030.2 In 2019, 5691 individuals died by suicide in England and Wales, an increase on the previous year by 271 deaths.3 A large percentage of people who die by suicide are in contact with the healthcare system in the year before their death, and there is evidence that help-seeking escalates in the weeks before death, providing opportunities for intervention.4–7 Reducing suicide rates and identifying those at high risk remains a priority for healthcare providers as opportunities for intervention rely on effective risk assessments.6

    Healthcare professionals routinely conduct suicide risk assessments across care settings with goals that differ by setting. For example, in accident and emergency departments the common goal is to determine if discharge or escalating the level of care is appropriate; in mental health inpatient settings risk assessment is part of an ongoing screening to monitor high-risk individuals.8 Evidence suggests greater screening for suicide risk is associated with increased detection of suicidal thoughts and feelings.9 However, Large et al10 conducted a meta-analysis of the sensitivity of suicide risk categorisation using the international body of literature, that is, not restricted by country or continent, and found that half of all suicides are likely to occur in a ‘low-risk’ category.10 Further, 95% of ‘high-risk’ individuals will not die by suicide.10 A systematic review of 21 studies (USA (9), UK (6), Canada (3), Australia (1) Sweden (1), Taiwan (1)) concluded that no one scale had sufficient evidence to support its use in clinical practice.11 A further meta-analysis determined no SRA tool is sufficiently accurate to determine an individual’s allocation to an intervention.12 Despite their limitations, suicide risk assessment tools are still widely used to estimate an individuals’s risk of suicide.6 Suicide risk assessment tools produce individual-level predicted probabilities of death by suicide or suicide-related behaviour, including self-harm, within a specific time period, often within the next year.4 Research has focused on identifying risk factors and developing suicide risk assessment tools to identify individuals at high risk, such as the Beck Hopelessness Scale13 and the SAD PERSONS scale,14 the predictive ability of which is low given the overall prevalence of suicide in the general population (0.01%). Such tools are, therefore, not useful as the sole basis for evaluating suicide risk.8

    To support healthcare professionals in assessing suicide risk, several organisations have produced guidelines which typically list common risk factors and potential ‘warning signs’.15 This results in a long lists of non-specific risk factors including having a mental illness diagnosis, a serious or chronic physical illness, life stressors, minority group status (such as being a member of a migrant or LGBTQ+ community), or having access to lethal means (including access to a high place).15 This lack of specificity means that the majority of the population may possess a combination of these risk factors at any given time without necessarily being at risk, making it very difficult for healthcare professionals to identify those high-risk individuals.15 This lack of predictive ability has led to suicide risk assessment tools being encouraged as an aid for decision-making and a guide to gathering information from individuals that help healthcare practitioners to develop care plans and identify service needs.4 8

    WHO guidelines suggest that suicide risk should be specifically evaluated with clinical interviews assessing psychological and social functioning.16 In the UK, the National Institute for Health and Care Excellence (NICE) guidelines emphasise clinical judgement, rather than relying on suicide risk assessment tools alone, to identify risk.17 Evidence suggests that there is considerable variation regarding how suicide risk assessments are conducted in practice in UK healthcare settings.18 There is also evidence of a lack of risk assessment training and a low awareness of suicide prevention guidance.18 At present, how widely these suicide risk assessments guidelines are implemented in the UK, including the extent to which suicide risk assessment tools are used.

    This paper reports findings from requests made under the Freedom of Information Act (2000) sent to NHS Trusts in England to identify the suicide risk assessment tools in use and the training of healthcare professionals in relation to these suicide risk assessment tools or assessment of suicide risk generally. NHS Trusts were targeted as, in the UK, over a third of people who die by suicide are in contact with a mental health service prior to death, with 13% attending emergency departments in the month before they die.5 7 Therefore, the nature of the interactions and the identification of suicide risk in hospital settings is critical for identification of those at highest risk and for suicide prevention. The focus on England was important for several reasons. In 2019, 5316 suicides were recorded in England and Wales, the highest recorded rate since the year 2000, and recorded male and female deaths by suicide have increased from the previous year (5.7% and 6.3%, respectively).19 The structure of NHS England includes NHS Trusts which are large organisations including hospitals, ambulance services, mental health services and social care, all delivered by a diverse range of healthcare professionals who interact with the public and have varying skills and training. Health spending and care provision in the UK has been transferred from central government to the individual countries (Scotland, Wales, Northern Ireland and England) and so the focus on England (rather than the whole of the UK) enabled the data to be easily integrated. Differing healthcare systems, for example, across countries or market-regulated, have different outcomes regarding continuity of care for people accessing mental health treatment.20 Further, differing attitudes towards public health between countries impacts on individual risk factors for suicide, for example, physical environment, and on healthcare spending.21 These factors, and others, will have impacts on which tools are used and how, and what training is implemented and how.

    Method

    Freedom of Information (FoI) requests were sent via email to 158 Trusts across England between October and November 2019. Email addresses were identified through the Trust websites. Trusts were asked to respond to six questions:

    1. Which suicide risk assessments do you use at your trust?

    2. What training do your staff undertake in using those suicide risk assessments?

    3. What training do your staff have in assessing risk of suicide in general?

    4. What is the procedure your staff follow when they identify a person at risk of suicide?

    5. How many people have been assessed for risk of suicide between 2012 and 2020, if possible, by month and department?

    6. How many people were identified as at risk of suicide between 2012 and 2020, if possible, by month and department?

    Responses were sent via email to an address created specifically for this project. Those that requested clarification on the initial request were sent a second e-mail to specify the information being requested.

    Patient involvement statement

    No patients were involved in the present study.

    Data handling and analysis

    Data were extracted from the FoI responses in relation to suicide risk assessment tools and training, and quantitative content analysis was conducted to explore the implementation of suicide risk assessment guidelines. Several responses were returned in the form of blocks of text rather than specific answers to individual questions and so a quantitative, deductive content analysis was selected as a systematic, objective means of describing and quantifying the surface meaning of text material.22 23 The coding scheme was developed by SF and RCD comprising code name, code definition, text examples and coding rules. The codes were derived from literature detailing how suicide risk assessments are conducted in the UK18 and NICE guidelines, and sought to explore how risk was assessed in practice. Each response to the FoI request was formatted to ensure consistency across documents, and each line treated as a single unit. As some sentences ran across more than one unit, coders agreed that all units containing information related to a single data point would be coded. Therefore, results would indicate how much of a single FoI response is dedicated to each code. For example, if a response constituted 20 units (lines) of data (text), analysis would determine how many of those units contained information about clinical judgement (code 5).

    The coding scheme was piloted by SF and OM coding one response independently. Authors then compared codes, discussed discrepancies, difficulties in applying the scheme and adjusted the coding scheme. SF and OM used the second version of the coding scheme to independently code seven responses, and Krippedorff’s Alpha test24 was used to determine intercoder reliability (test specifically developed for use in content analysis). In light of low intercoder reliability, the scheme was further revised following discussion between SF, RCD and OM. This third version of the coding scheme was used to code a second set of seven responses, with excellent intercoder reliability (α=0.82). This version was then used to code all the responses (table 1).

    View this table:
    Table 1

    Content analysis codebook: suicide risk assessment process and tools

    Results

    From the 158 FoI requests sent, 86 (54.43%) responses from Trusts were received, 9 of which did not provide any information for the study (eg, stating that they did not provide information for research purposes via FoI requests).

    Overview of tools used

    There was wide variation in the responses to question 1, ‘what suicide risk assessment tools do your staff use?’. A total of 77 responses were collected and a summary is provided in table 2. For brevity, responses relating to the use of a ‘locally developed’ tools or a ‘Proforma’ have been combined into the ‘locally developed risk assessments’ category.

    View this table:
    Table 2

    Overview of suicide risk assessment tools used by Trusts

    Several Trusts stated using more than one tool and some reported using tools that were not specifically for suicide risk assessments (see online supplemental file 1 for a full list of responses to question 1). Two Trusts stated they did not know what tools clinicians used to assess suicide risk or if a tool was used at all.

    Overview of training

    There were 72 FoI responses to question 3 relating to the training in suicide risk assessments provided to staff by Trusts. For brevity, a summary of responses have been categorised in table 3 (see online supplemental file 2 for full list of responses to question 3).

    View this table:
    Table 3

    Overview of training carried out by Trusts

    Data recording

    There were 74 responses for requests for information about the numbers of people assessed and identified as being at risk for suicide (questions 5 and 6). Of these, 62 stated that they did not centrally record the number of people assessed or identified as at risk and therefore it was not possible to supply the data requested. Six responded that these questions were not applicable. Five Trusts responded with the number of people assessed, and of those four recorded how many were identified as high risk. One Trust responded with the explanation that it is their policy not to identify a person as being or not being ‘at risk’ and prefer to be guided by Skills based Training on Risk Management (STORM) training and develop care and safety plans with the person.

    Content analysis

    Seventy-four responses were usable for content analysis (excluded responses include those that sent policy documents, a response to a separate FoI request from 2017 and a response stating they did not answer FoI requests for research). A total of 884 units were coded based on the returned responses. One hundred and twenty-one (13.6%) units were identified as discussing the wider assessment of risk, including evidence of the identification of clinical and demographic features known to be suicide risk factors, such as depression and hopelessness. One hundred and seventy-one (19.9%) units identified discussing sharing of risk between staff and departments, care planning and referrals. Eight (0.9%) of the identified units provided evidence of healthcare professionals using their experience and knowledge to steer the risk assessment process (see table 4 for detailed examples of coded responses).

    View this table:
    Table 4

    Summary of the responses by the different categories of suicide risk assessment reported being used

    Discussion

    Many people who die by suicide are in contact with healthcare services prior to their death, therefore understanding how suicide risk assessments are undertaken in practice is important given the potential for intervention to prevent suicide. This FoI study provides an overview of which suicide risk assessment tools are used in NHS Trusts in England, what associated training is provided to healthcare professionals, and how UK national (NICE) guidance for assessing the risk of suicide is implemented.

    Our key findings indicate that most trusts who responded to the FoI requests reported using locally developed suicide risk assessment tools. ‘Locally developed’ is a term frequently used in the literature to describe a suicide risk assessment tool that is not psychometrically validated. Such unvalidated tools may have been developed by the Trust for internal use, and such risk assessment tools appear to be used frequently in UK healthcare settings.18 Furthermore, in the current FoI study, 18 trusts (20.9%) reported using no tool at all and several stated using more than one. Ten (11.6%) trusts reported providing no training for any staff regarding assessing risk of suicide, 13 (15.1%) provided specific suicide risk assessment training and others were unsure about what training would equip their staff with this skill, stating ligature training or mental health first aid in response to a question about general suicide risk training for staff (see online supplemental file 2). Our data highlight wide variation in the use of suicide risk assessment tools and inconsistent staff training in how to carry out a suicide risk assessment across NHS Trusts in England. The lack of established standards in risk assessment may be in part due to their low predictability; however, it was beyond the scope of this work to investigate why individual trusts choose different approaches to suicide risk assessment and therefore would be inappropriate to draw conclusions. Inconsistency in tools and training across healthcare services is unsurprising given what we know from the existing literature. Data collected from across Europe, North America, South America, Asia and Australasia found over 200 commercially available suicide risk assessment tools and a further 200 institutionally or individually developed tools in use.25 There is evidence of similar inconsistency across mental health services in the UK in the use of risk assessment tools with almost 39% using locally developed tools mirroring the present study findings.26 There is documented variation of which suicide risk assessment tools are used and how they are used between and within NHS Trusts, and of low healthcare professional confidence in assessing suicide risk potentially due to a lack of training.18 The continued use of suicide risk assessment tools, despite explicit clinical guidance discouraging their use,16 17 is possibly due to the perceived lack of alternatives.26 Low levels of healthcare professional confidence in assessing suicide risk due to inadequate training could also be a factor in why healthcare professionals seek the reassurance of a tool.

    An important omission from the current suicide risk assessments process is the routine recording of how many risk assessments are carried out and how many people are consequently identified as being at risk. Almost 90% of mental health NHS Trusts are unable to identify which patients who had died by suicide had received the treatment options recommended by NICE guidance due to the information not being centrally recorded.27 Given the seriousness of suicide risk, the accurate recording of suicide risk data would help to identify patterns in treatments offered, the associated health outcomes, and the efficacy of risk management processes. A cross-government strategy that aimed to prevent suicides in England outlined the need to improve and expand the systematic collection of data pertaining to suicides, and while mortality statistics are routinely recorded and made available allowing for the identification of national trends, they do not facilitate detailed analysis.28 Routinely recording data related to the number of suicide risk assessments conducted, and subsequent care provision, is crucial for targeting prevention efforts more accurately, monitoring adherence to NICE guidance, and in evaluating the effectiveness of current practice.

    Implications for practice

    Our findings identify the need for greater parity in suicide risk assessment across and within English NHS Trusts. Currently, it is possible to receive two different suicide risk assessments within the same department at a specific Trust, resulting in different outcomes.18 This potentially puts people at risk because the treatment offered depends on the tool used and the level of healthcare professional training, rather than NICE-recommended needs-based assessment. The reason for such wide variation in suicide risk assessments tools and lack of specific suicide risk assessment training needs further investigation. Additionally, there is a need to explore how well frontline healthcare professionals follow Trust guidance or national recommendations for clinical practice when working with people who are suicidal. Specific training in suicide risk assessment and prevention is needed for the assessment process to be a meaningful exercise. Emphasis should be on using the risk assessment process not as a means of predicting suicide or suicidal behaviours in an individual but as a means of identifying needs and developing therapeutic engagement.

    Limitations

    The current sample represents 39.6% of the 217 NHS Trusts in England (as of April 2020). Many NHS Trust websites were not up to date in terms of the trust structure or current contact for FoI requests. Difficulty in determining how to make the requests was a barrier for this study and has wider implications for the public’s ability to access information about public authorities, such as the NHS. As responses to these questions were obtained via FoI requests which were responded to by information governance staff and not directly by healthcare professionals, the data presented here must be interpreted cautiously as they do not necessarily reflect the individual practice of healthcare professionals. How well frontline healthcare professionals follow Trust guidance or national recommendations for clinical practice when working with people who are suicidal is not clear based on the data presented here.

    Conclusion

    This FoI study highlights wide variation in suicide risk assessment tools and a lack of specific training in suicide risk assessment in NHS Trusts in England. The content analysis of Trust responses indicates that there is wider assessment of risk taking place, but practitioners’ clinical judgement does not appear to feature that prominently in the suicide risk assessment process. The implementation of specific training is necessary for the risk assessment process to identify patient needs and to be a meaningful suicide prevention exercise. Few Trusts appear to routinely record the number of risk assessments conducted and how many people are subsequently identified as being at risk, which limits the potential to evaluate how well the current assessment processes work, and if these are sufficient to manage risk. The introduction of routine data recording could determine the efficacy of the processes currently in place.

    Data availability statement

    All data relevant to the study are included in the article or uploaded as supplementary information.

    Ethics statements

    Patient consent for publication

    Ethics approval

    Not applicable.

    References

    1. ↵
      1. World Health Organization (WHO)
      . Suicide worldwide in 2019: global health estimates. 2021. Available: https://apps.who.int/iris/rest/bitstreams/1350975/retrieve
    2. ↵
      1. United Nations
      . Goal 3 | Department of economic and social affairs. n.d. Available: https://sdgs.un.org/goals/goal3
    3. ↵
    4. ↵
      1. Kessler RC,
      2. Bossarte RM,
      3. Luedtke A, et al
      . Suicide prediction models: a critical review of recent research with recommendations for the way forward. Mol Psychiatry 2020;25:168–79. doi:10.1038/s41380-019-0531-0
      OpenUrlCrossRefPubMed
    5. ↵
      1. Leavey G,
      2. Rosato M,
      3. Galway K, et al
      . Patterns and predictors of help-seeking contacts with health services and general practitioner detection of Suicidality prior to suicide: a cohort analysis of suicides occurring over a two-year period. BMC Psychiatry 2016;16:120. doi:10.1186/s12888-016-0824-7
    6. ↵
      1. Luoma JB,
      2. Martin CE,
      3. Pearson JL
      . Contact with mental health and primary care providers before suicide: a review of the evidence. Am J Psychiatry 2002;159:909–16. doi:10.1176/appi.ajp.159.6.909
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. John A,
      2. DelPozo-Banos M,
      3. Gunnell D, et al
      . Contacts with primary and secondary healthcare prior to suicide: case–control whole-population-based study using person-level linked routine data in Wales, UK, 2000–2017. Br J Psychiatry 2020;217:717–24. doi:10.1192/bjp.2020.137
      OpenUrlCrossRefPubMed
    8. ↵
      1. Sequeira L,
      2. Strudwick G,
      3. Bailey SM, et al
      . Factors influencing suicide risk assessment clinical practice: protocol for a scoping review. n.d. Available: http://bmjopen.bmj.com/
    9. ↵
      1. Boudreaux ED,
      2. Camargo CA,
      3. Arias SA, et al
      . Improving suicide risk screening and detection in the emergency department. Am J Prev Med 2016;50:445–53. doi:10.1016/j.amepre.2015.09.029
      OpenUrlCrossRefPubMed
    10. ↵
      1. Large M,
      2. Kaneson M,
      3. Myles N, et al
      . Meta-analysis of longitudinal cohort studies of suicide risk assessment among psychiatric patients: heterogeneity in results and lack of improvement over time. PLoS One 2016;11:e0156322. doi:10.1371/journal.pone.0156322
    11. ↵
      1. Saab MM,
      2. Murphy M,
      3. Meehan E, et al
      . Suicide and self-harm risk assessment: a systematic review of prospective research. Arch Suicide Res 2022;26:1645–65. doi:10.1080/13811118.2021.1938321
      OpenUrl
    12. ↵
      1. Carter G,
      2. Milner A,
      3. McGill K, et al
      . Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales. Br J Psychiatry 2017;210:387–95. doi:10.1192/bjp.bp.116.182717
      OpenUrlAbstract/FREE Full Text
    13. ↵
      1. Beck AT,
      2. Weissman A,
      3. Lester D, et al
      . The measurement of pessimism: the hopelessness scale 1. J Consult Clin Psychol 1974;42:861–5. doi:10.1037/h0037562
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Patterson WM,
      2. Dohn HH,
      3. Bird J, et al
      . Evaluation of suicidal patients: the SAD PERSONS scale. Psychosomatics 1983;24:343–5, doi:10.1016/S0033-3182(83)73213-5
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Franklin JC,
      2. Ribeiro JD,
      3. Fox KR, et al
      . Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull 2017;143:187–232. doi:10.1037/bul0000084
      OpenUrlCrossRefPubMed
    16. ↵
      1. World Health Organization
      . Preventing suicide preventing suicide. WHO Libr Cat Data 2014;89.
    17. ↵
      1. NICE
      . Self-harm in over 8s management. Nice, 2011: 1420–8.
    18. ↵
      1. Fedorowicz SE,
      2. Dempsey RC,
      3. Ellis N, et al
      . How is suicide risk assessed in healthcare settings in the UK? A systematic scoping review. PLoS One 2023;18:e0280789. doi:10.1371/journal.pone.0280789
    19. ↵
      1. Office for National Statistics
      . Suicides in England and Wales: 2019 registrations. Statistical bulletin. 2020.
    20. ↵
      1. Nicaise P,
      2. Giacco D,
      3. Soltmann B, et al
      . Healthcare system performance in continuity of care for patients with severe mental illness: A comparison of five European countries. In: Health policy. Elsevier Ireland Ltd, 2020: 25–36.
    21. ↵
      1. Jarvis T,
      2. Scott F,
      3. El-Jardali F, et al
      . Defining and classifying public health systems: a critical interpretive synthesis. Health Res Policy Syst 2020;18:68. doi:10.1186/s12961-020-00583-z
    22. ↵
      1. Burla L,
      2. Knierim B,
      3. Barth J, et al
      . From text to codings: intercoder reliability assessment in qualitative content analysis. Nurs Res 2008;57:113–7. doi:10.1097/01.NNR.0000313482.33917.7d
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Kleinheksel AJ,
      2. Rockich-Winston N,
      3. Tawfik H, et al
      . Demystifying content analysis. Am J Pharm Educ 2020;84:7113. doi:10.5688/ajpe7113
    24. ↵
      1. Krippendorff K
      . Computing Krippendorff’s alpha-reliability. n.d. Available: https://repository.upenn.edu/asc_papers/43
    25. ↵
      1. Singh JP,
      2. Desmarais SL,
      3. Hurducas C, et al
      . International perspectives on the practical application of violence risk assessment: a global survey of 44 countries. International Journal of Forensic Mental Health 2014;13:193–206. doi:10.1080/14999013.2014.922141
      OpenUrlCrossRefWeb of Science
    26. ↵
      1. Graney J,
      2. Hunt IM,
      3. Quinlivan L, et al
      . Suicide risk assessment in UK mental health services: a national mixed-methods study. Lancet Psychiatry 2020;7:1046–53. doi:10.1016/S2215-0366(20)30381-3
      OpenUrl
    27. ↵
      1. Geekie J,
      2. Read J,
      3. Renton J, et al
      . Do English mental health services know whether they followed N.I.C.E. guidelines with patients who killed themselves? Psychol Psychother 2017;90:797–800. doi:10.1111/papt.12141
      OpenUrl
    28. ↵
      1. Department of Health
      . Preventing suicide in England - a cross-government outcomes strategy to save lives. 2011. Available: www.dh.gov.uk/
      1. Savage A,
      2. Hyde R
      . Using freedom of information requests to facilitate research. International Journal of Social Research Methodology 2014;17:303–17. doi:10.1080/13645579.2012.742280
      OpenUrl

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Contributors SF: conceptualising, data gathering, analysis, interpretation, drafting, revising; RCD: conceptualising, analysis, interpretation, drafting, revising; NJE, CJG: conceptualising, interpretation, revising; OM: analysis, interpretation, drafting, guarantor.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.