Tools for assessing quality and risk of bias by levels of evidence

Sun Mi Lim; Ein Soon Shin; Sun Hee Lee; Kyung Hwa Seo; Yu Min Jung; Ji Eun Jang

doi:10.5124/jkma.2011.54.4.419

Journal List > J Korean Med Assoc > v.54(4) > 1042425

Go to TopGo to Top Go to BottomGo to Bottom

TOOLS

Lim, Shin, Lee, Seo, Jung, and Jang: Tools for assessing quality and risk of bias by levels of evidence

Healthcare Policy

Journal of the Korean Medical Association

Journal of the Korean Medical Association 2011; 54(4): 419-429.

Published online: 8 April 2011

DOI: https://doi.org/10.5124/jkma.2011.54.4.419

Tools for assessing quality and risk of bias by levels of evidence

Sun Mi Lim, PhD¹, Ein Soon Shin, PhD², Sun Hee Lee, MD², Kyung Hwa Seo, MBA¹, Yu Min Jung, RN², Ji Eun Jang, BS²

¹Research Institute for Healthcare Policy, Korean Medical Association, Korea.

²Department of Preventive Medicine, Ewha Womans University School of Medicine, Seoul, Korea.

Corresponding author: Ein Soon Shin, shin1121@ewha.ac.kr

Received 30 January 2011 Accepted 17 February 2011

(open-access):

This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Tools for assessing methodological quality or risk of bias in randomized controlled trials (RCTs) and non-randomized studies (NRS) were reviewed. The van Tulder scale and Cochrane's assessment of risk of bias are the two most useful methodological quality evaluation tools for RCTs. Cochrane's tool includes sequence generation, allocation of sequence concealment, blinding, incomplete outcome data, selective outcome reporting, and other potential sources of bias. The Cochrane Collaboration Group recommends the Downs and Black instrument and the Newcastle-Ottawa Scale for evaluating the quality of NRS. In conclusion, this study offers useful information to physicians about tools for assessing the quality of evidence in clinical guidelines. Further research is needed to provide an essential core for evidence-based decision making regarding levels and/or grades of recommendations.

Keywords: Jadad's scale, van Tulder scale, Cochrane's tool, Downs and Black instrument, Newcastle-Ottawa Scale

Figures and Tables

Table 1

Quality assessment tools of randomized controlled trials

CONSORT, consolidated standards of reporting trials; SIGN, Scottish Intercollegiate Guidelines Network; NICE, National Institute for Health and Clinical Excellence.

Table 2

Jadad scale

From Jadad AR, et al. Control Clin Trials 1996;17:1-12, Appendix with permission from Elsevier [23].

Table 3

van Tulder scale [24]

Table 4

Cochrane's assessment of risk of bias [1]

^*Minimization may be implemented without a random element, and this is considered to be equivalent to being random.

Table 5

Quality assessment tools of non-randomized studies

Table 6

Downs and Black scale: checklist for measuring study quality [28]

Table 7A

Newcastle-Ottawa quality assessment scale: case-control studies [29]

Table 7B

Newcastle-Ottawa quality assessment scale: cohort studies [29]

References

1. Higgins JP, Green S, editors. Cochrane handbook for systematic reviews of interventions. Ver. 5.1.0. The Cochrane Collaboration. 2011. cited 2011 Jan 7. Available from: http://www.cochrane-handbook.org.

2. Chalmers TC, Smith H Jr, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A. A method for assessing the quality of a randomized control trial. Control Clin Trials. 1981. 2:31–49.

3. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995. 16:62–73.

4. Barratt A. Evidence based medicine and shared decision making: the challenge of getting both evidence and preferences into health care. Patient Educ Couns. 2008. 73:407–412.

5. Kranke P. Evidence-based practice: how to perform and use systematic reviews for clinical decision-making. Eur J Anaesthesiol. 2010. 27:763–772.

6. Parrilla-Castellar ER, Almeyda R, Nogales E, Velez M, Ramos M, Rivera JE, Da Vila B, Torres V, Capriles J, Adamsons K. Evidence-based medicine as a tool for clinical decision-making in Puerto Rico. P R Health Sci J. 2008. 27:135–140.

7. Tiburi MF. Evidence-based medicine as viewed by key decision-makers of health plans in southern Brazil. Health Serv Manage Res. 2008. 21:185–191.

8. Stolba N, Nguyen TM, Tjoa AM. Towards sustainable decision-support system facilitating EBM. Conf Proc IEEE Eng Med Biol Soc. 2007. 2007:4355–4358.

9. Wyer PC, Silva SA. Where is the wisdom? I. a conceptual history of evidence-based medicine. J Eval Clin Pract. 2009. 15:891–898.

10. Collins J. Evidence-based medicine. J Am Coll Radiol. 2007. 4:551–554.

11. Anastasiu M, Strâmbu V, Popa F. Evidence-based medicine, conceptual challenge or the future of daily practice? Chirurgia (Bucur). 2007. 102:527–530.

12. Carter MJ. Evidence-based medicine: an overview of key concepts. Ostomy Wound Manage. 2010. 56:68–85.

13. Maluf-Filho F. The importance of evidence-based medicine concepts for the clinical practitioner. Arq Gastroenterol. 2009. 46:87–89.

14. Borgerson K. Valuing evidence: bias and the evidence hierarchy of evidence-based medicine. Perspect Biol Med. 2009. 52:218–233.

15. Isaac CA, Franceschi A. EBM: evidence to practice and practice to evidence. J Eval Clin Pract. 2008. 14:656–659.

16. Kruer MC, Steiner RD. The role of evidence-based medicine and clinical trials in rare genetic disorders. Clin Genet. 2008. 74:197–207.

17. Rogers W, Ballantyne A. Justice in health research: what is the role of evidence-based medicine? Perspect Biol Med. 2009. 52:188–202.

18. Kitto S, Petrovic A, Gruen RL, Smith JA. Evidence-based medicine training and implementation in surgery: the role of surgical cultures. J Eval Clin Pract. 2010. 08. 04. [Epub]. DOI: 10.1111/j.1365-2753.2010.01526.x.

19. Carretier J, Bataillard A, Fervers B. The patient's role in evidence-based medicine. J Chir (Paris). 2009. 146:537–544.

20. Soll RF. Evaluating the medical evidence for quality improvement. Clin Perinatol. 2010. 37:11–28.

21. Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, Petticrew M, Altman DG. International Stroke Trial Collaborative Group. European Carotid Surgery Trial Collaborative Group. Evaluating non-randomised intervention studies. Health Technol Assess. 2003. 7:iii–x. 1–173.

22. MacLehose RR, Reeves BC, Harvey IM, Sheldon TA, Russell IT, Black AM. A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies. Health Technol Assess. 2000. 4:1–154.

23. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996. 17:1–12.

24. van Tulder M, Furlan A, Bombardier C, Bouter L. Editorial Board of the Cochrane Collaboration Back Review Group. Updated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine (Phila Pa 1976). 2003. 28:1290–1299.

25. Newell SA, Sanson-Fisher RW, Savolainen NJ. Systematic review of psychological therapies for cancer patients: overview and recommendations for future research. J Natl Cancer Inst. 2002. 94:558–584.

26. Scottish Intercollegiate Guideline Network. SIGN 50: a guideline developer's handbook 2008 [Internet]. 2008. cited 2011 Jan 28. Edinburgh: Scottish Intercollegiate Guidelines Network;Available from: http://www.sign.ac.uk/pdf/sign50.pdf.

27. National Institute for Health and Clinical Excellence. The guidelines manual 2009 [Internet]. 2009. cited 2011 Jan 27. London: National Institute for Health and Clinical Excellence;Available from: http://www.nice.org.uk/media/5F2/44/The_guidelines_manual_2009_-_All_chapters.pdf.

28. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998. 52:377–384.

29. Wells GA, Shea B, O'Connell D, Peterson J, Welch V, Losos M, Tugwell P. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses [Internet]. c1996-2010. cited 2011 Mar 28. Ottawa: Ottwa Hospital Research Institute;Available from: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.

30. Reisch JS, Tyson JE, Mize SG. Aid to the evaluation of therapeutic studies. Pediatrics. 1989. 84:815–827.

31. Thomas H. Quality assessment tool for quantitative studies: effective public health practice project. 2003. Hamilton: McMaster University.

32. Zaza S, Wright-De Agüero LK, Briss PA, Truman BI, Hopkins DP, Hennessy MH, Sosin DM, Anderson L, Carande-Kulis VG, Teutsch SM, Pappaioanou M. Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task Force on Community Preventive Services. Am J Prev Med. 2000. 18:1 Suppl. 44–74.

TOOLS

Similar articles