Skip to main content
SpringerLink
Log in

Knowledge of Evidence-Based Services Questionnaire: Development and Validation of a Short Form

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

The Knowledge of Evidence-Based Services Questionnaire (KEBSQ) is an objective measure of therapist knowledge of practices derived from the evidence base for the treatment of youth psychopathology. However, the length of this measure (i.e., 40 items) and respondent demands associated with each item makes it burdensome for researchers and clinicians. This study developed and validated a Short Form of the KEBSQ using Item Response Theory measurement models. The Short Form consists of 17 items and generates two separate scores: Correct Endorsements and Correct Rejections. The Short Form was found to correlate highly with and perform similarly to the Full Form, providing preliminary validity evidence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research,6(2), 61–74. https://doi.org/10.1023/B:MHSR.0000024351.12294.65.

    Article  PubMed  PubMed Central  Google Scholar 

  • Becker-Haimes, E. M., Okamura, K. H., Wolk, C. B., Rubin, R., Evans, A. C., & Beidas, R. S. (2017). Predictors of clinician use of exposure therapy in community mental health settings. Journal of Anxiety Disorders,49, 88–94. https://doi.org/10.1016/j.janxdis.2017.04.002.

    Article  PubMed  PubMed Central  Google Scholar 

  • Beidas, R. S., Aarons, G. A., Barg, F., Evans, A., Hadley, T., Hoagwood, K., et al. (2013). Policy to implementation: Evidence-based practice in community mental health–study protocol. Implementation Science, 8(1), 38. https://doi.org/10.1186/1748-5908-8-38.

    Article  PubMed  Google Scholar 

  • Beidas, R. S., & Kendall, P. C. (2010). Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology Science and Practice,17(1), 1–30. https://doi.org/10.1111/j.1468-2850.2009.01187.x.

    Article  PubMed  Google Scholar 

  • Beidas, R. S., Marcus, S., Wolk, C. B., Powell, B., Aarons, G. A., Evans, A. C., et al. (2016). A prospective examination of clinician and supervisor turnover within the context of implementation of evidence-based practices in a publicly-funded mental health system. Administration and Policy in Mental Health and Mental Health Services Research, 43(5), 640–649. https://doi.org/10.1007/s10488-015-0673-6.

    Article  PubMed  Google Scholar 

  • Beidas, R. S., Williams, N. J., Green, P. D., Aarons, G. A., Becker-Haimes, E. M., Evans, A. C., et al. (2018). Concordance between administrator and clinician ratings of organizational culture and climate. Administration and Policy in Mental Health and Mental Health Services Research,45(1), 142–151. https://doi.org/10.1007/s10488-016-0776-8.

    Article  PubMed  Google Scholar 

  • Bliese, P. (2000). Within-group agreement, non-independence, and reliability: implications for data aggregation and analysis. In K. Klein & S. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349–380). San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • Brown, R. D., & Hauenstein, N. M. A. (1993). Rwg: an assessment of within-group interrater agreement. Organizational Research Methods,78(2), 306–309.

    Google Scholar 

  • Chernyshenko, O. S., Stark, S., Chan, K.-Y., Drasgow, F., & Williams, B. (2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research,36(4), 523–562. https://doi.org/10.1207/S15327906MBR3604_03.

    Article  CAS  PubMed  Google Scholar 

  • Choi, S. W. (2009). Firestar: Computerized adaptive testing simulation program for polytomous item response theory models. Applied Psychological Measurement,33(8), 644–645. https://doi.org/10.1177/0146621608329892.

    Article  Google Scholar 

  • Chorpita, B. F., Becker, K. D., & Daleiden, E. L. (2007). Understanding the common elements of evidence-based practice. Journal of the American Academy of Child & Adolescent Psychiatry, 46(5), 647–652. https://doi.org/10.1097/chi.0b013e318033ff71.

    Article  Google Scholar 

  • Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology,77(3), 566–579. https://doi.org/10.1037/a0014565.

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005). Modularity in the design and application of therapeutic interventions. Applied and Preventive Psychology,11(3), 141–156. https://doi.org/10.1016/j.appsy.2005.05.002.

    Article  Google Scholar 

  • Chorpita, B. F., Daleiden, E. L., Ebesutani, C., Young, J., Becker, K. D., Nakamura, B. J., et al. (2011). Evidence-based treatments for children and adolescents: An updated review of indicators of efficacy and effectiveness. Clinical Psychology Science and Practice,18(2), 154–172. https://doi.org/10.1111/j.1468-2850.2011.01247.x.

    Article  Google Scholar 

  • Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science. https://doi.org/10.1186/1748-5908-4-50.

    Article  PubMed  Google Scholar 

  • Dearing, J. W. (2009). Applying diffusion of innovation theory to intervention development. Research on Social Work Practice,19(5), 503–518. https://doi.org/10.1177/1049731509335569.

    Article  PubMed  PubMed Central  Google Scholar 

  • Do, M., Warnick, E., & Weersing, V. (2012). Examination of the psychometric properties of the therapy procedures checklist-family revised. Poster presented at the annual meeting of the Association for Behavioral and Cognitive Therapies (ABCT). Presented at the National Harbor, MD.

  • Edelen, M. O., & Reeve, B. B. (2007). Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Quality of Life Research,16(1), 5–18.

    Article  Google Scholar 

  • Efron, B. (1979). Bootstrap methods: Another look at the jack-knife. The Annals of Statistics,7(1), 1–26.

    Article  Google Scholar 

  • Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science. https://doi.org/10.1186/s13012-014-0157-1.

    Article  PubMed  Google Scholar 

  • Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Earlbaum.

    Google Scholar 

  • Garland, A. F., Bickman, L., & Chorpita, B. F. (2010). Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health and Mental Health Services Research,37(1–2), 15–26. https://doi.org/10.1007/s10488-010-0279-y.

    Article  PubMed  Google Scholar 

  • Glasgow, R. E., & Riley, W. T. (2013). Pragmatic measures. American Journal of Preventive Medicine,45(2), 237–243. https://doi.org/10.1016/j.amepre.2013.03.010.

    Article  PubMed  Google Scholar 

  • Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., et al. (2008). Assessing the Organizational Social Context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research,35(1–2), 98–113. https://doi.org/10.1007/s10488-007-0148-5.

    Article  PubMed  Google Scholar 

  • Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37(3), 230–244.

    Article  Google Scholar 

  • Kolko, D. J., Cohen, J. A., Mannarino, A. P., Baumann, B. L., & Knudsen, K. (2009). Community treatment of child sexual abuse: a survey of practitioners in the National Child Traumatic Stress Network. Administration and Policy in Mental Health and Mental Health Services Researc,36(1), 37–49.

    Article  Google Scholar 

  • Kreier, C. D., & Frisbie, D. A. (1989). Effectiveness of multiple true-false items. Applied Measurement in Education,2, 207–216.

    Article  Google Scholar 

  • Lewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015). Outcomes for implementation science: An enhanced systematic review of instruments using evidence-based rating criteria. Implementation Science. https://doi.org/10.1186/s13012-015-0342-x.

    Article  PubMed  Google Scholar 

  • Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., et al. (2018). From classification to causality: Advancing understanding of mechanisms of change in Implementation Science. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2018.00136.

    Article  PubMed  PubMed Central  Google Scholar 

  • Liew, T. M. (2019). The optimal short version of Montreal Cognitive Assessment in diagnosing mild cognitive impairment and dementia. Journal of the American Medical Directors Association,20, 1055e.1–1055e.8.

    Article  Google Scholar 

  • Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hilldale, NJ: Erlbaum.

    Google Scholar 

  • McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist,65(2), 73–84. https://doi.org/10.1037/a0018121.

    Article  PubMed  Google Scholar 

  • Mokken, R. J. (1971). A theory and procedure of scale analysis. Berlin, Germany: De Gruyter.

    Book  Google Scholar 

  • Moore, T. M., Scott, J. C., Reise, S. P., Port, A. M., Jackson, C. T., Ruparel, K., et al. (2015). Development of an abbreviated form of the penn line orientation test using large samples and computerized adaptive test simulation. Psychological Assessment,27(3), 955–964. https://doi.org/10.1037/pas0000102.

    Article  PubMed  PubMed Central  Google Scholar 

  • Nakamura, B. J., Higa-McMillan, C. K., Okamura, K. H., & Shimabukuro, S. (2011). Knowledge of and attitudes towards evidence-based practices in community child mental health practitioners. Administration and Policy in Mental Health and Mental Health Services Research,38(4), 287–300. https://doi.org/10.1007/s10488-011-0351-2.

    Article  PubMed  Google Scholar 

  • Okamura, K. H., Nakamura, B. J., Mueller, C., Hayashi, K., & McMillan, C. K. H. (2016). An exploratory factor analysis of the knowledge of evidence-based services questionnaire. The Journal of Behavioral Health Services & Research,43(2), 214–232. https://doi.org/10.1007/s11414-013-9384-5.

    Article  Google Scholar 

  • Okamura, K. H., Hee, P. J., Jackson, D., & Nakamura, B. J. (2018). Furthering our understanding of therapist knowledge and attitudinal measurement in youth community mental health. Administration and Policy in Mental Health and Mental Health Services Research,45(5), 699–708. https://doi.org/10.1007/s10488-018-0854-1.

    Article  PubMed  Google Scholar 

  • Powell, B. J., Beidas, R. S., Rubin, R. M., Stewart, R. E., Wolk, C. B., Matlin, S. L., et al. (2016). Applying the policy ecology framework to Philadelphia’s behavioral health transformation efforts. Administration and Policy in Mental Health and Mental Health Services Research,43(6), 909–926. https://doi.org/10.1007/s10488-016-0733-6.

    Article  PubMed  Google Scholar 

  • PracticeWise, LLC. (2017). PracticeWise evidence-base services database. Retrieved from https://www.practicewise.com

  • Reise, S. P., & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement,27(2), 133–144.

    Article  Google Scholar 

  • Roalf, D. R., Moore, T. M., Wolk, D. A., Arnold, S. E., Mechanic-Hamilton, D., Rick, J., et al. (2016). Defining and validating a short form montreal cognitive assessment (s-MoCA) for use in neurodegenerative disease. Journal of Neurology, Neurosurgery & Psychiatry,87(12), 1303–1310. https://doi.org/10.1136/jnnp-2015-312723.

    Article  Google Scholar 

  • Rogers, E. M. (2003). Diffusion of innovation (5th ed.). New York: Free Press.

    Google Scholar 

  • Rosenman, R., Tennekoon, V., & Hill, L. G. (2011). Measuring bias in self-reported data. International Journal of Behavioural and Healthcare Research,2(4), 320. https://doi.org/10.1504/IJBHR.2011.043414.

    Article  PubMed  Google Scholar 

  • Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Chicago, IL: Psychometric Society.

    Book  Google Scholar 

  • Sanders, M. R., Prinz, R. J., & Shapiro, C. J. (2009). Predicting utilization of evidence-based parenting interventions with organizational, service-provider and client variables. Administration and Policy in Mental Health and Mental Health Services Research,36(2), 133–143. https://doi.org/10.1007/s10488-009-0205-3.

    Article  PubMed  Google Scholar 

  • Seng, A. C., Prinz, R. J., & Sanders, M. R. (2006). The Role of training variables in effective dissemination of evidence-based parenting interventions. International Journal of Mental Health Promotion,8(4), 20–28. https://doi.org/10.1080/14623730.2006.9721748.

    Article  Google Scholar 

  • Stumpf, R. E., Higa-McMillan, C. K., & Chorpita, B. F. (2009). Implementation of evidence-based services for youth: Assessing provider knowledge. Behavior Modification,33(1), 48–65. https://doi.org/10.1177/0145445508322625.

    Article  PubMed  Google Scholar 

  • Ward, A. M., Regan, J., Chorpita, B. F., Starace, N., Rodriguez, A., Okamura, K., et al. (2013). Tracking evidence based practice with youth: Validity of the MATCH and standard manual consultation records. Journal of Clinical Child & Adolescent Psychology,42(1), 44–55.

    Article  Google Scholar 

  • Weersing, V. R., Weisz, J. R., & Donenberg, G. R. (2002). Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child & Adolescent Psychology,31(2), 168–180. https://doi.org/10.1207/S15374424JCCP3102_03.

    Article  Google Scholar 

  • Weist, M., Lever, N., Stephan, S., Youngstrom, E., Moore, E., Harrison, B., et al. (2009). Formative evaluation of a framework for high quality, evidence-based services in school mental health. School Mental Health,1(4), 196–211. https://doi.org/10.1007/s12310-009-9018-5.

    Article  Google Scholar 

  • Weisz, J. R., Jensen-Doss, A., & Hawley, K. M. (2006). Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist,61(7), 671–689. https://doi.org/10.1037/0003-066X.61.7.671.

    Article  PubMed  Google Scholar 

  • Weisz, J. R., Ugueto, A. M., Cheron, D. M., & Herren, J. (2013). Evidence-based youth psychotherapy in the mental health ecosystem. Journal of Clinical Child & Adolescent Psychology,42(2), 274–286. https://doi.org/10.1080/15374416.2013.764824.

    Article  Google Scholar 

  • Weisz, J. R., Kuppens, S., Ng, M. Y., Eckshtain, D., Ugueto, A. M., Vaughn-Coaxum, R., et al. (2017). What five decades of research tells us about the effects of youth psychological therapy: A multilevel meta-analysis and implications for science and practice. American Psychologist,72(2), 79–117. https://doi.org/10.1037/a0040360.

    Article  PubMed  Google Scholar 

Download references

Funding

This study was supported by the following grants from the US National Institute of Mental Health: K23 MH099179, Beidas; T32 MH109433.

Author information

Authors and Affiliations

  1. Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, 3535 Market St, 3rd Floor, Philadelphia, PA, 19104, USA

    Gwendolyn M. Lawson, Tyler M. Moore, Emily M. Becker-Haimes & Rinad S. Beidas

  2. Department of Medical Ethics and Health Policy, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA

    Rinad S. Beidas

  3. Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA

    Rinad S. Beidas

  4. State of Hawaiʻi Child and Adolescent Mental Health Division, Honolulu, HI, USA

    Kelsie H. Okamura

  5. Hall Mercer Community Mental Health, Pennsylvania Hospital, Philadelphia, PA, USA

    Emily M. Becker-Haimes

  6. Penn Implementation Science Center at the Leonard Davis Institute of Health Economics (PISCE@LDI), University of Pennsylvania, Philadelphia, PA, USA

    Rinad S. Beidas

Authors
  1. Gwendolyn M. Lawson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tyler M. Moore
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kelsie H. Okamura
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emily M. Becker-Haimes
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rinad S. Beidas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gwendolyn M. Lawson.

Ethics declarations

Conflicts of interest

Dr. Beidas reported receiving consulting fees from Merck Sharpe & Dohme and the Camden Coalition of Healthcare Providers; and royalties from the Oxford University Press. Drs. Lawson, Moore, Okamura, and Becker-Haimes declare that they have no conflicts of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 72 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lawson, G.M., Moore, T.M., Okamura, K.H. et al. Knowledge of Evidence-Based Services Questionnaire: Development and Validation of a Short Form. Adm Policy Ment Health 47, 581–596 (2020). https://doi.org/10.1007/s10488-020-01020-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-020-01020-7

Keywords

Search

Navigation