Skip to main content

Advertisement

SpringerLink
Log in

Implementing a Measurement Feedback System: A Tale of Two Sites

  • Original Paper
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

A randomized experiment was conducted in two outpatient clinics evaluating a measurement feedback system called contextualized feedback systems. The clinicians of 257 Youth 11–18 received feedback on progress in mental health symptoms and functioning either every 6 months or as soon as the youth’s, clinician’s or caregiver’s data were entered into the system. The ITT analysis showed that only one of the two participating clinics (Clinic R) had an enhanced outcome because of feedback, and only for the clinicians’ ratings of youth symptom severity on the SFSS. A dose–response effect was found only for Clinic R for both the client and clinician ratings. Implementation analyses showed that Clinic R had better implementation of the feedback intervention. Clinicians’ questionnaire completion rate and feedback viewing at Clinic R were 50 % higher than clinicians at Clinic U. The discussion focused on the differences in implementation at each site and how these differences may have contributed to the different outcomes of the experiment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. Note that this initial version was considered a beta version with software bugs identified and corrected throughout about the first year.

  2. As noted the methods section, participation in the quality improvement initiative was required of agency staff. In addition, the agency issued a mandate about midway through the project for both sites requiring the use of CFS for all eligible cases with CFS implementation data to be used in staff performance evaluations.

References

  • American Psychological Association (APA) Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285.

    Article  Google Scholar 

  • Arco, L. (1997). Improving program outcome with process-based feedback. Journal of Organizational Behavior Management, 17(1), 37–64.

    Article  Google Scholar 

  • Arco, L. (2008). Feedback for improving staff training and performance in behavioral treatment programs. Behavioral Interventions, 23(1), 39–64.

    Article  Google Scholar 

  • Athay, M. M., Riemer, M., & Bickman, L. (2012). The Symptoms and Functioning Severity Scale (SFSS): Psychometric evaluation and differences of youth, caregiver, and clinician ratings over time. Administration and Policy in Mental Health and Mental Health Services Research, 39(1–2), 13–29.

    Article  PubMed  PubMed Central  Google Scholar 

  • Baumberger, B. K. (2014). Understanding and promoting treatment integrity. In L. M. Hagermoser Sanetti & T. R. Kratochwill (Eds.), Treatment integrity: A foundation for evidence-based practice in applied psychology (pp. 35–54). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Bickman, L. (2008). Why don’t we have effective mental health services? Administration and Policy in Mental Health and Mental Health Services Research, 35, 437–439.

    Article  PubMed  Google Scholar 

  • Bickman, L. & Athay, M. I. (Eds.) (2012). Youth mental health measurement (special issue) Administration and Policy in Mental Health and Mental Health Services Research.

  • Bickman, L., Athay, M. M., Riemer, M., Lambert, E. W., Kelley, S. D., Breda, C., et al. (Eds.). (2010). Manual of the Peabody Treatment Progress Battery (2nd ed.). Nashville, TN: Vanderbilt University.

    Google Scholar 

  • Bickman, L., Kelley, D. S., & Athay, M. (2012). The technology of measurement feedback systems. Couple and Family Psychology, 1(4), 274–284.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bickman, L., Kelley, S. D., Breda, C., deAndrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatric Services, 62(12), 1423–1429.

    Article  PubMed  Google Scholar 

  • Bickman, L., Rosof, J., Salzer, M. S., Summerfelt, W. T., Noser, K., Wilson, S. J., & Karver, M. S. (2000). What information do clinicians value for monitoring adolescent client progress and outcome? Professional Psychology: Research and Practice, 31(1), 70–74.

    Article  Google Scholar 

  • Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015). Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy research, 25(1), 6–19.

    Article  PubMed  Google Scholar 

  • Bruns, E., Hoagwood, K., & Hamilton, J. (2008). State implementation of evidence-based practice for youths, Part I: Responses to the state of the evidence. Journal of the American Academy of Child and Adolescent Psychiatry, 47(4), 369–373.

    Article  PubMed  Google Scholar 

  • Cebul, R. D. (2008). Using electronic medical records to measure and improve performance. Transactions of the American Clinical and Climatological Association, 119, 65–75. discussion 75–76.

    PubMed  PubMed Central  Google Scholar 

  • Chorpita, B., Bernstein, A., & Daleiden, E. (2008). Driving with roadmaps and dashboards: Using information resources to structure the decision models in service organizations. Administration and Policy in Mental Health and Mental Health Services Research, 35, 114–123.

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., Bernstein, A. D., & Daleiden, E. L. (2011). Empirically guided coordination of multiple evidence-based treatments: An illustration of relevance mapping in children’s mental health services. Journal of Consulting and Clinical Psychology, 79, 470–480.

    Article  PubMed  Google Scholar 

  • Douglas, S., Button, S., & Casey, S. E. (2014). Implementing for sustainability: Promoting use of a measurement feedback system for innovation and quality improvement. Administration and Policy in Mental Health and Mental Health Services Research,. doi:10.1007/s10488-014-0607-8.

    PubMed  Google Scholar 

  • Duncan, E. A. S., & Murray, J. (2012). The barriers and facilitators to routine outcome measurement by allied health professionals in practice: A systematic review. BMC Health Services Research, 12, 96.

    Article  PubMed  PubMed Central  Google Scholar 

  • Duncan, K., & Pozehl, B. (2000). Effects of performance feedback on patient pain outcomes. Clinical Nursing Research, 9(4), 379–397.

    Article  CAS  PubMed  Google Scholar 

  • Fitzpatrick, M. (2012). Blurring practice–research boundaries using progress monitoring: A personal introduction to this issue of Canadian psychology. Canadian Psychology/Psychologie Canadienne, 53(2), 75–81.

    Article  Google Scholar 

  • Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute National Implementation Research Network (FMHI Publication #231).

  • Furman, C. E., Adamek, M. S., & Furman, A. G. (1992). The use of an auditory device to transmit feedback to student therapists. Journal of Music Therapy, 29(1), 40.

    Article  Google Scholar 

  • Gleacher, A. A., Nadeem, E., Moy, A. J., Whited, A. L., Albano, A., Radigan, M., et al. (2011). Statewide CBT training for clinicians and supervisors treating youth: The New York State Evidence Based Treatment Dissemination Center. Journal of Emotional and Behavioral Disorders, 19(3), 182–192.

    Article  Google Scholar 

  • Goebel, L. J. (1997). A peer review feedback method of promoting compliance with preventive care guidelines in a resident ambulatory care clinic. The Joint Commission Journal on Quality Improvement, 23(4), 196–202.

    CAS  PubMed  Google Scholar 

  • Goodman, J. D., McKay, J. R., & DePhilippis, D. (2013). Progress monitoring in mental health and addiction treatment: A means of improving care. Professional Psychology: Research and Practice, 44(4), 231–246.

    Article  Google Scholar 

  • Hannan, C., Lambert, M., Harmon, C., Nielson, S., Smart, D., Shimokawa, K., et al. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology, 61(2), 155–163.

    Article  PubMed  Google Scholar 

  • Hatfield,D., McCullough, L., Frantz, S. H. B., & Krieger, K., Do we know when our clients get worse? An investigation of therapists’ ability to detect negative client change. (2009). Clinical Psychology and Psychotherapy. doi:10.1002/cpp.656.

  • Hawley, K. M., & Weisz, J. R. (2003). Child, parent, and therapist (dis)agreement on target problems in outpatient therapy: The therapist’s dilemma and its implications. Journal of Consulting and Clinical Psychology, 71(1), 62–70.

    Article  PubMed  Google Scholar 

  • Holmboe, E., Scranton, R., Sumption, K., & Hawkins, R. (1998). Effect of medical record audit and feedback on residents’ compliance with preventive health care guidelines. Academic Medicine, 73(8), 901–903.

    Article  CAS  PubMed  Google Scholar 

  • Howe, A. (1996). Detecting psychological distress: can general practitioners improve their own performance? British Journal of General Practice, 46(408), 407–410.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child and Adolescent Psychology, 39, 885–896.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.

    Article  Google Scholar 

  • Lambert, M. J. (2001). Psychotherapy outcome and quality improvement: introduction to the special section on patient-focused research. Journal of Consulting and Clinical Psychology, 69(2), 147–149.

    Article  CAS  PubMed  Google Scholar 

  • Lambert, M. J., Hansen, N. B., & Finch, A. E. (2001). Patient-focused research: using patient outcome data to enhance treatment effects. Journal of Consulting and Clinical Psychology, 69(2), 159–172.

    Article  CAS  PubMed  Google Scholar 

  • Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing feedback to psychotherapists on their patients’ progress: Clinical results and practice suggestions—Review. Journal of Clinical Psychology, 61(2), 165–174.

    Article  PubMed  Google Scholar 

  • Leshan, L. A., Fitzsimmons, M., Marbella, A., & Gottlieb, M. (1997). Increasing clinical prevention efforts in a family practice residency program through CQI methods. The Joint Commission Journal on Quality Improvement, 23(7), 391–400.

    CAS  PubMed  Google Scholar 

  • Love, S. M., Koob, J. J., & Hill, L. E. (2007). Meeting the challenges of evidence-based practice: Can mental health therapists evaluate their practice? Brief Treatment and Crisis Intervention, 7(3), 184–193.

    Article  Google Scholar 

  • Martin, A. M., Fishman, R., Baxter, L., & Ford, T. (2011). Practitioners’ attitudes towards the use of standardized diagnostic assessment in routine practice: a qualitative study in two child and adolescent mental health services. Clinical Child Psychology and Psychiatry, 16(3), 407–420.

    Article  PubMed  Google Scholar 

  • Mazonson, P. D., Mathias, S. D., Fifer, S. K., Buesching, D. P., Malek, P., & Patrick, D. L. (1996). The mental health patient profile: does it change primary care physicians’ practice patterns? The Journal of the American Board of Family Practice, 9(5), 336–345.

    CAS  PubMed  Google Scholar 

  • Moløkken, K. J. & Jørgensen, M. (2003, September). A Review of Surveys on Software Effort Estimation International Symposium on Empirical Software Engineering (ISESE 2003).

  • Mortenson, B. P., & Witt, J. C. (1998). The use of weekly performance feedback to increase teacher implementation of a pre-referral academic intervention. School Psychology Review, 27, 613–627.

    Google Scholar 

  • New Freedom Commission on Mental Health. (2003). Achieving the Promise: Transforming Mental Health Care in America. Final Report. DHHS Pub. No. SMA-03-3832. Rockville, MD.

  • Noell, G. H., & Gansle, K. A. (2014). The use of performance feedback to improve intervention implementation in schools. In L. M. Hagermoser Sanetti & T. R. Kratochwill (Eds.), Treatment integrity: A foundation for evidence-based practice in applied psychology (pp. 161–184). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., et al. (2012). A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review, 69(2), 123–157.

    Article  PubMed  PubMed Central  Google Scholar 

  • Riemer, M., & Bickman, L. (2011). Using program theory to link social psychology and program evaluation. In M. M. Mark, S. I. Donaldson, & B. Campbell (Eds.), Social Psychology and Program/Policy Evaluation. New York: Guilford.

    Google Scholar 

  • Riemer, M., Rosof-Williams, J., & Bickman, L. (2005). Theories related to changing clinician practice. Child and Adolescent Psychiatric Clinics of North America, 14, 241–254.

    Article  PubMed  Google Scholar 

  • Robinson, M. B., Thompson, E., & Black, N. A. (1996). Evaluation of the effectiveness of guidelines, audit and feedback: improving the use of intravenous thrombolysis in patients with suspected acute myocardial infarction. International Journal for Quality in Health Care, 8(3), 211–222.

    Article  CAS  PubMed  Google Scholar 

  • Rokstad, K., Straand, J., & Fugelli, P. (1995). Can drug treatment be improved by feedback on prescribing profiles combined with therapeutic recommendations? A prospective, controlled trial in general practice. Journal of Clinical Epidemiology, 48, 1061–1068.

    Article  CAS  PubMed  Google Scholar 

  • Rose, D. J., & Church, R. J. (1998). Learning to teach: the acquisition and maintenance of teaching skills. Journal of Behavioral Education, 8(1), 5–35.

    Article  Google Scholar 

  • Rotheram-Borus, M. J., Swendeman, D., & Chorpita, B. F. (2012). Disruptive innovations for designing and diffusing evidence-based interventions. American Psychologist, 67(6), 463–476.

    Article  PubMed  PubMed Central  Google Scholar 

  • Sapyta, J., Riemer, M., & Bickman, L. (2005). Feedback to clinicians: Theory, research, and practice. Journal of Clinical Psychology, 61(2), 145–153.

    Article  PubMed  Google Scholar 

  • Southam-Gerow, M. A., & Prinstein, M. J. (2014). Evidence base updates: The evolution of the evaluation of psychological treatments for children and adolescents. Journal of Clinical Child and Adolescent Psychology, 43(1), 1–6.

    Article  PubMed  Google Scholar 

  • Tabenkin, H., Steinmetz, D., Eilat, Z., Heman, N., Dagan, B., & Epstein, L. (1995). A peer review programme to audit the management of hypertensive patients in family practices in Israel. Family Practice, 12(3), 309–312.

    Article  CAS  PubMed  Google Scholar 

  • Tuckman, B. W., & Yates, D. (1980). Evaluating the student feedback strategy for changing teacher style. The Journal of Educational Research, 74(2), 74–77.

    Article  Google Scholar 

  • Useem, J. (May 31, 2014). Business school, disrupted. The New York Times.

  • Wager, K. A., Lee, F. W., & Glaser, J. P. (2009). Health care information systems: A practical approach for health care management (2nd ed.). San-Francisco: Jossey-Bass.

    Google Scholar 

  • Weisz, J. R., Ng, M. Y., & Bearman, S. K. (2014). Odd couple? Reenvisioning the relation between science and practice in the dissemination-implementation era. Clinical Psychological Science, 2(1), 58–74.

    Article  Google Scholar 

  • Wolpert, M., Curtis-Tyler, K., & Edbrooke-Childs, J. (2014). A qualitative exploration of patient and clinician views on patient reported outcome measures in child mental health and diabetes services. Administration and Policy in Mental Health and Mental Health Services Research, doi:10.1007/s10488-014-0586-9.

Download references

Author information

Authors and Affiliations

  1. Vanderbilt University, Nashville, USA

    Leonard Bickman, Susan R. Douglas & Ana Regina Vides De Andrade

  2. Hope Network, Grand Rapids, MI, USA

    Michele Tomlinson

  3. New York University, New York, USA

    Alissa Gleacher, Serene Olin & Kimberly Hoagwood

Authors
  1. Leonard Bickman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Susan R. Douglas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ana Regina Vides De Andrade
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michele Tomlinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alissa Gleacher
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Serene Olin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Kimberly Hoagwood
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leonard Bickman.

Additional information

The authors received federal funding from the Agency on Healthcare Research and Quality (AHRQ, R18HS18036-01, Hoagwood & Bickman, PIs). Susan Douglas has been previously published as Susan Douglas Kelley.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bickman, L., Douglas, S.R., De Andrade, A.R.V. et al. Implementing a Measurement Feedback System: A Tale of Two Sites. Adm Policy Ment Health 43, 410–425 (2016). https://doi.org/10.1007/s10488-015-0647-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-015-0647-8

Keywords

Search

Navigation