Skip to main content

Advertisement

SpringerLink
Log in

Validation of an Expanded Measure of Integrated Care Provider Fidelity: PPAQ-2

  • Published:
Journal of Clinical Psychology in Medical Settings Aims and scope Submit manuscript

Abstract

This study aimed to validate the factor structure of the expanded Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ-2), which is designed to assess provider fidelity to both the Primary Care Behavioral Health (PCBH) and collaborative care management (CCM) models of integrated primary care. Two-hundred fifty-three integrated care providers completed self-reports of professional background, perceptions of clinic integration and related practice barriers, and the PPAQ-2. Confirmatory factor analyses were conducted to assess the theorized factor structure and criterion validity was assessed through correlational analysis. Factor analyses demonstrated adequate fit with the data and acceptable to excellent composite reliabilities across five PCBH domains and five CCM domains. Validity was demonstrated by correlations between adherence scores and measures of clinic integration and barriers to fidelity. The PPAQ-2 is a psychometrically sound measure that can be used in future integrated care dismantling studies to identify provider behaviors that best predict patient outcomes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Afthanorhan, W. M. A. B. W. (2013). A comparison of partial least square structural equation modeling (PLS-SEM) and covariance based structural equation modeling (CB-SEM) for confirmatory factor analysis. International Journal of Engineering Science and Innovation Technology,2, 198–205.

    Google Scholar 

  • Atkins, D., Kilbourne, A. M., & Shulkin, D. (2017). Moving from discovery to system-wide change: The role of research in a learning health care system: Experience from three decades of health systems research in the Veterans Health Administration. Annual Review of Public Health,38, 467–487.

    Article  Google Scholar 

  • Awang, Z., Ahmad, J. H., & Zin, N. M. (2010). Modelling job satisfaction and work commitment among lecturers: a case of UITM kelantan. Journal of Statistical Modeling and Analytics,1, 45–59.

    Google Scholar 

  • Bacon, L. D. (1999). Using LISREL and PLS to Measure Customer Satisfaction. Paper presented at the Sawtooth Software Conference Proceeding, La Jolla, CA.

  • Beehler, G. P., Funderburk, J. S., King, P. R., Wade, M., & Possemato, K. (2015). Using the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ) to identify variation in practice patterns. Translational Behavioral Medicine,5, 384–392.

    Article  Google Scholar 

  • Beehler, G. P., Funderburk, J. S., Possemato, K., & Dollar, K. M. (2013). Psychometric assessment of the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ). Translational Behavioral Medicine,3, 379–391.

    Article  Google Scholar 

  • Beehler, G. P., Funderburk, J. S., Possemato, K., & Vair, C. L. (2013). Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: A Delphi study. Implementation Science,8, 19.

    Article  Google Scholar 

  • Beehler, G. P., & Lilienthal, K. R. (2017). Provider perceptions of an integrated primary care quality improvement strategy: The PPAQ Toolkit. Psychological Services,14, 50–56.

    Article  Google Scholar 

  • Beehler, G. P., & Wray, L. O. (2012). Behavioral health providers’ perspectives of delivering behavioral health services in primary care: a qualitative analysis. BMC Health Services Research,12, 337. https://doi.org/10.1186/1472-6963-12-337.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., … Czajkowski, S. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology,23(5), 443–451. https://doi.org/10.1037/0278-6133.23.5.443.

    Article  PubMed  Google Scholar 

  • Bower, P., Gilbody, S., Richards, D., Fletcher, J., & Sutton, A. (2006). Collaborative care for depression in primary care: Making sense of a complex intervention: Systematic review and meta-regression. The British Journal of Psychiatry,189, 484–493.

    Article  Google Scholar 

  • Brawer, P. A., Brugh, A. M. T., Martielli, R. P., O’Connor, S. P., Mastnak, J., Scherrer, J. F., & Day, T. E. (2011). Enhancing entrance into PTSD treatment for post-deployment veterans through collaborative/integrative care. Translational Behavioral Medicine,1(4), 609–614.

    Article  Google Scholar 

  • Brawer, P. A., Martielli, R. P., Pye, P. L., Manwaring, J., & Tierney, A. (2010). St. Louis Initiative for Integrated Care Excellence (SLI2CE): Integrated collaborative care on a large scale model Families. Systems, & Health,28, 175–187.

    Article  Google Scholar 

  • Bryan, C. J., Corso, M. L., Corso, K. A., Morrow, C. E., Kanzler, K. E., & Ray-Sannerud, B. (2012). Severity of mental health impairment and trajectories of improvement in an integrated primary care clinic. Journal of Consulting and Clinical Psychology,80(3), 396–403.

    Article  Google Scholar 

  • Bryan, C. J., Morrow, C. E., & Appolonio, K. K. (2009). Impact of behavioral health consultant interventions on patient symptoms and functioning in an integrated family medicine clinic. Journal of Clinical Psychology,65, 281–293.

    Article  Google Scholar 

  • Butler, M., Kane, R. L., McAlpine, D., Kathol, R. G., Fu, S. S., Hagedorn, H., & Wilt, T. J. (2008). Integration of Mental Health/Substance Abuse and Primary Care No. 173. (AHRQ Publication No. 09-E003.). Rockville: Agency for Healthcare Research and Quality.

  • Carroll, C., Patterson, M., Wood, S., Booth, B., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science,2, 40.

    Article  Google Scholar 

  • Chaney, E. F., Rubenstein, L. V., Liu, C. F., Yano, E. M., Bolkan, C., Lee, M., … Uman, J. (2011). Implementing collaborative care for depression treatment in primary care: A cluster randomized evaluation of a quality improvement practice design. Implementation Science,6, 121.

    Article  Google Scholar 

  • Cigrang, J. A., Rauch, S. A., Mintz, J., Brundige, A. R., Mitchell, J. A., Najera, E., … STRONG STAR Consortium. (2017). Moving effective treatment for posttraumatic stress disorder to primary care: A randomized controlled trial with active duty military. Families, Systems & Health,35, 450–462.

    Article  Google Scholar 

  • Fauth, J., & Tremblay, G. (2011). The integrated care evaluation project: Antioch University.

  • Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research,18, 39–50.

    Article  Google Scholar 

  • Fortney, J., Enderle, M., McDougall, S., Clothier, J., Otero, J., Altman, L., & Curran, G. (2012). Implementation outcomes of evidence-based quality improvement for depression in VA community based outpatient clinics. Implementation Science,7, 30. https://doi.org/10.1186/1748-5908-7-30.

    Article  PubMed  Google Scholar 

  • Gilbody, S., Bower, P., Fletcher, J., Richards, D., & Sutton, A. J. (2006). Collaborative care for depression: A cumulative meta-analysis and review of longer-term outcomes. Archive of Internal Medicine,166, 2314–2321.

    Article  Google Scholar 

  • Hasson, H. (2010). Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implementation Science,5, 67. https://doi.org/10.1186/1748-5908-5-67.

    Article  PubMed  Google Scholar 

  • Hrisos, S., Eccles, M. P., Francis, J. J., Dickinson, H. O., Kaner, E. F., Beyer, F., & Johnston, M. (2009). Are there valid proxy measures of clinical behaviour? A systematic review. Implementation Science,4, 37. https://doi.org/10.1186/1748-5908-4-37.

    Article  PubMed  Google Scholar 

  • Hunter, C. L., & Goodie, J. L. (2010). Operational and clinical components for integrated-collaborative behavioral healthcare in the patient-centered medical home. Families, Systems & Health,28, 308–321. https://doi.org/10.1037/a0021761.

    Article  Google Scholar 

  • Hunter, C. L., Goodie, J. L., Oordt, M. S., & Dobmeyer, A. C. (2017). Integrated behavioral health in primary care: Step-by-step guidance for assessment and intervention (2nd ed.). Washington, DC: American Psychological Association.

    Book  Google Scholar 

  • Kearney, L. K., Dollar, K. M., Beehler, G. P., Goldstein, W. R., Grasso, J., Wray, L. O., & Pomerantz, A. S. (in press). Creation and implementation of a national interprofessional integrated primary care competency training program: Preliminary findings and lessons learned. Training and Education in Professional Psychology.

  • Kearney, L. K., Post, E. P., Zeiss, A., Goldstein, M. G., & Dundon, M. (2011). The role of mental and behavioral health in the application of the patient-centered medical home in the Department of Veterans Affairs. Translational Behavioral Medicine,1, 624–628.

    Article  Google Scholar 

  • Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation,24, 315–340.

    Article  Google Scholar 

  • Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage.

    Book  Google Scholar 

  • Newman, I., Lim, J., & Pineda, F. (2013). Content validity using mixed methods approach: Its application and development through the use of a Table of Specifications Methodology. Journal of Mixed Methods Research,7, 243–260.

    Article  Google Scholar 

  • Nunally, J. C., & Bernstien, I. H. (1994). Psychometric theory (3rd ed.). New York: Mcfgraw-Hill.

    Google Scholar 

  • Oslin, D. W., Ross, J., Sayers, S., Murphy, J., Kane, V., & Katz, I. R. (2006). Screening, assessment, and management of depression in VA primary care clinics. The Behavioral Health Laboratory. Journal of General Internal Medicine,21, 46–50.

    Article  Google Scholar 

  • Oxman, T. E., Schulberg, H. C., Greenberg, R. L., Dietrich, A. J., Williams, J. W. J., Nutting, P. A., & Bruce, M. L. (2006). A fidelity measure for integrated management of depression in primary care. Medical Care,44, 1030–1037.

    Article  Google Scholar 

  • Possemato, K., Johnson, E. M., Beehler, G. P., Shepardson, R. L., King, P., Vair, C. L., … Wray, L. O. (2018). Patient outcomes associated with primary care behavioral health services: A systematic review. General Hospital Psychiatry,53, 1–11.

    Article  Google Scholar 

  • Ray-Sannerud, B. N., Dolan, D. C., Morrow, C. E., Corso, K. A., Kanzler, K. E., Corso, M. L., & Bryan, C. J. (2012). Longitudinal outcomes after brief behavioral health intervention in an integrated primary care clinic. Families, Systems & Health,30(1), 60–71. https://doi.org/10.1037/a0027029.

    Article  Google Scholar 

  • Reiter, J. T., Dobmeyer, A. C., & Hunter, C. L. (2018). The Primary Care Behavioral Health (PCBH) Model: An overview and operational definition. Journal of Clinical Psychology in Medical Settings,25, 109–126.

    Article  Google Scholar 

  • Robinson, P. J., & Reiter, J. T. (2016). Behavioral consultation and primary care: A guide to integrating services (2nd ed.). New York, NY: Springer.

    Book  Google Scholar 

  • Ross, J. T., TenHave, T., Eakin, A. C., Difillippo, S., & Oslin, D. W. (2008). A randomized controlled trial of a close monitoring program for minor depression and distress. Journal of General Internal Medicine,23, 1379–1385.

    Article  Google Scholar 

  • Rubenstein, L. V., Chaney, E. F., Ober, S., Felker, B., Sherman, S. E., Lanto, A., & Vivell, S. (2010). Using evidence-based quality improvement methods for translating depression collaborative care research into practice. Families, Systems, & Health,28, 91–113.

    Article  Google Scholar 

  • Rubenstein, L. V., Williams, J. W., Danz, M., Shekelle, P., Suttorp, M., & Johnsen, B. (2009). Determining key features of effective depression interventions. Washington, DC: Department of Veterans Affairs.

    Google Scholar 

  • Shiner, B., Watts, B. V., Pomerantz, A., Groft, A., Scott, D., Street, B., & Young-Xu, Y. (2009). Access to what? An evaluation of the key ingredients to effective advanced mental health access at a VA Medical Center and its affiliated community-based outreach clinics. Military Medicine,174, 1024–1032.

    Article  Google Scholar 

  • Strosahl, K. (1998). Integrating behavioral health and primary care services: The primary mental health care model. In A. Blount (Ed.), Integrated primary care: The future of medical and mental health collaboration (pp. 139–166). New York: Norton.

    Google Scholar 

  • VA Health Services Research and Development Service. (2008). Collaborative Care for Depression in the Primary Care Setting: A Primer on VA’s Translating Initiatives for Depression into Effective Solutions (TIDES) Project, Boston, MA.

  • Williams, J. W., Jr., Gerrity, M., Holsinger, T., Dobscha, S., Gaynes, B., & Dietrich, A. (2007). Systematic review of multifaceted interventions to improve depression care. General Hospital Psychiatry,29(2), 91–116. https://doi.org/10.1016/j.genhosppsych.2006.12.003.

    Article  PubMed  Google Scholar 

  • Woltmann, E., Grogan-Kaylor, A., Perron, B., Georges, H., Kilbourne, A. M., & Bauer, M. S. (2012). Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: Systematic review and meta-analysis. American Journal of Psychiatry,169, 790–804.

    Article  Google Scholar 

  • Wong, K. K. (2010). Handling small survey sample size and skewed dataset with partial least square path modelling (pp. 20–23). Vue: The Magazine of the Marketing Research and Intelligence Association.

    Google Scholar 

  • Zanjani, F., Miller, B., Turiano, N., Ross, J., & Oslin, D. (2008). Effectiveness of telephone-based referral care management: A brief intervention to improve psychiatric treatment engagement. Psychiatric Services,59, 776–781.

    Article  Google Scholar 

Download references

Funding

This study was funded by the VA Center for Integrated Healthcare at the VA Western New York Healthcare System. The information provided in this study does not represent the views of the Department of Veterans Affairs or the United States Government.

Author information

Authors and Affiliations

  1. VA Center for Integrated Healthcare (116N), VA WNY Healthcare System, 3495 Bailey Ave, Buffalo, NY, 14215, USA

    Gregory P. Beehler, Paul R. King & Wade R. Goldstein

  2. School of Public Health and Health Professions, University at Buffalo, The State University of New York, Buffalo, NY, USA

    Gregory P. Beehler

  3. VA Center for Integrated Healthcare, Syracuse VA Medical Center, Syracuse, NY, USA

    Jennifer S. Funderburk, Kyle Possemato & Michael Wade

  4. Department of Psychology, Syracuse University, Syracuse, NY, USA

    Jennifer S. Funderburk & Kyle Possemato

  5. Department of Psychiatry, University of Rochester, Rochester, NY, USA

    Jennifer S. Funderburk & Kyle Possemato

  6. Department of Counseling, School, and Educational Psychology, University at Buffalo, The State University of New York, Buffalo, NY, USA

    Paul R. King

  7. Harry S. Truman Memorial Veterans’ Hospital, Columbia, MO, USA

    John A. Maddoux

Authors
  1. Gregory P. Beehler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jennifer S. Funderburk
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul R. King
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kyle Possemato
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. John A. Maddoux
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wade R. Goldstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Michael Wade
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gregory P. Beehler.

Ethics declarations

Conflict of interest

Gregory P. Beehler, Jennifer S. Funderburk, Paul R. King, Kyle Possemato, John A. Maddoux, Wade R. Goldstein, Michael Wade have no conflict of interest.

Ethical Approval

 All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Human and Animal Rights and Informed Consent

All procedures followed were in accordance with the ethical standard of the responsible committee on human experimentation of the VA Western New York Healthcare System and with the Helsinki Declaration of 1975, as revised in 2000. As an anonymous provider survey, this study was deemed exempt from Institutional Review Board review.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Beehler, G.P., Funderburk, J.S., King, P.R. et al. Validation of an Expanded Measure of Integrated Care Provider Fidelity: PPAQ-2. J Clin Psychol Med Settings 27, 158–172 (2020). https://doi.org/10.1007/s10880-019-09628-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10880-019-09628-0

Keywords

Search

Navigation