Development and Feasibility Testing of the Clinical-Community Linkage Self-Assessment Survey for Community Organizations

Introduction: Clinical-community linkages (CCLs) can improve health, but few instruments exist to evaluate these partnerships. To address this gap, we develop and test the Clinical-Community Linkage Self-Assessment Survey (CCL Self-Assessment).

Materials and Methods: We built on an existing framework and conducted a literature review to guide the design of our survey, and obtained feedback from academic, clinical, and community-based experts. To pretest the instrument, we conducted 10 think-aloud interviews with community-based health-promotion organizations. We performed feasibility testing with 38 staff from 20 community organizations, followed by criterion-validity testing.

Results: The 15-item final instrument includes five domains: Nature of the Relationship, Communication, Referral Process, Feedback Loop, and Timeliness. Expert feedback included keeping the CCL Self-Assessment brief and actionable. Think-aloud interviews produced a range of revisions related to item wording, instructions, brevity, and formatting. Feasibility testing showed high response rate and ease of administration. Sites scoring high on the CCL Self-Assessment also scored high on the criterion measure.

Discussion: We demonstrate feasibility, as well as face, content, construct, and criterion validity. Initial results suggest the CCL Self-Assessment survey may be used by community organizations to identify strengths and weaknesses of their linkages. Next steps include additional statistical validation and testing to determine how the CCL Self-Assessment survey works in the field as well as providing specific tools to improve linkages.

Introduction

Clinical-community linkages (CCLs) are collaborative relationships between clinical and community organizations aimed at improving population health (1). The field of CCLs is relatively emergent (2). Both the CDC and AHRQ define a linkage as a clinic and community organization engaging in at least one of Himmelman's strategies for working together (networking, coordinating, cooperating, and collaborating). The strategies range in formality, characteristics, and degree of resource sharing. Therefore, CCLs take many diverse forms, ranging from informally sharing information, to establishing a bi-directional feedback loop with HIPAA compliant communication about each patient (1–3).

Case Studies of CCLs

Some recent research identifies opportunities to create CCLs in diverse settings, including primary care (4), physical therapy clinics (5, 6), optometrists (7), pharmacists (8), medical teaching centers (9) and small independent practices (10). There are several reports of efforts in Hawaii focusing on a spectrum of contexts (11–13).

The literature provides case examples of CCLs that describe the various strategies and diverse settings, primarily focusing on a variety of patient outcomes. Examples include interventions focused on community health workers (14–16) system or clinical level interventions (17–21), pediatricians (22, 23), nursing students (24), and reviews describing screening and connection strategies (25). Some examples also provide descriptions of specific CCL protocols (26–28).

Evaluation of CCL

Many of these peer-reviewed case descriptions explicitly discuss what hindered and what expanded the partnerships. These generally include either an asset or gap in: time or resources; infrastructure; and trust, rapport, or awareness of the partner (29–41). Several articles present conceptual tools within very specific contexts. These include roadmaps to address social determinants of health (42), delineating approaches to evaluate partnerships and implementation (43), and a guide on how community-based organizations can help link seniors with chronic disease management (44).

Most of the concrete evaluative tools for CCLs that do exist are in the gray literature, generally aimed at the community-based organization. These include finance and contract tools (45, 46), innovative payment models (47), implementation roadmaps (48), questionnaires, tools, and guides both in the United States (49, 50) and outside (51, 52). These resources are also written for specific contexts, are not peer reviewed, and are not based on a widely accepted CCL framework.

More research is needed to assess the effectiveness of CCLs and help organizations improve CCL quality, particularly through identification of CCL components that are key to effectiveness. Many descriptions of tools to assess CCLs do not describe validation processes, and most evaluations center on patient-level outcomes, rather than focusing on aspects of the community-clinical relationship itself, which drive the outcomes (16). A recent systematic review found the most important components affecting healthcare collaboration to be structure and processes, and standard assessment tools help focus efforts on weak components (53).

Theoretical Framework

In order to better understand CCLs and to map existing available metrics of CCL quality, the Agency for Healthcare Research and Quality (AHRQ) created the Clinical-Community Relationships Measurement (CCRM) Framework and the CCRM Atlas (2, 3). The CCRM Framework is a comprehensive framework for evaluating CCLs that combines Etz et al.'s Bridging Model (54) with Donabedian's Structure-Process-Outcome (S-P-O) model (55)¹. The CCRM Atlas presents this measurement framework and provides a listing of existing measures of clinical-community relationships.

The CCRM Framework identifies S-P-O measurement domains for each element of a linkage. Specifically, it lists seven S-P-O measurement domains related to the linkage between a clinic or clinician and a community resource. Of these seven domains, one is structural, three are related to process, and three to outcomes. However, only one (feedback & communication) has existing evaluation measures. Even these existing measures have important limitations: they are single items taken from instruments developed for different purposes, have not been validated for evaluating CCLs, and do not address all of the components defined in the domains (2, 3).

Considering the lack of peer-reviewed, validated evaluative tools that focus directly in the Clinical-Community Linkage, and to fill the gaps in evaluation measures from the CCRM Atlas, we sought to develop a pragmatic and accessible instrument that assesses a CCL's strengths and weaknesses that adhered to the conceptual model presented in the CCRM Atlas. We believe this tool will facilitate the evaluation of CCLs by community organizations and identify areas for improvement.

The purpose of this manuscript is to describe the development and testing of the Clinical-Community Linkage Self-Assessment Survey (CCL Self-Assessment), a tool for community organizations to evaluate their linkages with clinical organizations. We test the instrument for feasibility, and demonstrate face, content, construct, and criterion validity. Future development includes testing the same survey design with clinical partners to include entire CCL dyads.

Materials and Methods

We conducted this work as part of a larger research project, Physical Therapists Recommending Enhance®Fitness to Expand Reach (PT-REFER). The project was conducted in partnership with YMCA of the USA and implemented in local YMCA Associations. A YMCA Association can consist of a single YMCA site (called branch), or be the umbrella organization for multiple branches in an area. The goal of PT-REFER was to develop and test an intervention facilitating CCLs between YMCA Associations and physical therapy clinics to increase referrals from the clinics to Enhance®Fitness, a physical activity program for older adults available through YMCA Associations (details are available at clinicaltrials.gov/ct2/show/NCT03139461) (26). The University of Washington Institutional Review Board approved this study.

We sought to develop an instrument that assesses a CCL's strengths and weaknesses and is also “pragmatic,” in terms of being: (1) actionable, (2) feasible, and (3) important to stakeholders (57). We used an established three-step method for instrument development and validation (58, 59). To ensure rigor, we developed the instrument in three sub-steps: a) we built an initial draft based on findings from a literature review and existing domains from the CCRM Framework, with its established content validity; b) we revised the draft based on feedback from academic and clinical and community-based practice experts; and c) we sought feedback from members of our intended audience—staff from community organizations that offered health promotion programs— via “think-aloud” interviews (60). We define “community-based” as organizations outside of clinical health care or public health agencies (61) and “health promotion programs” as an organized public health action that aims to enable people to increase control over, and to improve their health (62, 63). Hearing from community organization staff if and how the survey items made sense and capturing important aspects of CCLs helped establish face and construct validity. To test feasibility of the CCL Self-Assessment in practice, we administered the instrument to the 20 YMCA Associations enrolled in the PT-REFER study. We tested criterion validity by comparing the data obtained through the CCL Self-Assessment to a conceptually related but distinct survey on community outreach that also captured empirical practices related to partnership strength.

Note that we did not conduct common psychometric tests predicated on homogeneity among items, such as Cronbach's alpha and factor analysis, which are only appropriate under assumptions of a shared cause (64, 65). The CCL Self-Assessment is a formative measure, in which the domains collectively “form” the latent construct. These types of models have neither internal consistency, nor the same antecedents and consequences as reflective measures. Homogeneity of items is not important for cause indicators; however, the specific items included is important for cause indicators because the latent variable is caused by them, and therefore omitting an item may invalidate the indicator. Criterion validation is one of the few objective ways of evaluating cause indicators. Therefore, tests of homogeneity are at best irrelevant and at worst can produce spurious findings (65, 66).

First Step: Instrument Development

The CCRM Domains and Content Validity

Structure and process are considered the most actionable components of Donabedian's S-P-O model (67). Therefore, we focused our instrument on measuring the structure and process domains of CCLs, as described in the CCRM Framework, and excluded the outcome domains. Content validity refers to how well items taken together constitute an adequate operational definition of the construct, and the degree to which the items are relevant, representative, and comprehensive of the construct (68). Content validity was established in the rigorous development of the CCRM Framework domains, by identifying items through literature review and using domains from the CCRM Framework, followed by iterative feedback on the instrument from academic and clinical and community-based practice experts and members of our intended audience. We built the foundation of our conceptual model on these established, peer-reviewed, and widely accepted materials.

The structure and process domains, as they appear in the CCRM Framework (2, 3), are: (1) nature and strength of inter-organizational relationship, (2) feedback and communication, (3) referral process, and (4) timeliness. We present the definitions and how each of our final items align to them below (see Section Obtaining Expert Feedback).

Identifying Existing Instruments

To explore whether published CCL instruments existed for these domains, we conducted a two-part comprehensive literature review, searching PubMed and ScienceDirect for English language articles written since 2005. We began by searching for real-world example of formal CCLs, including reports, examples, applications, and evaluations. We moved on to a literature review related to the CCRM Framework domains listed above. Search terms included: clinical, communication, community, enrollment, evaluation, instrument, intake, linkages, organizational, partnerships, process, referral, relationships, survey, time, timeliness, and tool. A diagram of the process is depicted in Figure 1, and a summary of our citations are depicted in Table 1.

FIGURE 1

Figure 1. PRISMA flow diagram for the systematic review detailing the search process, number of sources identified, screened, and included.

TABLE 1

Table 1. Summary of publications included literature review.

The literature review identified few papers, none of which included information that could be used in the development of the instrument. Most papers we identified addressed organizational collaboration in unrelated contexts; focused on measuring patient-level outcomes; or dealt with case studies of CCLs, new methods of adding features to CCLs, and various strategies for improving CCLs.

We also reviewed three potentially relevant survey instruments referenced in the CCRM Atlas (69–71). Of these, only the Continuity of Care Practices Survey – Practice Level [CCPS-P] (71) included items related to our domains of interest. Because the instrument had been developed in the substance abuse context, we adapted six items from this instrument to reflect a broader referral process. We edited the text, and adjusted item order to follow the logical progression of the referral process. The adapted items are questions 3a, 3b, and 3c of the final CCL Self-Assessment (see Table 2: CCL Self-Assessment).

TABLE 2

Table 2. Complete clinical-community linkage self-assessment survey.

Generating New Items and Designing the Instrument

We developed an initial draft of the CCL Self-Assessment, which contained 29 draft items representing core concepts based on the definition of each of the CCRM Framework domains of interest. The draft items included an initial draft of the six items from the Continuity of Care Practices Survey – Practice Level mentioned above. Each item in our initial draft was a statement (e.g., The partnership with our clinical partner requires extensive sharing of responsibilities.), with a 7-point response scale, indicating to what extent statements applied to a single partnership, and an option “Not able to answer.” A score of 1 indicated “To little or no extent” and a score of 7 indicated “To a very great extent.” The draft instrument contained the following: Nature of the relationship (6 items); Referral process (7 items); Feedback and Communication (6 items); Timeliness (2 items).

Obtaining Expert Feedback

We solicited feedback on the draft instrument through open, in-person discussion from three expert panels affiliated with the PT-REFER study. Each discussion was led by an experienced facilitator while a second person took notes. The three panels included: project co-investigators not involved in instrument development (9 members), the Project Advisory Group (10 members), and the Scientific Advisory Board (5 members). Panel-member expertise included: research and practice (in the fields of physical therapy, psychiatry, public health, and communication); community organization operations (such as exercise, health, and evidence-based programs; and payer engagement strategies); dissemination and implementation science; organizational behavior and behavioral science; clinical practice and management; evaluation; statistics; and CCLs. We incorporated feedback after each panel review. The panels noted that the CCL Self-Assessment should be both useful and actionable, and suggested adding a component to guide improvement (i.e., a toolkit or implementation guide). We also received feedback related to the instrument's format, scoring, clarity, brevity, content, and flow.

We initially focused the instrument on a single partnership to enable community organizations to choose which partnership to evaluate. However, the experts on our panels advised that examining the partnerships globally would identify variability among partnerships, broad areas needing improvement, and reduce the danger of selection bias (from organizations only evaluating their weakest or strongest partnerships). Further, this would reduce burden, make instrument simpler, improve the likelihood of the CCL Self-assessment being used, and be more useful in general.

As a result of panel feedback, we reframed the instructions and items clarified that the survey focused on how the organization interacts with all their clinical partners, rather than evaluating a single clinical partner. We adjusted the formatting and wording to make items clearer and shorter, and to make the survey appear cleaner (i.e., making the document more accessible and inviting; formatted to facilitate consumption of information with less effort). We split the Feedback and Communication domain into two categories to reflect the logical flow and progression of referral activities. In addition, we identified items with a large degree of overlap, where we could eliminate items without compromising content validity, and shortened the instrument to a total of 15 items.

We also adjusted the survey-response scale to represent how consistently the action is taken with partners because it was more intuitive for respondents in thinking about their CCLs. The final draft moved from a seven-point Likert scale to a forced choice Likert scale with 4 options: (1) We do not do this; (2) We do this for less than half our partners, (3) We do this for more than half our partners, (4) We do this for nearly all our partners.

Final Items and Definitions

The final order of item categories was: nature of the relationship, communication, referral process, feedback, and timeliness. Every item in our instrument drew directly from these definitions of the domains. We present the definitions below, and the final items drawn from each. The final instrument is presented in Table 2: CCL Self-Assessment.

Structure Domain

Nature and strength of inter-organizational relationship was defined as: the level of intensity of a relationship between a clinic/clinician and community resource, based on Himmelman's (2002) definitions of networking, coordinating, cooperating, and collaborating. This includes the degree to which the relationship can overcome common barriers of working together— time, trust, and turf (72). Specifically, Himmelman defines:

Networking as exchanging information for mutual benefit.

Coordinating as exchanging information and altering activities for mutual benefit and to achieve a common purpose.

Cooperating as exchanging information, altering activities, and sharing resources for mutual benefit and to achieve a common purpose.

Collaborating as exchanging information, altering activities, sharing resources, and enhancing the capacity of another for mutual benefit and to achieve a common purpose.

The five final Nature of the Relationship in the CCL Self-Assessment items were:

a. Our organization and our clinical partners have a system in place to share information (electronic or otherwise). (e.g., We exchange information about how we each support patients.)

b. Our organization and our clinical partners change activities to make accessing each other's services or resources easier for patients. (e.g., We change our service schedules so patients can visit our organization directly after their clinic visit.)

c. Our organization and our clinical partners share resources to better connect patients with clinics and with our programs. (e.g., We share the costs of employing a part-time marketing and outreach coordinator.)

d. Our organization and our clinical partners enhance each other's capacity to support patients. (e.g., We provide skill-development training for each other's staff.)

e. Our organization and our clinical partners facilitate insurance reimbursement for patient participation in our programs. (e.g., We advocate together to help get our programs covered by the patient's health insurance.)

Process Domains

Feedback and communication was defined as: the level and means of communication between the community resource and the clinic/clinician (72).

The two final Communication items in the CCL Self-Assessment were:

a. Our organization and our partners have each designated specific people we can contact when needed.

b. Our organization has formal or informal procedures that enable consistent connection with our clinical partners, either on the phone, by email, fax, text, in writing, or in person.

The three final Feedback Loop items in the CCL Self-Assessment were:

a. After receiving a referral, our organization confirms with our clinical partners that the referred patient is attending the program.

b. After receiving a referral, our organization receives specific information from the clinical partners about a patient (e.g., if their condition changes).

c. After a referred patient enrolls, our organization regularly sends our clinical partners information about patients' outcomes (e.g., once every program cycle).

Referral process was defined in the CCRM Atlas as: data (e.g., frequency) related to the process of developing, obtaining, and confirming a referral (72).

The four final Referral process items in the CCL Self-Assessment were:

a. Our clinical partners notify our organization when they refer a patient, rather than giving the referral only to the patient.

b. Our clinical partners set up an appointment for referred patients with a member of our staff.

c. Our clinical partners review the patient's discharge summary or discuss health concerns with our organization prior to a patient's first session.

d. Our organization has the capacity to assess referral rates (i.e., how many patients are being sent) from our clinical partners.

Timeliness was defined in the CCRM Atlas as: the amount of time it takes for clinical preventive services to be delivered when clinicians make referrals to community resources (72).

The one final Timeliness items in the CCL Self-Assessment were:

a. After receiving a referral from our clinical partners, patients are able to enroll in our program within the shortest possible amount of time (e.g., less than 2 weeks).

Think-Aloud Interviews, Face Validity, and Construct Validity

To gain insight into the instrument's usability, we performed think-aloud interviews with the intended audience—community organization staff. This type of cognitive pre-testing reveals how users understand and navigate a survey and its questions, and identifies problems with wording or comprehension (73). It assesses face validity, the extent to which a test subjectively appears to measure its intended aims (i.e., the relevance of a test as it appears to test participants) (59). It also captures construct validity, the appropriateness of inferences made on the basis of tested items, specifically whether a test measures the intended construct (74, 75). To ensure we were building a pragmatic tool (57), we asked participants about the instrument's relevance to their jobs, the burden to complete, whether they would use it, and in what context, and how it could be more useful.

Think-Aloud Interview Procedures

We recruited interview participants using purposive sampling to gain the most information possible with efficient use of resources (76). We purposively sampled from (a) community organizations that offered health promotion programs; and (b) had received clinical referrals previously. Further, we were confident in our ability to engage with the interview participants as they had previously participated in related research. We sent an informational flier to community organizations that were in the greater Seattle area. We had no specific requirements about participant position in the organization (i.e., management vs. staff) because we were designing the survey to be completed by anyone at the organization. All organizations offered health-promotion programs (e.g., aging services, tobacco cessation, or breastfeeding support) and received clinical referrals. Interested participants responded via email.

We conducted each interview at the participant's office, with one researcher taking field notes and a second conducting the audio-recorded interviews. Interviews lasted 40–75 min, and participants received a $75 incentive (determined by the fair value and time commitment estimates in the Seattle areas, as determined by the University of Washington). In accordance with accepted think-aloud interview procedures, participants narrated their thoughts as they reviewed the entire printed CCL Self-Assessment. We probed as little as possible to avoid influencing responses but reminded participants to think aloud (73). After each page, we asked participants to explain questions in their own words. We asked explicitly about every item and prompt, heading, response option, and scale to determine if they were appropriate and intuitive, and whether anything should be reworded or re-ordered. At the end, we debriefed overall impressions, usefulness, and length, and asked about the context in which participants might use the instrument. We revised the instrument throughout the cognitive testing period.

Second Step: Feasibility Testing

Feasibility studies help determine if a proposed piece of research (e.g., the CCL Self-Assessment) can realistically be completed (77). We tested the survey (Table 2) for feasibility with staff of the 20 YMCA Associations already enrolled in the PT-REFER study. We administered the survey online using REDCap electronic data-capture tools (REDCap, RRID:SCR_003445) hosted at the University of Washington (78, 79) through grant support (UL1 TR002319, KL2 TR002317, and TL1 TR002318 from NCATS/NIH).

We sent an initial invitation via email to our primary contact at each of the 20 YMCA Associations and followed up with reminders. We received a response from every YMCA primary contact (n = 20). At the end of the survey, we asked the primary contact to provide the name and email address of a second person who worked at their YMCA Association and was knowledgeable about current clinical partnerships. Eighteen of the primary contacts were able to provide a second contact, to whom we sent the same survey email invitation and reminders (n = 18). This allowed us to increase the sample size and include multiple perspectives.

Scoring

In the final instrument (Table 2), each item was rated on a categorical scale from zero to three. The points assigned for each response option were: We do not do this = 0; We do this for less than half our partners = 1; We do this for more than half our partners = 2; We do this for nearly all our partners = 3. We summed item scores for each domain, and the final survey was summed across all five domains. The total survey had a score range from 0 to 45 points. Numbers of items and possible points, by domain, were: Nature of the Relationship (5 items, 15 points); Communication (2 questions, 6 points); Referral Process (4 items, 12 points); Feedback Loop (3 items, 9 points); and Timeliness (1 item, 3 points).

The respondent would sum their score across each domain. Domains with lower scores indicate linkages areas that could be strengthened. Domains with higher scores indicate linkage areas are already strong.

Third Step: Criterion Validation

Criterion Measure: Outreach Practices Survey

Criterion validity refers to how well operationalized scores correlate with a representation of non-test criteria (i.e., the criterion measure) (80). For the criterion measure, we calculated a summary score from sections of a conceptually related but distinct PT-REFER instrument, the Outreach Practices Survey. This instrument captures self-reported partnership activities and empirically measures practices of partnerships, among other concepts. Although it captures comparable activities, it is not a self-assessment of overall partnership strength and was not developed in collaboration with partner organizations, and therefore does not fill the same needs as the CCL Self-Assessment.

Scores were based on seven items: the number of clinical partners (3 categorical options), frequency of communication (6 categorical options), clinical partner's response to the organization's outreach efforts (6 categorical options), strategies used to maintain relationships (checklist of 7 items), level of support from senior leadership (6 categorical options), strategies used to facilitate outreach (checklist of 6 items), and major barriers (checklist of 6 items). Categorical options were assigned points from zero to 3, or zero to 6. Checklist items were assigned one point for each response; except barriers, which were assigned one negative point per response. Scores were summed across all items, such that higher scores indicated stronger connections. The mean score of this survey was 13.4, with a standard deviation of 6.0. High scores on the CCL Self-Assessment were expected to align with high scores on the Outreach Practices Survey.

Criterion Validation Analysis

We administered the CCL Self-Assessment online with 38 YMCA representatives in the PT-REFER study using REDCap electronic data-capture tools hosted at the University of Washington (REDCap, RRID:SCR_003445) (78). Initial invitations were sent via email, and followed up with 3 reminders sent every 4 days. We compared the summary scores from the Outreach Practices Survey to the scores from the CCL Self-Assessment in STATA (Stata, RRID:SCR_012763) version #13.1 using simple linear regression and Pearson's correlation coefficient.

As a secondary descriptive analysis, we tested each of the five CCL Self-Assessment domains against the summary score of the Outreach Practices Survey using the same analytical techniques. Because of the exploratory nature of the secondary analysis and the small number of repeated tests (n = 5), we did not adjust for multiple measures.

Results

First Step: Instrument Development

Think Aloud Interviews

Interviewees found the CCL Self-Assessment accessible, low-burden, and useful. Feedback included suggestions for a) rewording an item, scale, or set of instructions for clarity or precision; b) cutting words for brevity, accessibility, and reduction of visual clutter; and c) formatting to emphasize important points. By the 10th interview, no new insights were emerging, so no additional interviews were conducted.

All participants found the survey length to be appropriate. They suggested retaining all 15 items and offering the survey in both paper and electronic formats. No respondent suggested focusing the survey on single partnerships. All said the survey would be useful to evaluate the strength of their linkages and that they could see themselves using the survey. Respondents were also asked explicitly about each item, the set of prompts, if the scale seemed sensible, and if the answer choices fit. They were asked if the section headers made sense and were helpful.

In order to make the instrument more useful and actionable, participants wanted a separate toolkit with specific action steps that mapped to the instrument. Several participants noted the CCL Self-Assessment would be most useful and applicable to a program manager, rather than to staff working directly with program participants.

Second Step: Feasibility Testing

Thirty-eight staff from 20 YMCA Associations completed the CCL Self-Assessment online; two YMCA Associations had only one staff member participate. The sample was geographically diverse, representing 13 states across the U.S. Of the 38 participants, 34 (89.5%) were females. Five (13%) YMCA Association staff members reported having no clinical partnerships, 14 (37%) reported having 1–5 clinical partnerships, and half (50%) reported at least 6 clinical partnerships. Half of the YMCA Associations consisted of a single branch, while the other half consisted of multiple branches.

Summary data for the CCL Self-Assessment, its domains and specific items are presented in Table 3. All individual item scores predicted whether the corresponding domain score was below or above the midpoint of its range. Put differently, if the domain score was above the midpoint, all corresponding items were also above the midpoint. On all but two of the items, this small sample used the full scoring range (0–3). Standard deviations were generally similar across items, fitting the normal distribution. Timeliness scored highest on the CCL Self-Assessment, while Referral Process scored lowest.

TABLE 3

Table 3. Descriptive data and results of criterion validity testing of the clinical-community linkage self-assessment, 2017 (n = 38)^a.

Third Step: Criterion Validation

We found sites that scored high on the CCL Self-Assessment measure also scored high on the Outreach Practices Survey, and vice-versa. Our primary analysis of the summary scores found significant results using simple linear regression (β: 0.89, p < 0.001) and Pearson's correlation coefficient (0.71, p < 0.001) (Table 3). We also compared the scores on a scatter plot (Figure 2), which shows a roughly linear association.

FIGURE 2

Figure 2. Comparison of scores of clinical-community linkage self-assessment survey and outreach practices survey.

In our secondary analysis, we found that scores for four of the five domains (nature of the relationship, communication, referral process, feedback loop) correlated with the total score from the Outreach Practices Survey (Table 3).

Discussion

We developed and validated a 15-item CCL Self-Assessment that covered five essential CCL domains: nature of the relationship, communication, referral process, feedback, and timeliness. The instrument performed well in a feasibility test with a group of YMCA Association staff. We developed the instrument based on an existing framework with established content validity and demonstrated criterion validity when compared to results from a conceptually related but distinct survey that also captured empirical practices related to partnership strength.

Our comprehensive literature review and expert feedback yielded an initial product that was close to meeting the needs of community organization staff and ensured our instrument filled a gap in the available literature. Expert review and cognitive testing with the intended audience showed that the CCL Self-Assessment was a pragmatic, useful, and desired tool and demonstrated face and construct validity. Feasibility testing with a group of community organizations enrolled in a CCL development study showed that the instrument was easy to administer, and that organizations are willing and able to fill it out. Criterion testing provided evidence of validity for the full survey, and for four of the five domains. Although adding feedback from the linking partner is an important part of a holistic evaluation and should be the focus of future research, the linking partner assessment is beyond the scope of our current work.

Instrument Development and Think-Aloud Interviews

Including multiple perspectives and types of evidence throughout our process allowed us to identify stakeholder needs with more certainty. Reviewing the literature underscored the scarcity of CCL resources and the need for evaluative instruments. Think-aloud interviews echoed feedback heard during our expert review: that the instrument is important and useful to stakeholders, on the condition it be brief and actionable. Although respondents pointed out the survey would be most useful to leadership, it may be beneficial for organizations to assess differing perspectives throughout their workforce, regardless of position within the organization. Previous literature suggests that a key to successful partnerships is consistent awareness, understanding, and cohesiveness (e.g., high commitment, engagement and buy-in) of the partnership by staff at all levels of the organization (1).

Feasibility Testing

The range and distribution of domain scores (presented in Table 3), and the similarity of the standard deviation suggested that the survey captured both low and high performance, and that respondents did not exaggerate their responses to make their performance appear higher. Individual item scores predicted corresponding domain scores, suggesting that none of the items were contrary to the latent concept they represented. Among our respondents, capacity to enroll new program participants (as captured by the timeliness question) was their greatest strength, while establishing a standard referral process with their clinical partners needed the most improvement. This result is consistent with previous literature showing barriers to creating system-level infrastructures for referrals between clinical and non-clinical organizations (6, 81).

Criterion Testing

The CCL Self-Assessment showed validity in criterion testing for the overall survey and for four of the five domains. Testing the validity of all CCL Self-Assessment domains enabled us to diagnose potential problems in the assessment. The single-item timeliness domain had weak, non-significant correlations (β: 0.06, p = 0.23; Pearson's 0.35, p = 0.13). This suggests that timeliness may not have been captured in an appropriate level of detail or context in the Outreach Practices Survey, where it appeared as a single item in a checklist of barriers.

Identifying Next Steps

In this paper we described a step-by-step process for developing and validating a pragmatic, formative assessment, and this approach is likely applicable to a range of settings where a formative self-assessment tool is thought to be needed. The initial results of the CCL Self-Assessment are promising and meet a pressing need: respondents want to better understand and strengthen their relationships with clinical partners. Next steps include testing the CCL Self-Assessment in the field to understand how people use it, and determining if and when it contributes to stronger CCLs. An important future consideration is to further develop the tool to include the perspective of the clinical partner. This would involve testing the same survey design with CCL dyads. If the tool demonstrates utility in improving CCLs, researchers should identify how to disseminate it to community organizations and demonstrate how to use it, which will likely include working with community organization leadership. In order to maintain our commitment to provide stakeholders with an actionable instrument, an important addition will be developing and testing an accompanying toolkit that maps directly to the self-assessment and provides step-by-step guidance for strengthening partnerships.

Limitations

A danger with any formative scale is that it misses critical content and therefore has poor content validity. It is impossible to rule out the prospect that we failed to identify and include critical components of CCLs in our assessment. We attempted to limit this risk by initially basing this work on the CCRM Framework and a review of the related literature, followed by iteratively revising the assessment through a series of think-aloud interviews with our intended audience (community organization staff). This limits, but does not eliminate, the danger that one or more important domains of CCLs were missed. Additionally, we attempted to mitigate the drawbacks of the think-aloud interviews. Our sample was ultimately restricted to the greater Seattle area of Washington State. However, we attempted to provide more generalizable insights into how useful the CCL Self-Assessment would be to a broader group of people and situations by identifying diverse stakeholder needs by engaging staff at different levels in organizations, and from multiple sectors of community based organizations (i.e., faith-based organizations, health promotion programs, senior centers, cancer support, non-governmental organizations, community health programs, and the department of health). To address the fact that think-aloud interviews are prone to subject resistance (73), we trained the interviewer to guide the informant and probe appropriately. Of note, we conducted feasibility testing and criterion validation in the context of an ongoing study, so participants could differ in important ways from similar staff at community organizations in non-research settings. However, we asked for a second respondent, who was not the primary contact in the study, and may more closely represent in-field participation. This practice aligned well with study findings showing the importance of gaining multiple perspectives in an organization (82). Lastly, the scope of our work only included the community organization, and would be improved by the inclusion of the clinical entity.

Conclusion

We described the development of a self-assessment tool for community organizations to evaluate the strength and areas for improvement of their clinical partnerships. Our research showed that having an instrument to evaluate linkages is important (a key component to developing pragmatic measures) to stakeholders, if that instrument is brief, accessible, and pragmatic. We also demonstrated face, content, and construct validity of the CCL Self-Assessment survey and tested criterion validity with a conceptually related but distinct survey that captured empirical practices associated with partnership strength.

Initial results support the feasibility of the instrument, and suggest that the CCL Self-Assessment survey may be used by community organizations to identify the strengths and weaknesses of their linkages. Next steps may include additional statistical validation and testing to determine how the CCL Self-Assessment survey works in the field, and the inclusion of the clinical entity.

Data Availability Statement

The original contributions presented in the study are included in the article/supplementary files, further inquiries can be directed to the corresponding author/s.

Ethics Statement

The studies involving human participants were reviewed and approved by the University of Washington Institutional Review Board. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author Contributions

SF, JH, MP-P, and MK: study conception and design. SF: data collection and draft manuscript preparation. SF, JH, MP-P, MK, and CH: analysis and interpretation of results and manuscript revision. All authors reviewed the results and approved the final version of the manuscript.

Funding

This work was supported by a Health Promotion and Disease Prevention Research Center, Cooperative Agreement Number U48DP005013 from the Centers for Disease Control and Prevention, as well as grant support for REDCap electronic data-capture tools (UL1 TR002319, KL2 TR002317, and TL1 TR002318 from NCATS/NIH).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The findings and conclusions in this manuscript are those of the author(s) and do not necessarily represent the official position of the Centers for Disease Control and Prevention. This is part of a larger study registered at a public trials registry: Physical Therapists - Recommending Enhance Fitness to Expand Reach (PT-REFER); NCT03139461; https://clinicaltrials.gov/ct2/show/NCT03139461.

Footnotes

1. ^The agency also created several resources to guide research and evaluation of effective clinical-community resource relationships (56).

References

1. Centers for Disease Control and Prevention (CDC). Community-Clinical Linkages for the Prevention and Control of Chronic Diseases A Practitioner' s Guide. (2016).

Google Scholar

2. Dymek C, Johnson M, Mardon R, Hassell S, Carpenter D, McGinnis P, et al. Clinical-Community Relationships Measures (CCRM) Atlas. Agency for Healthcare Research & Quality. Rockville, MD (2015). Available online at: http://www.ahrq.gov/professionals/prevention-chronic-care/resources/clinical-community-relationships-measures-atlas/index.html (accessed October 1, 2021).

3. Buckley DI, McGinnis P, Fagnan LJ, Mardon R, Johnson M, Dymek C. Clinical-Community Relationships Evaluation Roadmap. Rockville, Maryland. (2015). Available online at: http://www.ahrq.gov/professionals/prevention-chronic-care/resources/clinical-community-relationships-eval-roadmap/index.html (accessed October 1, 2021).

4. Omura JD, Watson KB, Loustalot F, Fulton JE, Carlson SA. Primary care providers' awareness of physical activity-related intensive behavioral counseling services for cardiovascular disease prevention. Am J Health Promot. (2019) 33:208–16. doi: 10.1177/0890117118784226

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Vincenzo JL, Hergott C, Schrodt L, Perera S, Tripken J, Shubert TE, et al. Physical therapists as partners for community fall risk screenings and referrals to community programs. Front public Heal. (2021) 9:672366. doi: 10.3389/fpubh.2021.672366

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Fishleder S, Petrescu-Prahova M, Harris JR, Steinman L, Kohn M, Bennett K, et al. Bridging the gap after physical therapy: clinical–community linkages with older adult physical activity programs. Innov Aging. (2018) 2:igy006–igy006. doi: 10.1093/geroni/igy006

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Miyawaki CE, Mauldin RL, Carman CR. The potential of optometrists' referrals of older patients to community-based exercise programs: findings from a mixed-methods study. J Aging Phys Act. (2020) 28:194–207. doi: 10.1123/japa.2018-0442

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Alexander SJ, Mager NAD. Ohio community pharmacist interest and participation in community-clinical linkages. Inov Pharm. (2019) 10. doi: 10.24926/iip.v10i2.1760

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Martin JM, Tremblay B, Karlowicz K. Community health education re-envisioned: the value of partnership with the local food bank. J Prof Nurs Off J Am Assoc Coll Nurs. (2020) 36:417–23. doi: 10.1016/j.profnurs.2020.01.009

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Islam N, Rogers ES, Schoenthaler EDd A, Thorpe LE, Shelley D. A cross-cutting workforce solution for implementing community-clinical linkage models. Am J Public Health. (2020) 110:S191–3. doi: 10.2105/AJPH.2020.305692

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Sentell TL, Ching L, Cacal SL, Rowan K. Community-clinical linkages within health care in hawai'i: history, innovation, and future directions. Hawai'i J Med public Heal a J Asia Pacific Med Public Heal. (2019) 78:3–5. Available online at: https://hawaiijournalhealth.org/

PubMed Abstract | Google Scholar

12. Otsuki JA, Hedges JR, Masaki K. Interprofessional education in hawai'i to support community-clinical linkages. Hawai'i J Med public Heal a J Asia Pacific Med Public Heal. (2019) 78:52–4. Available online at: https://hawaiijournalhealth.org/

PubMed Abstract | Google Scholar

13. Trinacty CM, LaWall E, Ashton M, Taira D, Seto TB, Sentell T. Adding social determinants in the electronic health record in clinical care in hawai'i: supporting community-clinical linkages in patient care. Hawaii J Med Public Health. (2019) 78:46. Available online at: https://hawaiijournalhealth.org/

PubMed Abstract | Google Scholar

14. Lohr AM, Doubleday K, Ingram M, Wilkinson-Lee AM, Coulter K, Krupp K, et al. A community health worker-led community-clinical linkage model to address emotional well-being outcomes among latino/a people on the US-mexico border. Prev Chronic Dis. (2021) 18:E76. doi: 10.5888/pcd18.210080

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Quigley L, Matsuoka K, Montgomery KL, Khanna N, Nolan T. Workforce development in Maryland to promote clinical-community connections that advance payment and delivery reform. J Health Care Poor Underserved. (2014) 25:19–29. doi: 10.1353/hpu.2014.0062

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Lohr AM, Ingram M, Nuñez A V., Reinschmidt KM, Carvajal SC. Community–clinical linkages with community health workers in the united states: a scoping review. Health Promot Pract. (2018) 19:349–60. doi: 10.1177/1524839918754868

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Raffo JE, Titcombe C, Henning S, Meghea CI, Strutz KL, Roman LA. Clinical-community linkages: the impact of standard care processes that engage medicaid-eligible pregnant women in home visiting. Women's Heal issues Off Publ Jacobs Inst Women's Heal. (2021) 31:532–9. doi: 10.1016/j.whi.2021.06.006

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Felipe RA, Plescia M, Peterman E, Tomlin H, Sells M, Easley C, et al. A public health framework to improve population health through health care and community clinical linkages: the ASTHO/CDC heart disease and stroke prevention learning collaborative. Prev Chronic Dis. (2019) 16. doi: 10.5888/pcd16.190065

PubMed Abstract | CrossRef Full Text | Google Scholar

19. King RJ, Heisey-Grove DM, Garrett N, Scott KA, Daley MF, Haemer MA, et al. The childhood obesity data initiative: a case study in implementing clinical-community infrastructure enhancements to support health services research and public health. J Public Health Manag Pract. (2022) 28:E430–40. doi: 10.1097/PHH.0000000000001419

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Fiechtner L, Puente GC, Sharifi M, Block JP, Price S, Marshall R, et al. A community resource map to support clinical-community linkages in a randomized controlled trial of childhood obesity, Eastern Massachusetts, 2014-2016. Prev Chronic Dis. (2017) 14:E53. doi: 10.5888/pcd14.160577

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Holliday CS, Williams J, Salcedo V, Kandula NR. Clinical identification and referral of adults with prediabetes to a diabetes prevention program. Prev Chronic Dis. (2019) 16:E82. doi: 10.5888/pcd16.180540

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Beck AF, Marcil LE, Klein MD, Sims AM, Parsons AA, Shah AN, et al. Pediatricians contributing to poverty reduction through clinical-community partnership and collective action: a narrative review. Acad Pediatr. (2021) 21:S200–6. doi: 10.1016/j.acap.2021.04.010

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Beck AF, Henize AW, Kahn RS, Reiber KL, Young JJ, Klein MD. Forging a pediatric primary care-community partnership to support food-insecure families. Pediatrics. (2014) 134:e564–71. doi: 10.1542/peds.2013-3845

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Livsey KR, Sheppard FH, Danielle Martin J. High-impact community clinical experiences to promote health and enhance well-being of older adults. Nurs Educ Perspect. (2020) 41:126–7. doi: 10.1097/01.NEP.0000000000000442

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Lundeen EA, Siegel KR, Calhoun H, Kim SA, Garcia SP, Hoeting NM, et al. Clinical-community partnerships to identify patients with food insecurity and address food needs. Prev Chronic Dis. (2017) 14:E113. doi: 10.5888/pcd14.170343

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Petrescu-Prahova M, Kohn M, Leroux B, Steinman L, Fishleder S, Pike M, et al. Building community-clinical linkages to increase older adult physical activity: The PT-REFER trial protocol and participant baseline characteristics. Contemp Clin Trials Commun. (2019). 15:100373. doi: 10.1016/j.conctc.2019.100373

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Lim S, Wyatt LC, Mammen S, Zanowiak JM, Mohaimin S, Troxel AB, et al. Implementation of a multi-level community-clinical linkage intervention to improve glycemic control among south Asian patients with uncontrolled diabetes: study protocol of the DREAM initiative. BMC Endocr Disord. (2021) 21:233. doi: 10.1186/s12902-021-00885-5

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Lohr AM, Ingram M, Carvajal SC, Doubleday K, Aceves B, Espinoza C, et al. Protocol for LINKS (linking individual needs to community and clinical services): a prospective matched observational study of a community health worker community clinical linkage intervention on the US-Mexico border. BMC Public Health. (2019) 19:399. doi: 10.1186/s12889-019-6725-1

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Pirkle CM, Vu ND, Ilagan LSK, Cacal SL, Stupplebeen DA, Nett B. Greater community-clinical linkages and attention to patient life stage: recommendations to improve diabetes self-management education in Hawai'i. Hawaii J Med Public Health. (2019) 781:70–7.

PubMed Abstract | Google Scholar

30. Vogel MT, Petrescu-Prahova M, Steinman L, Clegg-Thorp C, Farmer C, Sarliker SE, et al. Partnerships for blood pressure control in Washington state, December 2016-July 2017. Health Promot Pract. (2021) 22:52–62. doi: 10.1177/1524839919853819

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Petrescu-Prahova M, Harris JR, Leroux B, Kohn M, Kava CM, Zeliadt SB, et al. Clinical-community linkages as a strategy for increasing evidence-based program reach: Results of the PT-REFER randomized controlled trial with older adults and YMCA associations. Contemp Clin trials Commun. (2022) 26:100888. doi: 10.1016/j.conctc.2022.100888

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Ayers C, Arch J. VA-academic partnerships: challenges and rewards for new VA mental health investigators. Commun Ment Health J. (2013) 49:709–13. doi: 10.1007/s10597-012-9575-1

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Ackermann RT. Description of an integrated framework for building linkages among primary care clinics and community organizations for the prevention of type 2 diabetes: emerging themes from the CC-Link study. Chronic Illn. (2010) 6:89–100. doi: 10.1177/1742395310364857

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Olmos-Ochoa TT, Miake-Lye IM, Glenn BA, Chuang E, Duru OK, Ganz DA, et al. Sustaining successful clinical-community partnerships in medically underserved urban areas: a qualitative case study. J Community Health Nurs. (2021) 38:1–2. doi: 10.1080/07370016.2021.1869423

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Rosen M. Impacting quadruple aim through sustainable clinical-community partnerships: best practices from a community-based organization perspective. Am J Lifestyle Med. (2020) 14:524–31. doi: 10.1177/1559827620910980

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Ramanadhan S, Daly J, Lee RM, Kruse GR, Deutsch C. Network-based delivery and sustainment of evidence-based prevention in community-clinical partnerships addressing health equity: a qualitative exploration. Front public Heal. (2020) 8:213. doi: 10.3389/fpubh.2020.00213

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Stupplebeen DA, Sentell TL, Pirkle CM, Juan B, Barnett-Sherrill AT, Humphry JW, et al. Community health workers in action: community-clinical linkages for diabetes prevention and hypertension management at 3 community health centers. Hawai'i J Med public Heal a J Asia Pacific Med Public Heal. (2019) 78:15–22. Available online at: https://hawaiijournalhealth.org/

PubMed Abstract | Google Scholar

38. Brandt HM, Vanderpool RC, Curry SJ, Farris P, Daniel-Ulloa J, Seegmiller L, et al. A multi-site case study of community-clinical linkages for promoting HPV vaccination. Hum Vaccin Immunother. (2019) 15:1599–606. doi: 10.1080/21645515.2019.1616501

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Payán DD, Sloane DC, Illum J, Vargas RB, Lee D, Galloway-Gilliam L, et al. Catalyzing implementation of evidence-based interventions in safety net settings: a clinical-community partnership in south Los Angeles. Health Promot Pract. (2017) 18:586–97. doi: 10.1177/1524839917705418

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Koch U, Stout S, Landon BE, Phillips RS. From healthcare to health: a proposed pathway to population health. Healthc. (2016) 4:291–7. doi: 10.1016/j.hjdsi.2016.06.007

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Braun KL, Stewart S, Baquet C, Berry-Bobovski L, Blumenthal D, Brandt HM, et al. The national cancer institute's community networks program initiative to reduce cancer health disparities: outcomes and lessons learned. Prog Community Health Partnersh. (2015) 9:21–32. doi: 10.1353/cpr.2015.0017

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Henize AW, Beck AF, Klein MD, Adams M, Kahn RS. A Road map to address the social determinants of health through community collaboration. Pediatrics. (2015) 136:e993–1001. doi: 10.1542/peds.2015-0549

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Lee RM, Ramanadhan S, Kruse GR, Deutsch C. A mixed methods approach to evaluate partnerships and implementation of the massachusetts prevention and wellness trust fund. Front public Heal. (2018) 6:150. doi: 10.3389/fpubh.2018.00150

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Coleman EA, Whitelaw NA, Schreiber R. Caring for seniors: how community-based organizations can help. Fam Pract Manag. (2014) 21:13–7. Available online at: https://www.aafp.org/fpm/2014/0900/p13.html

PubMed Abstract | Google Scholar

45. Aging Disability Business Institute,. Assessment Tools. Available online at: https://www.aginganddisabilitybusinessinstitute.org/assessment-tools/ (accessed Mar 11, 2022)

Google Scholar

46. The Commonwealth Fund,. Welcome to the Return on Investment (ROI) Calculator for Partnerships to Address the Social Determinants of Health. Available online at: https://www.commonwealthfund.org/roi-calculator (accessed Mar 11 2022)

Google Scholar

47. Meals on Wheels America. Effective Partnerships Between Community-Based Organizations Healthcare: A Possible Path to Sustainability. (2018). Available online at: https://www.mealsonwheelsamerica.org/docs/default-source/research/effective-partnerships-between-cbos-and-healthcare_full-report_final.pdf?sfvrsn=8e5fbf3b_2 (accessed Mar 11 2022)

Google Scholar

48. Health Leads. The Roadmap. Available online at: https://healthleadsusa.org/resource-library/roadmap/ (accessed Mar 11 2022)

Google Scholar

49. National Council on Aging. NCOA Partnership Assessment Tool. (2021). Available online at: https://www.ncoa.org/article/ncoa-partnership-assessment-tool (accessed Mar 11 2022)

Google Scholar

50. Partnership for Healthy Outcomes. Developed by Partnership for Healthy Outcomes Bridging Community-Based Human Services and Healthcare A collaboration of Partnership Assessment Tool for Health ® ®. (2017).

Google Scholar

51. Safer Care Victoria. Partnering in Healthcare Framework. (2019). Available online at: https://www.bettersafercare.vic.gov.au/support-and-training/partnering-with-consumers/pih (accessed Mar 11 2022)

Google Scholar

52. Center for the Advancement of Collaborative Strategies in Health. The Partnership Self-Assessment Tool. (2002). Available online at: https://atrium.lib.uoguelph.ca/xmlui/bitstream/handle/10214/3129/Partnership_Self-Assessment_Tool-Questionnaire_complete.pdf?sequence=1&isAllowed=y (accessed Mar 11 2022)

Google Scholar

53. Yousefian S, Sohrabizadeh S, Jahangiri K. Identifying the components affecting intra-organizational collaboration of health sector in disasters: Providing a conceptual framework using a systematic review. Int J Disast Risk Reduct. (2021) 57:102146. doi: 10.1016/j.ijdrr.2021.102146

CrossRef Full Text | Google Scholar

54. Etz RS, Cohen DJ, Woolf SH, Holtrop JS, Donahue KE, Isaacson NF, et al. Bridging primary care practices and communities to promote healthy behaviors. Am J Prev Med. (2008) 35:S390–7. doi: 10.1016/j.amepre.2008.08.008

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Donabedian A. The quality of care. How can it be assessed? JAMA. (1988) 260:1743–8. doi: 10.1001/jama.260.12.1743

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Agency for Healthcare Research and Quality. Clinical-Community Linkages. Rockville, MD. (2016). Available online at: https://www.ahrq.gov/professionals/prevention-chronic-care/improve/community/index.html (accessed Mar 24, 2017)

Google Scholar

57. Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. (2013) 45:237–43. doi: 10.1016/j.amepre.2013.03.010

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Hannon PA, Helfrich CD, Chan KG, Allen CL, Hammerback K, Kohn MJ, et al. Development and pilot test of the workplace readiness questionnaire, a theory-based instrument to measure small workplaces' readiness to implement wellness programs. Am J Heal Promot. (2017) 31:67–75. doi: 10.4278/ajhp.141204-QUAN-604

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Mitchell MS, Goodman JM, Alter DA, Oh PI, Faulkner GEJ. Development of the Health Incentive Program Questionnaire (HIP-Q) in a cardiac rehabilitation population. Transl Behav Med. (2015) 5:443–59. doi: 10.1007/s13142-015-0330-3

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Willis GB, Royston P, Bercini D. The use of verbal report methods in the development and testing of survey questionnaires. Appl Cogn Psychol. (1991) 5:251–67. doi: 10.1002/acp.2350050307

CrossRef Full Text | Google Scholar

61. Mazzucca S, Arredondo EM, Hoelscher DM, Haire-Joshu D, Tabak RG, Kumanyika SK, et al. Expanding implementation research to prevent chronic diseases in community settings. Ann Rev Public Health. (2020) 42:135–58. doi: 10.1146/annurev-publhealth-090419-102547

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Centers for Disease Control and Prevention. Program Performance and Evaluation Office (PPEO). (2021).

Google Scholar

63. Centers for Disease Control and Prevention (CDC). Health-Related Quality of Life (HRQOL). (2018).

Google Scholar

64. Nunnally J, Bernstein I. Psychometric Theory 3rd edition. MacGraw-Hill, New York. (1994).

PubMed Abstract | Google Scholar

65. Streiner DL, Norman GR, Cairney J. Health Measurement Scales. Oxford University Press. (2015) vol. 11. doi: 10.1093/med/9780199685219.001.0001

CrossRef Full Text | Google Scholar

66. DeVellis RF. Scale Development: Theory and Applications. Sage publications. (2016). Vol. 26.

Google Scholar

67. Hoenig H, Duncan PW, Horner RD, Reker DM, Samsa GP, Dudley TK, et al. Structure, process, and outcomes in stroke rehabilitation. Med Care. (2002) 40:1036–47. doi: 10.1097/00005650-200211000-00005

PubMed Abstract | CrossRef Full Text | Google Scholar

68. Shrotryia VK, Dhanda U. Content Validity of Assessment Instrument for Employee Engagement. SAGE Open. (2019). doi: 10.1177/2158244018821751

CrossRef Full Text | Google Scholar

69. Amoroso C, Proudfoot J, Bubner T, Jayasinghe UW, Holton C, Winstanley J, et al. Validation of an instrument to measure inter-organisational linkages in general practice. Int J Integr Care. (2007) 7. doi: 10.5334/ijic.216

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Fortinsky RH, Unson CG, Garcia RI. Helping family caregivers by linking primary care physicians with community-based dementia care services: the Alzheimer's service coordination program. Dementia. (2002) 1:227–40. doi: 10.1177/147130120200100208

CrossRef Full Text | Google Scholar

71. Schaefer JA, Cronkite R, Ingudomnukul E. Assessing continuity of care practices in substance use disorder treatment programs. J Stud Alcohol. (2004) 65:513–20. doi: 10.15288/jsa.2004.65.513

PubMed Abstract | CrossRef Full Text | Google Scholar

72. Himmelman AT. Collaboration Defined: A Developmental Continuum of Change Strategies. (2002).

Google Scholar

73. Dillman DA, Smyth JD, Christian LM. Internet, Phone, Mail, and Mixed-Mode Surveys, The Tailored Design Method. Hoboken, NJ: John Wiley & Sons. (2014). Available online at: https://play.google.com/books/reader?printsec=frontcover&output=reader&id=W5I_BAAAQBAJ&pg=GBS.PT12

Google Scholar

74. Üstün TB, Chatterji S, Kostanjsek N, Rehm J, Kennedy C, Epping-Jordan J, et al. Developing the world health organization disability assessment schedule 2.0. Bull World Health Organ. (2010) 88:815–23. doi: 10.2471/BLT.09.067231

PubMed Abstract | CrossRef Full Text | Google Scholar

75. Messick S. Test validity: a matter of consequence. Soc Indic Res. (1998) 45:35–44. doi: 10.1023/A:1006964925094

CrossRef Full Text | Google Scholar

76. Palinkas LA, Horwitz SM, Green CA, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Heal. (2015) 42:533–44. doi: 10.1007/s10488-013-0528-y

PubMed Abstract | CrossRef Full Text | Google Scholar

77. Orsmond GI, Cohn ES. The distinctive features of a feasibility study: objectives and guiding questions. OTJR Occup Particip Heal. (2015) 35:169–77. doi: 10.1177/1539449215578649

PubMed Abstract | CrossRef Full Text | Google Scholar

78. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. (2009) 42:377–81. doi: 10.1016/j.jbi.2008.08.010

PubMed Abstract | CrossRef Full Text | Google Scholar

79. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O'Neal L, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. (2019) 95:103208. doi: 10.1016/j.jbi.2019.103208

PubMed Abstract | CrossRef Full Text | Google Scholar

80. Piedmont RL. Criterion Validity BT - Encyclopedia of Quality of Life and Well-Being Research. In: Michalos AC, editor. Dordrecht: Springer Netherlands. (2014). p. 1348. doi: 10.1007/978-94-007-0753-5_618

CrossRef Full Text | Google Scholar

81. Valaitis R, Meagher-Stewart D, Martin-Misener R, Wong ST, MacDonald M, O'Mara L, et al. Organizational factors influencing successful primary care and public health collaboration. BMC Health Serv Res. (2018) 18:1–7. doi: 10.1186/s12913-018-3194-7

PubMed Abstract | CrossRef Full Text | Google Scholar

82. Weiner BJ. A theory of organizational readiness for change. Implement Sci. (2009) 4:67. doi: 10.1186/1748-5908-4-67

PubMed Abstract | CrossRef Full Text