Using CollaboRATE, a brief patient‐reported measure of shared decision making: Results from three clinical settings in the United States

Abstract Introduction CollaboRATE is a brief patient survey focused on shared decision making. This paper aims to (i) provide insight on facilitators and challenges to implementing a real‐time patient survey and (ii) evaluate CollaboRATE scores and response rates across multiple clinical settings with varied patient populations. Method All adult patients at three United States primary care practices were eligible to complete CollaboRATE post‐visit. To inform key learnings, we aggregated all mentions of unanticipated decisions, problems and administration errors from field notes and email communications. Mixed‐effects logistic regression evaluated the impact of site, clinician, patient age and patient gender on the CollaboRATE score. Results While CollaboRATE score increased only slightly with increasing patient age (OR 1.018, 95% CI 1.014‐1.021), female patient gender was associated with significantly higher CollaboRATE scores (OR 1.224, 95% CI 1.073‐1.397). Clinician also predicts CollaboRATE score (random effect variance 0.146). Site‐specific factors such as clinical workflow and checkout procedures play a key role in successful in‐clinic implementation and are significantly related to CollaboRATE scores, with Site 3 scoring significantly higher than Site 1 (OR 1.759, 95% CI 1.216 to 2.545) or Site 2 (z=−2.71, 95% CI −1.114 to −0.178). Discussion This study demonstrates that CollaboRATE can be used in diverse primary care settings. A clinic's workflow plays a crucial role in implementation. Patient experience measurement risks becoming a burden to both patients and administrators. Episodic use of short measurement tools could reduce this burden.


| INTRODUCTION
An emphasis in United States (US) health-care policy on improving patient experiences of care has led to increased focus on structural and process measures of health care, including patient satisfaction with health-care facilities, personnel and service delivery. [1][2][3][4] Recently, interest has grown in shared decision making, a process where patients are supported to participate in health-care decisions. Given this interest, CollaboRATE was developed as a process measure of shared decision making (SDM) between patients and clinicians. 5,6 Existing patient experience measurement typically involves lengthy surveys that ask patients to recall encounters occurring up to 6 months in the past. 7 In contrast, CollaboRATE is brief and focuses on a core construct, namely the perception of being informed and then involved in decision-making steps. This focus reduces survey burden and enables the use of efficient and inexpensive survey administration methods which minimize the delay between a patient experiencing a health-care interaction and responding to an evaluation survey (eg,. using text messages or automated telephone calls). This real-time survey administration allows patients to more easily recall their satisfaction with care, and Stull et al. 8 find recall of satisfaction to be optimal within days of a clinic visit.
Few studies measure outpatient care experiences close to the time of visit and those that do are often conducted in the context of a specific disease. 9,10 One exception is found in Tai In light of the emphasis placed on process measures of patient experience and the paucity of reports addressing feasibility and results of routine, real-time, patient-reported measurement of shared decision making across multiple sites, we aim to (i) provide insight on facilitators and challenges to implementing a real-time patient survey and (ii) evaluate CollaboRATE scores and response rates across multiple primary care clinical settings with varied patient populations.

| Field notes
Field notes regarding implementation challenges and solutions were kept on an ad hoc basis by study staff at all sites throughout the project as an integral part of project management processes. Investigators and research staff from all sites contributed observations through email messages and regular project meetings; meetings were held weekly at Site 1 and quarterly between Sites 1 and 2 and Sites 1 and 3.
All contributors were familiar with the study protocol and participated in the design of survey implementation at their respective sites. Field notes were collected, organized and held by the project coordinator based at the Lebanon, NH site. All instances of unanticipated decisions, problems and errors were documented.

| Overview
Setting Three geographically diverse US primary care practices were included in this study: Lebanon, NH (Site 1); Los Gatos, CA (Site 2); and Chelsea, MA (Site 3). Detailed information on each site is included below.

Participants
All adult patients ages 18 and older visiting the participating primary care teams were eligible to participate.

Data collection process
All sites allowed clinicians to describe the survey to patients, although this behaviour was eventually encouraged as a participationenhancing measure only at the Los Gatos site. No clinicians at any site personally delivered the survey to their patients.

Setting details
This setting included three primary care teams based in a rural academic medical centre in New Hampshire with approximately 16 000 patients. Clinicians included physicians, physicians' assistants, nurses, nurse practitioners and pharmacists. The patient population is more than 95% white and non-Hispanic, and 99% of patients speak English as a primary language.

Participant details
Parents and guardians of patients under 18 were also eligible to complete CollaboRATE on behalf of their children at this site.

Setting details
This primary care clinic in suburban Los Gatos, CA is one of many community-based clinics of the Palo Alto Medical Foundation, a large, not-for-profit health-care delivery system in the San Francisco Bay Area, and has a patient panel of approximately 13 000. The patient population is predominantly Caucasian (44%) and Asian (34%).
Approximately 90% of patients speak English as a primary language, with 46 other language groups also represented.

Data collection process
This site aimed to collect 300 completed CollaboRATE surveys during the February 2015-March 2015 study period. The study was reviewed by the Sutter Health Institutional Review Board and written participant consent requirements were waived. An initial week-long pilot period involved only the reception staff informing patients about the CollaboRATE survey as they arrived for their visit. After their visit, a receptionist invited all eligible patients to complete a paper survey containing CollaboRATE. The pilot period yielded very low response rates (2%). Recruitment in the subsequent data collection period included asking physicians to encourage participation, additional reminder signage, and medical assistants providing patients with surveys before they left the exam room. A locked collection box was placed at the clinic exit with a sign reminding patients to deposit the completed survey in the box. Thus, the physical clinic layout did not present a barrier to survey completion at this site. The study population at the Los Gatos, CA site included all patients visiting the participating clinics.

| Analysis
Returned surveys missing one or more CollaboRATE responses were considered incomplete and excluded from analysis. Descriptive statistics compared response rates and CollaboRATE scores across sites.
Because patient reported experience measures often show ceiling effects, we decided a priori to conduct a top score analysis which has been shown to enhance variation in scores 5,.14 Therefore, the CollaboRATE score represents the proportion of patients (minimum sample size of 25) responding with the highest possible score on all three questions. The unit of analysis was patients.  15 To assess effect modification by site, we reran the original mixed-effects logistic regression model described above, this time including two interaction terms accounting for associations between site and patient age or gender, respectively. Survey data was analysed with Stata 13 software. 16 To inform the key learnings and descriptions of site-level characteristics, we aggregated all mentions of unanticipated decisions, problems, and administration errors from field notes and email communications.

| Response rates by site
Response rates varied across sites, with Site 3 (Chelsea, MA) achieving the highest response rate of 73% compared to 25% overall at Site 1 (Lebanon, NH) and 30% at Site 2 (Los Gatos, CA). Site 1 saw variation in response rates across the various administration modes, described in detail elsewhere. 17 Of all surveys returned, missing CollaboRATE data were minimal: <0.5% at Site 1, <1% at Site 2, and <0.1% at Site 3.  Patient demographics also play a role.

| Key learnings and site-level characteristics
The following key learnings and other site-level characteristics may contribute to the observed variation in CollaboRATE response rates and CollaboRATE scores by site.

| Site 1: Lebanon, New Hampshire
The relatively novel patient survey administration procedures used in patient portal, IVR, and SMS modes presented logistical challenges.
We relied on the medical centre's information systems department for programming key aspects of survey administration. We were not able to negotiate priority status for our programming needs, given other competing deadlines in the organizational work schedule, and this led to delays and errors. For example, limitations of existing software meant survey format was not exactly as we had stipulated and we were unable to ensure that the invitation to complete the survey on the patient portal was sent from a neutral source and not from their own clinician. We concluded that unless collecting patient experience data was an organizational priority, other organizations' information systems may be reluctant to facilitate in-house digital methods of patient survey administration. Capacity for engaging external contractors or an on-staff programmer might have eliminated these delays, although the need for integration into existing administrative systems would still exist.
In the paper survey administration mode, patients who did not need to schedule follow-up appointments often did not make the effort to collect the CollaboRATE survey from the assigned staff person, despite this process being intended to occur for each patient. The location of the staff assigned to distribute surveys was not convenient, as their offices were not located near the clinic exit and access by patients often required exiting the consultation room and walking away from the clinic exit. As such, many patients may not have received or completed CollaboRATE due to the clinic layout.
Our attempts to use text messaging (SMS) on cellular telephones also revealed logistical challenges due to significant variation in subscriber plans and very limited reception in some rural areas. In the United States, some cellular telephone subscribers pay a fee (<$0.50 USD) for each text message sent or received. Some cellular service providers offer to deliver an organization's outgoing messages to their customers free of charge for an annual $25 000 USD up-front fee, although that cost was prohibitive in this project. Despite the cost to patients, we used the SMS approach to assess text message patient survey administration in this study.

| Site 2: Los Gatos, California
At the Los Gatos site, the clinic's physicians and operational leaders strongly supported the project. When the initial attempt to collect responses by asking receptionists to alert patients to the survey led to only 26 surveys completed over a two-week pilot period, we were able to consult with the two clinic leaders and change the survey administration workflow. The new workflow ensured that both clinicians and reception staff were requesting survey completion, supported by the medical assistants giving surveys to patients as they finished their clinic visits. Using the modified recruitment methods, 323 surveys were completed during a two-week data collection period.

| Site 3: Chelsea, Massachusetts
We observed significant staff commitment to the data collection process at this site. The clinic workflow and layout facilitated data collection; patients were required to make contact with administrative staff as they left the clinic, which provided an opportunity for staff to confirm receipt of the survey. We also believe that the additional financial incentive to MAs contributed to the 73% response rate achieved in this site.

| Principal findings
Variation was found across the three sites with regard to both response rates and CollaboRATE scores. Site-level factors were associated with scores at Site 3 (Chelsea) where response rates were highest, but these factors were not as influential as the clinician seen and the patient's gender in accounting for observed variation in

| Strengths and limitations
The participating sites represent a diverse group of rural, urban, and suburban primary care clinics in which CollaboRATE was successfully administered immediately following primary care clinic visits using Our lack of detailed patient demographic data may inflate estimates of site and clinician impact on CollaboRATE scores. [18][19][20][21] The variation in CollaboRATE score we observed between sites may therefore be due in part to the socio-demographic diversity of the respondent populations between sites or to selection bias due to differences in response rates between sites, rather than to other potential site-specific sources of variation such as clinic layout and workflow. Additionally, we lack the ability to link survey responses to individual patients and therefore cannot model survey response through regression analysis.
Demographic and performance data on clinicians, if it were available, may also help explain the existing relationship between clinician identity and CollaboRATE scores. Finally, we lack data on those patients at Los Gatos who were unable to participate because the survey was not available in their preferred language, although field notes did not include record of patients unable to complete the survey due to language restrictions. to lead to better response rates. Technological survey administration also helps boost response rates.

| Policy implications
There is a significant risk that patient experience measurement could However, CG-CAHPS lacks items directly assessing the process of SDM and takes the form of a lengthy questionnaire. 7 This level of measurement cannot be sustained unless there is interest in using the data for quality improvement, and it is helpful for the motivation for such use to arise from the organization undertaking the measurement.
Episodic use of short measurement tools could reduce this burden.
We conclude that collecting real-time data about key aspects of patient experience is not easy. Ensuring that the data are made available for clinicians and management so that it can be used for timely quality improvement is a research frontier yet to be explored. CollaboRATE was delivered to patients by telephone using an interactive voice response system, programmed by the medical centre's information systems department. An automated telephone call was made to each patient's cell phone number at 7:00 pm on the day of their clinic visit. Before initiating the survey, the respondent was asked to confirm that he or she was the individual who had visited the clinic that day. Upon confirmation, numerical keypad responses to CollaboRATE questions were requested. If any of the three CollaboRATE questions remained incomplete at 7:00 pm the following day, an identical automatic call was placed at that time.

CONFLICTS OF INTEREST
Mode 4: Short message service (SMS text messages) Text messages were sent to patient cell phones at 7:00 pm on the day of their clinical visits, programmed by the medical centre's information systems department. The first message introduced the survey and offered an opt out opportunity. Remaining messages each contained a single CollaboRATE question and response instructions. Subsequent messages were triggered by each further reply. If any of the three CollaboRATE questions remained incomplete at 7:00 pm the following day, the first introductory text message was re-sent.
Mode 5: Tablet and mail Using tablet computers, research assistants offered patients an opportunity to complete an online version of CollaboRATE as they left the clinic, hosted in Qualtrics (Qualtrics LLC, Provo, UT). Patients who declined the tablet opportunity were asked to complete and return a paper-based survey by mail in a postage-paid envelope. two degree-of-freedom tests of whether there was a difference from odds ratio of 1 for the two interaction contrasts whose effects are identified (Sites 2 and 3) against the baseline interaction contrast (of Site 1).