INTRODUCTION

Pragmatic clinical trials generate evidence of how health care interventions work in real-world settings. Many pragmatic clinical trials, such as those supported through the National Institutes of Health (NIH) Health Care Systems Research Collaboratory, are embedded within health care systems. That is, the identification and random assignment of participants and sites, intervention delivery, and outcome assessment occur in the context of usual clinical operations. Such trials raise a number of ethical issues that have been described previously.2 One issue relates to what we have coined “pragmatic clinical trial collateral findings” (PCT-CFs)—findings, discovered intentionally or unintentionally, that may have implications for patients’ health but were not generated to address the trial’s primary research question.5 For example, in a pragmatic clinical trial comparing the effectiveness of 2 medications for hypertension, researchers collecting data from the electronic health record might discover that a patient appeared to be taking 2 contraindicated medications.

There are important questions concerning the identification, assessment, and management of PCT-CFs. What constitutes a PCT-CF that should be reported? What are the roles and responsibilities within a health care system for managing PCT-CFs? To whom and how should findings be reported? The NIH Collaboratory conducted a multiphase project to understand the challenges of PCT-CFs and suggest policy recommendations for handling them. The first step was a conceptual analysis of ethical issues associated with PCT-CFs.5 The second step was a qualitative study of the perspectives of stakeholders in pragmatic clinical trials, including principal investigators and health care system leaders.6 The third step was a qualitative study of the views of the general public on the disclosure of PCT-CFs.1

Results of this previous work and input from a multistakeholder advisory panel revealed significant uncertainty about whether and how to communicate PCT-CFs to patients. Specifically, it is unclear who should notify the patient and whether to explain that the PCT-CF was discovered in the context of a pragmatic clinical trial. It is also unknown how the effects of these 2 factors might depend on the type of PCT-CF. Therefore, we conducted an experiment, using a nationally representative web-based survey, to determine how reactions to a letter communicating a PCT-CF relate to who signed the letter, the type of finding, and whether the letter specified that the finding arose from a pragmatic clinical trial.

METHODS

We conducted a web-based survey using a between-subjects design in which respondents were stratified by educational level (i.e., high school degree or less vs some college or more) and randomly assigned within strata to 1 of 16 hypothetical scenarios. The scenarios reflected all possible combinations of who signed a letter notifying the patient about a PCT-CF, whether the letter included a detailed description of the pragmatic clinical trial, and the type of collateral finding (Table 1). In each scenario, the respondent visits their doctor, Dr. Lee, at City Medical Center for 1 of 3 possible indications: treatment for high blood pressure, evaluation of back pain, or routine screening for colon cancer. Two years later, the respondent receives a letter in the mail from City Medical Center reporting a collateral finding that may be important to their health. The respondent then records their reactions and perceptions in response to a combination of open- and closed-ended questions.

Table 1 Experimental Factors and Levels
Full size table

Sample

Sample selection and administration of the survey were managed by Ipsos. During the field period from May 5 through May 14, 2020, 7635 English-speaking potential respondents 18 years and older were randomly sampled from the Ipsos KnowledgePanel of 55,000 US adults. The sample recruitment process uses an address-based sampling method from the latest Delivery Sequence File of the US Postal Service—a database with full coverage of all delivery points in the USA. Thus, samples from KnowledgePanel cover all households regardless of their access to a telephone, providing fully representative online samples. Adults recruited from households without internet access are provided a web-enabled device, such as a tablet computer, and free internet service. Once assigned to the survey, panel members received an email invitation to participate in the study.4 Nonresponders received an email reminder after 3 days. Respondents who completed the survey received $5 worth of points that could be redeemed for cash, gift cards, or other merchandise.

Development of Scenarios

The study team drafted the 16 scenarios and letters based on previous focus group findings.1 Draft letters were tested and revised on the basis of small-group cognitive interviews with 16 individuals of a variety of ages, genders, and races and ethnicities. Additional changes to the letters were made in response to individual cognitive interviews with a separate sample of 20 participants. To ensure that the scenarios and letters were realistic in terms of how the collateral findings would be reported to patients, the letters were reviewed by 4 health care system administrators who also served as members of the project’s advisory committee.

Data Collection

The study team drafted 16 versions of a survey, each of which began with a scenario and letter describing the finding (see Appendix 1 for example). Common to all survey versions was a series of open-ended, multiple-choice, and closed-ended questions that focused on the following topics regarding the hypothetical scenario and letter2: emotional reactions to receiving the letter5; why respondents had these reactions6; questions they had after reading the letter1; what their next steps would be4; perceptions of why they were being sent the letter; and3 thoughts about the communication itself (Appendix 2). Questions about the communication addressed the amount of information in the letter, how well respondents thought they understood the communication, the clarity of the letter, and the acceptability of receiving the communication by mailed letter rather than other modes. For some items, the survey provided open text fields to allow respondents to explain their answers. The survey also included questions about respondents’ perception of their health, whether they regularly see a health care provider, and whether they thought their survey answers were influenced by the ongoing COVID-19 pandemic. All survey items were evaluated and revised through 2 additional rounds of individual cognitive interviews with a total of 20 participants, who also provided feedback on the letters as noted earlier.

After the survey data were collected, pre-banked information that was previously collected by Ipsos on panel members’ demographic characteristics and backgrounds was added to the data set. Study consent and execution were conducted online by Ipsos. After agreeing to participate, respondents read the description of the scenario to which they were randomly assigned, including the letter reporting the PCT-CF, and answered the questions described above. The Institutional Review Boards of the Duke University Health System and Johns Hopkins Medicine approved this research.

Data Analysis

We prespecified 2 primary outcome variables: the action the respondent would take next and the respondent’s emotional reactions. The only action the letters asked the respondents to take was to contact their doctor immediately. Therefore, respondents’ answers to a question asking them what they would do next were coded as “yes” if they would contact their doctor immediately and “no” otherwise. For the questions about emotional reactions, exploratory analyses suggested that the best way to reflect these reactions was to categorize respondents as selecting all negative, all positive, mixed, or no feelings.

Respondents who sped through the survey (i.e., they took less than 2 min to complete it) or who were missing responses to more than one-third of the questions were omitted from the analytic sample. Patient characteristics were summarized using standard descriptive statistics on the unweighted sample. All remaining quantitative analyses used survey weights to allow inferences to the US English-speaking adult population. The primary outcome, whether the respondent would contact their doctor immediately, was analyzed using a multiple logistic regression model that included the role of the letter’s signatory, the presence of a trial description, the type of PCT-CF, and all 2-way and 3-way interactions among these variables. We used a threshold of P < .05 as a heuristic to derive a final model by first testing the 3-way interaction, then an omnibus term for all 2-way interactions, and then individual 2-way interactions, with testing of lower-order effects only if the value for the current step was P > .05. Results for all retained covariates are expressed in terms of odds ratios and corresponding 95% confidence intervals, along with model-predicted probabilities by the level of each covariate.

We prespecified a policy-relevant difference between groups in a dichotomous outcome as 15 percentage points. We conducted sensitivity analyses by estimating the same model for respondents who reported having someone (nurse practitioner, doctor, etc.) they see for regular health visits and separately for those in worse versus better self-reported overall health (“poor”/“fair”/“good” vs “very good”/“excellent”). Another sensitivity analysis counted respondents who indicated they would contact either their doctor or some “other health care professional” as taking the recommended action. We analyzed the secondary outcome of emotional reaction profile the same as for the primary outcome, but using a multinomial logistic regression instead of a binary model. We conducted similar sensitivity analyses for respondents who had their own doctor and based on self-reported health.

Two members of the research team (JB, EM) reviewed responses from the open text field for the survey item, “What questions, if any, would you have?” and developed a 14-item codebook based on commonly recurring themes. The codebook was iteratively revised by the coding team with input from other coauthors (GG, KW). Each survey response was independently coded by 2 members of the coding team, who subsequently discussed and resolved any coding discrepancies (JB, EM).

RESULTS

Of the 7635 people who were sampled and invited to participate, 4218 (55%) completed the study. After removing 138 respondents for speeding through the survey and/or missing more than one-third of responses, the final analytic data set included 4080 respondents. The median time for survey completion was 6 min. Table 2 shows the demographic characteristics of the unweighted sample. The sample reflects diversity with respect to all key demographic variables.

Table 2 Characteristics of Sample (Unweighted; N = 4080)
Full size table

In response to the question “Do you think you would have answered any of these questions differently if we weren’t in the middle of a coronavirus/COVID-19 pandemic?” 85% of respondents answered “no,” 3% answered “yes,” and 12% answered “maybe.”

Contacting a Doctor Immediately

Respondents’ intention to contact a doctor immediately or not did not vary by the role of the letter’s signatory (P = .80), but there was an interaction between whether the letter included a description of the trial and the type of PCT-CF (χ23 = 21.8; P < .001). Inclusion of a trial description was associated with a lower likelihood of contacting a doctor among respondents assigned to the A/B drug trial with a PCT-CF of contraindicated medications, and a higher likelihood among respondents in the colorectal cancer screening trial with a PCT-CF of an underperforming test kit (Supplemental Figure 1). There was no relationship between trial description and intention to contact a doctor for the other 2 types of findings (contraindicated medications and hematuria). However, as shown in Table 3, neither of the effects observed for trial description resulted in differences in predicted probabilities that exceeded our prespecified threshold for a policy-relevant difference.

Table 3 Predicted Probabilities of Intending to Contact a Doctor Immediately by Type of Finding and Level of Detaila
Full size table

In sensitivity analyses, the absence of effects exceeding the predefined policy-relevant threshold was consistent for respondents who had their own doctor and for those who self-reported worse and better overall health (data not shown). Broadening the correct answer to include immediately contacting another health professional or doctor also did not produce effects that exceeded the threshold.

Emotional Reactions

Respondents endorsed a range of emotional reactions after reading the letter, with “concerned” being the most frequent reaction (Table 4). In terms of the combination of emotions endorsed, 2869 (70%) respondents endorsed all negative emotions, 219 (5%) endorsed all positive emotions, 561 (14%) endorsed mixed emotions, and 423 (10%) endorsed no emotions. Membership in each of these categories was related to letter signatory (χ23 = 12.2; P < .007), trial description (χ23 = 9.9; P = .02), and type of PCT-CF (χ29 = 31.7; P < .001). There were no interactions among these factors. Supplemental Figure 2 shows the distribution of respondents in each emotional category by level of each experimental factor derived from the multinomial logistic regression model. For all 3 experimental factors, none of the differences in the percentage of respondents exceeded the predefined threshold for policy relevance. This lack of differences was consistent for respondents who had their own doctor and for those who self-reported worse and better overall health.

Table 4 Endorsement of Emotional Reactions After Reading Hypothetical Letter
Full size table

Questions Asked by Respondents

The number of respondents who asked a question was similar between those whose letters included a trial description (57%) and those whose letters did not (52%). Table 5 shows the distribution of question codes by whether the letter included a description of the trial. The distribution of codes was consistent across the 2 groups, with the largest difference not exceeding our predefined threshold of policy relevance of 15 percentage points. The most frequently asked questions in both groups addressed next steps that should be taken, health concerns, and a negative impact on trust or confidence.

Table 5 Questions Asked by the Pragmatic Clinical Trial Description (Yes/No)a
Full size table

Other Reactions

When asked about the main purpose of the letter, the 2 most frequently endorsed answers were “to give you information about your health” (57%) and “to tell you to contact your doctor” (46%). Almost one-third of respondents (31%) endorsed “to protect people at the medical center from legal trouble.”

Most respondents rated the letters as “very clear” (45%) or “somewhat clear” (47%). Similarly, 51% said they understood the information “completely,” and 46% understood “somewhat.” In 7 of 8 cases, inclusion of a description of the pragmatic clinical trial resulted in lower ratings of letter clarity and subjective understanding (data not shown). When describing the amount of information contained in the letter, half of respondents (54%) felt it was the “right amount of information,” whereas 38% felt it was “not enough” and 7% felt it was “too much.”

DISCUSSION

As more pragmatic clinical trials are embedded within health care systems, there is a need to anticipate and respond to PCT-CFs. This study was conducted to provide evidence to inform the most effective strategies for communicating PCT-CFs to patients. The results demonstrate that, while there were subtle influences on responses, people’s initial emotional reactions to a letter reporting collateral findings and their intention to contact their doctor immediately were not affected to a noteworthy degree by the role of the letter’s signatory or whether the pragmatic clinical trial was described in the letter. People’s subjective impressions of the letter’s clarity and of their own level of understanding were generally lower for versions of the letter that included a description of the trial.

Our findings have several implications for crafting effective communications to patients about PCT-CFs. The likelihood that patients would follow the recommendation to contact their doctor was generally consistent across scenarios—around 50–60%. Thus, with respect to patient welfare, we would not expect any impact related to who sends the letter (such as a personal physician rather than a researcher or administrator) or whether the letter describes the pragmatic clinical trial that generated the collateral finding. With respect to patient rights, it could be argued that transparency in how the PCT-CF was identified is important. Yet, our findings are less clear on this point. When a trial description was not included, we did not observe any examples of respondents asking questions that would have been answered by a trial description. Respondents were more focused on next steps and what the collateral finding meant for their health. Similarly, inclusion of a trial description did not appear to increase the number of questions or change the nature of the questions, though it did adversely affect clarity and subjective understanding. More work would be needed to develop clearer communication about pragmatic clinical trials, if a trial description were to be included in such a letter.

Our findings support our prior observation from our qualitative work1 that health care system leaders and clinicians should be prepared for negative reactions from patients who receive information about PCT-CFs. They should also be prepared for questions that focus on the range of issues described in Table 5, including questions about health concerns and next steps. The number of questions the health care system will need to field does not seem to be affected by who sends the letter or whether the pragmatic clinical trial was described. Also relevant for health care systems is the finding that one-third of respondents believed the primary purpose of the letter was to protect the health care institution from liability. These perceptions might stem from a broader mistrust of health care institutions. Future work could examine whether such reactions differ between traditional health care systems and those that are more integrated and structured as learning health systems.

Our study has several strengths. The large, representative sample allowed for statistically precise characterizations of experimental effects. To ensure the experiences of people with lower levels of education were well represented, we explicitly sampled and randomized within 2 levels of education. The letters we used were developed with qualitative input from members of the public and were vetted by health care system administrators for verisimilitude to current practice.

The study also has limitations. First, while responses to hypothetical vignettes can be informative,3,7 it is unknown how well the findings translate to real-world situations. Our expectation is that the relative effects we observed in terms of differences between types of letters are representative of the effects one would observe with real letters sent to real patients. We expect the absolute response levels (e.g., the percentage of people who would contact their doctor immediately) may not generalize as well as the relative effects. Finally, it was challenging to capture and report the initial emotional reactions of the participants. Emotional reactions are often complex and dynamic. For example, a person might report feeling concerned and then explain their feeling in terms of both being worried about a potential health problem and being angry about their inclusion in a study without their awareness. Our strategy for querying and analyzing these reactions was informed by our prior qualitative work with the public,1,6 but it is likely that significant aspects of these experiences are not adequately captured in web-based surveys.

Conclusion

In hypothetical scenarios in which patients were informed of collateral findings from a pragmatic clinical trial, the role of the letter’s signatory and the type of collateral finding did not substantially affect people’s intentions to take the recommended action of contacting their doctor immediately. Deciding whether to include a description of the pragmatic clinical trial that led to the collateral finding requires a trade-off between transparency and greater difficulty understanding the contents of the letter. Health care systems should prepare for negative reactions and questions from patients in response to such letters. Some questions might be avoided by developing better ways to describe pragmatic clinical trials in communications to patients. Future work should evaluate patients’ reactions to disclosures of PCT-CFs in real-world settings.