Research priorities for the pan-Canadian Oncology Symptom Triage and Remote Support practice guides: a modified nominal group consensus.

Introduction
The pan-Canadian Oncology Symptom Triage and Remote Support (costars) team is studying how to improve the quality and consistency of cancer symptom management.


Methods
A 1-day invitational meeting was held 24 October 2017 in Ottawa, Ontario, to review the current evidence from costars projects and to establish research priorities for a future largescale implementation study. The meeting included 36 participants who were clinicians from adult oncology, pediatric oncology, and homecare; policymakers from national, provincial, and regional organizations; researchers; and a patient. Half the day involved summarizing evidence from four costars studies and experiences with implementing the costars symptom practice guides. The second half of the day used a modified nominal group technique to generate research questions within small groups, presentation of research questions to all participants, and two rounds of voting to reach consensus on research priorities.


Results
Participants proposed 4 research categories:■ User-centred augmentation to enhance usability (for example, designing a mobile costars solution)■ Outcome measurement (for example, determining key competencies for clinicians)■ Regular renewal of costars to keep pace with evolving evidence (for example, updates for novel therapies)■ Integration into clinical practice (for example, meaningful engagement of patients and caregivers in study design).


Conclusions
Across categories, the top 3 priorities were effect on health services use, competency development, and a mobile costars solution. Future research will address identified priorities, reflecting the needs and perspectives of diverse stakeholders. Stakeholder collaboration will continue to guide our approach to operationalizing this priority research agenda.


INTRODUCTION
Prompt and effective symptom management is a cornerstone of cancer care and results in decreased symptom severity, improved quality of life, and less health services use 1,2 . Undermanaged symptoms represent an important safety issue for patients, because symptoms can otherwise become life-threatening. With a large percentage of cancer therapy being delivered on an outpatient basis, symptoms often occur when patients are outside of hospital 3 . For those patients, the telephone is the main and quickest route for access to cancer services 4,5 . Thus, an important supportive service is remote telephone or e-mail access to clinicians who can guide patient self-care and triage symptom severity to ensure that the appropriate level of care is received. To assist that service with standardized, evidence-based symptom recommendations, the pan-Canadian Oncology Symptom Triage and Remote Support (costars) team developed 15 evidence-informed practice guides for cancer treatment-related symptom management.
A phased, systematic, and integrated knowledge translation (kt) approach was used to develop the costars symptom management practice guides 6 . With funding from the Canadian Partnership Against Cancer, the costars research team and stakeholders developed the practice guides after a systematic review and use of the can-implement method 7 for the adaptation of existing clinical practice guidelines 8,9 . The costars practice guides, which are available in English and French, were updated in 2013 and 2016. The topics covered in the set of practice guides include anxiety, appetite loss, bleeding, breathlessness, constipation, depression, diarrhea, fatigue, febrile neutropenia, mouth sores, nausea and vomiting, pain, peripheral neuropathy, skin reaction from radiation, and sleep problems. The guides instruct nurses to assess symptom severity; triage patients based on greatest symptom severity; for non-severe symptoms, review medications being used for the symptom and review selfcare strategies; and document the plan agreed upon with the patient. A costars steering committee with members from 8 of 10 Canadian provinces assumes responsibility for evidence-based updates to the guides.
To facilitate costars implementation into the clinical setting, a training program for nurses was developed and evaluated [10][11][12] , and a number of survey tools and instruments were validated to evaluate the integration of the guides into clinical practice [13][14][15] . The Canadian Association of Nurses in Oncology has endorsed the practice guides and facilitated their dissemination online (http://www.cano-acio.ca/ triage-remote-protocols). Although the costars practice guides were implemented as part of a study in Halifax, Montreal, and Sudbury, they are also in various stages of implementation at several other Canadian oncology programs, including The Ottawa Hospital a,b , Sunnybrook Health Sciences Centre, and the Bayshore HealthCare out-of-hours telehealth program. Research studies have also indicated the importance of using costars in face-to-face encounters to ensure consistency in symptom management 14,c . More recently, the guides were adapted for use by radiation therapists to support an interprofessional team approach to symptom management 16 Given the strong evidence on which costars practice guides are based and findings from the implementation studies, there is an opportunity for scaling up implementation to broadly improve cancer symptom management and to optimize Canadian health services use. To begin developing a research program related to the costars guides and their widespread implementation, it is first necessary to consider key research priorities in those areas. The overall aim of the present study was to engage a broad range of stakeholders in identifying key areas for future costars research and to reach consensus on research priorities.

METHODS Design
A modified nominal group technique (ngt) was used to develop consensus about costars research priorities at a 1-day invitational meeting. The ngt method uses quantitative and qualitative data collection methods to pool judgments from a group of subject matter experts 19 . For the purposes of the present study, the ngt consisted of a highly structured face-to-face meeting with two facilitated rounds of voting on the perceived importance of costars research questions by consensus conference participants.

Participants and Setting
Using peer knowledge and expert consultation, key stakeholder groups were identified. Specifically, a core group of authors (DS, TT, LJ) with expertise in the field of cancer symptom management developed a list of potential meeting invitees based on research, education, and clinical activities. The stakeholder invitation list included researchers and knowledge users from Western Canada, Ontario, Quebec, and Eastern Canada to ensure pan-Canadian representation. This heterogeneous group of invitees varied in terms of cancer care experiences, profession, training, and setting. More specifically, representation aimed to include researchers who had evaluated use of the costars practice guides; health care professionals working with patients having symptoms from cancer treatments; patients who had experienced cancer symptoms and their family caregivers; health policymakers or senior leaders with influence on cancer symptom management; and educators involved with the training of health care professionals.

Procedure
One week before the meeting, each consensus conference participant was sent a schedule of events detailing planned meeting topics and objectives to review. This pre-workshop preparation allowed participants to begin considering priority research topics based on their personal and professional knowledge and experience.
The meeting started with introductions and an icebreaker exercise. The exercise was used to establish a rapport between participants that would lead to them feeling more comfortable presenting their ideas and feedback within the group.
Objectives of the first half of the day were to disseminate and discuss empirical and experiential evidence about implementing costars practice guides in ambulatory oncology programs and homecare agencies, and to identify gaps in the evidence relevant to implementation of the costars practice guides. To address those objectives, the agenda consisted of presentations by subject-matter experts (DS, TT, LJ, AD) and a personal experience presentation from a patient (CL). After the presentations, large-group discussions focused on the current costars evidence-base. Next, an expert panel (EG, IDG, CL, Kednapa Thavorn) commented on the clinical value of the practice guides, kt, economic evaluation, and administrative implications for future research.
The modified ngt procedure was conducted during the second half of the day. In the consensus meeting, participants proposed research category groups for subsequent consideration of research priorities for the costars practice guides. Each participant then joined a self-selected research category group. The patient representative rotated through the various groups. Categories were orally suggested by participants to the group at large based on their perception of areas with research and clinical gaps in cancer symptom management. No limitations were imposed on the number or type of categories that could be suggested. Each group was given 90 minutes to develop an unlimited list of research questions related to their associated category group. Questions and categories were displayed on a large poster board, such that all participants could see each question. A poster board that participants from all category groups could use to suggest research questions on any topic was also available during the process. Category group members selected a spokesperson to present their research questions to the group at large, and participants who added research questions to the poster board were invited to present their questions with a rationale for considering the question to be a priority.
Once all research questions emerging from all category groups had been presented, participants were given 15 minutes to consider the research questions and to endorse those that they identified as research priorities. Each participant was given 3 red stickers and was asked to endorse questions as priorities by placing one or more stickers next to the question written on the poster boards. A single participant could place multiple stickers on 1 question or could place their stickers on 3 different research questions. A facilitator of the ngt (LAJ) then calculated the number of endorsement stickers for each question and presented the results to all participants. After a brief large-group discussion about the priorities, participants were given another opportunity to highlight the reasons they felt that a specific research question should be considered a priority. In a second round of voting, participants then engaged in the same process with 3 blue stickers.
At the end of the day, participants completed an evaluation of the meeting. Evaluation items included 5 multiple choice questions rating whether the meeting objectives were met (response choices: Yes, No, Unsure), the comprehensiveness of information presented (response choices: Too much info, Just right, Not enough info), acceptability of time for discussion (response choices: Too much time, Just right, Not enough time), and overall satisfaction with the meeting (response choices: Excellent, Good, Fair, Poor). Participants were also asked in 3 open-ended questions to identify meeting "likes" and "dislikes" and to suggest improvements to the experience.

Analysis
After both rounds of voting, the total number of stickers assigned to each question was counted, and questions were ranked according to the endorsement received. Sticker count was based on the percentage of the total possible sticker-based endorsement. Descriptions for each research category were summarized using comments from the poster boards and detailed field notes recorded by 2 investigators (MC, LAJ). Descriptive statistics were used to summarize data from participant evaluation questionnaires. Two investigators (MC, LAJ) reviewed the qualitative data from the open-ended questions and organized them into themes that reflected participant likes and dislikes with the meeting.

Participant and Meeting Details
The meeting was held 24 October 2017 in Ottawa, Ontario. The 36 meeting participants were clinicians (for example, nurses, radiation therapists) and managers from tertiary cancer treatment centres or homecare; policymakers from regional, provincial, or federal cancer care organizations; career researchers; and a patient (  Although participants could have qualified for more than one characteristic in each group, the characteristic they self-selected was used. presentations conducted during the first half of the meeting. The ngt involved 30 participants (scheduling conflicts prohibited the involvement of 6 participants in the second half of the day). During the voting process, participants distributed 173 of a possible 180 stickers.

Research Priority Categories
Initially, 6 broad priority categories were identified by the participants: ■ costars gaps (for example, immunotherapy) ■ "Cancer symptom journey" ■ Radiation therapy ■ Outcomes and outcome measures ■ Homecare ■ Miscellaneous Participants then collapsed those 6 broad categories for research into the 4 categories presented in the subsections that follow and submitted a list of research questions within each category. Specifically, the "homecare" and "radiation" categories were merged with "cancer symptom journey." Table iii shows categories with questions that received at least 1 vote of endorsement (described in the relevant subsection). Table iv shows research questions that were developed within each category group, but that did not receive at least 1 vote of endorsement at the groupat-large level.

User-Centred Augmentation to Enhance Usability
Although the guides were highly regarded by all participants, changes to improve usability were discussed. First, participants identified the need to develop and evaluate a mobile costars solution-that is, an mHealth app. The app would contain all the symptom assessment questions, triaging, and symptom management strategies included in the costars practice guides. The app would also interface with hospital electronic health record systems to allow for seamless upload of data to patient medical charts. The development of both clinician and patient versions of the app would be a priority. The clinician version was suggested to support costars use by practitioners in all cancer care settings (for example, busy hospital clinics, patient homes). The patient version would support real-time symptom self-management by patients in their natural environments (home, work, school, and so on).
Second, research related to augmenting the "look and feel" (that is, the design) of the paper-version costars practice guides was prioritized. Suggestions to improve usability (including acceptability) so as to promote clinician use of the costars guides were to create engaging and attractive graphics and to apply human factors design principles to refine the layout of the text.
Third, research is needed to determine which elements should be added, removed, or adjusted within the guides to make them applicable to all sites where oncology care is provided (for example, homecare, primary care, longterm care, hospices, telehealth). In particular, it was recommended that the costars guides be adapted to ensure their utility to clinicians with different scopes of practice providing care within those environments (for example, nurse practitioners, physicians, medical radiation therapists, registered practical nurses).
Fourth, participants identified the need for longitudinal research to measure the effect of proactively using the costars practice guides to support symptom management throughout the disease trajectory. In contrast to the predominantly patient-driven costars model that is currently in effect (for example, patient-initiated telephone call, patient-reported Edmonton Symptom Assessment System score ≥ 4 at clinic visit), participants suggested an evaluation of the effect of clinician-driven symptom management. In that case, clinicians would contact patients at specified intervals to inquire about cancer symptoms and to provide management support as needed. That process was recommended to be longitudinal, with patients being contacted regularly by clinicians from diagnosis until cure or end of life. Participants further discussed using those regular symptom assessment contacts as an opportunity to support and encourage adherence by patients receiving oral chemotherapy.
Finally, participants prioritized patient-centred investigations into how the language used in the costars  practice guides could be modified to improve cultural sensitivity. To that end, participants recommended recruiting a broad and diverse group of oncology patients to determine culturally appropriate changes to the practice guides.

Outcome Measurement
First, participants felt that using economic evaluation methods to assess health services use (for example, use of ambulatory and emergency services) after costarsinformed health care interactions was important. Within the scope of health services use, participants recommended evaluating the patient experience with such use as it pertains to costars-informed care. For example, do patients have a greater sense of self-efficacy or an improved capacity to communicate with clinicians after costars telephone-based symptom management support? Second, developing and assessing key clinician competencies related to the costars practice guides was a research priority. Participants considered that the establishment of comprehensive, standardized competencies to measure in clinicians providing symptom assessment, triage, and management is essential to the effectiveness of costars. The development of a framework to assess those clinician competencies from the vantage points of trainers, clinicians, and patients was identified.
Third, assessing the acceptability, use, and perceived effectiveness of costars symptom management advice by patients was a priority. That is, do patients use, like, and find effective the advice they are given by nurses during telephone-based symptom support interactions?
Fourth, the use of data obtained from costarsinformed contact with clinicians to evaluate patient symptom complexity over time was recommended. Specifically, participants suggested longitudinal evaluation of symptom occurrence, symptom severity, symptom clustering, and the predictive, mediating, and moderating variables associated with symptom status in large, diverse groups of cancer patients. Finally, participants recommended investigating the effect of guideline use on clinician judgment and autonomous decision-making.

Regular Renewal of COSTaRS to Keep Pace with Evolving Evidence
First, given the rapidly changing face of oncologic care and the advent of novel treatments such as immunotherapy, participants recommended research into renewing the costars guides to improve their applicability to new treatments. Although the guides have already been updated twice, participants indicated that a sustainable plan for the regular renewal of the guides would help to ensure that symptom assessment and management advice keep pace with emerging evidence used in clinical practice (for example, symptom management for immunotherapy). Moreover, participants suggested that research into regular guide renewal be interdisciplinary and pan-Canadian in nature to improve the generalizability of the practice guides.

Integration of COSTaRS into Clinical Practice
A final priority category focused on research needed to support the integration of costars practice guide-based symptom care into clinical practice. Within that category, meeting participants first recommended research efforts that would evaluate how integrated kt principles could be used to meaningfully engage patients and caregivers in the design, development, evaluation, and implementation of symptom management practice guides. Second, to promote awareness and uptake in a broad group of stakeholders (for example, patients, caregivers, clinicians, administrators, policymakers), developing optimal strategies to disseminate information about symptom management practice guides was recommended. Third, to ensure seamless symptom assessment and management and to decrease duplication of data collection, participants prioritized determining how best to merge the various guideline tools used in practice (for example, costars, distress screens, toxicity assessments).

Meeting Evaluation
Overall, participants rated the meeting as excellent (n = 25, 83%) or good (n = 5, 17%). All participants (n = 30, 100%) agreed that the meeting objectives to disseminate and discuss empirical and experiential costars evidence (first half of meeting) and establish research priorities for the costars practice guides (second half of meeting) were met. The information presented to participants was considered comprehensive (n = 29, 97%) and novel (n = 30, 100%). Qualitative comments on meeting "likes" included "[the patient's] opening was powerful. Her story (or similar) would be a great addition to online or face-to-face [costars] education, grounding us in person-centred work" and "high calibre participants representing diverse stakeholder groups with excellent thoughts and perspectives on costars and cancer symptom management." Participants planned to take action after the meeting by "getting costars integrated into practice in our area" and "emphasizing importance to include patients as stakeholders in discussion, project plans, evaluation [at my hospital]." No "dislikes" were reported by participants.

DISCUSSION
Our consensus-building exercise was successful in developing priority research questions related to the costars practice guides that could be used to inform an ongoing program of costars research, including a scaling-up study. Consensus meeting participants were highly engaged in the process; represented a diverse stakeholder group in terms of profession or role, practice setting, and geographic base; and rated the meeting overwhelmingly positively. These were the overarching research priority categories identified by meeting participants: ■ User-centred augmentation to enhance usability ■ Outcome measurement ■ Regular renewal of costars to keep pace with evolving evidence ■ Integration of costars into clinical practice Within those categories, research questions broadly included ■ how can the utility and effectiveness of the costars practice guides be improved? ■ how best to assess the effect of costars on patient, clinician, and health system outcomes? ■ how should symptom management guidelines in general be developed, evaluated, and implemented into practice?
Taken together, those research categories and questions can be considered to be a research agenda for costars and relevant to other clinical care practice guidelines within the field of oncology.
Although there was consensus on the part of meeting participants that the costars guides were acceptable and clinically useful, recommendations to enhance usability were made. Specifically, participants recommended developing and evaluating an electronic platform for the deployment of costars. Electronic platforms-and, in particular, smartphone and tablet apps-are now regarded as useful media to improve the accessibility, acceptability, and potentially, the effectiveness of health interventions [23][24][25] . The development of a costars smartphone app might facilitate practice guide use in a variety of health care settings (for example, hospital, primary care, home). Further, if a future costars app had the capacity to interface with patient electronic health records, it could enable the collection of comprehensive and high-quality patient-oriented data that longitudinally track patient symptoms and responses. Previous research into improving access to oncology clinical guidelines has indicated that a mobile app could be a viable solution for interdisciplinary clinicians 26 . Consistent with recommendations for mHealth innovation development 27 , a future costars app should be developed using an iterative and user-centred design framework to help ensure applicability and "use-as-intended" for clinicians. User-centred design is consistent with an integrated kt approach by which relevant end users are involved in intervention design, evaluation, and implementation 6 . Involving end users is a strong predictor that an intervention will be used 6,28 . Hence, the user-centred method for app development should also be applied to the recommendation by participants to augment the design and cultural sensitivity of the paper-based costars guides.
The expanding role of settings other than ambulatory oncology programs for the provision of cancer care is increasingly being recognized 29,30 , and changes to guide content to fit those many diverse sites were also prioritized. The costars practice guides have great potential to support symptom management within alternative settings such as homecare, particularly if the guides are adapted to meet the needs and scope of practice within those settings. For example, cancer symptom management is primarily provided by registered nurses in cancer programs; in homecare, services to oncology clients are often provided by registered practical nurses 20,21 .
Changes whereby the guides would be proactively used by clinicians to assess and manage symptoms throughout the cancer care trajectory might help patients receive comprehensive symptom management. Such a model has previously been evaluated in cancer patients after chemotherapy administration, but compared with standard care, it did not result in significant improvements in the number of reported cancer symptoms, symptom distress, or satisfaction with care 31 . Those findings suggest that careful examination of the effects of the model, including effects on patient self-efficacy, time from symptom occurrence to management, and health care resource utilization, is necessary. Another important target population for outreach by clinicians is patients receiving oral chemotherapy, for whom concerns about adherence exist. A recent systematic review indicated that telephone outreach by nurses or pharmacists to patients receiving oral chemotherapy might reduce the severity of toxicities and improve adherence, making a costars research avenue in that area important to explore 32 . Investigations into the acceptability and feasibility (for example, cost, effect on workflow) of clinician-driven symptom assessment from the viewpoints of the various stakeholders involved must be conducted.
Research into the effects of the costars practice guides from the viewpoint of health services use, including economic and patient experience evaluations, was the highest endorsed priority. Our team is currently studying the effects of costars-guided symptom triage and management on emergency department use by patients. That work will begin to build an evidence base about the implications of practice-guide-led symptom management on cancer care service-level outcomes.
Participants also strongly endorsed research focused on costars outcome measurement and development of a set of key competencies for clinicians who will use the guides. The costars training workshops for nurses have already been used to support implementation of the guides into clinical settings 14 and have been associated with improved confidence in use of the guides by nurses 10 . Building a means to assess the competency of clinicians using the guides will ensure optimally effective guide use. Other identified priorities related to costars practice guide outcome assessment included evaluation of the acceptability, use, and perceived effectiveness of the practice guide symptom management advice and evaluation of symptom complexity over time using data from the practice guides. Answers to those research questions are needed to determine the "active ingredients" in the costars care model and to augment those ingredients as necessary 33 .
With respect to the effects of guideline use on clinician judgment and autonomous decision-making, a recent critical literature review revealed that, if a clinical decisionsupport guideline is considered to limit autonomy, physicians might not use it or might use it inappropriately 34 . That finding was echoed in previous work examining nurse-perceived barriers associated with costars: a perception that the practice guides would overshadow their clinical experience and nursing judgment. Evaluation of clinician perceptions related to loss of autonomy and education supporting the use of guidelines from clinical effectiveness and feasibility standpoints are needed to overcome that barrier.
The continuing emergence of novel therapeutic agents (for example, immunotherapy) was the impetus behind prioritizing research focused on a process of regularly renewing the guides. The costars team has already updated the practice guides twice, with the most recent version including 2 new symptom practice guides 9 . However, the changes did not include evidence for symptom management specific to immunotherapy or oral chemotherapy. Ever-changing advances in cancer therapies aim to improve survival and quality of life, but can be associated with toxicities requiring new management strategies and modalities 35 . A sustainable process to regularly renew costars is a patient safety priority and will help to make the practice guides applicable to the effective management of symptoms related to new treatments.
The meaningful inclusion of patients and caregivers in practice guide development and evaluation was a highly endorsed priority. That model has been integral to costars development to date and will continue to underpin future research efforts 36,37 . Previous research indicates that meaningful involvement of patients, caregivers, and clinicians is feasible and results in increased study enrolment rates, outcomes that are more relevant, and greater uptake of research results 6,38 . Investigations into how best to merge the various symptom assessment tools used in practice will ensure comprehensive assessment and decreased patient, clinician, and system burden. That outcome is important, because perceived clinician burden is a barrier to cancer symptom assessment 39 , and streamlining the process might improve adherence to routine assessment to inform better symptom management.
The idea-generation method used during the meeting resulted in the creation of a number of research questions, some of which received no priority votes. Those ideas might not have been prioritized because of the interests and insights of the particular body of meeting participants, but they might still represent viable suggestions for future costars research.
Limitations related to this priority-setting exercise should be noted. First, only 30 participants were involved in the priority-setting component at the 1-day invitational meeting, primarily because of limited finances available to host an in-person meeting. Although we endeavoured to recruit a heterogeneous group of participants, we might not have identified all costars priority research questions that would have emerged had a larger group of participants attended the meeting. Additionally, many of the research questions proposed by participants were broad in nature and could be difficult for researchers to interpret. To address that limitation, we will use an integrated kt approach during all stages of costars research to ensure grounding in the needs and opinions of relevant stakeholders.

CONCLUSIONS
This study successfully established future research priorities for the costars practice guides from the vantage point of a diverse group of stakeholders. Key findings included the need to enhance guide usability, to appraise costarsrelated health and service outcomes, to continually renew guide content, and general recommendations to integrate costars practice guides into clinical practice. Research addressing some of the identified priority questions is already under way (effect on emergency department use, for instance). Future endeavors will use the outputs of the consensus meeting to inform the design of a large-scale study to refine and broadly implement the guides in diverse practice settings. Continued stakeholder collaboration will guide the research approaches and the implementation of this priority agenda.