The use of virtual nominal groups in healthcare research: An extended scoping review

Introduction The Nominal Group Technique (NGT) is a consensus group method used to synthesize expert opinions. Given the global shift to virtual meetings, the extent to which researchers leveraged virtual platforms is unclear. This scoping review explores the use of the vNGT in healthcare research during the COVID-19 pandemic. Methods Following the Arksey and O’Malley’s framework, eight cross-disciplinary databases were searched (January 2020-July 2022). Research articles that reported all four vNGT stages (idea generation, round robin sharing, clarification, voting) were included. Media Synchronicity Theory informed analysis. Corresponding authors were surveyed for additional information. Results Of 2,589 citations, 32 references were included. Articles covered healthcare (27/32) and healthcare education (4/32). Platforms used most were Zoom, MS Teams and GoTo but was not reported in 44% of studies. Only 22% commented on the benefits/challenges of moving the NGT virtually. Among authors who responded to our survey (16/32), 80% felt that the vNGT was comparable or superior. Conclusions The vNGT provides several advantages such as the inclusion of geographically dispersed participants, scheduling flexibility and cost savings. It is a promising alternative to the traditional in-person meeting, but researchers should carefully describe modifications, potential limitations, and impact on results.


Introduction
Consensus group methods are often used to synthesize expert opinions when evidence is lacking or contradictory.They have been increasingly utilized in healthcare, business, engineering, and education [1][2][3][4].
The NGT was developed for idea generation and group decision-making [3].The structured format allows for effective generation and prioritization of ideas.The NGT has several key steps: presentation of the nominal question, silent generation of ideas in writing, roundrobin sharing of ideas from participants, group discussion and clarification of each idea, followed by anonymous individual voting to rank priority areas.Feedback of results followed by more discussion and re-rating may occur [13,14].
The NGT differs from other consensus techniques by employing an in-person meeting among 5-12 participants [15].This is touted by many to be a strength, as it not only allows discordant ideas on topics of mutual interest to be freely expressed and synthesized, but it also affords the opportunity to explore any differences in opinions.Further, the collaborative feature of the NGT may increase ownership of research among stakeholders and enhance the potential of informing policy or practice [3].However, the small number of participants is also regarded as a limitation, and the potential for dominant members to unduly influence group decision-making cannot be dismissed, even if NGT aims to counteract this effect in its structured procedure.
The coronavirus disease of 2019 (COVID-19) pandemic has fundamentally changed the way we work, learn, and conduct research [16] and NGT has not been exempt from the shift; many researchers transitioned the in-person NGT meeting to a virtual environment [17][18][19][20].For instance, Nelson et al. (2022) employed 3 nominal groups to identify burnout strategies in resident physicians, one being the traditional in-person meeting followed by two of which were held virtually via Zoom [17].Timmermans et al. (2022) conducted three virtual NGTs (vNGT) and provided recommendations on the transition to the synchronous, online environment, but their claims were drawn solely from the authors' subjective experiences [20].
There remains ambiguity regarding the extent to which other researchers have shifted the NGT to a virtual format, the types of platforms used, and modifications made to the technique.In addition to this exploratory inquisition, it is also important to consider whether researchers voiced any challenges, or perceived advantages or disadvantages.Taken collectively, these lessons learned would provide guidance for future users of vNGT.
The COVID-19 pandemic has supported an upsurge in information and communication technologies, which fundamentally changed how individuals interact.The Media Synchronicity Theory (MST) is a conceptual framework that considers the effectiveness of information and communication technologies in facilitating group work [21].Through the lens of MST, all communication activities are grouped into two simple processes: conveyance (transmission of new information to the receiver that enables the creation and revision of individual understanding of a problem) and convergence (mutual process that governs how individuals understand and negotiate a common ground for a problem), the latter of which may involve more authentic, rapid back and forth transmission of information.For the NGT, the idea generation phase would presumably require more conveyance, while the discussion and clarification phase emphasize more convergence.Both processes are necessary, but the proportion depends on the complexity and inherent characteristics of the research.
In summary, the NGT has been used in a variety of research settings, to inform important decisions.Although interaction between panel members is fundamental to the process, many in-person meetings have moved online due to the pandemic.At present, it is unclear to what extent the NGT has been undertaken virtually, and what implications this may have for the decision process.

Objective
The overarching purpose of this study is to explore the use of the virtual Nominal Group Technique (vNGT) in healthcare research.Specific objectives are to answer the following questions: 1) To what extent has the NGT been used virtually?2) What virtual communication platforms are used?3) What modifications to the technique were made to accommodate this online format?And 4) What advantages and disadvantages were noted by authors?

Methods
Following a study protocol that has been published [22], a scoping review was conducted.A scoping review was considered appropriate since this topic is poorly defined, and the purpose was to map the literature, find key concepts, types and sources of evidence and identify gaps in the literature.The study followed the Arksey and O'Malley framework [23] and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Review [24].Written ethics approval was granted from the University of Ottawa Research Ethics Board on September 26, 2022.All participants answering the survey provided written informed consent.

Step 1: Identifying the research question
Our study began with a broad objective: to explore the use of the vNGT in healthcare research.This study included all English-language published research in healthcare and healthcare education that used the NGT in a virtual format.This included using any non-in-person format, such as videoconferencing.The outcomes of interest included the author's description of how the vNGT was used, and the perceived success of the process, benefits, risks, and challenges.Informed by a preliminary search of the literature, we decided upon the following questions:

Step 2 & 3: Identifying relevant articles and article selection
We started with the following framework: Population, Concept, Context.The population included any published research studies using the nominal group technique, the concept entailed the use of virtual modalities to execute the nominal group technique, and context involved any study topic.
Several pilot searches were undertaken to define the search strategy.Details can be found in the published protocol [22].Final inclusion and exclusion criteria are noted in Table 1.
The final search strategies were executed by an information specialist (KF) and peer reviewed using the PRESS guideline [25].The searches were conducted July 15th, 2022, in: MEDLINE(R) ALL (OvidSP), Embase (OvidSP), CINAHL (EBSCOHost), ERIC (OvidSP), Education Source (EBSCOHost), APA PsycInfo (OvidSP), Web of Science, and Scopus to retrieve references published January 2020 to July 2022.This time period was selected to detect studies conducted during the COVID-19 pandemic to reflect the more rapid shift to virtual formats.No search filters, or language limits were used, but conference abstracts were removed since only full papers were of interest.The search strategies are included in S1 File.
Using the Covidence software [26], all titles and abstracts were reviewed in duplicate with varying pair combinations by three co-authors (SHL, SHM, MG and a research assistant).Those meeting inclusion criteria, or if not clearly conducted in person were pulled for full text review.Duplicate review of full text articles against inclusion/exclusion criteria was completed.All conflicts were resolved via consensus discussion with a third member of the team.

Step 4: Charting the data
Through an iterative process the data extraction form was developed.In addition to publication-level information and demographics, concepts related to the MST were explored.Details can be found in the published protocol [22].
Following best practices from the manual for evidence synthesis [27] at least two members (SL, SHM) independently reviewed 20% of articles applying the final data extraction form.Thereafter, one member of the research team carried out data extraction with verification from a second member.Any ambiguous items that arose were resolved through discussion with the senior author (SHM).
Step 5: Collating, summarizing, and reporting the results Both quantitative and thematic analyses were used to synthesize study results.Quantitative analysis focused on the nature (e.g., education, clinical research, guideline development) and distribution of relevant articles.Two members of the research team (SL, SHM) independently reviewed the data to identify preliminary themes as informed by the MST.Several group meetings with all team members were held to review the data and to agree on a final summary of findings. Step

6: Survey of the authors
Upon review of several studies, it became apparent that many of the articles did not comment on the "virtual" aspect of the NGT.As a result, we disseminated an online survey to the corresponding authors for articles included in the study.The survey sought to confirm which virtual platform was used and for which steps of the NGT, if additional functions were used (e.g., chats features) or modifications made to the NGT to accommodate the virtual platform, why the virtual platform was used, their general impressions and perceived benefits and challenges, comparing their experience with in-person NGTs, and any lessons learned.The survey is available in S2 File.The Survey.The Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist can be found in S3 File.

Results
A total of 2,589 records were identified through database searching, of which 1,754 were removed as duplicates (Fig 1).Excluding irrelevant titles and abstracts, 598 full-text articles were assessed for eligibility based on our exclusion criteria.Resulting in 32 full-text articles meeting inclusion criteria.
Research questions were more related to healthcare (87.5%; 28/32) than to healthcare education (12.5%; 4/32).Within healthcare, topics were varied and included items such as improving care for inmate dementia [30], home rehabilitation for stroke survivors [49], COVID-19 vaccine rollout in pharmacies [51] barriers to testing lipids and achieving disease control in rheumatoid arthritis [56,57].Patients were often central to the vNGT with topics including patient views on the treatment of osteoporosis [28], multiple myeloma [50] or osteoarthritis [43] and patient reported outcomes in heart failure [33].

vNGT participants
Five studies did not report the number of participants (15.6%) [6,38,41,46,47].For those that did, the total number of participants per NGT group ranged from 2-20, however it was not always clear how many participants were in each NGT group as some studies ran several NGT groups in parallel (Table 2).

vNGT modifications for virtual
Modifications to accommodate the technique virtually were often implied to have taken place, but rarely commented upon (25%; 8/32).Notably, the online break-out room functionality replaced physical meeting rooms to conduct multiple nominal group sessions [51].Investigators also implemented online sheets [51], interactive whiteboards, and slideshows [39] to complement the process.Asynchronous communication modalities, defined as non-interactive, non-simultaneous platforms such as email threads and e-survey, were noted in a quarter of the articles (25%; 8/ 32).The idea generation step was conducted asynchronously in 8 of 32 articles (25%).At the discussion and clarification stage, Bavelaar et al., 2022 [29] and Hoops et al., 2022 [31] offered asynchronous participation in conjunction with the interactive discussion in the form of email threads.
Five studies held in-person and virtual sessions in parallel, two of which allowed for both in-person and virtual within the same meeting based on context-dependent needs (e.g., geographic difference, member preference, technical challenges) [29,31].

Documentation
In many instances, the discussion phase of the NGT was formally documented either via audio or video recording (41%; 13/32) or transcription in the form of written notes during the meeting (69%; 22/32).Some opted to using both recording and live transcription (34%; 11/32).

Other findings
Our inclusion criteria required that all stages of the vNGT be virtual (i.e., no in-person components).However, during the screening we did note that 52 articles incorporated at least one virtual component.

Descriptive characteristics
Benefits and challenges of vNGT.Only eight articles (25%;8/32) provided comments on the benefits or challenges (see Table 4).Online participation afforded geographical and scheduling flexibility, allowing a more representative sample of participants to convene in nominal group sessions [29].One study suggested that the online nature of discussions may also have promoted the sharing of ideas among participants [51].Janssens et al., (2021) suggested for older participants that the telephone was preferable [50].The addition of asynchronous platforms was advantageous for one study as it provided more time for researchers to procure stakeholder perspectives, participants to understand study material through repeated review, and to efficiently prioritize and rank ideas.
In contrast, some comments directly addressed the challenges of adapting onsite activities to videoconferencing, email threads, and phone calls, which could have affected participant engagement.Two articles suggested that virtual modalities could have acted as a barrier to participation to some individuals, such as older participants [50,57].Prospective participants may

Benefits Challenges
Al-Yateem, 2021 [44] • asynchronous online discussions allowed time to gather different perspectives and participants to fully understand the process by reviewing the material as many times as they wished • online survey platform was a powerful tool to quickly prioritize the skills proposed by each participant.
• required extra effort from participants to support multiple meetings, asynchronous activities, and surveys • results may have been compromised to an extent by directing participants to a priori information online Bavelaar, 2021 [29] None reported • differences in how and when group discussions were conducted may have influenced the results (pre-pandemic in some countries vs pandemic in others) • adapting the group discussion from an onsite activity to a thread of emails, phone calls, or videoconferencing could have impacted the engagement process Beaudart, 2022 [28] • afforded geographical flexibility • webcam allowed for nonverbal communication • no associated travel cost or facility rental fees • safer than face-to-face meetings during waves of the pandemic • limited generalizability to lower socioeconomic participants due to access to e-devices • technical challenges prevailed despite premeeting checks • inherent restricted sample size in online meetings precluded specific subgroup analysis (i.e., lack of power) Fisher, 2021 [49] • allowed geographical distant members to be included • the use of parallel groups also helped to provide insight into replicability of results

None reported
Janssens, 2021 [50] • telephone interviews allowed for more flexibility in choosing • various dates for participation especially for older patients over online meetings • participants who were not comfortable with online discussions or telephone (e.g., older participants) were less likely to participate, hence had to be given the choice to attend in-person Michel, 2021 [51] • the completion and collation of the prepared excel sheets worked well • the online nature of discussions enhanced sharing of ideas participants • additional time for facilitators to compare and combine Excel sheets was not feasible due to differing time zones which led to perceived time constraints (only brief periods of time were possible for collation, clarification, and removal of duplicates) Owensby, 2020 [57] • online participation offers scheduling flexibility, limits transportation barriers, and allows for a potentially more representative sample to provide feedback • online participation may have acted as a barrier to participation Singh, 2022 [39] None reported • research studies using virtual NGT method are fewer, and more evidence is needed to be confident that this is analogous to in-person NGT *studies not listed did not comment on benefits or challenges of vNGT https://doi.org/10.1371/journal.pone.0302437.t004 have declined invitations to the study due to the extra effort required to partake in online meetings and asynchronous activities [44].Furthermore, differences in geographical time zones posed time constraints for participants, which could have influenced group discussions, and therefore the results of the study [51].One study mentioned that results could have been further compromised via asynchronous modalities as participants would be directed to a priori information online [44].Participant feedback.One article openly gathered feedback from participants regarding the virtual nature of the NGT [44].Participants expressed that the asynchronous format provided ample time to prepare ideas before of convening online.
Corresponding author survey results.The survey was completed by half of corresponding authors (50%; 16/32).Authors reported having been involved in in-person sessions 0 to 4 times in the past, and virtual sessions from 1 to 10 times.Participants were also asked their general impression of how the virtual worked compared to in-person NGT.Overall, 44% (7/ 16) felt that the vNGT was better, 36% (6/16) felt that both media performed similarly, and

Benefits
• Individualization of ideas due to lack of discussion among participants • Technological restrictions (i.e., having to unmute) allow for structured sharing of ideas and discussion • Better geographical representation, diversity, and accessibility of expert participants • Reduced effort on logistical planning (i.e., money, time, and accommodation) • Better engagement of quieter voices allowing for uniform contribution • Ease of moderation and participation • Platform capable of recording and transcription • Higher attendance • Enriched visualization of ideas

Concerns
• IChallenge for moderators and participants pre-pandemic as virtual platforms were not commonplace (i.e., mixmatch of computers, phones, in-person) • Technological limitations frequently present (i.e., Health IT access denial, positioning of devices to see physical flipchart in room) • Akin to face-to-face in terms of not being able to reach consensus on the first NGT meeting • Difficult to foster organic discussions • Requiring of reminders to be silent during voting (no different than F2F) • Greater confidentiality considerations • Less opportunity to observe non-verbal cues • More distractions from participant environment • Not as engaging as F2F that may diminish opportunity for rich discussions

Authors' recommendations
• Clear communication of ground rules (i.e., ensuring confidentiality, secure log-on environment) • Ample time to allow for preliminary introductions and participant access to platform • Designation of IT support crew in addition to session moderator/facilitator and provision of contact information to participants • Piloting and pre-meeting with participants prior to NGT • Creative measures to facilitate each stage (i.e., private chat functions virtual whiteboards for sharing, online group maps for idea generation, email or surveys for voting, etc.) • Consideration of issues with hybrid virtual-face-to-face sessions • Shortening of session time • Recruitment of an experienced moderator/facilitator • Technical issues present in pre-pandemic era are mostly resolved as of current https://doi.org/10.1371/journal.pone.0302437.t00519% (3/16) felt that vNGT was inferior to in-person NGT.Thematic analysis across the questions fell under benefits and concerns of using the vNGT as seen in Table 5.

Discussion
This study set out to explore the use of the vNGT.Our findings demonstrate researchers use virtual platforms considering restrictions for travel and the requirements for social distancing.Although 8 databases were included in our study, the vast majority of articles reporting use of a virtual platform were from medical journals and related to healthcare.
The pandemic has imposed unprecedented sanctions in the way research is conducted, necessitating the transition from physical to virtual interactive modalities.Recent guidelines have suggested that researchers should report any modifications made as a result of extenuating circumstances (e.g., COVID-19), report important modifications to the methods, including mitigating strategies and impact on the results [58].Our review demonstrated that most published research did not adhere to these guidelines; approximately a quarter of studies reported modifications to the method and just under a half which virtual platform was used.None considered the impact on outcomes.Overall, this is consistent with poor reporting seen in consensus research [4,59,60].
A few articles discussed the benefits and challenges of conducting the NGT virtually.Commonly cited benefits were increased accessibility for participants by overcoming barriers to participation, such as psychological discomfort in group settings [19], transportation issues [29,49], or time constraints [32].This advantage may have been amplified for healthcare professionals seeking to actively participate in research in the midst of battling the COVID-19 pandemic and has been previously reported [58].However, lack of access or ability to use digital platforms may have reduced accessibility for some participants, due to socioeconomic disadvantage [28] or advanced age [50,57].Previous literature has certainly described age, race and literacy-related disparities in the use of technology with online populations being younger and more affluent [61,62].However, some report that online focus group participants were more likely to be non-white, less educated, and less healthy than the in-person sample [63].Thus, online methods may lead to less representative participant demographics.
Several other limitations were noted such as timing of sessions if participants are from different time zones [51], and technical difficulties [28].Testing the technology prior to use and having designated staff to address technical issues has been recommended [20], but is not always sufficient [28].
From the perspective of MST, the virtual platforms offered a variety of options to enhance communication.The capacity of communication modalities to support synchronicity is further informed by qualities intrinsic to the media, including, but not limited to transmission velocity, which refers to the speed at which a message reaches the recipient; parallelism, the number of simultaneous transmissions at any given time, and symbol sets, meaning different ways in which a message can be encoded.In general, symbol sets and transmission velocity enhance synchronicity, whereas parallelism decreases it [21,64,65].
The MST helped to inform our understanding of vNGT in pragmatic terms.Studies employed a diverse array of modalities depending on the NGT stage.In idea generation, which predominantly comprises one-way input, several studies [29,31,35,36,44,48,53,54] adopted asynchronous modalities such as e-surveys or emails.On the other hand, discussion and clarification phases were conducted synchronously as demonstrated in all studies in our review with two also providing asynchronous additional options [29,31].This is in keeping with the idea that convergent communication benefits from higher transmission velocity.See  Concerns about engagement and interactivity were raised by several corresponding authors in our study, but none explored this in detail.Media richness theory suggests that computermediated group communication has lower social presence and less task focus [66] but this theory predates the current advances in platforms, which have improved our ability to see and interact with colleagues.One study that directly compared the web-based NGT to a traditional in-person format was using asynchronous technology with no visual capabilities [66].Not surprisingly, the online participants were significantly less satisfied with the decision process compared to the traditional session, but only marginal differences were found in the outcomes [66].Another study compared video to in person interviews.Interestingly Skype interviews yielded some of the most unguarded responses and richest data [67].
For this review, the vNGT was defined as having all stages in a virtual format.Several authors used a combination of virtual and in-person, but these were not included and are worthy of further exploration.

Conclusion
The vNGT offers several potential benefits; inclusion of geographically dispersed participants, reduced time and expense for travel/meeting accommodations, and flexible scheduling.Reduced engagement and reduced participation for technology-challenge individuals are considerations.Since research on the vNGT itself is lacking, authors should clearly report modifications made and risks/benefits as well as potential impact on the consensus decision.Any minor adjustments in research protocols can introduce unanticipated vulnerabilities, thus compromising results and improved reporting is essential.In addition, more research is required to directly compare in-person to vNGTs including potential implications for cognitive biases [68].

Fig 1 .
Fig 1. PRISMA flowchart of the literature search and study selection process for a scoping review of virtual nominal group techniques published between March 2020 and July 2022.Abbreviation: NGT, nominal group technique.https://doi.org/10.1371/journal.pone.0302437.g001 Fig 2 for considerations at each stage.