Development of a core outcome set for open lower limb fracture

Ethics and registration

The South-Central Research Ethics Committee (REC) granted ethical approval for this study on 1 March 2018 (REC reference: 18/SC/0051, IRAS project ID: 235150). The Core Outcomes for Open Lower Limb Fracture (CO-OLLF) study was prospectively registered on the Core Outcome Measures in Effectiveness Trials (COMET) database,¹¹ and was adopted by the National Institute for Health and Care Research (NIHR) Clinical Research Network (CRN) (Protocol number: 13257).

Design

Candidate outcomes were identified from a systematic literature review¹² and a secondary qualitative thematic analysis of in-depth patient interviews that explored the lived experience of recovery from open lower limb fracture. Candidate outcomes were fed back to stakeholders in structured discussion groups and a two-round Delphi survey before a final review at a consensus meeting. This study was designed and reported in accordance with the COMET Initiative guidance^7,9 and the Core Outcome Set-STAndards for Reporting statement,¹⁰ respectively.

Population

The population and health condition used for the CO-OLLF study was defined as all skeletally mature patients (age greater than 18 years) following open lower limb fracture of any grade distal to the acetabulum.¹² Health economic outcomes were considered out of scope.¹³

Information sources

Outcomes important to healthcare professionals and researchers were identified using a systematic literature review to collect, summarize, and categorize contemporary outcomes reported in clinical studies on skeletally mature patients following open lower limb fracture over ten years from 2009.⁵

Outcomes important to patients were identified through a secondary thematic analysis of 25 in-depth interview transcripts that explored the lived experience of recovery following open lower limb fracture. Interview participants were recruited as part of the WOLLF HTA-10/57/20 trial.⁴ A purposive sampling strategy was used to ensure a range of sex, age, mechanism of injury, the severity of open lower limb fracture, and time since injury. Interviews were conducted between two and four years post-injury to provide an overview of the recovery experience. The primary data collection, qualitative analysis, results, and further methodological detail regarding the research team, reflexivity, study design, participant demographics, and interview structure are published elsewhere.^3,4

A secondary thematic analysis of interview transcripts was undertaken. Open coding was used to identify and organize text extracts representing outcome codes. Outcome codes were interpreted to create outcome themes. Outcome themes were then grouped and organized under appropriate outcome domains using the COMET taxonomy of outcomes as a framework.¹⁴ Data were organized using NVivo V.12 (QSR International, UK). To facilitate reflection on the process of interpretation, a sample of five interview transcripts were coded in duplicate by ALA and ET before meeting to discuss and compare identified codes, themes, and interpretations. Rigour was demonstrated through trustworthiness.¹⁵ Outcome themes grouped under outcome in the COMET taxonomy outcome domains were presented to patients and healthcare professionals at structured discussion groups, a Delphi survey, and at a consensus meeting in an iterative process to enhance clarity. See Supplementary Methods for justification and a detailed summary of the thematic analysis methodology.

Outcomes identified from the systematic literature review and the qualitative analysis were combined and rationalized before being categorized and organized using the COMET taxonomy of outcomes to create an inventory of outcomes for open lower limb fracture. Outcomes were assigned a number to aid later discussion and tracking through the consensus process. The COMET taxonomy of outcomes is a categorization system that provides sufficient granularity and scope for classifying all medical outcomes in core outcome set development.¹⁴ The inventory of outcomes was presented to a purposive sample of healthcare professionals and researchers, followed by patients in two structured discussion groups. Structured discussion groups were used to refine and review outcomes for duplication, use of language, and understandability before the consensus process. The resulting list of outcomes was reviewed and refined by the study management group (consisting: ALA, ET, RF, MLC, and XLG) before being taken forward to the Delphi survey.

Delphi survey

A two-round Delphi survey was conducted using COMET DelphiManager (University of Liverpool, UK)¹⁶ between 9 February 2019 and 19 March 2019. Participants were recruited from three key stakeholder groups: patients, healthcare professionals, and researchers (clinical academics and people working in clinical musculoskeletal research, e.g. members of clinical trial teams). Study adoption by the NIHR CRN facilitated multisite recruitment. Patients were recruited from five UK major trauma centres (MTCs). Patients were eligible if they were over the age of 18 years and had sustained an open lower limb fracture (as defined above) at any point before participation. Healthcare professionals and researchers were recruited through cascade emails sent through the Major Trauma Network, the Orthopaedic Trauma Society, and the British Association of Plastic and Reconstructive Surgeons. Healthcare professionals and researchers were eligible for recruitment if their professional role involved the regular clinical care of patients following open lower limb fracture or if they actively participated in orthopaedic trauma research, respectively.

In each survey round, participants were presented with the inventory of outcomes for open lower limb fracture and asked to score the outcomes for importance using the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) 1 to 9 Likert scale.^17,18 The scale was displayed as 1 to 3 labelled ‘not important’, 4 to 6 labelled ‘important but not critical’, and 7 to 9 labelled ‘critical’. Participants could suggest additional outcomes at the end of round 1. The study management group reviewed proposed outcomes, and outcomes identified as novel were added to round 2. In round 2, participants were shown a bar chart of round 1 scores for each outcome by stakeholder group, overall score, and their first-round score. They were then asked to reflect on their scoring, considering the additional information before re-scoring each outcome. The consensus criteria used to identify outcomes as consensus-in, no-consensus, and consensus-out were those recommended by COMET and defined a priori (Table I).⁹ No outcomes were dropped between rounds.

Consensus meeting

The results of the Delphi survey were presented at a one-day face-to-face consensus meeting hosted at a purpose-built events centre on 26 March 2019. Patients, healthcare professionals, and researchers who had completed both rounds of the Delphi survey were invited to attend. Participants were sampled purposively using a sampling matrix to ensure a balanced representation of patients in terms of age, sex, injury severity, and time since injury, also including healthcare professionals and researchers to achieve a mixed sample of professional backgrounds. All participants were sent a detailed pre-meeting reading pack summarizing the project, results of the Delphi survey, and meeting agenda. The meeting structure and methods were guided by the COMET Handbook⁹ and adapted from the James Lind Alliance Priority Setting Partnerships methods for a final priority setting meeting.¹⁸

The meeting consisted of three separate sessions, and participants were split into three groups for sessions one and two, which were independently facilitated by members of the study management group (see Figure 1 for a schematic representation of the meeting structure). Initially, each group selected their top two or three outcomes from the ‘no-consensus’ outcomes identified in the Delphi survey for inclusion in the second session. Then, each group ranked a top ten list of outcomes from all ‘consensus-in’ outcomes identified in the Delphi survey second round plus outcomes selected for inclusion earlier. Finally, groups came together, and anonymous electronic voting was undertaken using TurningPoint software and handsets (Turning Technologies, USA). After removing duplicates, all outcomes from the groups’ top 10's were added to a Microsoft PowerPoint (Microsoft Corporation, USA) presentation template to facilitate electronic voting for inclusion in the final core outcome set. Electronic voting enabled the presentation of voting results on each outcome immediately to aid discussion. Participants were given the scoring options of 1 to 3: ‘not important’, ‘important but not critical’, and ‘critical’. Outcomes were voted on in two rounds with the facilitation of discussion on each outcome before repeated voting. Outcomes reaching ‘consensus-in’ were included in the core outcome set; all others were excluded. The core outcome set was then presented to the participants and ratified.

Fig. 1

Core Outcomes for Open Lower Limb Fracture study consensus meeting schematic flow diagram.

The meeting was chaired by a facilitator with experience in running consensus meetings (MLC). Group facilitators ensured that contrasting views were actively sought, and all participants had equal opportunity to contribute to the discussion throughout the day. Each participant was encouraged to share their opinions on an outcome before any voting or ranking exercise was undertaken. A modified nominal group technique (NGT) was used to facilitate consensus. NGT is a well-established consensus method used to make decisions efficiently while taking everyone’s views into account.^9,19 NGT eliminates duplicate ideas and allows all individual opinions to be considered, which influences the group intending to converge opinions to achieve a consensus.^9,19

Other analyses and statistical considerations

We analyzed participant attrition rate and outcome scoring changes between Delphi survey rounds. Medians and standard deviations were calculated for the score (1 to 3) of each outcome scored in the consensus meeting electronic-voting session using TurningPoint software.

There is currently no recommended minimum number of participants to include in a Delphi survey and no requirement to demonstrate a statistically representative sample. In previous studies, between ten and 15 participants per stakeholder group have yielded satisfactory results.^9,19 NGT does not depend on statistical power, and there is no robust or commonly used method for calculating the required number of participants.⁹

Information gathering

A systematic review describing outcomes reported in the literature has been previously published;⁵ 1,803 outcomes were identified from 786 studies and consolidated to 82 standardized outcome headings (Supplementary Figure a).

Thematic analysis of 25 interview transcripts identified 1,121 codes in the data representing an explicit or underlying outcome that may be important to measure in patients recovering from open lower limb fracture (see Supplementary Methods for examples of patient quotations representative of underlying outcome codes). Interpreting the outcome codes resulted in the development of 88 outcome themes and sub-themes (see the coding tree detailed in Supplementary Figure b). Duplicate data coding by ALA and ET demonstrated consistency over the identification of outcome codes in the data.

Outcome headings and themes from the systematic review and qualitative analysis, respectively, were brought together to create the inventory of outcomes for open lower limb fracture. The resulting inventory of outcomes consisted of a long list of 150 outcomes categorized using the COMET taxonomy of outcomes to 18 outcome domains within five core areas. Following rationalization of outcomes byALA, structured discussion groups, and a study management group meeting, 68 outcomes were shortlisted for the Delphi survey (see Figure 2 for a schematic diagram of outcome rationalization, with further detail provided in Supplementary Table i).

Fig. 2

Refinement of the inventory of outcomes for open lower limb fractures.

Delphi survey

A total of 136 participants completed both rounds of the Delphi survey, of whom 55 (40%), 73 (54%), and eight (6%) were patients, healthcare professionals, and researchers, respectively (see Table II for participant characteristics). Six and 11 outcomes achieved ‘consensus-in’ in the first and second rounds, respectively. No outcomes met the ‘consensus-out’ criteria. The remaining 59 outcomes were ‘no-consensus’ after the second round (see Supplementary Table ii for a complete list of outcomes and their scoring for ‘consensus-in’). Participants in the first round suggested 13 additional outcomes. The study management group rejected 11, and two were accepted for inclusion in the subsequent Delphi survey (see Supplementary Table iii for accepting or rejecting justification). Figure 3 shows a schematic flow diagram of the Delphi survey and consensus meeting results. The between round attrition rate was 27%, with attrition spread relatively evenly across the three stakeholder groups (26% patients, 28% healthcare professionals, and 27% for researchers).

Fig. 3

Schematic flow diagram of the Delphi survey and consensus meeting consensus results. COS, core outcome set.

Consensus meeting

A total of 15 patients, 14 healthcare professionals, 11 researchers, and one patient carer attended the consensus meeting (see Table II for participant characteristics). The majority of patients were male (nine (60%)), having sustained an open lower limb fracture of the tibia/fibula (14 (93%)) that required a skin graft or muscle flap (nine (60%)), and the mean age of patients was 53 years (26 to 76). See Table III for a complete list of outcomes by outcome number discussed at the consensus meeting, and Figure 3 for a schematic of results. The first session led to creating two new outcomes and the selection of ten outcomes out of the 59 ‘no-consensus’ outcomes for later consideration (Figure 3). The groups’ top ten outcomes included 18 outcomes after removing duplicates. In the workshop’s final session, walking, gait, and mobility, being able to return to life roles (e.g. caring, work, and military duty), quality of life, and pain or discomfort outcomes were agreed upon (see Table IV for electronic voting results in full).

Electronic voting in this session stimulated debate over whether the ‘bone healing’ and ‘deep infection’ outcomes should be included in the core outcome set. The patients felt that fundamentally, being able to walk, return to work, and maintain a good quality of life are the most important outcomes to achieve in recovery and that bone healing and infection were secondary. The inclusion of ‘deep infection’ was a sticking point for several consultant surgeons, arguing that minimizing the risk of infection is a guiding principle of modern practice and is fundamentally an essential component of achieving a positive treatment outcome. A healthcare professional raised a counterargument, pointing out that the core outcome set is intended for use in all research and clinical practice on patients following open lower limb fracture. Studies on rehabilitation or psychosocial support interventions would not consider infection a key outcome to measure. An additional point supported this argument that in any surgical intervention study, infection would be regarded as an adverse event; as such, its measurement would be mandatory. The group agreed that objective surgical outcomes, including ‘deep infection’ and ‘bone healing’, are important and need to be measured in surgical intervention trials but not across all open lower limb fracture studies. Therefore, they should not be included in the core outcome set.

The four ‘consensus-in’ outcomes were presented to the group and ratified as the core outcome set for open lower limb fracture (see Figure 3).