The Importance of Pragmatic Study Design to the Scholarly Influence of Surgical Hip Fracture Randomized Controlled Trials

Background: Surgical randomized controlled trials (RCTs) have potential drawbacks, leading some to question their role in filling the information gap in orthopaedic surgery. Pragmatism in study design was introduced to increase the clinical applicability of study results. The purpose of this study was to examine how pragmatism affects the scholarly influence of surgical RCTs. Methods: A search for surgical hip fracture-related RCTs published between 1995 and 2015 was done. Journal impact factor, citation number, research question, significance and type of outcome, number of centers involved, and the Pragmatic-Explanatory Continuum Indicator Summary-2 level of pragmatism score were recorded for each study. Scholarly influence was estimated by a study's inclusion into orthopaedic literature or guidelines or through the study's average yearly citation rate. Results: One hundred sixty RCTs were included in the final analysis. A multivariate logistic regression identified large study sample size as the only predictor of an RCT being used in clinical guidance texts. Large sample size and multicenter RCTs were predictors of high yearly citation rates. The level of pragmatism in study design did not predict scholarly influence. Conclusions: Pragmatic design is not independently associated with increased scholarly influence; however, large study sample size was the most important study characteristic affecting scholarly influence.

variables equally between treatment groups. [2][3][4][5] Since their introduction, RCTs have been increasing in quantity and quality, 6 with more than 40,000 RCTs currently being reported annually. 7 RCTs also have some notable drawbacks that limit their use. They can take several years to complete and are expensive to perform. [8][9][10] Furthermore, the results of RCTs may not be generalizable because of strict inclusion criteria, which limit the amount of patients who can be enrolled in a study. 9 Limits on patient enrollment can also result from studying a rare disease or not having researcher equipoise regarding enrollment of patients into a particular arm of a RCT. 11 Surgical RCTs have additional methodological drawbacks, such as difficulty in standardization of surgical procedures, difference in skill levels of participating surgeons, learning curves of new procedures, and various degrees of lack of blinding. 9,[12][13][14] A recent study found that compared with orthopaedic nonsurgical RCTs, surgical RCTs are lower in quality because of multiple factors, including difficulty in randomization and blinding. 8,15 Because of their importance to evidence-based medicine, and despite their limitations, the orthopaedic community has been called upon to produce well conducted and adequately powered surgical RCTs that address important therapeutic issues. 8,9,16 However, some still question if RCTs, particularly in the surgical field, should be the primary methodology to fill the information gaps in orthopaedic surgery. [17][18][19][20] Pragmatism in clinical trials was introduced to increase the clinical applicability of their results. 21 All trials, including RCTs, fall on a spectrum of being explanatory to being pragmatic. 21,22 The more explanatory the RCT, the more rigorous the inclusion/exclusion criteria are, and as a result, the study is able to measure the efficacy of a treatment in a very specific population but may fail to demonstrate efficacy if the treatment was applied to the general population. The more pragmatic the RCT, the more heterogenous the study population is, and as a result, the study is able to measure the efficacy of the intervention if it was applied to broad patient groups in real-world clinical practice. 21,23 However, the method by which more pragmatic RCTs fill knowledge gaps and inform clinical practice is not clear. The scholarly influence of a trial, as measured by its citation level and inclusion in various clinical guidelines and textbooks, may be one method by which surgical RCTs can inform clinical practice.
The purpose of this study was to examine how pragmatism affects the scholarly influence of surgical RCTs. Hip fracture surgical RCTs were used as an example.

Methods
This systematic review was registered (PROSPERO registration number 139858) and is reported following the Preferred Reporting Items for Systematic Reviews and Meta-analyses statement. 24 Data set is stored at the Dryad repository, DOI: 10.7272/Q6C24TPK.

Eligibility Criteria
All surgical hip fracture-related RCTs published between 1995 and 2015 were included in this study. A surgical hip fracture-related RCT was defined as an RCT where the therapeutic intervention under investigation was surgical. The choice of this time range was to maintain relevance to contemporary clinical practice by accounting for the last 2 decades of evidence-based medicine, but also allowing time for the RCT findings to be cited and incorporated into clinical guidelines and professional society recommendations. Observational studies, biomechanics studies, preclinical studies, case reports, reviews, systematic reviews, meta-analyses, and nonsurgical RCTs were excluded.

Search Strategy
We The search algorithm for Embase was (fractures: ti OR fracture: ti) AND (hip: ti OR intertrochanteric: ti OR trochanteric: ti OR subtrochanteric:ti OR peritrochanteric: ti OR "femur neck": ti OR "femoral neck": ti OR "neck of the femur": ti) AND (randomized: ti OR randomised: ti).

Study Selection
Abstracts of studies identified by the search strategy were reviewed by two investigators (G.G. and T.G.) independently. Eligibility criteria were applied. Disagreements were settled between the two investigators by consensus and were discussed with a third investigator (M.T.M.) when they did not reach an agreement.

Variable Collection
Data were collected independently by a pair of reviewers (G.G., T.G.) using a predefined electronic data extraction form. Disagreements were settled between the two investigators by consensus and were discussed with a third investigator (M.T.M.) when they did not reach an agreement.
For each included article, the following parameters were collected.
1. Study title, year of publication, primary country where research was done 2. Type of hip fracture studied 3. Type of primary research question 4. Journal name and impact factor (collected using our institutional subscription to InCites Journal Citation Reports by Clarivate Analytics (https:// clarivate.com/) 5. Number of centers involved in the study (single/multiple) 6. Funding source (as reported by the authors or recorded during study registration classified to for-profit, nonprofit, or both; not reported; no funding) 7. Study registration (classified to registered on ClinicalTrial.gov, registered on a non-American registry, or not registered). 8. Study groups and their sample sizes 9. Primary outcome (reported? yes/no, binary? Yes/no, significant difference found? Yes/no). Primary outcomes were those explicitly reported as such. If none was explicitly reported, we took the outcome as defined in the primary study objective. If the primary outcome was still not clearly identified, the article was excluded.

Assessment of Randomized Controlled Trial Pragmatism
The Pragmatic-Explanatory Continuum Indicator Summary (PRECIS-2) score for pragmatic/explanatory design was used to calculate the degree of pragmatism for each RCT. 25 We modified the PRECIS-2 score to exclude the "flexibility in adherence" dimension. This dimension refers to the degree of flexibility the participants have in adherence to the chosen intervention. However, in a surgical RCT, the intervention is completed during surgery and the participants have no choice but to "adhere" to it. Each of the remaining eight dimensions were given a score of 1 to 5, resulting in a maximal score (fully pragmatic study) of 40. The Modified PRECIS-2 score was scored independently by two reviewers (G.G., T.G.). Disagreements were settled between the two investigators by consensus and were discussed with a third investigator (M.T.M.) when they did not reach an agreement.

Figure 1
Flowchart demonstrating the manuscript review process. RCT = randomized controlled trial

Assessment of Scholarly Influence of a Randomized Controlled Trial
The scholarly influence of an RCT was judged by whether the study was included in clinical guidance texts and by the study citation rate. An RCT was considered as being included in clinical guidance texts if it was incorporated into recognized professional guidelines, included in recommended readings by professional societies, or included in the references in prominent professional textbooks. The American Academy of Orthopaedic Surgeons Evidence-based Clinical Practice Guidelines for management of hip fractures in the elderly, published in September 5, 2014, was used as an indication for incorporation into professional guidelines. 26 Recommended reading lists were taken from the Orthopaedic Trauma Association website 27 and AO Foundation surgical reference website. 28 Inclusion in prominent professional orthopaedic trauma textbooks was assessed in the reference lists of the 2019 edition of Rockwood and Green's Fractures in Adults 29 and 2019 edition of Skeletal Trauma. 30 Each study's citation rate was retrieved from Google Scholar and cross-referenced with the PubMed citation rate to ensure agreement. 31

Statistical Analysis
Descriptive statistics (counts, percentage, means, and SD) were used to describe the data. Continuous variables such as journal impact factor, yearly citation rate and sample size were converted into categorical variables of "high" or "low." A study was given a "high" value if its value was above the 75 th percentile for that variable. Binary outcomes (low versus High) were assessed using the chi-square test for categorical variables and the student's t-test for continuous variables. Significance was determined after making a Bonferroni adjustment for multiple comparisons. Linear regression was used to assess trends in the data over time. A multivariable logistic regression analysis with stepwise backward elimination and forward selection was done to identify independent variables predicting inclusion of an RCT in clinical guidance texts or having a high yearly citation level.
Statistics were calculated using a statistical package on R version 4.0.2 (2020-06-22). Copyright © 2020 The R Foundation for Statistical Computing.

Results
The initial search yielded 482 studies from PubMed and 382 studies from EMBASE. After removal of duplicates and application of eligibility criteria, 160 studies remained for final analysis (Figure 1). An upward trend (R 2 = 0.62) was observed in the number of hip fracture RCTs published each publication year during the study period ( Figure 2).
Eighty-five RCTs focused on trochanteric fractures, 73 on femoral neck fractures, two on femoral head fractures, and six on both femoral neck and trochanteric fractures. The most common research question sought to determine the best surgical implant, followed by comparing surgical techniques (Table 1)  Graph showing the randomized controlled trial publication trends throughout the study period.
constructs. Twenty-one (13.1%) of hip fracture RCTs were multicenter trials. Thirteen studies (8%) were done in North America, nine of them (6%) in the United States. Study parameters of RCTs in relation to their inclusion in clinical guidance texts are given in Table 2. The same characteristics in relation to a high yearly citation rate are given in Table 3.
Overall, we did not find a relationship between study pragmatism and increased scholarly influence. The modified PRECIS-2 scores were not found to be a predictor of both measurements of scholarly influence (inclusion in clinical guidance texts and a high average yearly citation rate). To further substantiate this finding, we conducted a sensitivity analysis by analyzing the scores as continuous variables, categorical variables, or a binary variable of whether the score was above the 75 percentile or not, did not change the results (Tables 2 and 3). In all analysis types, increased pragmatism did not correlate to increase scholarly influence.
A multivariate logistic regression analysis with stepwise backward elimination and forward selection identified only large study sample size as a statistically significant independent predictor of use of an RCT in clinical guidance texts (odds ratio [OR]: 2.5, confidence interval [CI]: 1.2 to 6.0, P = 0.02). Large sample size and multiple center RCTs were statistically significant predictors of high average yearly citation rates (OR: 4.7, CI: 1.9 to 11.9, P = 0.0008 and OR: 3.5, CI: 1.2 to 10.0, P = 0.017, respectively), while statistical significance in the primary outcome was suggested as a possible predictor of citation rates (OR: 2.3, CI: 0.95 to 5.8, P = 0.073).

Discussion
The purpose of this study was to examine how pragmatism affects the scholarly influence of surgical RCTs. This study demonstrated that a more pragmatic design, as measured using the PRECIS-2 score, did not result in increased scholarly influence. Large sample size was the primary study characteristic that was predictive of scholarly influence. This is the first study to examine the scholarly influence of pragmatism in surgical RCT study design.
The advent of RCTs had a revolutionary effect on medical science, with more than 40,000 RCTs being reported annually. 7 The ability of RCTs to reduce selection bias by distributing known and unknown confounding variables equally between treatment allocation groups has led them to be considered the highest level of evidence in therapeutic research. 1,2,17 Although RCTs are more ideally suited for medical trials, surgical RCTs have been increasing in number and quality over time. 4,6 This study demonstrated a 390% increase in the number of hip fracture RCTs between the first 3 years and the last 3 years of the study period. Higher quality hip fracture RCTs, as evidenced by the higher impact factor of the journals in which they were published, was not a significant predictor of scholarly influence in this study.
Surgical RCTs do not have a uniform clinical impact. 18 In a previous study, a distal radius fracture RCT was shown to influence clinical practice in England by virtue of its pragmatic design. 32 Other RCTs, may not have a similar impact on clinical practice because of conflicting results with similar RCTs or expertise bias. 18 Measurement of clinical impact of RCTs can be difficult and can only be done adequately using a national fracture registry. The low number of hip fracture RCTs in the United States 33 and the absence of a US national fracture registry 34 increase this difficulty. To circumvent this difficulty, in this study, we chose to assess whether the RCTs were used for scholarly influence and not assess whether this influence was impactful. This study's findings demonstrated that it is primarily large sample size that promoted the use of an RCT for scholarly influence. This finding is in accordance with the call to publish higher quality studies with larger sample size in orthopaedic trauma. 8,16,33 Interestingly, having a more pragmatic study design and reporting a significant primary outcome were not associated with increased scholarly influence. The PRECIS-2 tool was developed to help quantify the level of pragmatism in a given trial. 25 The separate domains of the PRECIS-2 tool focus on recruitment, delivery of the intervention, follow-up, and the choice of primary outcome. Therefore, trials can be deemed pragmatic about some domains, but rarely about all. As a result, challenges to designing a trial that is pragmatic in all domains have been identified. 21 The application of the PRECIS-2 tool to the RCTs included in this study identified the fact that aspects of pragmatic design may not be consistent with important surgical outcomes. For example, many hip fracture RCTs require long-term, resource intensive follow-up to compare one surgical intervention with another. However, this degree of follow-up would be deemed "nonpragmatic" or "explanatory" because it differs from typical hip fracture follow-up. Without the ability to report important surgical outcomes, a pragmatic trial is unlikely to have scholarly or clinical impact. This highlights the challenges of the pragmatic approach in surgical trials, where trial impact may need a more explanatory approach to have impact in the field. Citation analysis is commonly used for assessment of clinical impact and importance of research. 35,36 A number of studies have done such an analysis on hip treatment research. [37][38][39][40] A study on the 100 most cited publications in hip research demonstrated that only 5% of these studies were RCTs, and only 1% of studies considered "classic" were RCTs. 37 In another study on the 50 most cited hip surgery papers, 12% of the papers were RCTs. 40 Both of these studies suggest that, although desirable, a high level of evidence is not necessary for a study to have a high citation number. This study suggests that in order for RCTs to have a high citation rate, and ultimately a high absolute citation number, they should have a large sample size and be multicentered. Having a significant difference in the primary outcome variable trended toward statistical significance as a predictor. In two recent studies on the 50 and 100 most cited papers on hip and knee arthroplasty, respectively, most studies were published in high impact journals and originated in the United States. 38,39 In this study, only nine of 160 RCTs (6%) originated in the United States and only 13 of 160 (8%) originated in North America. This systematic review is the first study to examine whether a pragmatic design increases the scholarly influence of RCTs. This study also has some notable limitations. We decided to focus only on hip fracture RCTs because of their relative abundance and their applicability to many orthopaedic practices. Therefore, these findings may not be generalizable to other surgical RCTs. We did not examine the quality of the RCTs directly. Instead, we used journal impact factor as a surrogate for a high-quality paper. More detailed analysis of each study's methodology, and their specific research questions may uncover additional parameters that promote their use for scholarly influence. Finally, inclusion into clinical guidance texts, such as the American Academy of Orthopaedic Surgeons hip fracture guidelines, was determined using a limited number of texts, all of which are written in the English language and most of which originated in North America. Although these texts are widely recognized and respected, other continent-specific or country-specific texts exist and their inclusion may have influenced the results of this study.

Conclusions
This systematic review suggests that a pragmatic design, as measured by the PRECIS-2 score, is not independently associated with increased scholarly influence. However, since large study sample size was the most important study characteristic affecting scholarly influence, a pragmatic design with broad inclusion criteria may increase scholarly influence by increasing enrollment numbers. Future studies will need to address how scholarly influence of RCTs is translated to clinical impact and whether large sample size RCTs are able to fill information gaps in orthopaedic trauma.