Reporting and methodological quality of meta-analyses in urological literature

Purpose To assess the overall quality of published urological meta-analyses and identify predictive factors for high quality. Materials and Methods We systematically searched PubMed to identify meta-analyses published from January 1st, 2011 to December 31st, 2015 in 10 predetermined major paper-based urology journals. The characteristics of the included meta-analyses were collected, and their reporting and methodological qualities were assessed by the PRISMA checklist (27 items) and AMSTAR tool (11 items), respectively. Descriptive statistics were used for individual items as a measure of overall compliance, and PRISMA and AMSTAR scores were calculated as the sum of adequately reported domains. Logistic regression was used to identify predictive factors for high qualities. Results A total of 183 meta-analyses were included. The mean PRISMA and AMSTAR scores were 22.74 ± 2.04 and 7.57 ± 1.41, respectively. PRISMA item 5, protocol and registration, items 15 and 22, risk of bias across studies, items 16 and 23, additional analysis had less than 50% adherence. AMSTAR item 1, “a priori” design, item 5, list of studies and item 10, publication bias had less than 50% adherence. Logistic regression analyses showed that funding support and “a priori” design were associated with superior reporting quality, following PRISMA guideline and “a priori” design were associated with superior methodological quality. Conclusions Reporting and methodological qualities of recently published meta-analyses in major paper-based urology journals are generally good. Further improvement could potentially be achieved by strictly adhering to PRISMA guideline and having “a priori” protocol.


INTRODUCTION
A systematic review is a review of a clearly formulated question using systematic methods to identify, select and critically appraise relevant research. The systematic review may include a quantitative synthesis of results called meta-analysis, which summarizes all results of primary studies in order to obtain a combined estimate of the effect. Certain types of systematic review and meta-analysis are considered as the highest level of evidence (level 1a) (http: //www.cebm.net/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/). Also, well-conducted meta-analyses can sometimes resolve conflicting evidence and provide more reliable conclusions (Berlin & Golub, 2014). Meta-analyses are often appealing to both authors and journals as they are commonly highly cited publication. Rapidly expanding literature across all medical disciplines raise the increasing need to summarize and synthesis the currently available evidence. Such factors have contributed to the increased number of published meta-analyses in medical journals (Tunis et al., 2013;Zhang et al., 2016).
However, like original research articles, quantity does not mean quality (Adie et al., 2015;Berlin & Golub, 2014;Dechartres et al., 2014;Murad & Montori, 2013). It is imperative for both the medical and publishing community to aware the negative influence of flawed or low-quality meta-analyses (Berlin & Golub, 2014). Several statements or guidelines have been proposed and validated as the tools to assess the quality of published meta-analyses (Faggion, 2015;Liberati et al., 2009;Moher et al., 2009;Pieper et al., 2015;Shea et al., 2007a;Shea et al., 2007b;Shea et al., 2009;Stroup et al., 2000). The most well-known guideline is the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), which is actually a checklist recommended to follow when reporting meta-analyses Moher et al., 2009). An earlier initiative was the development of the AMSTAR, a measuring tool to assess the methodological or conducting quality of meta-analyses (Shea et al., 2007b;Shea et al., 2009). In other words, AMSTAR usually serves as a critical appraisal tool to identify the scope of bias in methodology at the review level.
Although still debatable, a number of studies in various surgical and medical fields have used the ''scores'' based on the fulfillments of PRISMA and AMSTA to assess the qualities of systematic reviews and meta-analyses (Adie et al., 2015;Gagnier & Kellam, 2013;Liu et al., 2017;Shea et al., 2007a;Tunis et al., 2013;Zhang et al., 2016). There exist several duplicate items in the two tools, generally they are considered separate tools and are commonly used together for the assessment, PRISMA for reporting quality and AMSTAR for methodological quality (Adie et al., 2015;Gagnier & Kellam, 2013;Liu et al., 2017;Shea et al., 2007a;Tunis et al., 2013;Zhang et al., 2016).
To date, no studies have comprehensively assessed the reporting and methodological quality of urological meta-analyses, in particular those published after the PRISMA initiative (2009). Considering meta-analyses are often influential and highly cited publications, there is a need to explore whether general characteristics (author, journal, and report) of the meta-analyses have an association with the overall quality. Therefore, in the present study, we specifically focused on meta-analyses published in major paper-based urology journals, with the aim to assess the reporting and methodological quality, as well as to identify relevant predictive or associated factors.

Eligibility criteria
To be eligible for inclusion, a meta-analysis had to meet the following inclusion criteria: (1) a study with the meta-analytic methodology pooling results from primary articles (including meta-analysis alone or systematic reviews containing meta-analyses); (2) published in the following 10 predetermined urology journals: British Journal of Urology International (BJUI), European Urology (EU), Journal of Endourology (JEU), Journal of Pediatric Urology (JPU), Journal of Urology (JU), Neurourology and Urodynamics (NUUD), Urology (URO), Urolithiasis (UL), formerly known as Urological Research, Urologic Oncology (UO) and World Journal of Urology (WJU); (3) published in the printed journal between January 1st 2011 to December 31th 2015 (excluding ''Epub ahead of print'').
Exclusion criteria were: (1) systematic review without meta-analysis; (2) original research article or original research article combined with a meta-analysis; (3) network meta-analysis or multiple group comparison meta-analysis; (4) meta-analysis of single proportions; (5) meta-analysis originally published in the Cochrane Database of Systematic Reviews. There were two reasons for our decision to exclude the network meta-analysis (multiple group comparison meta-analysis) and meta-analysis of single proportions. First, they were relatively uncommon compared to other ''traditional'' meta-analyses. Second, the methods and results reported by those meta-analyses were very heterogenic and different from ''traditional'' or pairwise meta-analyses (Bafeta et al., 2013;Bafeta et al., 2014). Some of the items in the PRISMA and AMSTAR do not perfectly apply to network meta-analyses and meta-analysis of single proportions.
Two investigators (LX, JX) independently screened the titles and abstracts of all the identified references. Full-text were then retrieved for potential eligible meta-analysis. Discrepancies were resolved by discussion between the two investigators.

Data extraction
We collected all data on general characteristics of the included meta-analyses, and the key reporting (PRISMA) and methodological (AMSTAR) components of the meta-analysis process. Two investigators (LX, JX) independently extracted the data. Any disagreements were resolved by discussion between the two investigators. Inter-observer reliability was examined using the kappa (κ) value.

General characteristics
We collected data on the following general characteristics: (1) corresponding author's region and country; (2) number of authors; (3) presence or absence of a professional with the background of epidemiology or statistics as a coauthor (including the acknowledgement part); (4) number of participating centers (department level); (5) subspecialties in urology (based on the American Urological Association classification); (6) presence or absence of any funding source; (7) the number of included studies; (8) type of the included studies (only RCTs or RCTs plus non-RCTs or only non-RCTs); (9) type of the meta-analyses (interventional, diagnostic, incidence related, prognostic or cannot classify); (10) type of the interventional meta-analyses (surgical or non-surgical); (11) attached a PRISMA checklist or not; (12) followed the PRISMA guideline or not (claimed this in the article or not); (13) provided the protocol and registration information or not, which also referred to the PRISMA item 5; (14) ''a priori'' design or not (claimed this in the article or not), which also referred to the AMSTAR item 1.

Assessment of key reporting components in the meta-analysis process
The PRISMA statement is a checklist of 27 items that are recommended to be included in systematic review and meta-analysis to ensure that the published report contains all relevant information (Supplemental Information). The present study focused only on metaanalyses and every item was applicable. Each PRISMA item was rated with a ''yes'' or ''no'' response. A ''yes'' response means that the item was reported, and a ''no'' response means that the item was not reported. For the purpose of data analysis, reported points were assigned as follows: ''yes'' = 1, ''no'' = 0. Therefore every included meta-analysis had an overall PRISMA score rated out of a maximum score of 27.

Assessment of key methodological components in the meta-analysis process
The AMSTAR tool is an 11-item questionnaire that was used to determine the methodological or conducting quality of systematic reviews and meta-analyses (Supplemental Information). The original tool had four responses with each item, ''yes,'' ''no,'' ''can't answer,'' or ''not applicable.'' Due to the fact that we focused on meta-analyses (excluded pure systematic reviews), every item was applicable. Each AMSTAR item was rated with a ''yes'' ''no'' or ''cannot answer'' response. A ''yes'' response means that the item is fulfilled, a ''no'' response means that the item is not fulfilled, a ''can't answer'' response means that it is inconclusive as to whether the item is fulfilled. For the purpose of data analysis, reported points were assigned as follows: ''yes'' = 1, ''no'' or ''can't answer'' = 0. Therefore, every included meta-analysis had an overall AMSTAR score rated out of a maximum score of 11.

Data analysis
Analyses, tables, and figures were configured by using a spreadsheet program (Excel 2013, Microsoft) and a statistical software (STATA 14.0, StataCorp LP). A descriptive analysis was performed for PRISMA and AMSTAR scores grouped by multiple categories. Shapiro-Wilk test was used to assess the normality of the PRISMA and AMSTAR scores (p = 0.376 and p = 0.057, respectively). Based on the distributions of PRISMA and AMSTAR scores and Shapiro-Wilk test results, we used the parametric tests to compare the qualities. Comparisons of mean qualities between dichotomous factors were conducted using the independent Student's t -test. Comparisons of study qualities between multifactor variables were conducted using the one-way analysis of variance (ANOVA), with the Tukey's HSD post hoc test. The PRISMA score and AMSTAR score were both divided into the superior and inferior quality groups with a cutoff value of 75% percentile of the respective ranges. Univariate logistic regression analysis was used to compare the differences between the superior and inferior groups with potential factors affecting study qualities. Variables included continent origin, country origin, number of authors, presence or absence of a professional with the background of epidemiology or statistics as a coauthor, number of participating centers, subspecialties, funding support, number of included studies, type of the included studies, interventional meta-analysis, type of interventional meta-analysis, followed the PRISMA guideline, and ''a priori'' design. Factors that found to be significant (p < 0.1) were then entered into the multivariate logistic regression analysis. A p < 0.05 was considered significant on statistical analyses. All the analyses were two-sided.

Search results
Figure 1 depicts a flow diagram of meta-analyses selection. The initial search identified 641 references with potential relevance. Screening the title and abstract excluded 422 references and another 36 references were excluded after reviewing the full-text. Finally, 183 metaanalyses were included for the final assessment and analysis (Supplemental Information).

General characteristics of the meta-analyses
The characteristics of the 183 meta-analyses are shown in Table 1. The number of authors and number of included studies were divided into two groups with the cutoff setting at median values (7 and 10, respectively). The number of patients included per metaanalysis ranged from 152 to 4082606. The number of patients in 14 studies could not be determined. EU (n = 44, 24%) published the largest number of included meta-analyses and JPU (n = 2, 1%) had the lowest number. The region where the largest number of included meta-analyses originated was Asia (n = 89, 49%). The most common countries of publication were China (n = 82, 45%), the USA (n = 22, 12%), and the UK (n = 20, 11%). Forty-four (24%) meta-analyses had at least one professional with the background of epidemiology or statistics as the coauthor (including the acknowledgement part). The most common subspecialty of the included meta-analyses was urologic oncology (n = 75, 41%). Most of the included meta-analyses could be categorized as interventional (n = 141, 77%), which were further subcategorized as surgical intervention (96/141, 68%) and non-surgical intervention (45/141, 32%). Sixty-two (34%) meta-analyses included only RCTs, 71 (39%) included only non-RCTs, and another 50 (27%) included both RCTs and non-RCTs. Fifty-five (30%) meta-analyses received funding support. Only two (1%) meta-analyses attached the PRISMA checklist. Fifty-five (30%) meta-analyses claimed followed the PRISMA guideline. Eight (4%) meta-analyses provided the protocol and registration information (PRISMA item 5) and 22 meta-analyses claimed the ''a priori'' design (AMSTAR item 1).

Reporting quality (PRISMA)
The overall mean PRISMA score of all the included meta-analyses was 22.74 ± 2.04 (84.2% of items adequately reported, on average). Most of the PRISMA items (25 out of 27) had a κ value more than 0.65 and none of them had a κ value less than 0.5. Table 1 shows mean PRISMA scores grouped with various factors. After excluding journals (JPU, NUUD, and UO) with less than 10 included meta-analyses, EU had the highest mean PRISMA score (23.52 ± 1.93). However, one-way ANOVA of PRISMA score showed no significant difference between the 7 journals (BJUI, EU, JEU, JU, URO, WJU, and UL), F (6,165) = 1.71, p = 0.12. Student's t -test showed no significant difference of PRISMA score between the meta-analyses from Asia and those from non-Asia region, t (181) = −0.50, p = 0.62. There was no significant difference in PRISMA scores between the meta-analyses from China and remaining countries, t (181) = −0.15, p = 0.88. Included meta-analyses in the subspecialty of urologic oncology had higher PRISMA scores than other specialties, t (181) = −2.09, p = 0.037. There was no significant difference in PRISMA scores between the included studies type (Only RCT vs. RCT & non-RCT vs. Only non-RCT), F (2,180) = 0.47, p = 0.63. There was no significant difference in PRISMA scores between the interventional and non-interventional meta-analyses, t (181) = 1.80, p = 0.07. All other two-group comparison test results are shown in Table 1. Figure 2A shows the PRISMA results on a per-item basis. Per-item PRISMA analysis revealed that five items had less than 50% adherence out of the 183 included meta-analyses (item 5, protocol and registration; items 15 and 22, risk of bias across studies; items 16 and 23, additional analysis). Item 8 (search) also had only 51% adherence.

Methodological quality (AMSTAR)
The overall mean AMSTAR score of all the included meta-analyses was 7.57 ± 1.41 (68.8% of items adequately reported, on average). Most of the AMSTAR items (10 out of 11) had a κ value more than 0.65 and none of them had a κ value less than 0.5. Table 1 shows mean AMSTAR scores grouped with various factors. After excluding journals (JPU, NUUD, and UO) with less than 10 included meta-analyses, EU had the highest mean AMSTAR score (7.98 ± 1.47). One-way ANOVA of AMSTAR score showed significant differences between the 7 journals (BJUI, EU, JEU, JU, URO, WJU, and UL), F (6,165) = 3.03, p = 0.008. Tukey's HSD post hoc test only showed that EU had higher AMSTAR score than URO, p = 0.034. Student's t -test showed no significant difference in AMSTAR score between the metaanalyses from Asia and those from non-Asia region, t (181) = 0.06, p = 0.95. There was no   significant difference of AMSTAR score between the meta-analyses from China and those from remaining countries, t (181) = −0.25, p = 0.80. Unlike PRISMA score, the AMSTAR score of included meta-analyses in the subspecialty of urologic oncology did not differ from other subspecialties, t (181) = −0.68, p = 0.50. There was a significant difference of AM-STAR score between the included studies type (Only RCT vs. RCT & non-RCT vs. Only non-RCT), F (2,180) = 4.98, p = 0.008. Tukey's HSD post hoc test showed that meta-analyses included only RCTs had higher AMSTAR score than those included only non-RCTs, p = 0.002.There was no significant difference of AMSTAR score between the interventional and non-interventional meta-analyses, t (181) = −1.23, p = 0.22. All the other two-group comparison test results are shown in Table 1. Figure 2B shows the AMSTAR results on a per-item basis. Per-item AMSTAR analysis revealed that 3 items had less than 50% adherence out of the 183 included meta-analyses (item 1, ''a priori'' design; item 5, list of studies; item 10, publication bias). Item 4 (gray literature) also had only 51% adherence.

Univariate and multivariate analyses
The 75% percentile of the respective ranges of PRISMA score and AMSTAR score was 24 and 9, respectively. The reporting quality (PRISMA) and methodological quality (AMSTAR) were divided by the cutoff value of 75% percentile into superior quality and inferior quality. The results of univariate and multivariate logistic regression analyses on the PRISMA and AMSTAR scores are presented in Tables 2 and 3, respectively. Univariate regression analyses demonstrated the following factors to be associated with superior reporting quality (PRISMA score ≥ 24) of the published meta-analyses: subspecialty of urologic oncology, funding support, non-interventional meta-analyses, following PRISMA guideline, and ''a priori'' design. Multivariate regression analyses confirmed the following factors to be associated with superior reporting quality (PRISMA score ≥ 24) of the published meta-analyses: funding support and ''a priori'' design.  Univariate regression analyses demonstrated the following factors to be associated with superior methodological quality (AMSTAR score ≥ 9) of the published meta-analyses: number of authors, subspecialty of urologic oncology, following PRISMA guideline, and ''a priori'' design. Multivariate regression analyses confirmed the following factors to be associated with superior methodological quality (AMSTAR score ≥ 9) of the published meta-analyses: following PRISMA guideline, and ''a priori'' design.

DISCUSSION
Our study demonstrates that both reporting and methodological qualities of recently published meta-analyses in major urology journals were generally good. Also, there were no significant variations of the qualities between major urology journals. On average, PRISMA score was 22.74 (84.2%) out of 27 and AMSTAR score was 7.57 (68.8%) out of 11. However, there still may be room for improvement based on the per-item results (Fig. 2). More importantly, several potential predictive factors for superior quality of urological meta-analyses were identified, including funding support, following PRISMA guideline, and ''a priori'' design. Knowledge and identification of variables predictive of high-quality meta-analysis are not only useful to readers, but also would be useful for journal reviewers and editors.
Reporting and methodological qualities of meta-analyses in other medical disciplines were evaluated with similar methods (Liu et al., 2017;Zhang et al., 2016). Zhang et al. (2016) focused on meta-analyses of surgical interventions in year 2013 and showed the mean PRISMA and AMSTAR adherences (by items) were 22.3 and 7.9, respectively. A recent study showed the mean PRISMA and AMSTAR adherences (by items) in the leading gastroenterology and hepatology journals were 20.8 and 7.6, respectively (Liu et al., 2017). Generally speaking, the quality of meta-analyses in major urology journals are good and consistent with previous studies in other medical fields.
Strengths of our study include the focused search and selection of meta-analyses, comprehensive assessment, and planned logistic regression analyses. We only included meta-analyses because they are different from qualitative systematic reviews in several ways and often have a more consistent format. In addition, some of the assessment items only applied to meta-analyses, such as PRISMA items 14,15,16,21,22,23, and AMSTAR items 9 and 10. By excluding systematic reviews without meta-analyses, our results would have more credibility and our conclusions would have a more specific implication. Also, we only focused on meta-analyses published starting from 2011, which is one year and a half after the publication of PRISMA statement (July 2009) . Since the AMSTAR tool was first published in 2007, this timeline setting would possibly minimize the confounding from authors' unavailability of PRISMA checklist and AMSTAR tool themselves.
There are some limitations to our study. First, the cumulative scores calculated from PRISMA checklist and AMSTAR tool may not be valid or truly reflect the reporting and methodological quality. However, at least for now, the scoring method seems to be the best option to quantify the quality of and meta-analyses (Adie et al., 2015;Gagnier & Kellam, 2013;Tunis et al., 2013;Zhang et al., 2016). Second, we limited our search to 10 predetermined major paper-based urology journals, which could cause the omitting of meta-analyses attached the PRISMA checklist (Tewari et al., 2012;Van Die et al., 2014). In addition, having a PRISMA checklist makes the peer review process more efficient and more informed.
Another important predictor of high-quality meta-analyses is ''a priori'' design. As one of the AMSTAR items, ''a priori'' design can predict both the reporting quality and methodological quality. In PRISMA checklist, item 5 is ''protocol and registration'', which can be considered as the higher standard of ''a priori'' design. It is not hard to understand that ''a priori'' design can make sure the researchers have a clear thinking and well-organized action. In addition, having a protocol or ''a priori'' design can partially obligate the authors from post hoc modification of inclusion criteria and analytic methods (Tunis et al., 2013). However, only 8 meta-analyses fulfilled the PRISMA item 5 and only 22 meta-analyses claimed ''a priori'' design. In the medical publication, requiring the protocol and registration information for RCTs is very common. As for systematic reviews, only Cochrane reviews require the authors to publish a peer-reviewed protocol before conducting the review. Previous studies have shown that Cochrane reviews appear to have higher methodological quality than systematic reviews or meta-analyses published in paper-based journals. Another common registration platform for systematic reviews and meta-analyses is PROSPERO (Booth et al., 2012). It would be very difficult for paper-based journals ask for prospective registration or peer-reviewed protocol for every meta-analysis. Attaching a study protocol written ''a priori'' might be a good start (Reeves et al., 2015).

CONCLUSIONS
Reporting and methodological qualities of recently published meta-analyses in major urology journals are generally good, however there are areas for potential improvement. Further improvement could potentially be achieved by strictly adhering to PRISMA guideline and preparing ''a priori'' protocol.

ADDITIONAL INFORMATION AND DECLARATIONS Funding
This work was supported by The Linda and Joel Appel Urologic Oncology Research Fund and The Honickman Family Urologic Research Fund. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.