Critical appraisal of systematic reviews of intervention studies in periodontology using AMSTAR 2 and ROBIS tools

Background Systematic reviews of intervention studies are used to support treatment recommendations. The aim of this study was to assess the methodological quality and risk of bias of systematic reviews of intervention studies in in the field of periodontology using AMSTAR 2 and ROBIS. Material and Methods Systematic reviews of randomized and non-randomized clinical trials, published between 2019 and 2020, were searched at MedLine, Embase, Web of Science, Scopus, Cochrane Library, LILACS with no language restrictions between October 2019 to October 2020. Additionally, grey literature and hand search was performed. Paired independent reviewers screened studies, extracted data and assessed the methodological quality and risk of bias through the AMSTAR 2 and ROBIS tools. Results One hundred twenty-seven reviews were included. According to AMSTAR 2, the methodological quality was mainly critically low (64.6%) and low (24.4%), followed by moderate (0.8%) and high (10.2%). According to ROBIS, 90.6% were at high risk of bias, followed by 7.1% low, and 2.4% unclear risk of bias. The risk of bias decreased with the increased in the impact factor of the journal. Conclusions Current systematic reviews of intervention studies in periodontics were classified as low or critically low methodological quality and high risk of bias. Both tools led to similar conclusions. Better adherence to established reporting guidelines and stricter research practices when conducting systematic reviews are needed. Key words:Bias, evidence-based dentistry, methods, periodontics, systematic review.


Introduction
Systematic reviews (RSs) of intervention studies are considered of high level of scientific evidence, being used to raise evidence that can support treatment recommendations and public health strategies (1).As other study designs, SRs are subject to biases that can compromise their validity and quality of evidence (2).Some tools were developed to assess the methodological quality and risk of bias of SRs, such as AMSTAR 2 (A Measurement Tool to Assess Systematic Reviews 2) (3) an updated version of ASMTAR (Risk of Bias in Systematic Reviews) (4), the Cochrane Collaboration tool for risk of bias of SRs (5).Some overviews in the periodontal field have assessed the methodological quality of SRs through AMSTAR, showing inconstant quality (6)(7)(8)(9).One overview assessed the methodological quality of SRs using the AMSTAR 2 and the risk of bias through ROBIS, and demonstrated very low overall quality (10).Among 23 SRs, only 3 SRs on peri-implantitis therapy had high quality according to AMSTAR 2, and only one were judged as low risk of bias according to ROBIS (10).This low overall quality raised questions about the general quality of the available evidence from RSs in periodontology.Hence, this overview aimed to: 1) describe the characteristics of SRs in periodontology; 2) assess if the certainty of the evidence is reported in these reviews; 3) assess the methodological quality using the AMSTAR 2; 4) assess the risk of bias using the ROBIS.

Material and Methods
This methodological survey was designed and performed following the recommendations of the Cochrane Handbook for Systematic Reviews of Interventions (11) and was reported in accordance with the PRISMA checklist (12).
-Research question What is the methodological quality and risk of bias of the SRs of intervention studies in periodontology published in 2019-2020?-Eligibility criteria Inclusion criteria were SRs of intervention studiesrandomized (RCTs) and non-randomized clinical trials (nRCTs) -with or without meta-analysis, in the field of periodontology, indexed between October 1st, 2019 to October 1st, 2020.SRs that authors classified the studies as having prospective design were included as nRCTs.According to the Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I), nRCTs are cohort studies in which intervention groups were allocated during the usual course of treatment instead of randomization (13).To be consistent, all non-randomized studies, nominated by authors as clinical trials, controlled clinical trials, prospective controlled trials, non-randomized prospective studies, prospective clinical studies, prospective controlled clinical studies and retrospective cohort studies, were classified as nRCTs.Exclusion criteria were: (a) SRs not related to the field of periodontology, (b) narrative or scope reviews, clinical guidelines, editorials or expert opinion papers, SRs of case-control and cross-sectional studies with PECO question, case reports and case series, pilot, in vitro and/ or animal studies.
-Search in databases An expert in SRs (CCM) designed and verified the strategies searches, and one reviewer (AGP) searched the following databases: MedLine (Pubmed), Embase (Elsevier), Web of Science, Scopus, Cochrane Library and LILACS for articles indexed between October 1st, 2019 to October 1st, 2020, with no language restrictions.This time length is enough to represent the current status of the quality of evidence in periodontology in the previous years, as the average time between the last search for a SR and its publication varies between 8 (14) to 15 months (15), and the mean time between the protocol's publication and the SR's publication is about 16 months (16).Grey literature was searched in OpenGrey, GreyLit and Google Scholar.A hand search was performed in the references list of selected articles, and in the main journals of periodontology found in the Journal Citation Reports (JCR) in the category "Dentistry, Oral Surgery and Medicine": Journal of Clinical Periodontology, Journal of Periodontology, Journal of Periodontal Research, International Journal of Periodontics & Restorative Dentistry, Journal of Periodontal and Implant Science, Periodontology 2000.Additional information of search strategies, including search terms, is detailed in the supplementary material (Supplement 1) (http://www.medicinaoral.com/medo-ralfree01/aop/jced_60197_s01.pdf).
-Studies selection Two pairs of independent reviewers screened studies based on titles and abstracts and then full text (AGP and SFF; JRC and LCMC).The reviewers were trained with a set of 10% of studies in each phase.In cases of less than 80% of agreement, additional rounds of training were carried out until reaching the necessary standard for each step.After reviewers achieved at least 80% of agreement, they underwent the screening process with the remaining of studies.The Rayyan platform (17) was used for studies screening.In cases of disagreement, an expert reviewer was consulted (CCM).
-Data extraction and assessment of methodological quality Data extraction, assessment of methodological quality and risk of bias were performed through the AMSTAR 2 (3) and ROBIS (5) tools by four pairs of independent reviewers (AGP and SFF; AGP and LCMC; JRC and SQN; CCM and TPP), using Excel spreadsheet editor.Reviewers were trained by two reviewers (AGP and CCM), the second one with broad experience in syste-matic reviews methodology.Again, the reviewers underwent as many rounds of training as necessary, until reaching 80% of agreement.All disagreements were solved by discussion and consensus.If consensus was not achieved, the principal investigator made the final decision.General data were extracted from the articles, and the list of the extracted data is available in the supplementary material (Supplement 2) (http://www.medicinaoral.com/medoralfree01/aop/jced_60197_s02.pdf).We uploaded the SRs protocols from the registration platform to compare with the published review, and extracted the JCR impact factor and the h-5 index of the journals from the JCR and Google Scholar Metrics, respectively.Disagreements during this step were resolved between the pair of reviewers.If disagreement persisted, the principal investigator was responsible for reaching a final consensus.Two reviews in Mandarin were translated using a translation tool.
-Statistical analysis Data was entered in IBM SPSS Statistics for Windows version 25 (Armonk, NY: IBM Corp.) for descriptive analyses.We calculated the relative and absolute frequencies for categorical variables, and mean, standard deviation and minimum/maximum values were provided for continuous variables.Analyses were performed considering all SRs and stratified by: SRs with RCTs and nRCTs, SRs with RCTs only and the impact factor of the journals (<3, ≥3 <6, ≥6).
-Protocol, register and PRISMA More than a third of the SRs (n=48; 37.8%) did not mention a study protocol.Among the 79 SRs that reported a protocol, 71 (89.9%) and 8 (10.1%) had registered and non-registered protocols, respectively.The most common registration platform was the International Prospective Register of Systematic Reviews (PROSPERO) (n=61; 85.9%).
-AMSTAR 2 The overall AMSTAR 2 methodological quality of all SRs was classified as critically low (n=82, 64.6%), low (n=31, 24.4%), moderate (n=1, 0.8%) and high (n=13, 10.2%) (Table 1, 1 cont.).The items 1 (components of PICO), 5 and 6 (study selection and data extraction in duplicate), 9 (satisfactory assessment of risk of bias), 11 (appropriate methods for meta-analysis), 14 (satisfactory discussion of heterogeneity) and 16 (report of sources of conflict of interest) received positive answers in more than 70% of SRs.The items with the highest percentage of overall negative responses were: 3 (reason for selection of certain study designs; 87.4%), 10 (report of funding sources for the included studies; 67.7%) and 4 (careful search of the literature; 65.4%).Five items considered critical according to AMSTAR 2 had large percentage of negative assessments: 2 (presence of protocol and justification for its modifications; 37.8%), 4 (careful literature search; 65.4%), 7 (list of excluded articles with justifications; 41.7%), 13 (consideration of the risk of bias in individual studies; 40.9%) and 15 (investigation and discussion of the impact of publication bias; 40.2%).
It is important to note that when analyses were performed considering the impact factor of the journal, the overall methodological quality was classified as high in ~30% of SRs in journals with impact factor ≥6.A high percentage of positive answers were also observed in the higher impact factor journals (Table 2-2 cont.-1).No expressive differences were observed when evaluating SRs according to the design of the included studies (Table 1, 1 cont.).
-ROBIS The overall ROBIS evaluations considered 113 (90.6%)SRs to be at high risk of bias, 11 (7.1%) at low risk and 3 (2.4%) at unclear risk of bias (Table 3          No expressive differences were observed when SRs were evaluated according to the design of the included studies (Tables 3).However, the risk of bias decreased with the increase of the impact factor of the journal (Table 4).Detailed ROBIS assessments such as concerns regarding study eligibility criteria, methods used to collect data and appraise studies and the synthesis and findings also decreased with the increase of the impact factor of the journal (

Discussion
The majority of SRs were classified as high risk of bias according to the ROBIS that agreed with the low methodological quality of the AMSTAR 2. It seems that both tools can indicate similar results as they point out in the same direction.This is in accordance with a recent review classifying SRs in dentistry as low and critically low quality (18) A wide variety of methodological deficiencies resulted in the classification of SRs as having high risk of bias.The absence or unjustified changes of the study protocol was the most important issue according to both tools.The prior creation and registration of a protocol is essential for ensuring the transparency of study methods and allowing adequate peer review of the proposed methodology, thus avoiding the selective reporting bias (11).
The deficiency of search strategies was another important identified bias.Search strategies for SRs should be as extensive as possible, without unjustifiable restrictions, including searches in the references of selected studies and in clinical trial registries.Additionally, complementary searches constitute an important source for the identification of potential studies.Its absence or unjustified restrictions increases the possibility of publication, language and selection biases, among others (11).Among the nine SRs that included a librarian on the research team, 77.8% had high methodological quality searches when assessed by the AMSTAR 2, in contrast to 17.8% of high-quality searches in SRs not including librarians.The inclusion of librarians, although not mandatory, is beneficial as it provides guidance at various stages of the research, such as in the processes of designing search strategies and is associated with more reproducible searches and improved methodological reporting in dental medicine SRs (19).
The processes of selection, data extraction and assessment of the risk of bias, which should be ideally carried out independently by more than one reviewer, were presented incompletely in most of the SRs.Cross-checking or duplicate selection processes, data extraction and as-e692 sessment of risk of bias can reduce biases, as well as the potential subjectivity of one single reviewer (20).
In addition to factors associated with methodological processes, the lack of robustness of the results and excessive bias in primary studies also lead to negative classification through the ROBIS assessment.Findings from SRs, especially those with meta-analyses, must be evaluated through complementary tests to assess its robustness, such as sensitivity tests, subgroup analysis, meta-regression and funnel plots (5).Few studies have proven the robustness of their findings, and the absence of such tests can result in false positive inferences in a meta-analysed result, leading the reader to believe in ineffective treatments.
It was reported that 68% of RCTs in the field of dentistry had an unclear or high risk of bias, according to the Cochrane risk of bias domains (21).If SRs do not test the robustness by meta-analytic approaches such as sensitivity analysis and meta-regression, the overall evidence may be biased.The inclusion of non-randomized intervention studies in the SRs might be considered an indication of acceptance of less-than-adequate research designs for intervention studies leading to low methodological quality or high risk of bias classifications.Nevertheless, no expressive differences were observed when SRs were evaluated according to the design of the included studies.The vast majority of SRs were of low and critically low quality when assessed by the AMSTAR 2 and judged as high risk of bias by the ROBIS.Overall, these two instruments led to similar conclusions in 93.7% of the assessments, although they are intended for different purposes.The first one is designed to assess the methodological quality of RS, or if the important aspects of the methods are being full filled (3).The second one can detect the risk of bias, so, although the SRs had full filled one item, it does not mean that is free of bias (5).This high agreement is probably due to the overlapping questions between these instruments (22), as well as the low general methodological quality of the SRs analysed.
The main source of SRs was the collaboration among authors from different continents (26.8%) and most SRs (97.6%) were published in English.This trend demonstrates the globalization of world science with authors from different countries resulting in international partnerships, exchange of knowledge and resources between research groups and a greater visibility of scientific research (23).
Regarding the scope of the journals, 44.9% of SRs were published in general dental journals.This can be partially explained by the high percentage of studies (26%) whose interventions aimed at improving oral hygiene habits (plaque reduction and gingivitis), areas of common interest in most dental specialties.In addition, it is important to note that some periodontology journals are no longer accepting submissions of reviews.It was recently reported that there are no significant differences between moderate/high and low/critically low methodological quality SRs in dentistry regarding publication year, continent, dental specialty and the impact factor of the journal (18).On the contrary, the increase in the impact factor of the journal decreased the risk of bias according to ROBIS in the present study.Few RSs (7.9%) did not mention conflict of interest in the paper at all or did not mention about funding (21.3%).
The presence of financial ties can be associated with positive outcomes in RCTs (24).In addition, a survey of 3,247 scientists funded by the US National Institutes of Health showed that 15.5% admitted to altering a study's design, methods, or results in response to pressure from funding sources (25).Thus, reporting potential conflicts of interest and funding sources is mandatory in scientific publications, as they aim to demonstrate the transparency and impartiality of the researchers who carry out the studies (11).Only a quarter of SRs assessed the certainty of evidence using the GRADE approach.The assessment of the certainty of the evidence is important to help interpreting the results.As it is a more conservative approach, it can help to avoid misleading conclusions (26).Therefore, any SRs of intervention, independent of the field of science, should add the analysis of the certainty of the body of evidence in their methods (26).Methodological and structural variability among systematic reviews have been observed and the quality of some studies is expected to vary (7).Notwithstanding the systematic and stringent approaches, not all systematic reviews are conducted and reported in the same manner and high methodological quality are uncommon according to specific checklists (7)(8)(9)(10).Quality assessments of systematic reviews are quite recent and researches should consider some guidelines when designing, conducting and reporting their reviews.
In the contemporary scientific scenario, it has been speculated that some issues may influence the quality, reliability and bias of currently scientific research such as the pressure for scientific publication, large volume of articles, predatory journals, quality of the peer review process, among others (27)(28)(29)(30)(31)(32)(33).It was also reported that the dental literature has been increasingly reviewed on various topics leading to SRs with questionable clinical or scientific value in terms of up-to-date information to advance knowledge (34).Overall, researches should critically reflect on these issues in order to their scientific production be aligned to core principles of evidence-based dentistry.Guidelines and quality assessment tools may be helpful to identify topics to be improved.Some limitations of the present study should be discussed.It had three pairs of independent reviewers, which may have resulted in different classifications by the e693 peers.However, in order to establish solid classification criteria and to achieve high levels of agreement, four training and calibration sessions were conducted using the guidance documents of AMSTAR 2 (3) and ROBIS (5).
A certain degree of variability in inter-examiner agreement was previously demonstrated (22,35).This methodological review is strong as it is the first one that raised the methodological quality using the new AMSTAR 2 together with ROBIS for risk of bias.Moreover, we extracted data of several characteristics of included SRs that are detailed in the supplementary material.

Conclusions
Most SRs of intervention studies in periodontology were classified as low methodological quality and high risk of bias.Methodological quality increased and risk of bias decreased with the increase in the impact factor of the journals.Although designed for different purposes, both AMSTAR 2 and ROBIS could lead to similar directions.Efforts should be direct to better adherence to reporting guidelines and stricter research practices when conducting SRs.AMSTAR 2 and ROBINS could help the authors to plan the protocol and the reporting of their SRs.

Protocol and register
This study protocol was registered a priori at PROSPERO (#CRD42020215676; "Quality assessment of systematic reviews and meta-analysis of periodontal intervention studies: an overview") and no changes were deemed necessary after the start of the study.

Source of Funding
This study was supported by grants from Conselho Nacional de Desenvolvimento Científico -CNPq, Brazil (grant #302251/2019-7), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior -CA-PES, Brazil and Pró-Reitoria de Pesquisa da UFMG (PIBIT/PRPq), Brazil.Funding agencies had no participation in the research design and data interpretation.

e694
commentary on current publishing trends in the field of temporomandibular disorders and bruxism.J Oral Rehabil.2019;46:1-4.35.Gates M, Gates A, Duarte G, Cary M, Becker M, Prediger B, et al.Quality and risk of bias appraisals of systematic reviews are inconsistent across reviewers and centers.J Clin Epidemiol.2020;125:9-15.
-3 cont.-2).Did the research questions and inclusion criteria for the review include the components of PICO?Did the report of the review contain an explicit statement that the review methods were established prior to conduct of the review and did the report justify any significant deviations from the protocol?Did the review authors report on the sources of funding for the studies included in the review?
9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?

Table 1 :
Methodological quality assessment through AMSTAR 2 according to the type of studies included in the systematic reviews.If meta-analysis was performed did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?
e683 12.15.If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?16.Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?

Table 1 cont . :
Methodological quality assessment through AMSTAR 2 according to the type of studies included in the systematic reviews.Did the research questions and inclusion criteria for the review include the components of PICO?
3. Did the review authors explain their selection of the study designs for inclusion in the review?

Table 2 :
Methodological quality assessment through AMSTAR 2 according to the impact factor of the journals.
11.If meta-analysis was justified did the review authors use appropriate methods for statistical combination of results?(Only complete this item if meta-analysis of other data synthesis techniques were reported)12.If meta-analysis was performed did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?

Table 2 cont . :
Methodological quality assessment through AMSTAR 2 according to the impact factor of the journals.

Table 2 cont.-1:
Methodological quality assessment through AMSTAR 2 according to the impact factor of the journals.
Q2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?

Table 3 :
Risk of bias assessment through ROBIS according to the type of studies included in the systematic reviews.Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?

Table 3 cont . :
Risk of bias assessment through ROBIS according to the type of studies included in the systematic reviews.

Table 3 cont.-1:
Risk of bias assessment through ROBIS according to the type of studies included in the systematic reviews.

Table 3 cont.-2:
Risk of bias assessment through ROBIS according to the type of studies included in the systematic reviews.

Table 4 :
Risk of bias assessment through ROBIS according to the impact factor of the journals.

Table 4 cont . :
Risk of bias assessment through ROBIS according to the impact factor of the journals.Q1.4 Were any restrictions in eligibility criteria based on study characteristics appropriate (e.g.date, sample size, study quality, outcomes measured)?Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?Were sufficient study characteristics available for both review authors and readers to be able to interpret the results?

Table 4 cont.-1:
Risk of bias assessment through ROBIS according to the impact factor of the journals.Was the synthesis appropriate given the nature and similarity in the research questions, study designs and outcomes across included studies?

Table 4 cont.-2:
Risk of bias assessment through ROBIS according to the impact factor of the journals.