Evidence base for recommendations for writing evidence‐based syntheses
Abstract
Objectives
This is a protocol for a Cochrane Review (methodology). The objectives are as follows:
To identify and assess studies of the effectiveness of interventions to create and implement recommendations for the writing of narrative summaries of evidence syntheses. We will do this separately for systematic reviews and CPGs.
Background
Health professionals rely on evidence synthesis, such as clinical practice guidelines (CPGs) and systematic reviews, in their day‐to‐day practice. Therefore, it is crucial for evidence syntheses to be written clearly and accurately, both for the professionals and consumers. There are several guidance documents for writing different sections of systematic reviews or CPGs, such as those from the National Institute for Health and Care Excellence (NICE) guidelines (NICE 2020) and American Academy of Family Physicians (AAFP) guidelines (AAFP 2017) for the recommendations in CPGs, PLEACS for plain language summaries in Cochrane Reviews (Pitcher 2022), author guidance for writing reviews from the Cochrane Effective Practice and Organisation of Care (EPOC) Group and for choosing images for sharing Cochrane news or evidence (Cochrane 2020), and GRADE guidelines on informative statements to communicate the findings of systematic reviews (Santesso 2019). However, many guidance documents do not report the evidence behind them or their effectiveness. The Cochrane Handbook of Systematic Reviews of Interventions provides guidance on how to write the conclusions of a systematic review but advises authors not to make recommendations (Higgins 2022). The aim of this Cochrane Methodology Review is to synthesize current knowledge about writing textual summaries of evidence synthesis and textual recommendation for health practice.
Description of the methods being investigated
Health guidance, in terms of expert opinion, has existed as long as medicine itself, but what we now perceive as a health guideline is closely connected to, and has developed with, evidence‐based medicine since the late 1980s (Eddy 1990). A health guideline summarizes the highest quality, up‐to‐date evidence and defines decision options and outcomes, with strength of recommendation usually being worded with a single verb, or graded and indicated for each recommendation (CEMB 2009; Guyatt 2008).
Likewise, a systematic review synthesizes the data relevant to a health question and produces an answer to the question, with critical review of the quality of the included studies and conclusions about the answer to the posed question. Language is important in clear communication of findings from CPGs and systematic reviews so that they are understood and used in practice. It has been shown that the use of deontic terms (i.e. words such as 'must', 'should' and 'may') are perceived to have different levels of obligation when used in CPGs (Lomotan 2010). However, there is a lack of evidence about which wording is superior in conveying the strength of clinical recommendation (Akl 2012).
One systematic meta‐review showed that easily understandable guidelines have a greater chance of implementation (Francke 2008). This is in line with the findings of a realist review of guideline uptake, which favours simple, clear, and persuasive language and format (Kastner 2015).
How these methods might work
By having clear writing guidelines and following principles of good written communication and clearly conveying the strength of evidence, CPGs and systematic reviews are expected to give clear answers to questions asked and be implementable in practice. Writing standards include avoiding highly technical language, using active verbs, avoiding a negative approach and personalizing the message (Coulter 1998; Grimshaw 2001; Michie 2005). When a NICE public health guideline was rewritten by applying these standards, patients had stronger intentions to implement its recommendations, had more positive attitudes towards them and greater behavioural control over their implementation (Gupta 2016).
Why it is important to do this review
A large amount of experts' time and resources are invested into making a CPG or systematic review. Their implementation in everyday practice has its barriers and facilitators, with lack of clarity being one of the significant barriers (Correa 2020). This review will provide current evidence for writing recommendations in CPGs and conclusions and summaries in systematic reviews, and thus help create better guidance for writing textual summaries of the evidence synthesis and recommendations from evidence synthesis. The review will also help identify areas where further research is needed to improve the understandability and usability of textual summaries of evidence syntheses. In this way, evidence synthesis in health will be more applicable in everyday practice, both by medical professionals and patients, resulting in a higher number of users, better clinical outcomes, and higher patient and medical personnel satisfaction.
Objectives
To identify and assess studies of the effectiveness of interventions to create and implement recommendations for the writing of narrative summaries of evidence syntheses. We will do this separately for systematic reviews and CPGs.
Methods
Criteria for considering studies for this review
Types of studies
We will include randomized trials, but also other study designs, considering that this research topic is interdisciplinary, so it may involve aspects of social sciences and humanities. These will include, but will not be limited to:
-
non‐randomized controlled study designs, such as controlled before‐after studies, interrupted time series and regression discontinuity designs;
-
non‐controlled observational studies, such as before‐after studies.
There will be no date or language restrictions. We will exclude studies without evaluation of data.
Types of data
We will include studies which have measured the effects of one or more interventions to change the use of, or intention to use, textual summaries of evidence syntheses or recommendations in CPGs or systematic reviews in practice.
We will exclude studies dealing with summary of findings tables, as we consider them tabular and not textual summaries. In this way, there will be no overlap with the ongoing Cochrane Methodology Review "Summary of findings tables for communicating key findings of systematic reviews" (Conway 2017).
Textual summaries will be defined as the text describing the findings or recommendations (or both) in health evidence syntheses. Examples of such textual summaries include, but are not limited to:
-
scientific abstracts;
-
plain language summaries and conclusions in Cochrane Reviews; and
-
professional and plain language summaries and recommendations in CPGs.
The participants will be any medical professionals, including medical students, and patients who either write or read evidence syntheses in CPGs or systematic reviews.
We will consider for inclusion any intervention which directly or indirectly affects participants' attitude, opinions, knowledge or understanding of textual summaries of evidence synthesis, as well as intention to change behaviour or actual change in behaviour related to textual summaries of evidence syntheses in CPGs or systematic reviews. Examples include interventions to increase the understanding or readability of the textual summaries or recommendations, including computerized textual analysis.
Types of methods
We will assess comparison of outcomes in intervention versus non‐intervention groups or before versus after the intervention. We will assess the groups for baseline comparability, such as age, gender and professional level, as well as other factors which may have influence on the outcome levels.
Types of outcome measures
We will use different outcome measures, depending on the types of studies included and unit of analysis. If the unit of analysis is textual, examples include the results of textual analysis of the recommendations, such as computerized text analysis using LIWC (LIWC2015, liwc.app/static/documents/LIWC2015%20Manual%20-%20Operation.pdf; Pennebaker Conglomerates, Austin, Texas, USA) or IBM Watson Tone Analyzer (www.ibm.com/watson). On the text level, we will consider outcomes such as change in the number of words or length, and change in formulation, readability or sentiment. If the unit of analysis is a person, outcome measures will be attitude, behaviour, opinion, intention to use or actual use of the textual summary.
Primary outcomes
-
Effectiveness of interventions to improve the writing of textual summaries and recommendations in evidence syntheses. These may include, but will not be restricted to:
-
application of textual (narrative) synthesis of CPGs and systematic reviews in everyday practice;
-
level of readability and intelligibility of the textual summaries of evidence synthesis;
-
consumer's satisfaction with textual summaries of evidence synthesis;
-
intention to use textual summaries of evidence synthesis; and
-
language analysis of textual summaries of evidence synthesis.
-
Secondary outcomes
Considering the lack of research in the area, we will consider all outcomes measured in the included studies.
Search methods for identification of studies
We will focus on electronic searches of article databases and reference lists of articles included in the review. We will have no date or language restrictions. Two review authors (JM, LU) will independently assess the retrieved documents for eligibility, initially reviewing titles and abstracts, and, if the publication is eligible, will review the full text. Any disagreements will be resolved by a third review author (RT).
Electronic searches
We will use the following bibliographical databases for the search of the literature: the Cochrane Library, MEDLINE, SCOPUS, ERIC, Web of Science and PsycINFO. We created the initial search strategy for MEDLINE in collaboration with a librarian, which we will adapt for other databases (Appendix 1).
Searching other resources
We will search the reference lists of included studies; the World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP), including the ClinicalTrials.gov; Open Science Framework; Prospero database; Base‐search.net; Google Scholar; Opengrey.org; Campbell Collaboration Library and Science.gov databases. We will also search the websites of professional societies who write guidelines/systematic reviews, and preprints via medRxiv.org, Lancet Preprints and Preprints.org.
Data collection and analysis
After our execution of the search strategies, we will collate the identified records (titles and available abstracts) in an EndNote database for removal of duplicates. We will use this data set for the screening of records and extraction of data from included articles or sources.
Selection of studies
Two review authors (JM and LU) will independently screen titles and abstracts of articles retrieved by our searches. We will retrieve the full‐text articles for those studies which are evaluated as potentially relevant. Two review authors (JM and LU) will independently examine the full text of the eligible articles for eligibility, using a priori defined eligibility criteria. We will resolve any disagreements by consulting a third review author (RT). We will list studies excluded at this stage in the 'Characteristics of excluded studies' table.
We will record the study selection process in sufficient detail to produce a PRISMA flow diagram.
We will collate multiple reports of the same study, so that each study, rather than each report, is the unit of interest in the review.
Data extraction and management
Two review authors will independently extract data using EndNote (Clarivate). Two other review authors will compare the two sets of extracted data against each other, and identify any disagreements, which will then be resolved by consensus. The data will be entered into a specially designed form, which will be developed. The review authors will not be blinded to the authors, interventions or results of the included studies. We will extract the following data and compile a 'Characteristics of included studies' table.
-
Study design (e.g. randomized trial, controlled before‐after study, etc.), date and length of follow‐up.
-
Participants (type of participants (students, health professionals, patients, policymakers, etc.), sample size, inclusion and exclusion criteria, demographic characteristics of participants (age, sex, country of origin, ethnicity, gender, field of research, professional, academic or research experience)).
-
Setting (type of institution or broader setting for the intervention, geographical location).
-
Intervention (type of intervention, duration of interventions, comparisons).
-
Outcomes (description of outcomes of interest, including the timing and method of outcome measurements).
If we find outcome measures that were not anticipated, we will extract data for these. We will pilot our data extraction form before use. If we find data in graphical format, we will contact study authors to request numbers. If we do not manage to obtain numbers, we will use Plot Digitizer software to extract data from figures (Jelicic Kadic 2016).
Assessment of risk of bias in included studies
For randomized trials, we will use the Cochrane RoB 2 tool for randomized trials (Higgins 2022; Sterne 2019). Our effect of interest will be the effect of adherence (per protocol). We will tabulate the risk of bias for each of the domains below, and provide a judgement of 'low risk of bias', 'some concerns' or 'high risk of bias', for our key outcomes.
-
Bias arising from the randomization process
-
Bias due to deviations from intended interventions
-
Bias due to missing outcome data
-
Bias in measurement of the outcome
-
Bias in selection of the reported result
We will use the variants of RoB 2 for cluster‐RCTs and cross‐over RCTs if we identify trials with these study designs (Sterne 2019).
For RoB 2, we will consider the overall risk of bias to be 'low' if all domains are at 'low risk'; 'some concerns' if at least one domain is of 'some concern' and no domain is at 'high' risk of bias; and 'high' if there is at least one domain considered at 'high', or several domains at 'some concerns' (Higgins 2022; Sterne 2019).
For other types of studies, we will use the ROBINS‐I tool (Higgins 2022; Sterne 2016), which consists of the following domains, and reach domain‐level judgements as 'low', 'moderate', 'serious' or 'critical' risk of bias, for our key outcomes.
-
Bias due to confounding
-
Bias in selection of participants into the study
-
Bias in classification of interventions
-
Bias due to deviations from intended interventions
-
Bias due to missing data
-
Bias in measurement of the outcome
-
Bias in selection of the reported result
Our effect of interest will be the effect of adherence (per protocol). For ROBINS‐I, we will consider the overall risk of bias to be 'low' if all domains are at 'low risk'; 'moderate' if all domains are at 'low risk' or 'moderate risk', 'serious' if there is at least one domain judged as serious risk but not at 'critical risk' of bias in any domain of bias; and 'critical' if there is at least one domain considered to be at 'critical risk' of bias (Higgins 2022; Sterne 2016).
We will address the risk of bias in a meaningful and transparent way, using appropriate instruments for the specific study features.
We will record each piece of information extracted for the risk‐of‐bias tools together with the source of this information in the source document. Two review authors (JM and RT) will independently test data collection forms and assessments of the risk of bias on a sample of articles. They will not be blinded to the names of the authors, institutions, journals or results of a study.
In assessing the certainty of the evidence with GRADE, we will take the assessment of the risk of bias into account.
Measures of the effect of the methods
As the starting point, we will identify the data type for each of the outcome measurements. We will analyze dichotomous data as risk ratios (RRs) with 95% confidence intervals (CIs). We will analyze continuous data as mean differences (MDs) for trials that measure outcomes the same way. For trials that measure the same outcome with a different method, we will use standardized mean differences (SMDs) to combine the results. For data reported as medians and interquartile ranges (IQR), we will provide a narrative description.
We will undertake meta‐analyses only when the research questions and the studies are sufficiently similar for meaningful pooling.
Unit of analysis issues
The unit of analysis will be the individual study participant or the individual textual document (CPG or systematic review) or its part.
Dealing with missing data
For continuous outcomes, we will calculate the MD or SMD based on the number of participants analyzed without and with (or before and after) intervention. If the number of participants analyzed is not presented for each time point, we will use the number of participants in each group at baseline. For missing summary data, we will use the available data to calculate relevant data, such as calculating missing standard deviations from other statistics (standard errors, CIs or P values), according to the methods recommended in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2022). Whenever possible, we will contact the original investigators to request missing data. We will address the possible impact of missing data that could not be recovered in the discussion of the review.
Assessment of heterogeneity
We plan to test results for heterogeneity across studies using the I2 statistic where an I2 value of 50% to 80% = moderate and worthy of investigation; 80% to 95% = severe and worthy of understanding; 95% to 100% = aggregate with major caution (Thorlund 2012).
Assessment of reporting biases
We will examine within‐study selective outcome reporting as a part of the overall risk of bias assessment and check for protocols of published articles to compare their outcomes. We will create funnel plots to assess small‐study biases (such as publication bias) if we identify at least 10 studies for each analysis. If we find asymmetry of the funnel plot either by visual inspection (Palmer 2008; Peters 2008) or statistical tests (Egger 1997; Harbord 2006), we will address that issue in our interpretation of results. For the statistical analysis, we will use R (R).
Data synthesis
Two review authors (JM, LU) will enter the data into Review Manager Web software (RevMan Web 2022). We will synthesize data by conducting a meta‐analysis using a random‐effects model on the data from suitable studies. If meta‐analysis is not possible, we will present the analysis by grouping the findings according to thematic concepts, such as the type of textual summary of evidence synthesis assessed (abstract, plain language summary, etc.), the target population (patients, clinicians, policymakers, etc.), the type of outcome measure and the type of assessment. Our primary analysis of RCTs will include results that have been judged at 'low risk of bias' or 'some concerns' by RoB 2; for other study types, our main analysis will include results with low, moderate, or serious risk of bias as judged by ROBINS‐I (Assessment of risk of bias in included studies). We will not pool data from RCTs and non‐randomized studies.
We will report measures of intervention effect from the studies that were modified for potential confounding variables over reported estimations that were not adjusted for potential confounding. If the studies have used multiple follow‐up periods, we will use data from the longest study follow‐up. We will synthesize findings by outcome, and, within the review, we will synthesize effects by comparison. We will include a summary of findings table to present the outcomes identified in the review. The table will be generated according to the recommendations of the Cochrane Handbook for Systematic Reviews of Interventions and will include a list of up to seven major outcomes in the review, a description of the intervention effect, the number of participants for each outcome and the GRADE for the certainty of the overall body of evidence for each outcome (Higgins 2022).
Subgroup analysis and investigation of heterogeneity
We will analyze evidence syntheses in systematic reviews and recommendations in guidelines as separate subgroups. If applicable, we will independently analyze the studies conducted with student, consumer (patient), health professionals and policymakers' population, and report the SMD for each analysis. For data in textual format, we will analyze the texts separately according to populations they presented.
Sensitivity analysis
For decisions which were arbitrary or unclear, we will repeat the primary analysis with alternative decisions or ranges of values to determine if the findings depend on such decisions.
Assessment of bias in conducting the systematic review
We will conduct the review according to this published protocol, and report any deviations from it in the 'Differences between protocol and review' section of the review.
