Six years of systematic literature reviews in software engineering: An updated tertiary study
Introduction
In 2004, Kitchenham et al. [14] introduced the concept of evidence-based software engineering (EBSE) as a promising approach to integrate academic research and industrial practice in software engineering. Following this paper, Dybå et al. [8] presented EBSE from the point of view of the software engineering practitioner, and Jørgensen et al. [20] complemented it with an account of the teaching aspects of EBSE to university students.
By analogy with evidence-based medicine [25], five steps are needed to practice EBSE:
- 1.
Convert the need for information (about the practice of software engineering) into answerable questions.
- 2.
Identify, with maximum efficiency, the best evidence with which to answer these questions.
- 3.
Appraise the evidence critically: assess its validity (closeness to the truth) and usefulness (practical applicability).
- 4.
Implement the results of this appraisal in software engineering practice.
- 5.
Evaluate the performance of this implementation.
The preferred method for implementing Steps 2 and 3 is systematic literature review (SLR). Kitchenham [15] adapted guidelines for performing SLRs in medicine to software engineering. Later, using concepts from social science [23], Kitchenham and Charters updated the guidelines [16]. The literature differentiates several types of systematic reviews [23], including the following:
- •
Conventional SLRs [23], which aggregate results about the effectiveness of a treatment, intervention, or technology, and are related to specific research questions such as Is intervention I on population P more effective for obtaining outcome O in context C than comparison treatment C? (resulting in the PICOC structure [23]). When sufficient quantitative experiments are available to answer the research question, a meta-analysis (MA) can be used to integrate their results [10].
- •
Mapping (or scoping) studies (MS) [1] aim to identify all research related to a specific topic, i.e., to answer broader questions related to trends in research. Typical questions are exploratory, e.g., What do we know about topic T?
Greenhalgh [11] emphasises that evidence-based practice is not only about reading papers and summarising their results in a comprehensive and unbiased way. It involves reading the right papers (those both valid and useful) and then changing behaviour in the practice of the discipline (in our case, software engineering). Therefore, EBSE is not only about performing high-quality SLRs and making them publicly available (Steps 2 and 3). All five steps should be performed for a practice to be considered evidence-based. Nevertheless, SLRs can play an important role in supporting research and education, and informing practice on the impact or effect of technology. Therefore, information about how many SLRs are available in software engineering, where they can be found, which topic areas have been addressed, and the overall quality of available studies can greatly benefit the academic community as well as practitioners.
In this article, we performed a mapping study of SLRs in software engineering published between 1st July 2008 and 31st December 2009. Our goal was to analyse the available secondary studies and integrate our findings with the results of the two previous studies discussed in Section 3. Our work is classified as a tertiary study because we performed a review of secondary studies. The study protocol is presented in Section 3. Sections 4 Data extraction results, 5 Discussion of research questions discuss the extracted data and their analysis. Finally, the conclusions are presented in Section 6.
Section snippets
Previous studies
Two previous tertiary studies have been performed aiming to assess the use of SLRs in software engineering research and, indirectly, to investigate the adoption of EBSE by software engineering researchers [18], [19]. Additionally, our team recently performed a critical appraisal of the SLRs reported in these two studies with respect to the types of research questions asked in published reviews [7]. In this section, we briefly describe these three studies and their relationships.
The first study,
Method
The research group that developed the OS and FE reviews intended to repeat their study at the end of 2009 to “track the progress of SLRs and evidence-based software engineering” [18]. During the 14th International Conference on Evaluation and Assessment in Software Engineering (EASE’2010) in May 2010, at Keele University, we discussed this extension with two members of the group, Pearl Brereton and David Budgen, and they said that the extension had not yet been performed. We informed them of
Data extraction results
A summary of the data collected from the 67 SLRs in the above processes is shown in Table 2. Regarding the nature of the references to the EBSE papers and SLR Guidelines, similarly to the findings reported by Kitchenham et al. [18], [19], all papers that cited the EBSE papers or the guidelines did so as a methodological justification for their study, so we considered all SLRs to be EBSE-positioned.
Table 3 shows the quality scores for each assessment question. We ordered the studies by the final
Discussion of research questions
In this section, we address the research questions presented in Section 3.1. We show the results of our study (SE), compare them with the findings of OS/FE, and integrate the results (OS/FE + SE).
Limitations of this study
Two major problems in SLRs are finding all the relevant studies and assessing their quality. In our study, we employed a mixed process approach to find relevant studies that combined an automatic search in search engines, manual search on relevant journals and conference proceedings, and backward search, that is, searching for relevant studies in the references of previously selected studies. We checked the coverage of our automatic search and only failed to recover one study in a set of 51,
Conclusions
This tertiary study analysed 1455 articles, of which 67 were considered to be systematic literature reviews in software engineering with acceptable quality and relevance. Among these studies, 15 appeared relevant to the undergraduate educational curriculum, 40 appeared of possible interest to practitioners, and 26 were directed mainly to researchers. Furthermore, the 67 studies addressed 24 different software engineering topics, covering 33% (15/46) of the SWEBOK sections.
Our study shows three
Acknowledgements
Fabio Q.B. da Silva holds a research Grant from the Brazilian National Research Council (CNPq) #314523/2009-0. Sérgio Soares is partially supported by CNPq (305085/2010-7) and FACEPE (APQ-0093-1.03/08) André L.M. Santos and Sérgio Soares are partially supported by the National Institute of Science and Technology for Software Engineering (INES), Grants 573964/2008-4 (CNPq) and APQ-1037-1.03/08 (FACEPE). A. César C. França and Cleviton V.F. Monteiro are doctoral students at the Centre for
References (28)
Systematic literature reviews in software engineering – a systematic literature review
Information and Software Technology
(2009)Literature reviews in software engineering – a tertiary study
Information and Software Technology
(2010)Capture–recapture in software inspections after 10 years research – theory, evaluation and application
JSS
(2004)- et al.
Scoping studies: towards a methodological framework
International Journal of Social Research Methodology
(2005) Guide to software engineering body of knowledge
IEEE Computer Society
(2004)- et al.
Evaluation Approaches for Software Architectural Documents: A Systematic Review. IDEAS
(2006) - J. Biolchini et al., Systematic review in software engineering. Technical Report RT-ES 679-05. PESC, COPPE/UFRJ,...
- Centre for Reviews and Dissemination, What are the Criteria for the Inclusion of Reviews on DARE?, 2007...
- Centre for Reviews and Dissemination, About DARE, 2010. <http://www.york.ac.uk/inst/crd/darefaq.htm> (accessed...
Critical Appraisal of Systematic Reviews in Software Engineering: a Tertiary Study, ESEM’2010
(2010)