Six years of systematic literature reviews in software engineering: An updated tertiary study

https://doi.org/10.1016/j.infsof.2011.04.004Get rights and content

Abstract

Context

Since the introduction of evidence-based software engineering in 2004, systematic literature review (SLR) has been increasingly used as a method for conducting secondary studies in software engineering. Two tertiary studies, published in 2009 and 2010, identified and analysed 54 SLRs published in journals and conferences in the period between 1st January 2004 and 30th June 2008.

Objective

In this article, our goal was to extend and update the two previous tertiary studies to cover the period between 1st July 2008 and 31st December 2009. We analysed the quality, coverage of software engineering topics, and potential impact of published SLRs for education and practice.

Method

We performed automatic and manual searches for SLRs published in journals and conference proceedings, analysed the relevant studies, and compared and integrated our findings with the two previous tertiary studies.

Results

We found 67 new SLRs addressing 24 software engineering topics. Among these studies, 15 were considered relevant to the undergraduate educational curriculum, and 40 appeared of possible interest to practitioners. We found that the number of SLRs in software engineering is increasing, the overall quality of the studies is improving, and the number of researchers and research organisations worldwide that are conducting SLRs is also increasing and spreading.

Conclusion

Our findings suggest that the software engineering research community is starting to adopt SLRs consistently as a research method. However, the majority of the SLRs did not evaluate the quality of primary studies and fail to provide guidelines for practitioners, thus decreasing their potential impact on software engineering practice.

Introduction

In 2004, Kitchenham et al. [14] introduced the concept of evidence-based software engineering (EBSE) as a promising approach to integrate academic research and industrial practice in software engineering. Following this paper, Dybå et al. [8] presented EBSE from the point of view of the software engineering practitioner, and Jørgensen et al. [20] complemented it with an account of the teaching aspects of EBSE to university students.

By analogy with evidence-based medicine [25], five steps are needed to practice EBSE:

  • 1.

    Convert the need for information (about the practice of software engineering) into answerable questions.

  • 2.

    Identify, with maximum efficiency, the best evidence with which to answer these questions.

  • 3.

    Appraise the evidence critically: assess its validity (closeness to the truth) and usefulness (practical applicability).

  • 4.

    Implement the results of this appraisal in software engineering practice.

  • 5.

    Evaluate the performance of this implementation.

The preferred method for implementing Steps 2 and 3 is systematic literature review (SLR). Kitchenham [15] adapted guidelines for performing SLRs in medicine to software engineering. Later, using concepts from social science [23], Kitchenham and Charters updated the guidelines [16]. The literature differentiates several types of systematic reviews [23], including the following:

  • Conventional SLRs [23], which aggregate results about the effectiveness of a treatment, intervention, or technology, and are related to specific research questions such as Is intervention I on population P more effective for obtaining outcome O in context C than comparison treatment C? (resulting in the PICOC structure [23]). When sufficient quantitative experiments are available to answer the research question, a meta-analysis (MA) can be used to integrate their results [10].

  • Mapping (or scoping) studies (MS) [1] aim to identify all research related to a specific topic, i.e., to answer broader questions related to trends in research. Typical questions are exploratory, e.g., What do we know about topic T?

Greenhalgh [11] emphasises that evidence-based practice is not only about reading papers and summarising their results in a comprehensive and unbiased way. It involves reading the right papers (those both valid and useful) and then changing behaviour in the practice of the discipline (in our case, software engineering). Therefore, EBSE is not only about performing high-quality SLRs and making them publicly available (Steps 2 and 3). All five steps should be performed for a practice to be considered evidence-based. Nevertheless, SLRs can play an important role in supporting research and education, and informing practice on the impact or effect of technology. Therefore, information about how many SLRs are available in software engineering, where they can be found, which topic areas have been addressed, and the overall quality of available studies can greatly benefit the academic community as well as practitioners.

In this article, we performed a mapping study of SLRs in software engineering published between 1st July 2008 and 31st December 2009. Our goal was to analyse the available secondary studies and integrate our findings with the results of the two previous studies discussed in Section 3. Our work is classified as a tertiary study because we performed a review of secondary studies. The study protocol is presented in Section 3. Sections 4 Data extraction results, 5 Discussion of research questions discuss the extracted data and their analysis. Finally, the conclusions are presented in Section 6.

Section snippets

Previous studies

Two previous tertiary studies have been performed aiming to assess the use of SLRs in software engineering research and, indirectly, to investigate the adoption of EBSE by software engineering researchers [18], [19]. Additionally, our team recently performed a critical appraisal of the SLRs reported in these two studies with respect to the types of research questions asked in published reviews [7]. In this section, we briefly describe these three studies and their relationships.

The first study,

Method

The research group that developed the OS and FE reviews intended to repeat their study at the end of 2009 to “track the progress of SLRs and evidence-based software engineering” [18]. During the 14th International Conference on Evaluation and Assessment in Software Engineering (EASE’2010) in May 2010, at Keele University, we discussed this extension with two members of the group, Pearl Brereton and David Budgen, and they said that the extension had not yet been performed. We informed them of

Data extraction results

A summary of the data collected from the 67 SLRs in the above processes is shown in Table 2. Regarding the nature of the references to the EBSE papers and SLR Guidelines, similarly to the findings reported by Kitchenham et al. [18], [19], all papers that cited the EBSE papers or the guidelines did so as a methodological justification for their study, so we considered all SLRs to be EBSE-positioned.

Table 3 shows the quality scores for each assessment question. We ordered the studies by the final

Discussion of research questions

In this section, we address the research questions presented in Section 3.1. We show the results of our study (SE), compare them with the findings of OS/FE, and integrate the results (OS/FE + SE).

Limitations of this study

Two major problems in SLRs are finding all the relevant studies and assessing their quality. In our study, we employed a mixed process approach to find relevant studies that combined an automatic search in search engines, manual search on relevant journals and conference proceedings, and backward search, that is, searching for relevant studies in the references of previously selected studies. We checked the coverage of our automatic search and only failed to recover one study in a set of 51,

Conclusions

This tertiary study analysed 1455 articles, of which 67 were considered to be systematic literature reviews in software engineering with acceptable quality and relevance. Among these studies, 15 appeared relevant to the undergraduate educational curriculum, 40 appeared of possible interest to practitioners, and 26 were directed mainly to researchers. Furthermore, the 67 studies addressed 24 different software engineering topics, covering 33% (15/46) of the SWEBOK sections.

Our study shows three

Acknowledgements

Fabio Q.B. da Silva holds a research Grant from the Brazilian National Research Council (CNPq) #314523/2009-0. Sérgio Soares is partially supported by CNPq (305085/2010-7) and FACEPE (APQ-0093-1.03/08) André L.M. Santos and Sérgio Soares are partially supported by the National Institute of Science and Technology for Software Engineering (INES), Grants 573964/2008-4 (CNPq) and APQ-1037-1.03/08 (FACEPE). A. César C. França and Cleviton V.F. Monteiro are doctoral students at the Centre for

References (28)

  • T. Dybå

    Evidence-based software engineering for practitioners

    IEEE Software

    (2005)
  • S.M. Easterbrook

    Selecting empirical methods for software engineering research

  • G. Glass

    Integrating findings: the meta-analysis of research

    Review of Research in Education

    (1977)
  • T. Greenhalgh

    How to Read a Paper: The Basics of Evidence-based Medicine

    (2007)
  • Cited by (0)

    View full text