Systematic review in software engineering: where we are and where we should be going

In 2004 Kitchenham et al. first proposed the idea of evidence-based software engineering (EBSE). EBSE requires a systematic and unbiased method of aggregating empirical studies and has encouraged software engineering researches to undertake systematic literature reviews (SLRs) of Software Engineering topics and research questions. As software engineers began to use the SLR technology, they also began to comment on the SLR process itself. Brereton et al (2007) was one of the first papers that commented on issues connected with performing SLRs and many such papers have followed since covering topics such as: The use of SLRs in education; Experiences of novices using SLRs; The adoption of mapping and scoping studies; The repeatability of SLRs; Improving the search and selection processes; Quality assessment of primary studies; Improving aggregation processes.

It therefore seems appropriate to identify the current status of such studies in software engineering, and identify whether there is evidence for revising and/or extending the guidelines for performing systematic literature reviews (Kitchenham and Charters, 2007). This keynote will report the current results of an ongoing systematic literature review that aims: A1: To identify and categorise papers investigating the SLR process and the claims relating to that process; A2: To identify the extent to which the claims of repeatability, lack of bias, and openness are supported; A3: To identify any areas where current guidelines need to be amended or extended to reflect current knowledge of applying SLRs in the context of software engineering.