In PracticeSoftware-testing education: A systematic literature mapping
Introduction
“Software is eating the world” (Andreessen, 2018). In other words, software systems have penetrated almost all industries and all aspects of our personal and professional lives. Furthermore, many industrial sources are reporting that software systems are getting increasingly complex (Khushu, 2019; Algaze, 2017; Etheredge, 2018). With the increasing complexity and scale of software systems, there is an ever-increasing demand for sophisticated and cost-effective software quality assurance. Software testing is a fundamental, and also the most widespread, activity to assure the quality of software systems. A 2013 study by Cambridge University (Britton et al., 2013) reported that the global cost of locating and removing defects from software systems has risen to $312 billion annually, and removing defects comprises, on average, about half of the development costs of a typical software project.
To meet the ever-growing demand for cost-effective software testing, there is the increasing need for a highly-skilled software testing work-force in the industry. But in the non-peer-reviewed literature (grey-literature) such as online blogs and articles, many industrial sources are also reporting the shortage of software testers, e.g., (ComputerWeekly, 2019, Murrary, 2019, uTest Community Management 2010). Furthermore, in the context of software testing talent shortage, it is often recruiting “quality” software testers (i.e., with the right skillset), not just recruiting “quantity” (number of people available in the job market) (Baker, 2019) that is the challenge.
To address the above needs and to train more highly-skilled software testers, many software engineering (SE) and computer science (CS) university programs worldwide have started to include software testing in their education curricula. This is achieved by either including distinct and separate software testing courses, or alternatively blending (integrating) software testing concepts into programming or other courses (Association for Computing Computing Machinery (ACM), 2019). To share their experience from such efforts, university educators write and publish papers about their software-testing education activities. Papers are written in this area to discuss and present more effective pedagogical approaches (e.g., how to better teach testing), new course proposals (e.g., course designs), and specific tools for testing education. Also, based on the educators’ experience, many papers have identified a large number of challenges which educators and students could face when teaching and learning about testing, as well as insights into how to overcome those challenges.
Given such a large body of experience and knowledge in the area of software-testing education, there is the need for a systematic review in this area, since it is often not possible for the individual educator to study all the papers and to synthesize all the evidence presented. For example, for a new educator who wants to teach a (new) course in software testing (in her/his university), it would be very valuable to know, before teaching, about the challenges faced by educators when teaching testing and also about insights on how to overcome those challenges. Thus, it would be useful to synthesize and summarize reported experience and evidence, as well as research topics and research questions (RQs) in this area. Our objective and goal in this paper are to provide such a synthesis and summary.
To address the above goal, we conducted a systematic literature mapping (SLM) to synthesize what the community of educators has published on this topic. Based on a systematic SLM process, we systematically select a pool of 204 papers, and by investigating nine RQs, we categorize and analyse various aspects of the subject under study.
The contributions of this review paper are three-fold:
- •
The classification of the studies in this area, performed through a systematic mapping, via RQs 1-8
- •
The synthesis of challenges faced during testing education (via RQ 9.1) together with the synthesis of insights (recommendations) for testing education (via RQ 9.2).
- •
The development of an index (repository) of the studies in this area, which is accessible in an online Google spreadsheet (goo.gl/DcEpMv)
The remainder of this paper is structured as follows. Section 2 provides background and related work. Section 3 describes the research method, and then the design and execution phases of the SLM. Section 4 presents the results of the literature review. Section 5 summarizes the findings and discusses the lessons learned. Finally, in Section 6, we draw conclusions, and suggest areas for future research.
Section snippets
Background and related work
In this section, we provide a brief overview on the state of software-testing education in universities versus software testing training in industry. We then briefly review related work, i.e., existing survey (review) papers in the areas of software engineering and software-testing education.
Design and execution of the SLM
Our research method in this paper is SLM. We present an overview to our SLM process, the research questions of the SLM, and then other aspects related to the design and execution of the SLM.
Using the well-known guidelines for conducting SMS and SLR studies in SE (e.g., (Petersen et al., 2015; Wohlin, 2014; Petersen et al., 2008; Kitchenham and Charters, 2007)) and also based on our past experience in SLR studies, e.g., (Zhi et al., 2015; Garousi et al., 2015; Doğan et al., 2014; Häser et al.,
Findings of the SLM
Through Section 4, we present results of the study's RQs.
Discussions
In this section, we first summarize the research findings and discuss implications and recommendations for educators (Section 5.1). Then, we present suggestions for further education research in this area in Section 5.2, and finally we discuss potential threats to validity of this review (Section 5.3).
Conclusions and future work
We conducted a systematic literature mapping (SLM) to identify the state-of-the-art in the area of software-testing education. Our SLM provides a classification of studies in this area, a synthesis of both challenges faced during testing education and insights for testing education, and an index to studies in this area. All three contributions can benefit educators in the design, delivery and assessment of software testing courses in university settings, and can provide a foundation of previous
Pool of studies in the systematic literature mapping
[P1] S. Tiwari, V. Saini, P. Singh, and A. Sureka, ``A case study on the application of case-based learning in software testing,'' in Proceedings of Innovations in Software Engineering Conference, 2018, p. 11. [P2] W. Zheng, Y. Bai, and H. Che, ``A computer-assisted instructional method based on machine learning in software testing class,'' Computer Applications in Engineering Education, vol. 26, no. 5, pp. 1150-1158, 2018. [P3] K. Buffardi and S. H. Edwards, ``A formative study of influences on
CRediT authorship contribution statement
Vahid Garousi: Conceptualization, Methodology, Data curation, Validation, Writing - original draft. Austen Rainer: Methodology, Data curation, Validation, Writing - original draft. Per Lauvås: Conceptualization, Methodology, Data curation, Validation, Writing - original draft. Andrea Arcuri: Conceptualization, Methodology, Data curation, Validation, Writing - original draft.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgment
Andrea Arcuri is supported by the Research Council of Norway (project on Evolutionary Enterprise Testing, grant agreement No 274385). We thank the anonymous reviewers and the editor for their constructive comments.
Vahid Garousi is a Senior Lecturer (Associate Professor) of Software Engineering in Queen's University Belfast, Northern Ireland, UK. He has an international profile, as he previously worked as an academic and consultant in three continents: the Netherlands (2017-2019), Turkey (2015-2017), and Canada (2001-2014). Dr. Garousi received his Ph.D. in Software Engineering in Carleton University, Canada, in 2006. His research expertise are: software engineering, software testing, empirical studies,
References (107)
- et al.
Gamification in software engineering education: A systematic mapping
J. Syst. Softw.
(2018) - et al.
Graphical user interface (gui) testing: systematic mapping and repository
Inf. Softw. Technol.
(2013) - et al.
On automated grading of programming assignments in an academic institution
Comput. Edu.
(2003) - et al.
Six years of systematic literature reviews in software engineering: An updated tertiary study
Inf. Softw. Technol.
(2011) - et al.
Web application testing: a systematic literature review
J. Syst. Softw.
(2014) - et al.
When and what to automate in software testing? A multivocal literature review,
Inf. Softw. Technol.
(2016) - et al.(2016)
- et al.
A survey of software testing practices in Canada
J. Syst. Softw.
(2013) - et al.
A systematic mapping study of web application testing
Elsevier J. Inf. Softw. Technol.
(2013) - et al.
Software test maturity assessment and test process improvement: A multivocal literature review
Inf. Softw. Technol.
(2017)
Systematic literature reviews in software engineering–a tertiary study
Inf. Softw. Technol.
A systematic review of UML model consistency management
Inf. Softw. Technol.
Guidelines for conducting systematic mapping studies in software engineering: An update
Inf. Softw. Technol.
Cognitive load during problem solving: effects on learning
Cogn. Sci.
Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies
Systematic Rev. J. Artic.
A survey of automated assessment approaches for programming assignments
Comput. Sci. Edu.
A systematic review of the application and empirical investigation of searchbased test case generation
IEEE Trans. Softw. Eng.
Production programming in the classroom
ACM SIGCSE Bull.
Dereferee: Exploring pointer mismanagement in student code
ACM SIGCSE Bull.
Why software is eating the world
Wall Str. J
A survey of the literature in computer science education since curriculum'68
Commun. ACM
Constructivism in computer science education
J. Comput. Math. Sci. Teach.
Revealing the programming process
ACM SIGCSE Bull.
Dimensions of software engineering course design
Teaching for quality learning at university: What the student does
McGraw-hill education (UK)
The act of discovery
Harvard Edu. Rev.
Effective and ineffective software testing behaviors by novice programmers
A survey of methods used to evaluate computer science teaching
ACM SIGCSE Bull.
A systematic survey of games used for software engineering education
Mod. Appl. Sci.
Factors limiting industrial adoption of test driven development: A systematic review
How is exploratory testing used? A state-of-the-practice survey
Experiences in using an automated system for improving students’ learning of computer programming
The Handbook of Research Synthesis and Meta-Analysis
Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies
BMC Med. Res. Method.
The CDIO approach,
Rethinking Engineering Education
Synthesizing evidence in software engineering research
Recommended Steps for Thematic Synthesis in Software Engineering
Proc. International Symposium on Empirical Software Engineering and Measurement
Research synthesis in software engineering: A tertiary study
Inf. Softw. Technol.
A systematic mapping study on game-related methods for software engineering education
Inf. Softw. Technol.
An automated learning system for Java programming
IEEE Trans. Educ.
A survey of evidence for test-driven development in academia
Acm SIGCSE Bull.
Selecting Empirical Methods for Software Engineering Research
Adding software testing to programming assignments
Cited by (0)
Vahid Garousi is a Senior Lecturer (Associate Professor) of Software Engineering in Queen's University Belfast, Northern Ireland, UK. He has an international profile, as he previously worked as an academic and consultant in three continents: the Netherlands (2017-2019), Turkey (2015-2017), and Canada (2001-2014). Dr. Garousi received his Ph.D. in Software Engineering in Carleton University, Canada, in 2006. His research expertise are: software engineering, software testing, empirical studies, action-research, and industry-academia collaborations. Dr. Garousi was selected as a Distinguished Speaker for the IEEE Computer Society from 2012 to 2015. He is a member of the IEEE and the IEEE Computer Society, and is also a licensed professional engineer (PEng) in the Canadian province of Alberta. In parallel to his academic career, he is a practicing software engineering consultant and coach, and has provided consultancy and corporate training services in several countries in the areas of software testing and quality assurance, model-driven development, and software maintenance. During his career so far, Vahid has served as the principal investigator (PI) for several research grants and has been active in initiating a number of major R&D software engineering projects with software companies in several countries including Canada and Turkey. He has been involved as an organizing or program committee member in many international conference, such as ICST, ICSP, CSEE&T, MoDELS and the Turkish National Software Engineering Conference. Among his awards is the prestigious Alberta Ingenuity New Faculty Award in June 2007.
Austen Rainer is a Professor in the School of Electronics, Electrical Engineering and Computer Science at Queen's University Belfast in Northern Ireland. He co-authored the first disciple-specific book on case study research in software engineering, and co-founded Software Innovation NZ, the national network of software researchers in New Zealand. He has multidisciplinary teaching and research interests, with particular focus on team innovation, human values and the impact of software and software engineering on society.
Per Lauvås is an associate professor at Kristiania University College in Oslo where he has been a faculty member since 2013. Prior to that, he worked as a software developer for nine years after graduating from the University of Oslo in 2004. Per teaches courses within software development. His main research interests are IT education and didactics of informatics.''
Andrea Arcuri is a Professor of Software Engineering at Kristiania University College, Oslo, Norway. His main research interests are in software testing, especially test case generation using evolutionary algorithms. Having worked 5 years in industry as a senior engineer, a main focus of his research is to design novel research solutions that can actually be used in practice. Dr. Arcuri is the main-author of EvoMaster and a co-author of EvoSuite, which are open-source tools that can automatically generate test cases using evolutionary algorithms. He received his Ph.D. in software testing from the University of Birmingham, UK, in 2009.