In Practice
Software-testing education: A systematic literature mapping

https://doi.org/10.1016/j.jss.2020.110570Get rights and content

Highlights

  • A systematic literature mapping (SLM) of 204 papers in software-testing education.

  • We review a summary of pedagogical approaches (how to best teach testing).

  • We review the challenges in testing education and insights on how to overcome those.

Abstract

Context

With the rising complexity and scale of software systems, there is an ever-increasing demand for sophisticated and cost-effective software testing. To meet such a demand, there is a need for a highly-skilled software testing work-force (test engineers) in the industry. To address that need, many university educators worldwide have included software-testing education in their software engineering (SE) or computer science (CS) programs. Many papers have been published in the last three decades (as early as 1992) to share experience from such undertakings.

Objective

Our objective in this paper is to summarize the body of experience and knowledge in the area of software-testing education to benefit the readers (both educators and researchers) in designing and delivering software testing courses in university settings, and to also conduct further education research in this area.

Method

To address the above need, we conducted a systematic literature mapping (SLM) to synthesize what the community of educators have published on this topic. After compiling a candidate pool of 307 papers, and applying a set of inclusion/exclusion criteria, our final pool included 204 papers published between 1992 and 2019.

Results

The topic of software-testing education is becoming more active, as we can see by the increasing number of papers. Many pedagogical approaches (how to best teach testing), course-ware, and specific tools for testing education have been proposed. Many challenges in testing education and insights on how to overcome those challenges have been proposed.

Conclusion

This paper provides educators and researchers with a classification of existing studies within software-testing education. We further synthesize challenges and insights reported when teaching software testing. The paper also provides a reference (“index”) to the vast body of knowledge and experience on teaching software testing. Our mapping study aims to help educators and researchers to identify the best practices in this area to effectively plan and deliver their software testing courses, or to conduct further education-research in this important area.

Introduction

“Software is eating the world” (Andreessen, 2018). In other words, software systems have penetrated almost all industries and all aspects of our personal and professional lives. Furthermore, many industrial sources are reporting that software systems are getting increasingly complex (Khushu, 2019; Algaze, 2017; Etheredge, 2018). With the increasing complexity and scale of software systems, there is an ever-increasing demand for sophisticated and cost-effective software quality assurance. Software testing is a fundamental, and also the most widespread, activity to assure the quality of software systems. A 2013 study by Cambridge University (Britton et al., 2013) reported that the global cost of locating and removing defects from software systems has risen to $312 billion annually, and removing defects comprises, on average, about half of the development costs of a typical software project.

To meet the ever-growing demand for cost-effective software testing, there is the increasing need for a highly-skilled software testing work-force in the industry. But in the non-peer-reviewed literature (grey-literature) such as online blogs and articles, many industrial sources are also reporting the shortage of software testers, e.g., (ComputerWeekly, 2019, Murrary, 2019, uTest Community Management 2010). Furthermore, in the context of software testing talent shortage, it is often recruiting “quality” software testers (i.e., with the right skillset), not just recruiting “quantity” (number of people available in the job market) (Baker, 2019) that is the challenge.

To address the above needs and to train more highly-skilled software testers, many software engineering (SE) and computer science (CS) university programs worldwide have started to include software testing in their education curricula. This is achieved by either including distinct and separate software testing courses, or alternatively blending (integrating) software testing concepts into programming or other courses (Association for Computing Computing Machinery (ACM), 2019). To share their experience from such efforts, university educators write and publish papers about their software-testing education activities. Papers are written in this area to discuss and present more effective pedagogical approaches (e.g., how to better teach testing), new course proposals (e.g., course designs), and specific tools for testing education. Also, based on the educators’ experience, many papers have identified a large number of challenges which educators and students could face when teaching and learning about testing, as well as insights into how to overcome those challenges.

Given such a large body of experience and knowledge in the area of software-testing education, there is the need for a systematic review in this area, since it is often not possible for the individual educator to study all the papers and to synthesize all the evidence presented. For example, for a new educator who wants to teach a (new) course in software testing (in her/his university), it would be very valuable to know, before teaching, about the challenges faced by educators when teaching testing and also about insights on how to overcome those challenges. Thus, it would be useful to synthesize and summarize reported experience and evidence, as well as research topics and research questions (RQs) in this area. Our objective and goal in this paper are to provide such a synthesis and summary.

To address the above goal, we conducted a systematic literature mapping (SLM) to synthesize what the community of educators has published on this topic. Based on a systematic SLM process, we systematically select a pool of 204 papers, and by investigating nine RQs, we categorize and analyse various aspects of the subject under study.

The contributions of this review paper are three-fold:

  • The classification of the studies in this area, performed through a systematic mapping, via RQs 1-8

  • The synthesis of challenges faced during testing education (via RQ 9.1) together with the synthesis of insights (recommendations) for testing education (via RQ 9.2).

  • The development of an index (repository) of the studies in this area, which is accessible in an online Google spreadsheet (goo.gl/DcEpMv)

The remainder of this paper is structured as follows. Section 2 provides background and related work. Section 3 describes the research method, and then the design and execution phases of the SLM. Section 4 presents the results of the literature review. Section 5 summarizes the findings and discusses the lessons learned. Finally, in Section 6, we draw conclusions, and suggest areas for future research.

Section snippets

Background and related work

In this section, we provide a brief overview on the state of software-testing education in universities versus software testing training in industry. We then briefly review related work, i.e., existing survey (review) papers in the areas of software engineering and software-testing education.

Design and execution of the SLM

Our research method in this paper is SLM. We present an overview to our SLM process, the research questions of the SLM, and then other aspects related to the design and execution of the SLM.

Using the well-known guidelines for conducting SMS and SLR studies in SE (e.g., (Petersen et al., 2015; Wohlin, 2014; Petersen et al., 2008; Kitchenham and Charters, 2007)) and also based on our past experience in SLR studies, e.g., (Zhi et al., 2015; Garousi et al., 2015; Doğan et al., 2014; Häser et al.,

Findings of the SLM

Through Section 4, we present results of the study's RQs.

Discussions

In this section, we first summarize the research findings and discuss implications and recommendations for educators (Section 5.1). Then, we present suggestions for further education research in this area in Section 5.2, and finally we discuss potential threats to validity of this review (Section 5.3).

Conclusions and future work

We conducted a systematic literature mapping (SLM) to identify the state-of-the-art in the area of software-testing education. Our SLM provides a classification of studies in this area, a synthesis of both challenges faced during testing education and insights for testing education, and an index to studies in this area. All three contributions can benefit educators in the design, delivery and assessment of software testing courses in university settings, and can provide a foundation of previous

Pool of studies in the systematic literature mapping

[P1]S. Tiwari, V. Saini, P. Singh, and A. Sureka, ``A case study on the application of case-based learning in software testing,'' in Proceedings of Innovations in Software Engineering Conference, 2018, p. 11.
[P2]W. Zheng, Y. Bai, and H. Che, ``A computer-assisted instructional method based on machine learning in software testing class,'' Computer Applications in Engineering Education, vol. 26, no. 5, pp. 1150-1158, 2018.
[P3]K. Buffardi and S. H. Edwards, ``A formative study of influences on

CRediT authorship contribution statement

Vahid Garousi: Conceptualization, Methodology, Data curation, Validation, Writing - original draft. Austen Rainer: Methodology, Data curation, Validation, Writing - original draft. Per Lauvås: Conceptualization, Methodology, Data curation, Validation, Writing - original draft. Andrea Arcuri: Conceptualization, Methodology, Data curation, Validation, Writing - original draft.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

Andrea Arcuri is supported by the Research Council of Norway (project on Evolutionary Enterprise Testing, grant agreement No 274385). We thank the anonymous reviewers and the editor for their constructive comments.

Vahid Garousi is a Senior Lecturer (Associate Professor) of Software Engineering in Queen's University Belfast, Northern Ireland, UK. He has an international profile, as he previously worked as an academic and consultant in three continents: the Netherlands (2017-2019), Turkey (2015-2017), and Canada (2001-2014). Dr. Garousi received his Ph.D. in Software Engineering in Carleton University, Canada, in 2006. His research expertise are: software engineering, software testing, empirical studies,

References (107)

  • B. Kitchenham

    Systematic literature reviews in software engineering–a tertiary study

    Inf. Softw. Technol.

    (2010)
  • F.J. Lucas et al.

    A systematic review of UML model consistency management

    Inf. Softw. Technol.

    (2009)
  • K. Petersen et al.

    Guidelines for conducting systematic mapping studies in software engineering: An update

    Inf. Softw. Technol.

    (2015)
  • J. Sweller

    Cognitive load during problem solving: effects on learning

    Cogn. Sci.

    (1988)
  • J. Adams

    Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies

    Systematic Rev. J. Artic.

    (2016)
  • K.M. Ala-Mutka

    A survey of automated assessment approaches for programming assignments

    Comput. Sci. Edu.

    (2005)
  • Algaze, B., ``Software is increasingly complex. that can be dangerous,''...
  • S. Ali et al.

    A systematic review of the application and empirical investigation of searchbased test case generation

    IEEE Trans. Softw. Eng.

    (2010)
  • E. Allen et al.

    Production programming in the classroom

    ACM SIGCSE Bull.

    (2003)
  • A. Allevato et al.

    Dereferee: Exploring pointer mismanagement in student code

    ACM SIGCSE Bull.

    (2009)
  • M. Andreessen

    Why software is eating the world

    Wall Str. J

    (2018)
  • Association for Computing Machinery (ACM), ``ACM curricula recommendations,''...
  • R.H. Austing et al.

    A survey of the literature in computer science education since curriculum'68

    Commun. ACM

    (1977)
  • Baker, J., ``2018′s software engineering talent shortage— It's quality, not just quantity,''...
  • M. Ben-Ari

    Constructivism in computer science education

    J. Comput. Math. Sci. Teach.

    (2001)
  • J. Bennedsen et al.

    Revealing the programming process

    ACM SIGCSE Bull.

    (2005)
  • M. Bernhart et al.

    Dimensions of software engineering course design

  • J.B. Biggs

    Teaching for quality learning at university: What the student does

    McGraw-hill education (UK)

    (2011)
  • Bradford, L., ``11 steps to becoming a software engineer (without a CS degree),''...
  • Britton, T., Jeng, L., Carver, G., Cheak, P., and Katzenellenbogen, T., ``Reversible debugging software,'' University...
  • J.S. Bruner

    The act of discovery

    Harvard Edu. Rev.

    (1961)
  • K. Buffardi et al.

    Effective and ineffective software testing behaviors by novice programmers

  • A. Carbone et al.

    A survey of methods used to evaluate computer science teaching

    ACM SIGCSE Bull.

    (1998)
  • C. Caulfield et al.

    A systematic survey of games used for software engineering education

    Mod. Appl. Sci.

    (2011)
  • A. Causevic et al.

    Factors limiting industrial adoption of test driven development: A systematic review

  • D. Pfahl et al.

    How is exploratory testing used? A state-of-the-practice survey

  • M. Choy et al.

    Experiences in using an automated system for improving students’ learning of computer programming

  • ComputerWeekly, ``Industry warned it faces a ``dire shortage of IT testers'',''...
  • H. Cooper et al.

    The Handbook of Research Synthesis and Meta-Analysis

    (2009)
  • C. Cooper et al.

    Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

    BMC Med. Res. Method.

    (2018)
  • E.F. Crawley et al.

    The CDIO approach,

    Rethinking Engineering Education

    (2014)
  • D.S. Cruzes et al.

    Synthesizing evidence in software engineering research

  • D.S. Cruzes et al.

    Recommended Steps for Thematic Synthesis in Software Engineering

    Proc. International Symposium on Empirical Software Engineering and Measurement

    (2011)
  • D.S. Cruzesa et al.

    Research synthesis in software engineering: A tertiary study

    Inf. Softw. Technol.

    (2011)
  • R.d.A. Mauricio et al.

    A systematic mapping study on game-related methods for software engineering education

    Inf. Softw. Technol.

    (2018)
  • C. Daly et al.

    An automated learning system for Java programming

    IEEE Trans. Educ.

    (2004)
  • C. Desai et al.

    A survey of evidence for test-driven development in academia

    Acm SIGCSE Bull.

    (2008)
  • S. Easterbrook et al.

    Selecting Empirical Methods for Software Engineering Research

  • S.H. Edwards

    Adding software testing to programming assignments

  • Engström, E., Storey, M.-A., Runeson, P., Höst, M., and Baldassarre, M.T., ``A review of software engineering research...
  • Cited by (0)

    Vahid Garousi is a Senior Lecturer (Associate Professor) of Software Engineering in Queen's University Belfast, Northern Ireland, UK. He has an international profile, as he previously worked as an academic and consultant in three continents: the Netherlands (2017-2019), Turkey (2015-2017), and Canada (2001-2014). Dr. Garousi received his Ph.D. in Software Engineering in Carleton University, Canada, in 2006. His research expertise are: software engineering, software testing, empirical studies, action-research, and industry-academia collaborations. Dr. Garousi was selected as a Distinguished Speaker for the IEEE Computer Society from 2012 to 2015. He is a member of the IEEE and the IEEE Computer Society, and is also a licensed professional engineer (PEng) in the Canadian province of Alberta. In parallel to his academic career, he is a practicing software engineering consultant and coach, and has provided consultancy and corporate training services in several countries in the areas of software testing and quality assurance, model-driven development, and software maintenance. During his career so far, Vahid has served as the principal investigator (PI) for several research grants and has been active in initiating a number of major R&D software engineering projects with software companies in several countries including Canada and Turkey. He has been involved as an organizing or program committee member in many international conference, such as ICST, ICSP, CSEE&T, MoDELS and the Turkish National Software Engineering Conference. Among his awards is the prestigious Alberta Ingenuity New Faculty Award in June 2007.

    Austen Rainer is a Professor in the School of Electronics, Electrical Engineering and Computer Science at Queen's University Belfast in Northern Ireland. He co-authored the first disciple-specific book on case study research in software engineering, and co-founded Software Innovation NZ, the national network of software researchers in New Zealand. He has multidisciplinary teaching and research interests, with particular focus on team innovation, human values and the impact of software and software engineering on society.

    Per Lauvås is an associate professor at Kristiania University College in Oslo where he has been a faculty member since 2013. Prior to that, he worked as a software developer for nine years after graduating from the University of Oslo in 2004. Per teaches courses within software development. His main research interests are IT education and didactics of informatics.''

    Andrea Arcuri is a Professor of Software Engineering at Kristiania University College, Oslo, Norway. His main research interests are in software testing, especially test case generation using evolutionary algorithms. Having worked 5 years in industry as a senior engineer, a main focus of his research is to design novel research solutions that can actually be used in practice. Dr. Arcuri is the main-author of EvoMaster and a co-author of EvoSuite, which are open-source tools that can automatically generate test cases using evolutionary algorithms. He received his Ph.D. in software testing from the University of Birmingham, UK, in 2009.

    View full text