Skip to main content
SpringerLink
Log in

Product backlog rating: a case study on measuring test quality in scrum

  • Original Paper
  • Published:
Innovations in Systems and Software Engineering Aims and scope Submit manuscript

Abstract

Agile software development methodologies focus on software projects which are behind schedule or highly likely to have a problematic development phase. In the last decade, Agile methods have transformed from cult techniques to mainstream methodologies. Scrum, an Agile software development method, has been widely adopted due to its adaptive nature. This paper presents a metric that measures the quality of the testing process in a Scrum process. As product quality and process quality correlate, improved test quality can ensure high-quality products. Also, gaining experience from 8 years of successful Scrum implementation at SoftwarePeople, we describe the Scrum process emphasizing the testing process. We propose a metric Product Backlog Rating (PBR) to assess the testing process in Scrum. PBR considers the complexity of the features to be developed in an iteration of Scrum, assesses test ratings and offers a numerical score of the testing process. This metric is able to provide a comprehensive overview of the testing process over the development cycle of a product. We present a case study which shows how the metric is used at SoftwarePeople. The case study explains some features that have been developed in a Sprint in terms of feature complexity and potential test assessment difficulties and shows how PBR is calculated during the Sprint. We propose a test process assessment metric that provides insights into the Scrum testing process. However, the metric needs further evaluation considering associated resources (e.g., quality assurance engineers, the length of the Scrum cycle).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. http://www.enfatico.com/.

References

  1. Abrahamsson P, Salo O, Ronkainen J, Warsta J (2002) Agile software development methods: review and analysis. VTT, Espoo

    Google Scholar 

  2. Rech J (2007) Handling of software quality defects in agile software development. In: Agile software development quality assurance 90

  3. Churchville D (2008) Agile thinking: leading successful software projects and teams. ExtremePlanner Software, San Diego

    Google Scholar 

  4. Chan K (2013) Agile adoption statistics 2012. http://www.onedesk.com/2013/05/agile-adoption-statistics-2012/

  5. Wailgum T (2007) From here to agility. http://www.cio.com.au/article/193933/from_here_agility/

  6. Ceschi M, Sillitti A, Succi G, De Panfilis S (2005) Project management in plan-based and agile companies. IEEE Softw 22:21–27

    Article  Google Scholar 

  7. Mann C, Maurer F (2005) A case study on the impact of scrum on overtime and customer satisfaction. In: Proceedings on agile conference 2005, p 70–79

  8. Schwaber C (2007) The truth about agile processes. http://www.rallydev.com/sites/default/files/The_Truth_About_Agile_Processes_Forrester_white_paper.pdf

  9. Schwaber K (2004) Agile project management with Scrum. O’Reilly Media, Inc, Sebastopol

    MATH  Google Scholar 

  10. Alliance S (2007) Scrum alliance membership survey shows growing scrum adoption and project success. http://prn.to/13t1V2V

  11. Schatz B, Abdelshafi I (2005) Primavera gets agile: a successful transition to agile development. IEEE Softw 22:36–42

    Article  Google Scholar 

  12. Upender B (2005) Staying agile in government software projects. In: Proceedings on agile conference 2005, p 153–159

  13. Mahnic V, Drnovscek S (2005) Agile software project management with scrum. In: EUNIS 2005 conference-session papers and tutorial abstracts

  14. Smith G, Sidky A (2009) Becoming agile in an imperfect world. Manning, Greenwich

    Google Scholar 

  15. Anderson DJ (2003) Agile management for software engineering: applying the theory of constraints for business results. Prentice Hall Professional, Upper Saddle River

    Google Scholar 

  16. Desharnais JM, Kocatürk B, Abran A (2011) Using the cosmic method to evaluate the quality of the documentation of agile user stories. In: IEEE software measurement, 2011 joint conference of the 21st international workshop on and 6th international conference on software process and product measurement (IWSM-MENSURA), 269–272

  17. Petersen K, Wohlin C (2010) Software process improvement through the lean measurement (spi-leam) method. J Syst Softw 83:1275–1287

    Article  Google Scholar 

  18. Lacovelli A, Souveyet C (2008) Framework for agile methods classification. In: MoDISE-EUS, Citeseer p 91–102

  19. Hartmann D, Dymond R (2006) Appropriate agile measurement: using metrics and diagnostics to deliver business value. In: IEEE agile conference 2006

  20. Jabangwe R, Petersen K, Mite D (2013) Visualization of defect inflow and resolution cycles: before, during and after transfer. In: IEEE software engineering conference (APSEC, 2013 20th Asia-Pacific), vol. 1, p 289–298

  21. Petersen K (2012) A palette of lean indicators to detect waste in software maintenance: a case study. In: Agile processes in software engineering and extreme programming. Springer, p 108–122

  22. Petersen K, Wohlin C (2011) Measuring the flow in lean software development. Softw Pract Exp 41:975–996

    Article  Google Scholar 

  23. Fuggetta A (2000) Software process: a roadmap. In: Proceedings of the conference on the future of software engineering, ACM 25–34

  24. Sommerville I, Kotonya G (1998) Requirements engineering: processes and techniques. Wiley, New York

    Google Scholar 

  25. Cugola G, Ghezzi C (1998) Software processes: a retrospective and a path to the future. Softw Process Improv Pract 4:101–123

    Article  Google Scholar 

  26. Kitchenham B, Pfleeger SL (1996) Software quality: the elusive target [special issues section]. IEEE Software 13:12–21

    Article  Google Scholar 

  27. Schwaber K, Sutherland J (2013) The definitive guide to scrum: the rules of the game. http://www.scrumguides.org/

  28. Shore J et al (2008) The art of agile development. O’Reilly, Sebastopol

    Google Scholar 

  29. Gannon M (2013) An agile implementation of scrum. In: IEEE aerospace conference 2013, p 1–7

  30. Rising L, Janoff NS (2000) The scrum software development process for small teams. IEEE Software 17:26–32

    Article  Google Scholar 

  31. Boehm BW (1984) Software engineering economics. IEEE Trans Softw Eng 4–21

  32. Ramamoorthy C, Prakash A, Tsai WT, Usuda Y (1984) Software engineering: problems and perspectives. Computer 17:191–209

    Article  Google Scholar 

  33. Kayes M (2011) Test case prioritization for regression testing based on fault dependency. In: 2011 3rd international conference on electronics computer technology (ICECT), vol 5, 48–52

  34. Beizer B (1995) Black-box testing: techniques for functional testing of software and systems. Wiley, New York

    Google Scholar 

  35. Perry WE (1999) Softw Testing. Wiley, New York

    Google Scholar 

  36. Majchrzak TA (2012) Improving software testing: technical and organizational developments. Springer, New York

    Book  Google Scholar 

  37. Feyh M, Petersen K (2013) Lean software development measures and indicators-a systematic mapping study. In: Lean enterprise software and systems. Springer 32–47

  38. Agarwal M, Majumdar R (2012) Tracking scrum projects tools, metrics and myths about agile. Int J Emerg Technol Adv Eng 2:97–104

    Google Scholar 

  39. Petersen K, Roos P, Nyström S, Runeson P (2014) Early identification of bottlenecks in very large scale system of systems software development. J Softw Evol Process 26:1150–1171

    Article  Google Scholar 

  40. Staron M, Meding W (2011) Monitoring bottlenecks in agile and lean software development projects—a method and its industrial use. In: Product-focused software process improvement. Springer 3–16

  41. Kniberg H (2007) In: Scrum and XP from the trenches: enterprise software development. Lulu.com

  42. Paetsch F, Eberlein A, Maurer F (2003) Requirements engineering and agile software development. In: Proceedings of twelfth IEEE international workshops on enabling technologies: infrastructure for collaborative enterprises (WET ICE 2003) 308–313

  43. Nuseibeh B, Easterbrook S (2000) Requirements engineering: a roadmap. In: Proceedings of the conference on the future of software engineering. ICSE ’00, New York, ACM 35–46

  44. Nokhbeh Zaeem R, Khurshid S (2012) Test input generation using dynamic programming. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering. FSE ’12, New York, ACM 34:1–34:11

  45. Samuel P, Surendran A (2010) Forward slicing algorithm based test data generation. In: 2010 3rd IEEE International Conference on computer science and information technology (ICCSIT), vol 8, 270–274

  46. Alakeel AM (2010) An algorithm for efficient assertions-based test data generation. J Softw 5:644–653

    Article  Google Scholar 

  47. Jafri R (2012) Database testing properties of a good test data and test data preparation techniques. http://www.softwaretestinghelp.com/database-testing-test-data-preparation-techniques/

  48. Chemuturi M (2006) Test effort estimation. Chemuturi Consultants, April

  49. Sangal N, Jordan E, Sinha V, Jackson D (2005) Using dependency models to manage complex software architecture. SIGPLAN Not. 40:167–176

    Article  Google Scholar 

  50. Gall H, Hajek K, Jazayeri M (1998) Detection of logical coupling based on product release history. In: Proceedings of the international conference on software maintenance 1998, 190–198

  51. Cataldo M, Mockus A, Roberts JA, Herbsleb JD (2009) Software dependencies, work dependencies, and their impact on failures. IEEE Trans Softw Eng 35:864–878

    Article  Google Scholar 

  52. O’reilly T (2007) What is web 2.0: Design patterns and business models for the next generation of software. Communications & strategies 17

  53. Zimmermann T, Nagappan N (2008) Predicting bugs from history. Springer, Berlin

    Book  Google Scholar 

  54. Kaner C, Bond WP (2004) Software engineering metrics: What do they measure and how do we know? Methodology 8:6

    Google Scholar 

  55. Pick G (2001) Data migration concepts & challenges. http://bit.ly/13OsRdS

  56. Fiedler RL, Kaner C (2009) Putting the context in context-driven testing (an application of cultural historical activity theory). In: Conference of the association for software testing

  57. Kaner C, Bach J, Pettichord B (2008) Lessons learned in software testing. Wiley, New York

    Google Scholar 

  58. Moore JW (1998) Software engineering standards. Wiley Online Library, New York

    Google Scholar 

  59. Kaner C (2003) What is a good test case? In: Software testing analysis and review conference

  60. Huang CY, Lin CT (2006) Software reliability analysis by considering fault dependency and debugging time lag. IEEE Trans Reliab 55:436–450

    Article  Google Scholar 

  61. Stark GE, Oman P, Skillicorn A, Ameele A (1999) An examination of the effects of requirements changes on software maintenance releases. Journal of Software Maintenance 11:293–309

    Article  Google Scholar 

  62. Gorschek T, Davis AM (2008) Requirements engineering: In search of the dependent variables. Inf Softw Technol 50:67–75

    Article  Google Scholar 

  63. Fowler M (2002) Patterns of Enterprise Application Architecture. Addison-Wesley Longman Publishing Co. Inc., Boston

    Google Scholar 

  64. Boehm B, Bose P, Horowitz E, Lee MJ (1994) Software requirements as negotiated win conditions. In: Requirements Engineering, 1994., Proceedings of the First International Conference on. 74–83

  65. Grady R (1994) Successfully applying software metrics. Computer 27:18–25

    Article  Google Scholar 

  66. Jung HW, Kim SG, Chung CS (2004) Measuring software product quality: A survey of iso/iec 9126. Software, IEEE 21:88–92

    Article  Google Scholar 

  67. Cavano JP, McCall JA (1978) A framework for the measurement of software quality. In: ACM SIGMETRICS Performance Evaluation Review. Volume 7., ACM 133–139

  68. Bertoa MF, Troya JM, Vallecillo A (2006) Measuring the usability of software components. J Syst Softw 79:427–439

    Article  Google Scholar 

  69. Nagappan N, Williams L, Vouk M (2003) Towards a metric suite for early software reliability assessment. In: International Symposium on Software Reliability Engineering. FastAbstract, Denver, CO, pp 238–239

  70. Goel AL (1985) Software reliability models: Assumptions, limitations, and applicability. Software Engineering, IEEE Transactions on 1411–1423

  71. Oman P, Hagemeister J (1992) Metrics for assessing a software system’s maintainability. In: Software Maintenance, 1992. Proceerdings., Conference on, IEEE 337–344

  72. Coleman D, Ash D, Lowther B, Oman P (1994) Using metrics to evaluate software system maintainability. Computer 27:44–49

    Article  Google Scholar 

  73. Washizaki H, Yamamoto H, Fukazawa Y (2003) A metrics suite for measuring reusability of software components. In: Software Metrics Symposium, 2003. Proceedings. Ninth International, IEEE 211–223

  74. Henry S, Kafura D (1981) Software structure metrics based on information flow. Software Engineering, IEEE Transactions on 510–518

  75. Paulk MC, Weber CV, Curtis B, Chrissis ME (1995) The capability maturity model: Guidelines for improving the software process, vol 441. Addison-wesley Reading, Boston

    Google Scholar 

  76. Paulk MC (1993) Comparing iso 9001 and the capability maturity model for software. Software Qual J 2:245–256

    Article  Google Scholar 

  77. Ahern DM, Clouse AA, Turner RA (2004) CMMI distilled: a practical introduction to integrated process improvement. Addison-Wesley Professional, Boston

    Google Scholar 

  78. Team CP (2002) Capability maturity model integration (cmmi sm), version 1.1. Software Engineering Institute, Carnegie Mellon University/SEI-2002-TR-012. Pittsburg, PA

  79. Emam KE, Melo W, Drouin JN (1997) SPICE: the theory and practice of software process improvement and capability determination. IEEE Computer Society Press, Washington

  80. Jung HW, Hunter R (2001) The relationship between iso/iec 15504 process capability levels, iso 9001 certification and organization size: an empirical study. J Syst Softw 59:43–55

    Article  Google Scholar 

  81. Basili V, Green S (1994) Software process evolution at the sel. IEEE Software 11:58–66

    Article  Google Scholar 

  82. McFeeley B (1996) Ideal: A user’s guide for software process improvement. Technical report, DTIC Document

  83. Salo O, Abrahamsson P (2007) An iterative improvement process for agile software development. Softw Process Improv Pract 12:81–100

    Article  Google Scholar 

  84. Cockburn A, Highsmith J (2001) Agile software development, the people factor. Computer 34:131–133

    Article  Google Scholar 

  85. Cockburn A, Williams L (2003) Agile software development: It’s about feedback and change. Computer 36:0039–43

    Google Scholar 

  86. Bratthall L, Wohlin C (2000) Understanding some software quality aspects from architecture and design models. In: Program Comprehension, 2000. Proceedings. IWPC 2000. 8th International Workshop on, IEEE 27–34

  87. Budgen D (2003) Software design. Pearson Education, New York

    MATH  Google Scholar 

  88. Crosby PB (1979) Quality is free: The art of making quality certain, vol 94. McGraw-Hill, New York

    Google Scholar 

  89. Galin D (2004) Software quality assurance: from theory to implementation. Pearson education, New York

    Google Scholar 

  90. Watkins J (2009) Agile testing: how to succeed in an extreme testing environment. Cambridge University Press, Cambridge

    Book  Google Scholar 

  91. Stamelos IG, Sfetsos P (2007) Agile software development quality assurance. Igi Global, Hershey

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science and Engineering, University of South Florida, Tampa, FL, USA

    Imrul Kayes

  2. SoftwarePeople, Copenhagen, Denmark

    Mithun Sarker

  3. Electrical and Computer Engineering, University of Alabama, Tuscaloosa, AL, USA

    Jacob Chakareski

Authors
  1. Imrul Kayes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mithun Sarker
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jacob Chakareski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Imrul Kayes.

Additional information

Imrul Kayes: Most of the work was done when the author was a software engineer at SoftwarePeople.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kayes, I., Sarker, M. & Chakareski, J. Product backlog rating: a case study on measuring test quality in scrum. Innovations Syst Softw Eng 12, 303–317 (2016). https://doi.org/10.1007/s11334-016-0271-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11334-016-0271-0

Keywords

Search

Navigation