Skip to main content
SpringerLink
Log in

Benchmarking software development productivity of CMMI level 5 projects

  • Published:
Information Technology and Management Aims and scope Submit manuscript

Abstract

In this paper, data envelopment analysis variable returns to scale (DEA VRS) model is applied to data collected on 79 software development projects from a leading CMMI level 5 organization. We divide overall software effort into software development effort, software quality conformance effort (EoC), and software maintenance non-conformance (EoNC) effort due to poor software quality at delivery time. Partitioning effort into software development and software quality metrics provides us a comprehensive model to measure productivity of software projects and to identify best practice projects. Some of positive productivity drivers from the DEA best practice efficient projects point to good customer rapport and application familiarity. Inefficient projects had problems such as customer requirements volatility, and the use of unfamiliar technology. The DEA results identify 12 “best practice” projects that can be emulated for software process improvement. Additionally, our results point to approximately 50 % potential for productivity improvement in software projects to get to the level of “best practice” projects. This study shows that including EoC and EoNC as inputs has a positive impact on the best practice frontier.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Hill PR (2010) Practical software project estimation: a toolkit for estimating software development effort & duration. McGraw-Hill, San Francisco

    Google Scholar 

  2. Jalote P (2004) CMM in practice: processes for executing software projects at Infosys. Pearson, Delhi

    Google Scholar 

  3. Flitman A (2003) Towards meaningful benchmarking of software development team productivity. Benchmarking Int J 10(4):382–399

    Article  Google Scholar 

  4. Paradi JC, Reese DN, Rosen D (1997) Applications of DEA to measure the efficiency of software production at two large Canadian banks. Ann Oper Res 73:91–115

    Article  Google Scholar 

  5. Parthasarathy S, Anbazhagan N (2008) Evaluating ERP projects using DEA and regression analysis. Int J Bus Inf Syst 3(2):140–157

    Google Scholar 

  6. Stensrud E, Myrtveit I (2003) Identifying high performance ERP projects. IEEE Trans Comput 29(5):398–416

    Google Scholar 

  7. Yang Z, Paradi JC (2004) DEA evaluation of a Y2K software retrofit program. IEEE Trans Eng Manag 51(3):279–287

    Article  Google Scholar 

  8. Card DN, El Emam K, Scalzo B (2001) Measurement of object-oriented software development projects. Software Productivity Consortium NFP, Herndon

    Google Scholar 

  9. Basili VR, Briand LC, Melo WL (1996) How reuse influences productivity in object-oriented systems. Commun ACM 39(10):104–116

    Article  Google Scholar 

  10. Blackburn JD, Scudder GD, Van Wassenhove LN (1996) Improving speed and productivity of software development: a global survey of software developers. IEEE Trans Softw Eng 22(12):875–885

    Article  Google Scholar 

  11. Briand LC, El-Emam K, Bomarius F COBRA (1998) A hybrid method for software cost estimation, benchmarking and risk assessment. In: 20th international conference on software engineering, pp 390–399

  12. Jeffery R, Ruhe M, Wieczorek I (2001) Using public domain metrics to estimate software development effort. In: Proceedings of METRICS’01, pp 16–27

  13. Boehm B (1981) Software engineering economics. Prentice-Hall, Englewood Cliffs

    Google Scholar 

  14. Boehm B, Clark B, Horowitz E, Westland C, Madachy R, Selby R (1996) The Cocomo 2.0 software cost estimation model—a status report. Am Program. 9:2–17

    Google Scholar 

  15. Banker RD, Kemerer CF (1989) Scale economies in new software development. IEEE Trans Softw Eng 15(10):1199–1205

    Article  Google Scholar 

  16. Pendharkar PC, Rodger JA, Subramanian GH (2008) An empirical study of the Cobb–Douglas production function properties of software development effort. Inf Softw Technol 50:1181–1188

    Article  Google Scholar 

  17. Kneuper R (2008) Improving software and systems development processes using capability maturity model integration (CMMI-DEV). Rocky Nook, Santa Barbara

    Google Scholar 

  18. Goldenson DR, Gibson DL (2003) Demonstrating the impact and benefits of CMMI: an update and preliminary results. CMU/SEI-2003-SR-009. Carnegie Mellon Software Engineering Institute, Pittsburgh

    Google Scholar 

  19. Charnes A, Cooper WW, Lewin AY, Seiford LM (1994) Data envelopment analysis: theory, methodology, and applications. Kluwer, Boston

    Book  Google Scholar 

  20. Charnes A, Cooper WW, Golany B, Seiford L, Stutz J (1985) Foundations of data envelopment analysis for Pareto-Koopmans efficient empirical production functions. J Econom 30:91–107

    Article  Google Scholar 

  21. Emrouznedjad A, Parker BR, Tavares G (2008) Evaluation of research in efficiency and productivity: a survey and analysis of the first 30 years of scholarly literature in DEA. Soc Econom Plan Sci 44:151–157

    Article  Google Scholar 

  22. Seiford LM (1996) Data envelopment analysis: the evaluation of the state of the art (1978–1995). J Prod Anal 9:99–137

    Article  Google Scholar 

  23. Seiford LM, Thrall RM (1990) Recent developments in DEA: the mathematical programming approach to frontier analysis. J Econom 46:7–38

    Article  Google Scholar 

  24. Tavares G (2002) A bibliography of data envelopment analysis (1978–2001). RUTCOR, Rutgers University, Newark

    Google Scholar 

  25. Charnes A, Cooper WW, Rhodes E (1978) Measuring the efficiency of decision making units. Eur J Oper Res 2:429–444

    Article  Google Scholar 

  26. Banker RD, Charnes A, Cooper WW (1984) Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manag Sci 30(9):1078–1092

    Article  Google Scholar 

  27. Banker RD, Datar SM, Kemerer CF (1991) A model to evaluate variables impacting the productivity of software maintenance projects. Manag Sci 37(1):1–18

    Article  Google Scholar 

  28. Elam J (1991) Evaluating the efficiency of IS organizations using data envelope analysis. In: Proceedings of the International Function Point Users Group

  29. Banker RD, Chang H, Kemerer CF (1994) Evidence on economies of scale in software development. Inf Softw Technol 36(5):275–282

    Article  Google Scholar 

  30. Mahmood MA, Pettingell KJ, Shaskevich AI (1996) Measuring productivity of software projects: a data envelope analysis approach. Decis Sci 27(1):57–80

    Article  Google Scholar 

  31. Myrtveit I, Stensrud E (1999) Benchmarking COTS projects using data envelopment analysis. In: Proceedings of METRICS’99, Washington, D.C., pp 269–278

  32. Zhang S, Wang Y, Tong J, Zhou J, Ruan L (2006) Evaluation of project quality: a DEA-based approach. Lect Notes Comput Sci 3966:88–96

    Article  Google Scholar 

  33. Asmild M, Paradi JC, Kulkarni A (2006) Using data envelopment analysis in software development productivity measurement. Softw Process Improv Pract 11:561–572

    Article  Google Scholar 

  34. Wray B, Mathieu R (2008) Evaluating the performance of open source software projects using data envelopment analysis. Inf Manag Comput Secur 16(5):449–462

    Google Scholar 

  35. Maxwell K, Forselius P (2000) Benchmarking software development productivity. Softw IEEE 17(1):80–88

    Article  Google Scholar 

  36. Pendharkar PC, Rodger J (2009) The Relationship between software development team size and software development cost. Commun ACM 52(1):141–144

    Article  Google Scholar 

  37. McCall JA, Richards PK, Walters GF (1977) Factors in software quality. Volumes 1, 2, 3, US Rome Air Development Center Reports NTIS AD/A-049 014, 015, 055

  38. Boehm BW, Brown JR, Kaspar JR, Lipow M, MacCleod GJ, Merrit MJ (1978) Characteristics of software quality. North-Holland, Amsterdam

    Google Scholar 

  39. Fenton NE (1994) Software measurement: a necessary scientific basis. IEEE Trans Softw Eng 20(3):199–206

    Article  Google Scholar 

  40. Kemayel L, Mili A, Ouederni I (1991) Controllable factors for programmer productivity: a statistical study. J Syst Softw 16(2):151–163

    Article  Google Scholar 

  41. Zhu J (2003) Quantitative models for performance evaluation and benchmarking: data envelopment analysis with spreadsheets. Kluwer, Boston

    Book  Google Scholar 

  42. Bowlin WF (1998) Measuring performance: an introduction to data envelopment analysis (DEA). J Cost Anal 7:3–27

    Article  Google Scholar 

  43. Chen Y, Zhu J (2004) Measuring information technology’s indirect impact on firm performance. Inf Technol Manag 5(1):9–22

    Article  Google Scholar 

  44. Soteriou AC, Zenios SA (1998) Data envelopment analysis: an introduction and an application to bank branch performance assessment. In: Marcoulides G (ed) Modern methods for business research. Lawrence Erlbaum Associates, London

    Google Scholar 

  45. Drake L, Howcroft B (1994) Relative efficiency in the branch network of a UK bank: an empirical study. Omega 22(1):83–90

    Article  Google Scholar 

  46. Cooper WW, Seiford LM, Zhu J (2004) Handbook on data envelopment analysis. Springer, New York

    Book  Google Scholar 

  47. Sexton TR, Silkman RH, Hogan AJ (eds) (1986) Data envelopment analysis: Critique and extensions, new directions for program evaluation. Jossey-Bass, San Francisco

    Google Scholar 

  48. Andersen P, Petersen NC (1993) A procedure for ranking efficient units in data envelopment analysis. Manag Sci 39:1261–1264

    Article  Google Scholar 

  49. Seiford LM, Zhu J (1999) Infeasibility of super efficiency data envelopment analysis models. INFOR 37(2):174–187

    Google Scholar 

  50. Torgersen AM, Førsund FR, Kittelsen SA (1996) Slack adjusted efficiency measures and ranking of efficient units. J Prod Anal 7:379–398

    Article  Google Scholar 

  51. Boh WF, Slaughter SA, Espinosa JA (2007) Learning from experience in software development: a multilevel analysis. Manag Sci 53(8):1315–1331

    Article  Google Scholar 

  52. Keil M, Cule PE, Lyytinen K, Schmidt RC (1998) A framework for identifying software project risks. Commun ACM 41(11):76–83

    Article  Google Scholar 

  53. Mantle MW, Lichty R (2012) Managing the unmanageable: rules, tools, and insights for managing software people and teams. Addison-Wesley, Boston

    Google Scholar 

  54. Kittelsen S (1999) Monte Carlo simulations of DEA efficiency measures and hypothesis tests. Department of Economics, University of Oslo, Oslo

    Google Scholar 

  55. Pendharkar PC, Rodger JA (2007) An empirical study of the impact of team size on software development effort. Inf Technol Manag 8:253–262

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Business Administration, Pennsylvania State University at Harrisburg, 777 West Harrisburg Pike, Middletown, PA, 17057, USA

    Dinesh R. Pai, Girish H. Subramanian & Parag C. Pendharkar

Authors
  1. Dinesh R. Pai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Girish H. Subramanian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Parag C. Pendharkar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dinesh R. Pai.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pai, D.R., Subramanian, G.H. & Pendharkar, P.C. Benchmarking software development productivity of CMMI level 5 projects. Inf Technol Manag 16, 235–251 (2015). https://doi.org/10.1007/s10799-015-0234-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10799-015-0234-4

Keywords

Search

Navigation