Skip to main content
SpringerLink
Log in

Review participation in modern code review

An empirical study of the android, Qt, and OpenStack projects

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Software code review is a well-established software quality practice. Recently, Modern Code Review (MCR) has been widely adopted in both open source and proprietary projects. Our prior work shows that review participation plays an important role in MCR practices, since the amount of review participation shares a relationship with software quality. However, little is known about which factors influence review participation in the MCR process. Hence, in this study, we set out to investigate the characteristics of patches that: (1) do not attract reviewers, (2) are not discussed, and (3) receive slow initial feedback. Through a case study of 196,712 reviews spread across the Android, Qt, and OpenStack open source projects, we find that the amount of review participation in the past is a significant indicator of patches that will suffer from poor review participation. Moreover, we find that the description length of a patch shares a relationship with the likelihood of receiving poor reviewer participation or discussion, while the purpose of introducing new features can increase the likelihood of receiving slow initial feedback. Our findings suggest that the patches with these characteristics should be given more attention in order to increase review participation, which will likely lead to a more responsive review process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://source.android.com/

  2. http://qt-project.org/

  3. http://www.openstack.org/

  4. https://wiki.qt.io/Qt_Contribution_Guidelines

  5. http://docs.openstack.org/infra/manual/developers.htmld#peer-review

  6. https://en.wikipedia.org/wiki/Android_version_history, https://en.wikipedia.org/wiki/List_of_Qt_releases, https://wiki.openstack.org/wiki/Releases

  7. https://codereview.qt-project.org/#/c/29921/

  8. http://sailhome.cs.queensu.ca/replication/review_participation/

  9. http://sailhome.cs.queensu.ca/replication/review_participation/

  10. https://android-review.googlesource.com/#/c/39850

  11. https://android-review.googlesource.com/#/c/39881

  12. An example in the Qt project: https://codereview.qt-project.org/#/c/27218 and https://codereview.qt-project.org/#/c/30591. An example in the OpenStack project: https://review.openstack.org/#/c/36808 and https://review.openstack.org/#/c/36832.

  13. https://review.openstack.org/#/c/36901/

  14. https://codereview.qt-project.org/#/c/29856

  15. https://bugreports.qt.io/browse/QTBUG-22625

  16. https://codereview.qt-project.org/#/c/101316

  17. https://review.openstack.org/#/c/35074/

  18. https://review.openstack.org/#/c/36448/

  19. http://sailhome.cs.queensu.ca/replication/review_participation/

  20. http://sailhome.cs.queensu.ca/replication/review_participation/

  21. http://sailhome.cs.queensu.ca/replication/review_participation/

References

  • Abelein U, Paech B (2013) Understanding the influence of user participation and involvement on system success a systematic mapping study. Empir Softw Eng (EMSE) 20(1):28–31

    Article  Google Scholar 

  • Bacchelli A, Bird C (2013) Expectations, Outcomes, and Challenges of Modern Code Review. In: Proceedings of the 35th International Conference on Software Engineering (ICSE), pp 712–721

  • Bacchelli A, Lanza M, Robbes R (2010) Linking E-Mails and Source Code Artifacts. In: Proceedings of the 32nd International Conference on Software Engineering (ICSE), pp 375–384

  • Balachandran V (2013) Reducing Human Effort and Improving Quality in Peer Code Reviews using Automatic Static Analysis and Reviewer Recommendation. In: Proceedings of the 35th International Conference on Software Engineering (ICSE), pp 931–940

  • Bavota G, Russo B (2015) Four Eyes Are Better Than Two: On the Impact of Code Reviews on Software Quality. In: Proceedings of the 31st International Conference on Software Maintenance and Evolution (ICSME), pp 81–90

  • Baysal O, Kononenko O, Holmes R, Godfrey MW (2012) The Secret Life of Patches: A Firefox Case Study. In: Proceedings of the 19th Working Conference on Reverse Engineering (WCRE), pp 447–455

  • Baysal O, Kononenko O, Holmes R, Godfrey MW (2015) Investigating technical and non-technical factors influencing modern code review. Empirical Software Engineering (EMSE) 1–28

  • Beller M, Bacchelli A, Zaidman A, Juergens E (2014) Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?. In: Proceedings of the 11th Working Conference on Mining Software Repositorie (MSR), pp 202–211

  • Bettenburg N, Hassan AE, Adams B, German DM (2013) Management of community contributions - A case study on the Android and Linux software ecosystems. Empirical Software Engineering (EMSE) 1–38

  • Bird C, Gourley A, Devanbu P, Gertz M, Swaminathan A (2006) Mining Email Social Networks. In: Proceedings of the 3rd International Workshop on Mining Software Repositories (MSR), pp 137–143

  • Bird C, Gourley A, Devanbu P (2007) Detecting Patch Submission and Acceptance in OSS Projects. In: Proceedings of the 4th International Workshop on Mining Software Repositories (MSR), pp 26–29

  • Bird C, Nagappan N, Murphy B, Gall H, Devanbu P (2011) Don’t Touch My Code! Examining the Effects of Ownership on Software Quality. In: Proceedings of the 8th joint meeting of the European Software Engineering Conference and the Internaltional Symposium on the Foundations of Software Engineering (ESEC/FSE), pp 4–14

  • Bosu A, Carver JC (2013) Impact of Peer Code Review on Peer Impression Formation: A Survey. In: Proceedings of the 7th International Symposium on Empirical Software Engineering and Measurement (ESEM), pp 133–142

  • Cramėr H (1999) Mathematical methods of statistics. University Press, Princeton, p 9

    MATH  Google Scholar 

  • Efron B (1986) How biased is the apparent error rate of a prediction Rule? J Am Stat Assoc 81(394):461–470

    Article  MathSciNet  MATH  Google Scholar 

  • Fagan ME (1999) Design and code inspections to reduce errors in program development. IBM Syst J 38(2-3):258–287

    Article  Google Scholar 

  • Fowler M, Foemmel M (2006) Continuous integration. http://www.thoughtworks.com/ContinuousIntegration.pdf

  • Gousios G, Pinzger M, van Deursen A (2014) An Exploratory Study of the Pull-based Software Development Model. In: Proceedings of the 36th International Conference on Software Engineering (ICSE), pp 345–355

  • Guzzi A, Bacchelli A, Lanza M, Pinzger M, Van Deursen A (2013) Communication in Open Source Software Development Mailing Lists. In: Proceedings of the 10th International Working Conference on Mining Software Repositories (MSR), pp 277–286

  • Hamasaki K, Kula RG, Yoshida N, Erika CCA, Fujiwara K, Iida H (2013) Who does what during a Code Review? An extraction of an OSS Peer Review Repository. In: Proceedings of the 10th International Working Conference on Mining Software Repositories (MSR), pp 49–52

  • Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiological Society of North America 143(1):29–36

    Google Scholar 

  • Harrell FE Jr (2002) Regression Modeling Strategies: With Application to Linear Models, Logistic Regression, and Survival Analysis, 1st. Springer, Berlin

    Google Scholar 

  • Harrell FE Jr (2015) rms: Regression Modeling Strategies. http://biostat.mc.vanderbilt.edu/rms

  • Hassan AE (2008) Automated Classification of Change Messages in Open Source Projects. In: Proceedings of the 23rd Symposium on Applied Computing (SAC), pp 837–841

  • Hassan AE (2009) Predicting Faults Using the Complexity of Code Changes. In: Proceedings of the 31st International Conference on Software Engineering (ICSE), pp 78–88

  • Hinkle DE, Wiersma W, Jurs SG (1998) Applied statistics for the behavioral sciences, 4th edn. Houghton Mifflin, Boston

    Google Scholar 

  • Jiang Y, Adams B, German DM (2013) Will My Patch Make It? And How Fast? Case Study on the Linux Kernel. In: Proceeding of the 10th International Working Conference on Mining Software Repositories (MSR), pp 101–110

  • Kamei Y, Shihab E, Adams B, Hassan AE, Mockus A, Sinha A, Ubayashi N (2013) A large-scale empirical study of just-in-time quality assurance. Trans Softw Eng (TSE) 39(6):757–773

    Article  Google Scholar 

  • Kim S, Whitehead EJ Jr, Zhang Y (2008) Classifying software changes: Clean or buggy? Trans Softw Eng (TSE) 34(2):181–196

    Article  Google Scholar 

  • Kononenko O, Baysal O, Guerrouj L, Cao Y, Godfrey MW (2015) Investigating Code Review Quality : Do People and Participation Matter ?. In: Proceedings of the 31st International Conference on Software Maintenance and Evolution (ICSME), pp 111–120

  • McIntosh S, Kamei Y, Adams B, Hassan AE (2014) The Impact of Code Review Coverage and Code Review Participation on Software Quality. In: Proceedings of the 11th International Working Conference on Mining Software Repositorie (MSR), pp 192–201

  • McIntosh S, Kamei Y, Adams B, Hassan AE (2015) An Empirical Study of the Impact of Modern Code Review Practices on Software Quality. Empirical Software Engineering (EMSE)

  • Mishra R, Sureka A (2014) Mining Peer Code Review System for Computing Effort and Contribution Metrics for Patch Reviewers. In: Proceedings of the 4th Workshop on Mining Unstructured Data (MUD), pp 11–15

  • Mockus A, Votta LG (2000) Identifying Reasons for Software Changes using Historic Databases. In: Proceedings of the 16th International Conference on Software Maintainance (ICSM), pp 120–130

  • Morales R, Mcintosh S, Khomh F (2015) Do Code Review Practices Impact Design Quality? A Case Study of the Qt, VTK, and ITK Projects. In: Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER)

  • Mukadam M, Bird C, Rigby PC (2013) Gerrit software code review data from android. In: Proceedings of the 10th International Working Conference on Mining Software Repositories (MSR), pp 45–48

  • Nagappan M, Zimmermann T, Bird C (2013) Diversity in Software Engineering Research. In: Proceedings of the 9th joint meeting of the European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE), pp 466–476

  • Nagappan N, Zeller A, Zimmermann T, Herzig K, Murphy B (2010) Change bursts as defect predictors. In: Proceedings of the 21st international Symposium on Software Reliability Engineering (ISSRE), pp 309–318

  • Nurolahzade M, Nasehi SM, Khandkar SH, Rawal S (2009) The Role of Patch Review in Software Evolution: An Analysis of the Mozilla Firefox. In: Proceedings of the join International and Anual ERCIM Workshops on Principles of Software Evolution and Software Evolution Workshop (IWPSE-Evol), pp 9–17

  • Porter A, Siy H, Mockus A, Votta L (1998) Understanding the sources of variation in software inspections. Trans Softw Eng Methodol (TOSEM) 7(1):41–79

    Article  Google Scholar 

  • Ratcliffe JG (2009) Moving Software Quality Upstream: The Positive Impact of Lightweight Peer Code Review. In: Pacific NW Software Quality Conference, pp 1–10

  • Raymond ES (1999) The cathedral and the bazaar. Knowl, Technol Policy 12 (3):23–49

    Article  Google Scholar 

  • Rea LM, Parker RA (2014) Designing and conducting survey research: A comprehensive guide. John Wiley & Sons

  • Rigby PC, Bird C (2013) Convergent Contemporary Software Peer Review Practices. In: Proceedings of the 9th joint meeting of the European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE), pp 202–212

  • Rigby PC, Storey MA (2011) Understanding Broadcast Based Peer Review on Open Source Software Projects. In: Proceeding of the 33rd International Conference on Software Engineering (ICSE), pp 541–550

  • Rigby PC, German DM, Storey MA (2008) Open Source Software Peer Review Practices: A Case Study of the Apache Server. In: Proceedings of the 30th International Conference on Software Engineering (ICSE), pp 541–550

  • Rigby PC, Cleary B, Painchaud F, Storey MA, German DM (2012) Contemporary peer review in action: Lessons from open source development. IEEE Softw 29(6):56–61

    Article  Google Scholar 

  • Rigby PC, German DM, Cowen L, Storey MA (2014) Peer review on open-source software projects: Parameters, Statistical Models, and Theory. Transactions on Software Engineering and Methodology (TOSEM) 23(4):Article No. 35

  • Sarle WS (1990) The VARCLUS procedure, 4th edn. SAS Institute, Inc

  • Sauer C, Jeffery DR, Land L, Yetton P (2000) The effectiveness of software development technical reviews: a behaviorally motivated program of research. Trans Softw Eng (TSE) 26(1):1–14

    Article  Google Scholar 

  • Shihab E, Jiang ZM, Hassan AE (2009) Studying the Use of Developer IRC Meetings in Open Source Projects. In: Prceedings of the 25th International Conference on Software Maintenance (ICSM), pp 147–156

  • Shull F, Basili V, Boehm B, Brown AW, Costa P, Lindvall M, Port D, Rus I, Tesoriero R, Zelkowitz M (2002) What We Have Learned About Fighting Defects. In: Proceedings of the 8th International Software Metrics Symposium (METRICS), pp 249–258

  • Tantithamthavorn C, McIntosh S, Hassan AE, Ihara A, Matsumoto K (2015) The Impact of Mislabelling on the Performance and Interpretation of Defect Prediction Models. In: Proceedings of the 37th International Conference on Software Engineering (ICSE), pp 812–823

  • Tao Y, Dang Y, Xie T, Zhang D, Kim S (2012) How Do Software Engineers Understand Code Changes?: An Exploratory Study in Industry. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (FSE), pp 51:1–51:11

  • Thongtanunam P, Yang X, Yoshida N, Kula RG, Ana Erika CC, Fujiwara K, Iida H (2014) ReDA: A Web-based Visualization Tool for Analyzing Modern Code Review Dataset. In: The proceeding of the 30th International Conference on Software Maintenance and Evolution (ICSME), pp 606–609

  • Thongtanunam P, McIntosh S, Hassan AE, Iida H (2015a) Investigating Code Review Practices in Defective Files: An Empirical Study of the Qt System. In: Proceedings of the 12th International Working Conference on Mining Software Repositories (MSR), pp 168–179

  • Thongtanunam P, Tantithamthavorn C, Kula RG, Yoshida N, Iida H, Matsumoto K (2015b) Who Should Review My Code? A File Location-Based Code-Reviewer Recommendation Approach for Modern Code Review. In: Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp 141–150

  • Thongtanunam P, McIntosh S, Hassan AE, Iida H (2016) Revisiting Code Ownership and its Relationship with Software Quality in the Scope of Modern Code Review. In: Proceedings of the 38th International Conference on Software Engineering (ICSE), pp 1039–1050

  • Tsay J, Dabbish L, Herbsleb J (2014) Let’s Talk About It: Evaluating Contributions through Discussion in GitHub. In: Proceedings of the 22nd International Symposium on the Foundations of Software Engineering (FSE), pp 144–154

  • Ukkonen E (1985) Algorithms for approximate string matching. Inf Control 64 (1-3):100–118

    Article  MathSciNet  MATH  Google Scholar 

  • Weißgerber P, Neu D, Diehl S (2008) Small Patches Get In !. In: Proceedings of the 2008 international working conference on Mining software repositories (MSR’08), pp 67–75

  • Zanjani M, Kagdi H, Bird C (2015) Automatically recommending peer reviewers in modern code review. IEEE Trans Softw Eng:1–13

    Google Scholar 

Download references

Acknowledgments

This research was supported by the Grant-in-Aid for JSPS Fellows (Numbers 16J02861).

Author information

Authors and Affiliations

  1. Software Design and Analysis Lab (SDLAB), Nara Institute of Science and Technology, Nara, Japan

    Patanamon Thongtanunam & Hajimu Iida

  2. Department of Electrical and Computer Engineering, McGill University, Montréal, Canada

    Shane McIntosh

  3. Software Analysis and Intelligence Lab (SAIL), Queen’s University, Kingston, Canada

    Ahmed E. Hassan

Authors
  1. Patanamon Thongtanunam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shane McIntosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed E. Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hajimu Iida
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patanamon Thongtanunam.

Additional information

Communicated by: Jeffrey C. Carver

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Thongtanunam, P., McIntosh, S., Hassan, A.E. et al. Review participation in modern code review. Empir Software Eng 22, 768–817 (2017). https://doi.org/10.1007/s10664-016-9452-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-016-9452-6

Keywords

Search

Navigation