Alex Borges
Sergio Soares
ABSTRACT
Context: Empirical studies are gaining recognition in the Software Engineering (SE) research community, allowing improved quality of research and accelerating the adoption of new technologies in the software market. However, empirical studies in this area are still limited. In order to foster empirical research in SE, it is essential to understand the resources available to aid these studies. Goal: Identify support mechanisms (methodology, tool, guideline, process, etc.) used to conduct empirical studies in the Empirical Software Engineering (ESE) community. Method: We performed a systematic mapping study that included all full papers published at EASE, ESEM and ESEJ since their first editions. Were selected 891 studies between 1996 and 2013. Results: A total of 375 support mechanisms were identified. We provide the full list of mechanisms and the strategies that uses them. Despite this, we identified a high number of studies that do not cite any mechanism to support their empirical strategies: 433 studies (48%). Experiment is the strategy that has more resources to support their activities. And guideline was the most used type of mechanism. Moreover we observed that the most mechanisms used as reference to empirical studies are not specific to SE area. And some mechanisms were used only in specific activities of empirical research, such as statistical and qualitative data analysis. Experiment and case studies are the strategies most applied. Conclusions: The use of empirical methods in SE has increased over the years. Despite this, many studies did not apply these methods and do not cite any resource to guide their research. Therefore, the list of support mechanisms, where and how they were applied is a major asset to the SE community. Such asset can encourage empirical studies aiding the choice regarding which strategies and mechanisms to use in a research, as well as pointing out examples where they were used, mainly to novice researchers. We also identified new perspectives and gaps that foster other research for the improvement of empirical research in this area.
- Sjoberg, D. I., Dyba, T., and Jorgensen, M. 2007. The Future of Empirical Methods in Software Engineering Research. In 2007 Future of Software Engineering (May 23--25, 2007). International Conference on Software Engineering. IEEE Computer Society, Washington, DC, 358--378. Google Scholar
Digital Library
- N. Juristo and A. M. Moreno. Basics of software engineering experimentation. Springer Publishing Company, Incorporated, 2010. Google ScholarDigital Library
- C. Wohlin, P. Runeson, M. Hst, M. C. Ohlsson, B. Regnell, and A. Wessln. Experimentation in software engineering. Springer Publishing Company, Incorporated, 2012. Google ScholarDigital Library
- V. R. Basili. The role of experimentation in software engineering: past, current, and future. In Proceedings of the 18th international conference on Software engineering, pages 442--449. IEEE Computer Society, 1996. Google ScholarDigital Library
- V. R. Basili, R. W. Selby, and D. H. Hutchens. Experimentation in software engineering. Software Engineering, IEEE Transactions on, (7): 733--743, 1986. Google ScholarDigital Library
- G. H. Travassos, P. S. M. dos Santos, P. Neto, and J. Biolchini. An environment to support large scale experimentation in software engineering. In Engineering of Complex Computer Systems, 2008. ICECCS 2008. 13th IEEE on, pages 193--202. IEEE, 2008. Google ScholarDigital Library
- B. Kitchenham, L. Pickard, and S. L. Pfleeger. Case studies for method and tool evaluation. Software, IEEE, 12(4): 52--62, 1995. Google ScholarDigital Library
- W. F. Tichy. Should computer scientists experiment more? Computer, 31(5): 32--40, 1998. Google ScholarDigital Library
- Kitchenham, B., and S. Charters. Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical Report. Keele University and University of Durham. 2007.Google Scholar
- S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian. Selecting empirical methods for software engineering research. In Guide to advanced empirical software engineering, pages 285--311. Springer, 2008.Google ScholarCross Ref
- Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., Linkman, S. Systematic literature reviews in software engineering - A systematic literature review. J. of Information and Software Technology. 51, 1 (Jan. 2011), 7--15. Google ScholarDigital Library
- T. Dyba, B. A. Kitchenham, and M. Jørgensen. Evidence-based software engineering for practitioners. Software, IEEE, 22(1): 58--65, 2005. Google ScholarDigital Library
- R. K. Yin. Case study research: Design and methods, volume 5. Sage, 2009.Google Scholar
- M. Jørgensen. A strong focus on low price when selecting software providers increases the likelihood of failure in software outsourcing projects. In Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering, pages 220--227. ACM, 2013. Google ScholarDigital Library
- V. R. B. G. Caldiera and H. D. Rombach. The goal question metric approach. Encyclopedia of software engineering, 2: 528--532, 1994.Google Scholar
- B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. El Emam, and J. Rosenberg. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, 28(8): 721--734, 2002. Google ScholarDigital Library
- Kitchenham, B. 2004. Procedures for performing systematic reviews. Technical Report. Keele University at Staffordshire.Google Scholar
- J. Corbin and A. Strauss. Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage, 2008.Google Scholar
- B. G. Glaser and A. L. Strauss. The discovery of grounded theory: Strategies for qualitative research. Transaction Books, 2009.Google Scholar
- V. R. Basili, F. Shull, and F. Lanubile. Building knowledge through families of experiments. Software Engineering, IEEE Transactions on, 25(4): 456--473, 1999. Google ScholarDigital Library
- C. Robson. Real world research: A resource for social scientists and practitioner-researchers, volume 2. Blackwell Oxford, 2002.Google Scholar
- A. Jedlitschka and D. Pfahl. Reporting guidelines for controlled experiments in software engineering. In Empirical Software Engineering, 2005. 2005 International Symposium on, pages 10-pp. IEEE, 2005.Google Scholar
- Seaman, C. B. 1999. Qualitative Methods in Empirical Studies of Software Engineering. IEEE Transactions on Software Engineering, 25(4), 557--572. Google ScholarDigital Library
- P. Runeson, M. Host, A. Rainer, and B. Regnell. Case study research in software engineering: Guidelines and examples. Wiley, 1 edition, April 10, 2012. Google ScholarDigital Library
- S. L. Plfeeger and B. A. Kitchenham. Principles of survey research: part 1: turning lemons into lemonade. ACM SIGSOFT Software Engineering Notes, 26(6): 16--18, 2001. Google ScholarDigital Library
- A. Gordon. Surveymonkey.com A web-based survey and evaluation system: http://www.surveymonkey.com. The Internet and Higher Education, 5(1): 83--87, 2002.Google ScholarCross Ref
- M. Hammersley and P. Atkinson. Ethnography: Principles in practice. Psychology Press, 1995.Google Scholar
- D. M. Fetterman. Ethnography: Step-by-step, volume17. Sage, 2010.Google Scholar
- D. E. Avison, F. Lau, M. D. Myers, and P. A. Nielsen. Action Research. Communications of the ACM, vol. 42, pp. 94--97, 1999. Google ScholarDigital Library
- R. L. Baskerville. Investigating information systems with action research. Communications of the AIS, 2(3es): 4, 1999. Google ScholarDigital Library
- C. Passos, D. S. Cruzes, T. Dyba, and M. Mendon ca. Challenges of applying ethnography to study software practices. In Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement, pages 9--18. ACM, 2012. Google ScholarDigital Library
Index Terms
- Support mechanisms to conduct empirical studies in software engineering: a systematic mapping study
Recommendations
Support mechanisms to conduct empirical studies in software engineering
ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and MeasurementContext: Empirical studies are gaining recognition in the Software Engineering (SE) research community. In order to foster empirical research, it is essential understand the environments, guidelines, process, and other mechanisms available to support ...
Empirical software engineering for agent programming
AGERE! 2012: Proceedings of the 2nd edition on Programming systems, languages and applications based on actors, agents, and decentralized control abstractionsEmpirical software engineering is a branch of software engineering in which empirical methods are used to evaluate and develop tools, languages and techniques. In this position paper we argue for the use of empirical methods to advance the area of agent ...
Empirical software product line engineering: a systematic literature review. an IST journal publication
SPLC '21: Proceedings of the 25th ACM International Systems and Software Product Line Conference - Volume AThe adoption of Software Product Line Engineering (SPLE) is usually only based on its theoretical benefits instead of empirical evidences. In fact, there is no work that synthesizes the empirical studies on SPLE. This makes it difficult for researchers ...
Comments