Skip to main content

Advertisement

SpringerLink
Log in

Towards a decision-making structure for selecting a research design in empirical software engineering

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Several factors make empirical research in software engineering particularly challenging as it requires studying not only technology but its stakeholders’ activities while drawing concepts and theories from social science. Researchers, in general, agree that selecting a research design in empirical software engineering research is challenging, because the implications of using individual research methods are not well recorded. The main objective of this article is to make researchers aware and support them in their research design, by providing a foundation of knowledge about empirical software engineering research decisions, in order to ensure that researchers make well-founded and informed decisions about their research designs. This article provides a decision-making structure containing a number of decision points, each one of them representing a specific aspect on empirical software engineering research. The article provides an introduction to each decision point and its constituents, as well as to the relationships between the different parts in the decision-making structure. The intention is the structure should act as a starting point for the research design before going into the details of the research design chosen. The article provides an in-depth discussion of decision points in relation to the research design when conducting empirical research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. This is an example of the different opinions related to terminology. Others may refer to this as “research type”.

References

  • Adolp A, Hall W, Kruchten P (2011) Using grounded theory to study the experience of software development. J Empir Softw Eng 16(4):487–513

    Article  Google Scholar 

  • Allison I, Merali Y (2007) Software process improvement as emergent change: a structurational analysis. Inf Softw Technol 49(6):668–681

    Article  Google Scholar 

  • Barney S, Mohankumar V, Chatzipetrou P, Aurum A, Wohlin C, Angelis L (2014) Software quality across borders: three case studies on company internal alignment. Inf Softw Technol 56(1):20–38

    Article  Google Scholar 

  • Basili V (1993) The experimental paradigm in software engineering. Proceedings of the International Workshop on Experimental Software Engineering Issues: Critical Assessment and Future Directions. Springer-Verlag, LNCS 706, London, UK, pp 3–12 Link: http://link.springer.com/content/pdf/10.1007/3-540-57092-6_91.pdf

  • Baskerville R (2008) What design science is not. Eur J Inf Syst 17:441–443

    Article  Google Scholar 

  • Benbasat I, Goldstein DK, Mead M (1987) The case research strategy in studies of information systems. MIS Q 11(3):369–386

    Article  Google Scholar 

  • Bertelsen OW (1997) Towards a unified field of se research and practice. IEEE Softw 14:87–88

    Article  Google Scholar 

  • Bhattacherjee A (2012) Social science research: principles, methods, and practices. USF Open Access Textbooks Collection. Book 3 University of South Florida Link http://scholarcommons.usf.edu/oa_textbooks/3

  • Birks DF, Fernandez W, Levina N, Nasirin S (2013) Grounded theory method in information systems research: its nature, diversity and opportunities. Guest editorial. Eur J Inf Syst 22:1–8

    Article  Google Scholar 

  • Boell S, Cecez-Kecmanivic D (2010) Literature reviews and the hermeneutic circle. Aust Acad Res Libr 41(2):129–144

    Article  Google Scholar 

  • Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101

    Article  Google Scholar 

  • Brooke C (2002) What does it mean to be ‘critical’ in IS research? J Inf Technol 17(2):49–57

    Article  Google Scholar 

  • Butler T (1998) Towards a hermeneutic method for interpretive research in information systems. J Inf Technol 13:285–300

    Article  Google Scholar 

  • Carver J, Seaman C, Jeffery R (2004) Using qualitative methods in software engineering. International Advanced School of Empirical Software Engineering (IASESE04), August 18, 2004, LA CA Link: http://chess.cs.umd.edu/class/fall2004/cmsc735/CMSC735%2011a%20Qualitative%20Analysis.pdf

  • Cecez-Kecmanovic D (2011) Doing critical information systems research–arguments for a critical research methodology. Eur J Inf Syst 20(4):440–455

    Article  Google Scholar 

  • Checkland P (1981) Systems thinking, systems practice. Wiley, UK

    Google Scholar 

  • Chen WS, Hirschheim R (2004) A paradigmatic and methodological examination of information systems research from 1991 to 2001. Inf Syst J 14(3):197–235

    Article  Google Scholar 

  • Coleman G, O’Connor R (2007a) Investigating software process in practice: a grounded theory perspective. J Syst Softw 81:772–784

    Article  Google Scholar 

  • Coleman G, O’Connor R (2007b) Using grounded theory to understand software process improvement: a study of Irish software product companies. Inf Softw Technol 49(6):654–667

    Article  Google Scholar 

  • Collis J, Hussey R (2009) Business research. Palgrave MacMillan, UK

    Google Scholar 

  • Creswell J (2013) Research design: qualitative, quantitative and mixed methods approach. Sage Publication, Thousand Oaks

    Google Scholar 

  • Davison RM, Martinsons MG, Kock N (2004) Principles of canonical action research. Inf Syst J 14(1):65–86

    Article  Google Scholar 

  • Dunne C (2011) The place of the literature review in grounded theory research. Int J Soc Res Methodol 14(2):111–124

    Article  MathSciNet  Google Scholar 

  • Dybå T, Prikladnicki R, Rönkkö K, Seaman CB, Sillito J (2011) Qualitative research in software engineering. Empir Softw Eng 16(4):425–429

    Article  Google Scholar 

  • Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. In: Shull F, Singer J, Sjøberg DIK (eds) Guide to advanced empirical software engineering, Springer Germany, pp 285–311

  • Eisenhardt KM (1989) Building theories from case study research. Academy of management. Acad Manag Rev 14(4):532–550

    Google Scholar 

  • Engel RJ, Schutt RK (2010) The practice of research in social work. Sage Publication, Thousand Oaks

    Google Scholar 

  • Ghanam Y, Maurer F, Abrahamsson P (2012) Making a leap to a software platform strategy: issues and challenges. Inf Softw Technol 54(9):968–984

    Article  Google Scholar 

  • Ghapanchi AH (2011) Dynamic capabilities and project characteristics contributing to the success of open source software projects. PhD Dissertation Thesis. The University of New South Wales, Sydney, NSW Australia

  • Gilbert GN (1995) Dagstuhl seminar on social science microsimulation: a challenge to computer science, SchjoB, May 1–5, 1995

  • Glaser BG (1992) Emergence vs. forcing: basics of grounded theory analysts. Sociology Press, California

    Google Scholar 

  • Glaser BG, Strauss AL (1967) The discovery of grounded theory: strategies for qualitative research. Aldine, New York

    Google Scholar 

  • Gregg DG, Kulkarni UR, Vinze AS (2001) Understanding the philosophical underpinnings of software engineering research in information systems. Inf Syst Front 3(2):169–183

    Article  Google Scholar 

  • Grix J (2002) Introducing students to the generic terminology of social research. Politics 22(3):175–186

    Article  Google Scholar 

  • Hannah DR, Lautsch BA (2011) Counting in qualitative research: why to conduct it, when to avoid it, and when to closet it. J Manag Inq 20(1):14–22

    Google Scholar 

  • Hansen S, Rennecker J (2010) Getting on the same page: collective hermeneutics in a systems development team. Inf Organ 20(1):44–63

    Article  Google Scholar 

  • Harrison RJ, Lin Z, Caroll GR, Carley KM (2007) Simulation modeling in organization and management research. Acad Manag Rev 32(4):1229–1245

    Article  Google Scholar 

  • Hevner AR, March ST, Park J, Ram S (2004) Design science in information systems research. MIS Q 28(1):75–105

    Google Scholar 

  • Jafarzadeh H, Aurum A, D’Ambra J (2011) Factors affecting the success of businesses in effective utilization of search engine advertising. International Conference on Information Systems, 3–7 December, Shanghai, China, 2011, sigIQ pre-ICIS workshop, pp 1 Link: http://www.northeastern.edu/iqworld/papers/Papers/Factors%20Affecting%20the%20Success%20of%20Businesses%20in%20Effective%20Utilization%20of%20Search%20Engine%20Advertising_abstract.pdf

  • Jafarzadeh H, Aurum A, D’ambra J, Abedin B (2013) Determinant of intention to use search engine advertising: a conceptual model. Int J Enterp Inf Syst 9(3):22–38

    Article  Google Scholar 

  • Johnson CF (1996) Deductive versus inductive reasoning: a closer look at economics. Soc Sci J 33(3):287–299

    Article  Google Scholar 

  • Johnson RB, Onwuegbuzie AJ (2004) Mixed methods research: a research paradigm whose time has come. Educ Res 33(7):14–26

    Article  Google Scholar 

  • Kachigan SK (1986) Statistical analysis: an interdisciplinary introduction to univariate & multivariate methods. Radius Press, New York

    Google Scholar 

  • Kitchenham BA, Pfleeger SA (2002) Principles of survey research part 2: designing a survey. ACM SIGSOFT Softw Eng Notes 27(1):18–20

    Article  Google Scholar 

  • Kitchenham BA, Pickard LM, Pfleeger SL (1995) Case studies for method and tool evaluation. IEEE Softw 12(4):52–62

    Article  Google Scholar 

  • Kitchenham BA, Pfleeger SL, Pickard LM, Jones PW, Hoaglin DC, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734

    Article  Google Scholar 

  • Klein HK, Myers MD (1999) A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Q 23(1):67–89

    Article  Google Scholar 

  • Kline RB (2011) Principles and practice of structural equation modeling. Guilford Press, New York

    MATH  Google Scholar 

  • Kontio J, Lehtola L, Bragge J (2004) Using the focus group method in software engineering: obtaining practitioner and user experience. Proceedings of International Symposium of Empirical Software Engineering, Los Angeles, CA, USA, August 2004, IEEE Computer Society Washington, DC, USA, pp 271–280

  • Kyburz-Graber R (2007) Does case–study methodology lack rigour? The need for quality criteria for sound case–study research, as illustrated by a recent case in secondary and higher education. Environ Educ Res 10(1):53–65

    Article  Google Scholar 

  • Law AM (2007) Simulation modeling and analysis, volume 4. McGraw-Hill, New York

    Google Scholar 

  • Lee AS (1989) A scientific methodology for MIS case studies. MIS Q 13(1):33–54

    Article  Google Scholar 

  • Lethbridge TC, Sim SE, Singer J (2005) Studying software engineers: data collection techniques for software field studies. J Empir Softw Eng 10(3):311–341

    Article  Google Scholar 

  • Lyons H (2009) Case Study research methodology for publishing developments in ICT-facilitated learning in higher education—a perspective approach. Innov Educ Teach Int 46(1):27–39

    Article  Google Scholar 

  • Mack N, Woodsong C, MacQueen K, Guest G, Namey E (2005) Qualitative research methods: a data collector’s field guide. Family Health International, Research Triangle Park

    Google Scholar 

  • Marascuilo LA, Serlin RC (1988) Statistical methods for the social and behavioral sciences. W.H. Freeman and Company, New York

    MATH  Google Scholar 

  • March ST, Smith GF (1995) Design and natural science research on information technology. Decis Support Syst 15(4):251–266

    Article  Google Scholar 

  • McKernan J (1996) Curriculum action research: a handbook of methods and resources for the reflective practitioner. Kogan Page, London

    Google Scholar 

  • McLeod L, MacDonell SG, Doolin B (2011) Qualitative research on software development: a longitudinal case study methodology. J Empir Softw Eng 16(4):430–459

    Article  Google Scholar 

  • Mingers J (2001) Combining IS research methods: towards a pluralist methodology. Inf Syst Res 12(3):240–259

    Article  Google Scholar 

  • Mkansi M, Acheampong EA (2012) Research philosophy debates and classification: students’ dilemma. Electron J Bus Res Methods 10(2):132–140

    Google Scholar 

  • Moe NB, Aurum A, Dybå T (2012) Challenges of shared decision-making: a multiple case study of agile software development. Inf Softw Technol 54(8):853–865

    Article  Google Scholar 

  • Müller M, Pfahl D (2008) Simulation methods. In: Shull F, Singer J, Sjøberg DIK (eds) Guide to advanced empirical software engineering. Springer, Germany

    Google Scholar 

  • Myers MD (1995) Dialectical hermeneutics: a theoretical framework for the implementation of information systems. Inf Syst J 5(1):51–70

    Article  Google Scholar 

  • Myers MD (1997) Qualitative research in information systems. MIS Q 21(2):241–242

    Article  Google Scholar 

  • Myers MD, Klein HK (2011) A set of principles for conducting critical research in information systems. MIS Q 35(1):17–36

    Google Scholar 

  • Nunamaker JF Jr, Chen M, Purdin TDM (1991) Systems development in information systems research. J Manag Inf Syst 7(3):89–106

    Google Scholar 

  • Orlikowski WJ, Baroudi JJ (1991) Studying information technology in organizations: research approaches and assumptions. Inf Syst Res 2(1):1–28

    Article  Google Scholar 

  • Ostrowski L, Helfert M (2011) Commonality in various design science methodologies. Proceedings of the Federated Conference on Computer Science and Information Systems, pp 317–320. ISBN 978-83-60810-39-2

  • Perry C (1998) Processes of a case study methodology for postgraduate research in marketing. Eur J Mark 32(9/10):785–802

    Article  Google Scholar 

  • Perry DE, Porter AA, Votta LG (2000) Empirical studies of software engineering: a roadmap. In: Finkelstein A (ed) The future of software engineering. ACM Press, New York

    Google Scholar 

  • Pinsonneault A, Kraemer K (1993) Survey research methodology in management information systems: an assessment. J Manag Inf Syst 10(2):75–105

    Google Scholar 

  • Rossi P, Freeman HF (1993) Evaluation: a systematic approach. Sage Publication, USA

    Google Scholar 

  • Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. J Empir Softw Eng 14(2):131–164

    Article  Google Scholar 

  • Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering: guidelines and examples. Wiley, USA

    Book  Google Scholar 

  • Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng 25(4):557–573

    Article  Google Scholar 

  • Shanks G, Parr A (2003) Positivist, single case study research in information systems: a critical analysis. Proceedings of the 11th European Conference on Information Systems, Naples, Italy 16-21 June 2003, pp 1760–1774 Link: http://is2.lse.ac.uk/asp/aspecis/20030140.pdf

  • Shaw M (2002) What makes good research in software engineering? Int J Softw Tools Technol Transfer 4(1):1–7

    Article  Google Scholar 

  • Shaw M (2003) Writing good software engineering research papers. Proceedings of 25th Int Conference on Software Engineering, Portland, Oregon, USA, May 2003, IEEE Computer Society Washington, DC, USA, pp 726–736

  • Shull F, Singer J, Sjøberg DIK (2008) Guide to advanced empirical software engineering. Springer, Germany

    Book  Google Scholar 

  • Shye S (1988) Inductive and deductive reasoning: a structural reanalysis of ability tests. J Appl Psychol 73(2):308–311

    Article  MathSciNet  Google Scholar 

  • Siegel S, Castellan NJ Jr (1988) Nonparametric statistics for the behavioral sciences. Mcgraw-Hill Book Company, New York

    Google Scholar 

  • Sjøberg DIK, Dybå T, Jørgensen M (2007) The future of empirical methods in software engineering. IEEE Proceedings of Future of Software Engineering (FOSE)

  • Smite D, Wohlin C, Gorschek T, Feldt R (2010) Empirical evidence in global software engineering: a systematic review. Empir Softw Eng Int J 15(1):91–118

    Article  Google Scholar 

  • Smite D, Wohlin C, Aurum A, Jabangwe R, Numminen E (2013) Offshore insourcing in software development: structuring the decision-making process. J Syst Softw 86(4):1054–1067

    Article  Google Scholar 

  • Staron M, Kuzniarz L, Wohlin C (2006) Empirical assessment of using stereotypes to improve comprehension of UML models: a set of experiments. J Syst Softw 79(5):727–742

    Article  Google Scholar 

  • Strauss AL, Corbin JM (1998) Basics of qualitative research: techniques and procedures for developing grounded theory. Sage Publication, Thousand Oaks

    Google Scholar 

  • Susman G, Evered R (1978) An assessment of the scientific merits of action research. Adm Sci Q 23(4):582–603

    Article  Google Scholar 

  • Tichy WF, Lukowicz P, Prechelt L, Heinz EA (1995) Experimental evaluation in computer science: a quantitative study. J Syst Softw 28(1):9–18

    Article  Google Scholar 

  • Tom E, Aurum A, Vidgen R (2013) An exploration of technical debt. J Syst Softw 86(6):1498–1516

    Article  Google Scholar 

  • Urquhart C (2002) Regrounding grounded theory–or reinforcing old prejudices? a brief reply to bryant. J Inf Technol Theory Appl 4(3):43–54

    Google Scholar 

  • Walker D, Myrick F (2006) Grounded theory: an exploration of process and procedure. Qual Health Res 16(4):547–559

    Article  Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, ISBN 978-3-642-29043-5

  • Wynekoop JL, Russo NL (1997) Studying system development methodologies: an examination of research methods. Inf Syst J 7(1):47–65

    Article  Google Scholar 

  • Yin RK (2002) Case study research. Sage Publication, CA

    Google Scholar 

  • Zelkowitz MV, Wallace D (1997) Experimental validation in software engineering. Inf Softw Technol 31(5):23–31

    Google Scholar 

Download references

Acknowledgments

The Knowledge Foundation in Sweden partially funded this work under a research grant for the Blekinge Engineering Software Qualities (BESQ+) research environment.

Author information

Authors and Affiliations

  1. Blekinge Institute of Technology, Karlskrona, Sweden

    Claes Wohlin

  2. Sydney, NSW, Australia

    Aybüke Aurum

Authors
  1. Claes Wohlin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aybüke Aurum
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Claes Wohlin.

Additional information

Communicated by: Jeffrey C. Carver

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wohlin, C., Aurum, A. Towards a decision-making structure for selecting a research design in empirical software engineering. Empir Software Eng 20, 1427–1455 (2015). https://doi.org/10.1007/s10664-014-9319-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-014-9319-7

Keywords

Search

Navigation