Skip to main content
SpringerLink
Log in

Large language models for qualitative research in software engineering: exploring opportunities and challenges

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

The recent surge in the integration of Large Language Models (LLMs) like ChatGPT into qualitative research in software engineering, much like in other professional domains, demands a closer inspection. This vision paper seeks to explore the opportunities of using LLMs in qualitative research to address many of its legacy challenges as well as potential new concerns and pitfalls arising from the use of LLMs. We share our vision for the evolving role of the qualitative researcher in the age of LLMs and contemplate how they may utilize LLMs at various stages of their research experience.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data Availability

No datasets were generated or analysed during the current study.

Notes

  1. https://www.csiro.au/en/news/all/articles/2023/june/humans-and-ai-hallucinate.

  2. https://www.cyberdaily.au/digital-transformation/9779-researchers-apologies-to-big-4-consultancy-firms-for-false-ai-based-accusations.

  3. https://c3.ai/glossary/data-science/model-drift/.

  4. https://www.sciencedirect.com/journal/information-and-software-technology/publish/guide-for-authors.

References

  • Alkaissi, H., McFarlane, S.I.: Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus 15, 192 (2023)

    Google Scholar 

  • Arora, C., John, G., Mohamed, A.: Advancing requirements engineering through generative AI: assessing the role of LLMs. (2023) arXiv preprint arXiv:2310.13976.

  • Balel, Y.: The role of artificial intelligence in academic paper writing and its potential as a co-author’, Euro. J. Therap.. (2023)

  • Bano, M., Didar Z., Jon W.: Exploring qualitative research using LLMs. (2023) arXiv preprint arXiv:2306.13298.

  • Bender, E.M., Timnit G., Angelina M.-M., Shmargaret S.: On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp 610–23 (2021)

  • Byun, C., Piper, V., Kevin, S.: Dispensing with Humans in Human-Computer Interaction Research. In: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–26 (2023)

  • Easterbrook, S., Singer, J., Storey, M.A., Damian, D.: Selecting empirical methods for software engineering research. Guide to Adv. Emp. Softw. Eng. 8, 285–311 (2008)

    Article  Google Scholar 

  • Ebert, C., Louridas, P.: Generative AI for software practitioners. IEEE Softw. 40, 30–38 (2023)

    Article  Google Scholar 

  • Emmert-Streib, F.: Importance of critical thinking to understand ChatGPT. Europ. J. Human Genet. 15, 1–2 (2023)

    Google Scholar 

  • Gentles, S.J., Cathy, C., Jenny, P., Ann Mckibbon, K.: Sampling in qualitative research: insights from an overview of the methods literature. Qual. Rep. 20, 1772–1789 (2015)

    Google Scholar 

  • Hoda, R.: Socio-technical grounded theory for software engineering. IEEE Transaction Software Engineering. 48, 3808–3832 (2021)

    Article  Google Scholar 

  • Hou, X., Yanjie, Z., Yue, L., Zhou, Y., Kailong, W., Li, L., Xiapu, L., David, L., John, G., Haoyu, W.: Large language models for software engineering: a systematic literature review. arXiv preprint arXiv:2308.10620

  • Jalil, S., Suzzana, R., Thomas, D.L., Kevin, M., Wing, L.: Chatgpt and software testing education: Promises & perils. In: 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 4130–37. IEEE (2023)

  • Jiang, D., Xiang R., Bill Y-L.: LLM-blender: Ensembling large language models with pairwise ranking and generative fusion. (2023) arXiv preprint arXiv:2306.02561.

  • Kitchenham, B.: Procedures for performing systematic reviews. Keele UK Keele Univ. 33(2004), 1–26 (2004)

    Google Scholar 

  • Kuhail, M.A., Sujith, S.M., Ashraf, K., Jose, B., Syed J.S.: Will I be replaced? Assessing chatgpt’s effect on software development and programmer perceptions of Ai tools. Assessing Chatgpt’s Effect on Software Development and Programmer Perceptions of Ai Tools.

  • Navigli, R., Simone, C., and Björn, R.: Biases in large language models: origins, inventory and discussion. ACM J. Data Inform. Qual. (2023)

  • Nguyen-Duc, A., Beatriz C.-D., Adam, P., Chetan, A., Dron, K., Tomas, H., Usman, R., Jorge, M., Eduardo, G., Kai-Kristian K.: Generative artificial intelligence for software engineering–a research Agenda, (2023) arXiv preprint arXiv:2310.18648.

  • Ozkaya, I.: Application of large language models to software engineering tasks: opportunities. Risks Implicat. IEEE Software. 40, 4–8 (2023)

    Google Scholar 

  • Polonsky, M.J., Jeffrey D.R.: Should artificial intelligent agents be your co-author? Arguments in favour, informed by ChatGPT. In: 91–96. SAGE Publications Sage UK: London, England (2023)

  • Rudolph, J., Tan, S., Tan, S.: ChatGPT: bullshit spewer or the end of traditional assessments in higher education? J. Appl. Learn. Teach. 24, 6 (2023)

    Google Scholar 

  • Scoccia, G.L.: Exploring Early Adopters’ Perceptions of ChatGPT as a Code Generation Tool. In: 2023 38th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW), pp 88–93 (2023)

  • Treude, C., Hideaki H.: She Elicits Requirements and he tests: software engineering gender bias in large language models. (2023) arXiv preprint arXiv:2303.10131.

  • Watkins, R.: Guidance for researchers and peer-reviewers on the ethical use of large language models (LLMs) in scientific research workflows. AI Ethics 16, 1–6 (2023)

    Google Scholar 

  • Watson, C.: Unreliable narrators?’Inconsistency’(and some inconstancy) in interviews. Qual. Res. 6, 367–384 (2006)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CSIRO’s Data61, Clayton, Australia

    Muneera Bano & Didar Zowghi

  2. Monash University, Melbourne, Australia

    Rashina Hoda

  3. University of Melbourne, Melbourne, Australia

    Christoph Treude

Authors
  1. Muneera Bano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rashina Hoda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Didar Zowghi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christoph Treude
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MB, RH, and DZ contributed to the ideation. MB and RH wrote the main manuscript text. DZ and CT reviewed, updated, and improved the manuscript. MB and RH prepared the final version.

Corresponding author

Correspondence to Muneera Bano.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bano, M., Hoda, R., Zowghi, D. et al. Large language models for qualitative research in software engineering: exploring opportunities and challenges. Autom Softw Eng 31, 8 (2024). https://doi.org/10.1007/s10515-023-00407-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10515-023-00407-8

Keywords

Search

Navigation