Methodologies for Performing Empirical Studies: Report from the International Workshop on Empirical Studies of Software Maintenance

Abstract

The International Workshop on Empirical Studies of Software Maintenance workshop took place following the International Conference on Software Maintenance in Monterey, California. The focus of the workshop was on experimental quantitative and qualitative studies of software maintenance processes. Of particular interest were the design of empirical studies, their underlying methodologies and techniques, and the lessons learned from them. This is the paper resulting from the charge to the group ‘Methodologies for Performing Empirical Studies’. A description of each paper in the group is presented, along with a summary of the discussion. The sessions were summarized as follows:

continue to address so-called ‘stale’ research questions

carefully define all research constructs and models

offer insights on the perceived level of generality of the research results

use research methods well-suited to the problem, and use them rigorously

combine methods where this would add significant additional insight—collaborate where necessary to achieve this goal

don't ignore factors relating to maintainers (e.g., ability and experience) despite the known difficulties in their measurement

maintain a strong linkage to practice to ensure the research's continued relevance.

The group concluded with a high degree of agreement and encouragement that the field was moving in appropriate directions.