Multi-criteria decision making approach for the selection of software effort estimation model

Software development with minimum effort has become a challenging task for the software developers. Software effort may be defined as the prediction process of the effort required to develop any software. Many software effort estimation models have been developed in the past, but it is observed that none of them can be applied successfully in all kinds of projects in different environments that raise the problem of the software effort estimation model selection. To select the suitable software effort estimation model, many conflicting selection criteria must be considered in the decision process. The present study emphasizes on the development of a fuzzy multi-criteria decision making approach by integrating Fuzzy Set Theory and Weighted Distance Based Approximation. To show the consistency of the proposed approach, methodology validation is also performed by making comparison with existing methodologies and to check the criticality of the selection criterion, sensitivity analysis is also performed.


Introduction
In today scenario, software has grown to be a very significant component for all kinds of systems and software effort estimation is of vital importance for the well balanced management of imminent software.The management, in terms of software effort, is necessary due to the speedy increase in the need of complex software systems at a large scale.Software effort estimation is referred as a prediction process to calculate the most rational amount of efforts required to develop any software based on imperfect and undecided inputs.Effort estimates can be used as input at various stages of the software development process as project plans, budgets, iteration plans, and investment analyses.The prime motive of the software developers is to develop software with better quality by minimizing the required effort.So, it can be stated that the success of any software to some extent relies on the perfect effort estimation.The inaccurate estimation of the required effort for the development of any software will affect adversely to both the customers and developers of the software because it can result in bad quality software, delay in the delivery time, misuse of software and contact cancellation, etc.
During the past decades, a number of models like expert estimation, formal estimation models and combinational estimation models have been developed by the various researchers to portray the software effort estimation.The main center of attention of most of the researchers is based on the creation of new software effort estimation models that results in a new emerging research area as selection of software effort estimation model.Although, the available software effort models are capable of calculating the total effort required to develop any software, but the accuracy differences of various software effort estimation models put forward that no model can be considered as the best one in all the cases.Due to the aforementioned requirements, the researchers focus on an emerging research problem termed as selection of software effort estimation model/technique.A comprehensive systematic review of the past literature enforces that the problem of selection and ranking of software effort estimation models can be represented in the form of multi-criteria decision making problem.The multi-criteria decision making (MCDM) problems are those problems in which a set of existing alternatives are evaluated on a pre-identified set of evaluation criteria and the information about that "which alternative is best" is obtained as the result.The present research argues on the problem "evaluation and selection of software effort estimation models" by modeling it as an MCDM problem.Further, a set of selection criteria is proposed that can contribute in the evaluation process of various software effort estimation models.The rest of the paper is ordered as: section 2 includes the literature review in support to the present problem, the proposed methodology in section 3, Evaluation framework development in section 4 is provided.Section-5 contains model demonstration to show the utility of the proposed methodology followed by the methodology validation and sensitivity analysis in section 6.The results are analyzed in section 7 under major findings and discussions of the paper and section 8 contains conclusion drawn from the present research.

Literature Review
An extensive literature survey into two parts was carried out in the context of the present problem of software effort estimation model selection.In the first part, the various models developed by the various researchers were studied with their associated advantages and disadvantages.It is inferred from that the software effort estimation models provided in the past are mainly categorized into two categories namely (i) Algorithmic and (ii) Non-Algorithmic models (Abbas et al. 2012;Pandey, 2013;Ramesh & Karunanidhi 2013).In the second part, the various evaluation metrics that contribute in the evaluation process of the software effort estimation models were analyzed.Leung and Fan provide selection criteria as mean relative error (MRE), mean absolute relative error (MARE), balance relative error (BREbias) to solve the present problem (Leung & Fan, 2002).Moløkken-Østvold et al. (2004) provided mean relative error (MRE), balance relative error bias (BREbias), balance relative error (BRE) and accuracy (ACC) as the selection criteria for the evaluation of software effort estimation models.Tim Menzies et al. (2006) proposed a methodology based on heuristic rejection rules named Coseekmo to rank the various software effort estimation models by considering mean relative error (MRE), mean magnitude of relative error (MMRE) and prediction (PRE) as selection criteria.Basha and Ponnurangam (2010) proposed several evaluation metrics as mean relative error (MRE), mean magnitude of relative error (MMRE), prediction (PRE), root mean square (RMS), relative root mean square (RRMS) for the ranking of software effort estimation models.In the contemporary work, Kaur et al. (2010) used some attributes as mean magnitude of relative error (MMRE), mean square error (MSE), root mean square error (RMSE) and root mean square error (RMSSE) as the selection criteria.Sehra et al. (2013) proposed a model based on Fuzzy Analytic Hierarchy Process by accounting reliability (REL), mean magnitude of relative error (MMRE), Prediction (PRE) and uncertainty (UNC) as selection criteria.Malathi and Sridhar (2012) proposed different selection criteria for the selection and raking of effort estimation models such as prediction (PRED), value accounted for (VAF), variance absolute relative error (VARE), mean absolute relative error (MARE), magnitude of relative error (MRE), root mean square error (RMSE).Wen et al. (2012) provide prediction (PRED), mean magnitude of relative error (MMRE) and median magnitude of relative error (MDMRE) for the purpose of selection and evaluation of models.Garcia-Diaz et al. (2013) developed a methodology based on fuzzy logic for the comparison of two fuzzy logic models for software development effort estimation.Prediction (PRE), mean error relative (MER), (mean magnitude of error relative) MMER were used as selection criteria in this research.Mittas and Angelis (2013) ranked various effort estimation models by using different criteria as mean absolute error (MAE), magnitude of relative error (MRE) and mean magnitude of relative error (MMRE).Preeth et al. (2014) used three selection criteria for the evaluation of effort estimation model such as magnitude of relative error (MRE), mean magnitude of relative error (MMRE) and prediction (PRED).Nayebi et al. (2015) used different criteria such as prediction (PRED), correlation coefficient (CORR) and Bayesian information correlation (BIC) for the selection of different models.It can be inferred that the problem of selection of software effort estimation model can be represented as a multi-criteria decision making (MCDM) problem.The use of the MCDM approaches to solve the real life problems such as vendor selection (Jarial and Garg, 2012), inventory policy selection (Amit et al., 2014), Software engineering matrices selection (Garg et al., 2013), have been widely accepted in the present time.

Proposed Methodology
In the present research, two well-known approaches namely, Fuzzy Set Theory (FST) and Weighted Distance Based Approximation are adopted first time to solve the problem of software effort estimation model selection.FST was first proposed by Zadeh (1965) to deal with the vagueness of the data.A fuzzy set is an ordered pair (F, x) where F is a set and : [0,1] x F  i.e. membership function.Fuzzy set theory has already applied in various areas, i.e. scientific environment, control theory, robotics, etc.A triangular fuzzy number (TFN) is usually used to deal with the vagueness of the data which is related to the performance of alternatives with respect to each index.A TFN defined by triplet T = (u, v, w) can be expressed as Eq. ( 1).The representation of TFN membership function is shown in Fig. 1.  (u1, v1, w1) and T2 = (u2, v2, w2).Some algebraic operations of the TFNs T1 and T2 can be represented as: Addition, Subtraction and Multiplication ' ' can be represented as: Weighted Distance Based Approximation (WDBA) measures the distance from the optimal point (best value of alternatives) and non-optimal point (worst value of alternatives).Finally, the alternatives are ranked according to their suitability index.The alternative having the least value of suitability index is ranked at first position and with the maximum value at last.WDBA approach has already been applied in many fields as plant layout design (Rao & Singh, 2012), E-learning websites selection (Jain et al. 2016), COTS component selection (Garg et al., 2016).The steps of WDBA method are given below.
Step 2: Calculate the weighted decision rating matrix (W) by using the Eq. ( 2), where wj is the weight of j th index.
Step 4: Calculate the distance matrix and composite distance matrix.Finally, the value of suitability index (SI) is obtained by using the Eq. ( 6).

Development of Evaluation Framework
To meet the main concern of the present research, a hierarchical structured evaluation framework for the evaluation and selection of software effort estimation model (SEEM) is being portrayed based on Fuzzy Weighted Distance based Approximation i.e.FWDBA method considering various evaluation criteria/sub-criteria and their interactions that is capable to work out with the multi-criteria decision making problems in an effective and efficient way.The hierarchy framework has three levels; the first level is a goal level.In the present problem, the goal is to select the best software effort estimation model among several models.The second level contains evaluation criteria against which software effort estimation models are evaluated.Finally, the third level contains the various software effort estimation models being evaluated.The complete hierarchical framework developed in the present research is shown in Fig. 2.

Model Demonstration
The step-wise working of the developed evaluation framework for the evaluation and selection of software effort estimation model in the present research follows the three steps as described in next subsections.

Identification and Selection of Models/Criteria
The first step in the evaluation process involves the identification of various software effort estimation models and the various selection criteria against which the models can be evaluated.After going through an extensive literature survey, total eleven software effort estimation models falling under the category of algorithmic models and total eleven selection criteria were identified and selected for the evaluation purpose.The description of these software effort estimation models and the selection criteria is provided in Table 1.

Data collection
Due to lack of maturity in the available secondary data, primary data was collected through the well designed questionnaires.Firstly, a team of five experts (E1, E2, E3, E4, E5) was established deliberately from academia, laboratories as well as IT industries.All the experts have more than 15 years experience in the field of the software effort estimation/software development.Secondly, two questionnaires were designed to collect the data regarding the priority weights of the selection criteria and the performance ratings of the software effort estimation models.The experts were asked to provide this data in the linguistic terms on a 7-point fuzzy scale that is shown in Table 2.The data collection in the present research was carried out mainly in two phases as (i) Selection criteria weights data collection and (ii) performance ratings of the models data collection.

Selection criteria weights
The experts provide the data about the priority weights of the selection criteria in the linguistic terms and then the data so obtained was converted in triangular fuzzy numbers and finally a crisp score was calculated by applying average operation.The data obtained about the priority weights is provided in Table 3.As described in the above section 5.2.1, the same procedure was adopted to collect the data concerning to the performance rating of the various software effort estimation models against a set of selection criteria.The data so obtained in linguistic terms from the experts is provided in Table 4. Now, the data provided in Table 5 is converted into single numeric value known as crisp value to get the final performance ratings of each software effort estimation model that is provided in Table 5.

Evaluation, Selection and Ranking
In the present research, the various software effort estimation models are ranked according to the suitability index value obtained by implementing the proposed methodology i.e.Distance Based Approximation.In the evaluation, selection and ranking procedure, the decision rating matrix is formulated first by taking priority weights of the selection criteria and the performance ratings of the models together.
(3-5) as: Finally, the value of suitability index (SI) is obtained using equation ( 6), and the final ranking of eleven software effort estimation models based on their suitability index is shown in Table 6 and Fig. 3.  To validate the results obtained from proposed approach i.e.F-WDBA, the present software effort estimation model selection problem is also solved using the same dataset presented in illustrated example is solved using two well-known MCDM approaches namely, AHP and TOPSIS.The comparative ranking statistics of the software effort estimation models abbreviated as (M1-M11) based on eleven selection criteria (C1-C11) using AHP, TOPSIS and F-WDBA along-with ranking differences are given in Table 7 and the comparison of the ranking obtained from three approaches as WDBA, AHP and TOP-SIS are represented in Fig. 4.
Further, Spearman's rank correlation test is also performed to test the existence of relationship between the rankings obtained from F-WDBA, AHP and TOPSIS.The test statistics for the Spearman's rank correlation test are given in Table 8.According to Spearman's test, any value of Spearman's Rank greater than 0.6 is considered good and shows the positive significant relationship between the results.It is observed that the value of Spearman's rank obtained in present work is 0.955 and 0.918 that affirms the strong positive relationship between the rankings obtained from WDBA, AHP and TOPSIS.In order to check the criticality of the selection criteria, one-way sensitivity analysis is also carried out in the present research.The criticality of selection criteria here means that which selection criteria is most important to which alternative i.e. software effort estimation model and which one is least important for the alternatives.To perform the sensitivity analysis in the present work, the performance rating of each software effort estimation model is increased by 10% for each selection criteria one by one and the impact of these changes in the performance ratings on the suitability index of each software effort estimation model is analyzed that is represented in Figure 5 that provides very useful information about the criticality of the selection criteria for each software effort estimation model.

Major findings and discussions
The present research emphasizes on the evaluation and selection of eleven software effort estimation models based on eleven selection criterion by implementing a well known MCDM approach namely, F-WDBA.Based on the results so obtained, some major findings of the research were discussed as given below.
In the present research, Fuzzy Weighted Distance Based Approach i.e.F-WDBA is applied to solve the problem software effort estimation model selection.According to proposed approach, the software effort estimation having the smallest value of suitability index is ranked at top position i.e. number-1 and model with largest value of suitability index is ranked at last position.It is depicted in

Fig. 3 .
Fig. 3. Rankings of Software Effort Estimation Models obtained from F-WDBA

Fig. 5 .
Fig. 5. Sensitivity Analysis Statistics of Software Effort Estimation Models

Table 1
Description of Software Effort Estimation Models and Selection Criteria

Table 3
Selection Criteria Weights Calculation

Table 4
Performance Ratings of Software Effort Estimation Models (Linguistic Terms)

Table 5
Performance Ratings of Software Effort Estimation Models

Table 6
Rankings of Software Effort Estimation Models using F-WDBA Method

Table 7
Comparison of the rankings obtained from AHP, TOPSIS and F-WDBA method

Table 8
Spearman's Rank Correlation Test Statistics Table3and Fig.2that Cocomo Model abbreviated as (M8) is ranked at top i.e. number-1 having least value of suitability index as 0.401528 followed by Jenson Model (M11) at number-2 with 0.492655 whereas Boeing Model (M4) is ranked at last position i.e. number-11 having the largest suitability index as 0.717730.To validate the results obtained by implementing proposed approach in the present research, a comparison is made with the existing MCDM approaches as AHP and TOPSIS.Further, Spearman's Rank Correlation test is carried out that depicts a strong positive relationship between the rankings obtained from