COTS evaluation using modified TOPSIS and ANP

https://doi.org/10.1016/j.amc.2005.11.006Get rights and content

Abstract

This paper models the COTS evaluation problem as an MCDM problem and proposes a five-phase COTS selection model, combining the technique of ANP (analytic network process) and modified TOPSIS (technique for order performance by similarity to idea solution). This article discusses using the ANP to determine the relative weights of multiple evaluation criteria. The modified TOPSIS approach is used to rank competing products in terms of their overall performance. To illustrate how the approach is used for the COTS evaluation problem, an empirical study of a real case is conducted. The case study demonstrates the effectiveness and feasibility of the proposed evaluation procedure.

Introduction

Given the high interest in motivation to the use of commercially available software, the evaluation and selection of commercial-off-the-self (COTS) products is an important activity in the software development projects. Selecting an appropriate COTS product is often a non-trivial task in which multiple criteria need to be careful consideration. According to our observation, many decision makers select COTS products according to their experience and intuition. This approach is obviously subjective and its weakness has been addressed in several previous studies [8], [9], [10], [12].

Alternatively, multiple criteria decision making (MCDM) approach is used in ranking or selecting one or more COTS products from a set of available alternatives with respect to multiple evaluation criteria. MCDM provides an effective framework for COTS products comparison involving the evaluation of multiple criteria. For example, Kontio [8] developed a systematic approach called OTSO (off-the-self-option) to demonstrate that MCDM can be effectively used to compare various COTS products from multiple dimensions. The OTSO approach uses the analytic hierarchy process (AHP) to consolidate the evaluation data for decision making process. The AHP method is now widely used by both researchers and practitioners in COTS selection problems [5], [6], [8], [11]. The methodology is only useful when the decision making framework has a unidirectional hierarchical relationship among decision levels. However, Carney and Wallnau [2] points out the evaluation criteria for COTS product are not always independent of each other, but often interact. An invalid result can be made in the face of this complexity. Moreover, AHP is not practically usable if the number of alternatives and criteria is large since the repetitive assessments may cause fatigue in decision makers [1]. The prior proposed MCDM techniques for COTS product evaluation are useful but have restricted application.

Another popular method for solving MCDM problems is the TOPSIS (technique for order performance by similarity to idea solution) which was first developed by Hwang and Yoon [7]. The TOPSIS bases upon the concept that the optimal alternative should have the shortest distance from the positive idea solution (PIS) and the farthest distance from the negative idea solution (NIS). Although the concept of TOPSIS is rational and understandable, and the computation involved is uncomplicated, the inherent difficulty of assigning reliable subjective preferences to the criteria is worth of note.

This paper models the COTS evaluation problem as an MCDM problem and proposes a five-phase COTS selection model, combining the technique of analytic network process (ANP) and modified TOPSIS. The ANP method is used in obtaining the relative weights of criteria but not the entire evaluation process to reduce the large number of pairwise comparison. As for the performance corresponding to each alternative, the modified TOPSIS approach using a new defined weighted Euclidean distance is conducted to rank competing products in terms of their overall performance on multiple evaluation criteria. The method presents here does not account for deriving the evaluation criteria for COTS selection. However, the proposed model may provide organizations a way to devise and refine adequate criteria and alleviate the risk of selecting sub-optimal solutions.

The rest of this paper is structured as follows: In the next section, the proposed COTS evaluation procedure is presented and an overview of the techniques used in our model is given. Section 3 will discuss the procedure and results of an empirical study. In Section 4, we present our conclusions on the results reported in this paper.

Section snippets

Proposed model

The evaluation procedure of this study consists of several steps as shown in Fig. 1. The first step is to identify the multiple criteria that are considered in the decision making process for the decision makers to make an objective and unbiased decision. Briand [1] points out that the selection of appropriate criteria is context dependent and cannot be part of a general COTS selection methodology. Then a relationship between criteria that shows the degree of interdependence relationship is

Illustrative example

To illustrate the proposed COTS evaluation process, an application based on practical experience and implementation in an electronic company is presented in this session. A team of four is been charged in an “off-line production data analysis system” selection project. All the team members have previous experience in information systems evaluation.

To conduct the empirical study, we spent more than two weeks to gather enough information through interviews with users and managers, observation of

Conclusions

The purpose of this paper is to present an effective framework using both ANP and modified TOPSIS methods for performing COTS evaluation. To address the criteria interdependence problem, ANP method is used in obtaining the relative weight of criteria. As the results shown in the empirical study, we find that the proposed method is practical for ranking competing COTS products in terms of their overall performance with respect to multiple interdependence criteria.

While we believe that the

References (17)

There are more references available in the full text version of this article.

Cited by (163)

  • Intelligent triage method for early diagnosis autism spectrum disorder (ASD) based on integrated fuzzy multi-criteria decision-making methods

    2023, Informatics in Medicine Unlocked
    Citation Excerpt :

    The FWIZC includes five steps to weight the set of criteria in a fuzzy manner; the five steps consist of selecting the specialists and developing the weighting questionnaire to prepare for the fuzzification processes. In general, various MCDM techniques were presented for weighting criteria in different fields, including the analytical hierarchy process (AHP) [36], the best-worst method (BWM) [37], and the analytical network process (ANP) [38]. The superior AHP, ANP, and BWM methods suffer from inconsistency issues that will dramatically increase with the increase in the number of criteria in a pairwise comparison.

  • Fuzzy decision by opinion score method

    2020, Applied Soft Computing Journal
    Citation Excerpt :

    The second challenge corresponds to the distance measurement between the ideal solution and alternatives. TOPSIS typically estimates the benefits and costs of each attribute based on the Euclidean distance between separate attributes and the ideal solution [55,56]. Having two different references for comparing alternatives creates another problem as discussed in the literature.

View all citing articles on Scopus
View full text