A systematic review of COTS evaluation and selection approaches

Article history: Received September 5, 2016 Received in revised format November 11 2016 Accepted January 2


Introduction
The effective development of the software at a very large scale has been always challenging task to the software developers.The various methods like structure programming, object oriented programming etc. have been employed already for the efficient and effective development of any software system.To achieve an effective software development, the software developers now move towards the implementation of reusability concept.In the recent years, the idea to develop the software at a large scale by the use of reusable components has become more and more popular.The concept of using the reusable components in the open literature is well-known to all as component based development (CBD).The process of the component based development leads to great benefits such as better software quality, reduction of development cost, and reduction in software complexity as well as development time.In the component based development, the required software system is firstly divided in the modules.Once the modules are identified, the appropriate reusable components known as commercial of the shelf (COTS) available in market are needed to be identified for each module.After the identification of the COTS components, they are finally integrated into the target software system.COTS may be defined as a software component easily available in the market for sale, lease to the general public provided by various vendors.COTS component may be directly integrated in the desired software system without making any modification.Simply, it can be stated about the COTS that these are the readymade reusable components and can be used "as it is" without making any change.In CBD, the success of the software development is completely dependent on the reusable components (COTS) that are going to be integrated.Therefore, the selection of the COTS components is a very crucial step for the effective software development.The improper selection of the COTS can result in the development of software with low quality and more cost.The selection of the COTS component has turn out to be a challenging task for the software developers now a day.The huge availability of COTS components in the market makes the selection of COTS very difficult.The research on the COTS selection has a great importance because the appropriate selection of COTS components is vastly desirable for the better software development.The rest of the paper is organized as: A literature review about the various existing COTS selection approaches is provided in the section named as literature review "section-2", the brief description of the existing COTS selection approaches along with their advantages and disadvantages is provided under the section of COTS selection approaches "section-3".

Literature Review
COTS component selection has become a major issues for the success of the component based software engineering.Researchers have provided various COTS selection approaches involving a number of selection criteria.This section tries to summarize the existing COTS selection approaches.Kontio (1999) developed an approach for the COTS evaluation and selection named as OTSO by considering value (functionalities, quality, architectural issues and strategic issues) and cost (development, acquisition and integration cost) as the evaluation criteria.In this research, the two techniques namely Weighted Sum Method (WSM) and Analytical Hierarchy Processing (AHP) were employed for the development of OTSO approach.Morisio and Tsoukis (1997) developed a 2-phase approach named as IusWare for the COTS selection problem based on the multi-criteria decision making concept.The two phases in the developed COTS selection approach consists of the development of an evaluation model and then its implementation for the COTS evaluation and selection.The evaluation model development phase contains the identification of the relevant evaluation criteria is performed and then the COTS components are evaluated on the basis of this identified set of evaluation criteria.Analytical Hierarchy Processing (AHP) and Weighted Sum Methods were deployed in this IusWare COTS selection approach for the purpose of aggregation.
An approach called PRISM (Portable, Reusable, and Integrated Software Modules) based on progressive filtering comprising of two parts namely product evaluation process (PEP) and generic component architecture formation for evaluation and selection of COTS was proposed by Lichota et al. (1997).Tran and Liu (1997) found that the multiple COTS components selection and evaluation in COTS rigorous systems is of great importance.They proposed a waterfall style approach CISD for the identification and evaluation of COTS components and then their integration in to the system for the software development.Maiden and Ncube (1998) proposed a new process PORE that addresses the problem arising due to the non-proper acquirement of requirements in the COTS selection process.An iterative process between the system requirements and the selection of COTS was defined in this approach.Kunda and Brooks (1999) presented STACE, which stands for "Social-Technical Approach to COTS evaluation".STACE comprises of several issues including social issues, non-technical issues, political and economic factors, organizational behavior and characteristics, and customer participation in many researches.Alves and Castro (2001) presented a systematic, repeatable and requirement oriented procedure CRE (COTS-Based Requirement Engineering) for COTS evaluation and selection using non-functional requirements e.g.security and performance, etc.The focus of this work was on the non-functional requirements, which came in existence to improve the inequity between competing COTS components when the functionality of the competing COTS products are exactly same.Phillips and Polen (2002) developed CEP, Comparative Evaluation Technique, as the modification in OTSO method.In this approach, a spread sheet model was designed for the evaluation of the COTS components.It provides the facility to the decision maker to compare similar COTS components for various evaluation criteria.Dorda et al. (2002) produced a COTS evaluation process, which is skilled to suit a variety of projects.PECA solved the problem of various COTS components evaluation and selection problem by determining the suitability of alterative components for the use in final system.The proposed approach consists of three steps as starting from evaluation planning to the data analysis by the establishment of evaluation criteria.Tello and Perez (2002) addressed the issue of COTS component selection among various similar COTS components and proposed BAREMO (Balanced Reuse Model) as an application of AHP to help software developers for suitable selection of COTS for the development of software.Gregor et al. (2002) developed STORYBOARD approach for the COTS selection by focusing on the understanding of customer's requirements with the use of use cases and screen captures.Burgues et al. (2002) proposed Combined Selection (CS), a two level (Local and Global) approach for the multiple COTS component selection.The selection is carried out in COTS intensive Systems.Boehm et al. (2003) presented a risk-driven approach i.e.Win-Win approach based on classical software development spiral model.Win-Win approach consists of an iterative process acquiring requirements in parallel to COTS candidate's evaluation.Erol and Ferrel-Jr (2003) proposed an approach based on fuzzy QFD (Quality Function Deployment) for the COTS component selection by incorporating the COTS selection from a set of products having multiple objectives and a set of qualitative as well as quantitative data.Further, the goal programming was suggested to reach the optimal selection for the COTS components.Chung and Cooper (2004) used functional and non-functional requirements in accumulation to the architecture of components (AC) for ranking and selection of COTS.They produced a well-disciplined and systematic approach CARE (COTS-Aware Requirement Engineering) for the COTS component selection.This approach improves the confidence in COTS component selection process by thoroughly employing the non-functional requirements (NFR) Framework for evaluation of COTS.Grau et al. (2004) presented DesCOTS that stands for Description, Evaluation and Selection of COTS components to be used for the selection and ranking of COTS with the help of software quality models.The wellknown software quality models as ISO 9126 etc. and the AHP technique were used in this approach.Shyur (2006) proposed an effective framework for the component selection and evaluation using ANP and TOPSIS.Mohamed et al. (2007) proposed a decision support system MiHOS for the COTS component selection by handling miss-match between the COTS and the requirements.The main motive of this approach was to improve the fitness of COTS components with the requirements specified.Cortellessa et al. (2008) introduced CODER framework for the COTS component selection.It was composed of a model builder and solver and UML case tool.Kwong et al. (2010) addressed the COTS component selection by focusing on the efficient measurement of cohesion and coupling of the modules, as the result they provide an optimization model for COTS selection using genetic algorithm.Ibrahim et al. (2011) proposed a hybrid model UnHOS (Uncertainty Handling in COTS Selection) by combining AHP and BBN followed by the sensitivity analysis to solve the COTS component selection problem.Gupta et al. (2012) introduced fuzzy mathematical programming (FMP) for the COTS component evaluation and selection.Ravichandran et al. (2012) adopted Neuro-fuzzy approach and presented an effective methodology ANFIS for the selection and evaluation of COTS components.Subhankar Sarkar (2012) provided a comprehensive framework ACT (Architecture Centric Tradeoff) for the COTS selection.Gupta et al. (2013) has developed a systematic and interactive Fuzzy Optimization framework for the solution of bi-objective fuzzy COTS selection.This approach enables the decision makers to modify the feasibility levels interactively and is continued until acceptable solution is reached.The various existing COTS selection approaches are provided in Table 1.

COTS Selection Approaches
It is observed from the available literature that a number of COTS evaluation and selection approaches have been developed by various researchers.A brief description about some of the existing approaches is provided in this section along with their advantages and disadvantages.

A. Brief Description OTSO
Off-The-Shelf Option (OTSO) method was developed to provide a basis for the selection of reusable components.This method compares the COTS components based on these two factors (i) cost and (ii) value.The cost was estimated by integration cost, development costs, and acquisition cost.A hierarchical evaluation criteria was used which consist of strategic concern, qualities, functionalities, and architectural constraints in this approach.To find the fitness of each component against evaluation criteria, GQM selection methodology was used.AHP is used for the purpose of evaluation technique.

IusWare
IusWare (IUStitia softWARis) is designed for the evaluation of COTS components in a rigorous and formal way which is based on two phase approach.First phase is to design the evaluation model.Second phase is to apply the evaluation model.IusWare method mainly relies on multi-criteria decision aid for the COTS evaluation and selection.

PRISM
PRISM that stands for Portable, Reusable, and Integrated Software Modules is basically focused on stand-alone test phase with an addition of evaluation approach.Selection methodology is used in PRISM were product evaluation process (PEP) and component architecture that can be considered for the evaluation of COTS process.First step of this method is the identification of the criteria.Second step is screening to identify the best fitness in one or more products.Third step is carried out to evaluate the products against the requirements, reusability and reliability.Fourth step is the integration test to estimate how the product is integrated in the system.

CISD
COTS-based Integrated Systems Development is designed for the identification, evaluation, selection and integration of COTS components in the final system based on a waterfall -style approach.The COTS products are evaluated based on 'performance of individual products', 'interoperability of products', and 'individual functionality'.The main steps involves in CISD are product identification, evaluation and product integration.

PORE
Procurement-Oriented Requirement Engineering (PORE) approach is based on the concept of requirements engineering phase of the COTS components.PORE approach acquires the requirements at the same time as acquired by the COTS components.PORE approach defines three phases for the evaluation of COTS components.Phase 1 gives guidance for acquiring the product information and customer requirements from COTS vendor information.Phase 2 gives guidance for acquiring the product information and customer requirements from vendors-led demonstration.Phase 3 gives guidance for acquiring the product information and customer requirements from customer-led product exploration.

STACE
Social-Technical Approach to COTS Evaluation (STACE) approach mainly relies on the non-technical issues that can contribute a lot in COTS selection such as human, social and organizational characteristics when evaluation criteria are defined and to conduct the evaluation process.STACE approach decomposed the high-level requirements into the small and measurable attributes.The COTS components are evaluated, selected and finally ranked based on the social-technical criteria.STACE approach requires more efforts.

CRE CRE (COTS-based Requirements Engineering
) focuses on the non-functional requirement (NFR) when COTS components are compared.For detailed evaluation, selects the best COTS components.CRE uses non-functional requirement (NFR) framework to model the non-functional requirement for this purpose.CRE approach involves the identification of COTS components, description, evaluation and selection of COTS components and their acceptance.

CEP
Comparative Evaluation process (CEP) is an enhancement of OTSO approach.This approach introduces the use of confidence factor (CF). CEP approach involves evaluation criteria as strategic performance, architectural, management, functional, and functional characteristic.The evaluation process of CEP approach is based on weighted average.

PECA
Plan, Establish, Collect and Analyze (PECA) technique is based on the flexibility to hold the realities of system.PECA involves two evaluation criteria are support services and quality of reputation.PECA approach involves four steps (i) Plan the evaluation includes create charter, identify stakeholders, pick the approach and estimating resource and schedule (ii) To establish the criteria includes identification and construction of criteria (iii) collect the data (iv) analyze the data using techniques.

BAREMO
Balanced Reuse Model focuses on the selection of COTS components and how to use AHP technique.BAREMO approach is the specification of the project objectives, to construct a decision tree, generation of comparison matrices for criteria and then calculation of final value using weighted addition.

STORYBOARD
The main focus of the storyboard approach is on the understanding of customer requirements by incorporating screen captures and use cases during the requirement engineering phase.Basically, the prime goal in this approach is to identify the appropriate COTS components according to the customer requirements that can be integrated finally in the target system.The various steps which are followed in this approach are (i) Identification of COTS components by determining the customer requirements (ii) development of the use cases based on customer requirements that are identified in step 1 (iii) Evaluation of COTS components.

CS
Combined-Selection (CS) is used for the evaluation selection of various COTS components that fulfills the requirements all together.Combined-Selection approach works on two levels to perform their activities.First level is local level and second is global level.The local level selects the individual COTS components that can be joint at the global level.The global level tries to find out the best combination of COTS components and address the process of the combined selection.

CARE
COTS-Aware Requirements Engineering (CARE) approach primarily focuses on how to define the goal, agent and requirement engineering phase of the process.Mainly, two requirements are defined by the CARE, which is native and foreign.CARE tries to make a gap between the native and foreign by asking the customers about the change the requirements or goal.The various steps involves in this approach are (i) define goals (ii) match goals (iii) rank components (iv) negotiate changes and (v) select components.This model is based on this evaluation criteria (i) functional (ii) non-functional and (iii) architectural effectiveness.

DesCOTS
Description, evaluation and selection of COTS components approach is based on the set of tools that can be used for the evaluation and selection of COTS components based on ISO/IEC9126.The following steps are involved in desCOTS system for defining the evaluation criteria (i) Determination of quality characteristic and sub-characteristic (ii) to refine the sub-characteristics (iii) refining the derived attributes to essential ones and (iv) to determine the metrics.

MiHOS
The focus of Mismatch-Handling aware COTS Selection (MiHOS) approach is to handle the mismatch between the requirements and COTS candidates.To handle the mismatches this method provides the guidelines that are found at the time of COTS selection process and uses some decision support techniques to hold the mismatches of the COTS selection process and provides a study to resolve these mismatches on the selection process.MiHOS approach has two main components (i) COTS selection and (ii) mismatch handling.

UnHOS
Uncertainty Handling in Commercial Off-The-Shelf (UnHOS) approach uses the Analytical Hierarchy Process (AHP) for the purpose of evaluation, selection and ranking of COTS components and BBN (Bayesian Belief Network) to show the uncertainty related to inaccurate information about the COTS candidates.

ANFIS
Adaptive Neuro-Fuzzy Inference Systems (ANFIS) approach develops an automated process of COTS component selection by using 14 parameters namely, Portability, Stability, Completeness, Consistency, Reliability, Interface and Structural Complexity, Security, Usability, Understandability of Software Documents, Accuracy, Performance, Compatibility, Customizable and Serviceability.This approach is a hybrid-learning algorithm to train and identify the membership function, which produces a single output.

ACT
Architecture Centric Tradeoff approach is a decision support approach for COTS components evaluation and selection.This approach comprises three parts (i) Meta-model (ii) Heuristics and (iii) Processor.Meta-model works on three layers namely conceptual, logical and physical layers.The Heuristic process is used to compute the Cost of Ownership.The Processor is used to find out the optimal tradeoff through COTS candidate solutions.

B. Advantages and Disadvantages
A lot of COTS evaluation and selection approaches have been developed by various researchers.No one approach is essentially better or worse than the other, in fact, their strengths and weaknesses are often admiring to each other.The advantages and disadvantages allied with various COTS selection approaches are provided in Table 2.

IusWare
The main focus of this approach is on quality attributes.
 Assumption of already existence of requirements is made. No support for multiple COTS selection and missmatch handling. AHP/WSM -weakness.PRISM  This approach addresses the issues related to the integration of COTS into the system. As prototyping is introduced in it, is of great importance.

CS
Support for multiple COTS selection is provided.
 No formal procedure is adopted for the guidance of integration. No mechanism for identification of conflict between selected COTS.

WIN-WIN Approach
Support for multiple COTS selection.Risk is addressed first time in this approach for COTS selection.A decision framework was provided.
No procedure to handle miss-matches between COTS before and after selection.
Fuzzy QFD Addresses qualitative data for COTS selection.
 Assumption of already existence of requirements is made. No support for multiple COTS selection and missmatch handling. AHP/WSM -weakness.

CARE
 COTS miss-match problem is partially addresses in it. The use of flexible requirements is made in this approach.
The main issues which was not resolved in this approach are:  Influence of miss-match on COTS selection decision. How to handle identified mismatches?

DesCOTS
 Consideration of quality attributes for COTS selection. Applicability in various domains.
No support for multiple COTS selection and missmatch handling is provided.
CODER Supports build vs. buy decision.Cost minimization.More effort required in this approach.

ANP+Modifi ed TOPSIS
Interdependence problem is addressed in this approach which leads to precise COTS selection.
Matrix operation of ANP cannot be applied directly on traditional fuzzy concept.

MiHOS
 Capability to handle miss-match before and after COTS selection. It uses interactive decision support techniques which makes COTS selection decision more precise.
 More effort required.
 No support for multiple COTS selection.

GAP Theory
 Identification of gaps between COTS and system requirements. Consideration of both functional and non-functional requirements.
Gap measurement for each selection criteria for each COTS component is very time consuming.

GA
Addresses the COTS selection mainly with coupling and cohesion of modules.
Subjective judgments such as determination of the scores of interaction and function ratings.

UnHOS
Uncertainty issue for COTS selection is addressed.Less number of selection attributes / criteria.

ANFIS
A large no. of selection criteria were adopted for COTS selection.More accuracy.
Derivation of FDR with respect to each proposed architecture makes it time consuming.
ACT Consideration of architectural orientation and cost of ownership.
In actual, it provides a 4-phase framework, which is not fully capable for COTS selection.
FMOP  Cost minimization. Addresses the incompatibility issue between various COTS components.
Assumption of triangular possibility distribution leads to imprecise coefficients for objective function.

Table 2
COTS Selection Approaches Advantages and Disadvantages

Table 2
COTS Selection Approaches Advantages and Disadvantages (Continued)