Eliciting opinions of experts using uncertainty

. Multiple criteria decision-making (MCDM) methods designed for evaluation of attractiveness of available alternatives, whenever used in decision-aid systems, imply active participation of experts. They participate in all stages of evaluation: casting a set of criteria, which should describe an evaluated process or an alternative; estimating level of importance of each criterion; estimating values of some criteria and sub-criteria. Social and economic processes are prone to laws of statistics,which are described and could be forecasted using the theory of probability. Weights of criteria, which reveal levels of their importance, could rarely be estimated with the absolute level of precision. Uncertainty of evaluation is characterised by a probability distribution. Aiming to elicit evaluation from experts we have to ﬁnd either a distribution or a density function. Statistical simulation method can be used for estimation of evaluation of weights and/or values of criteria by experts. Alternatively, character of related uncertainty can be estimated by an expert himself during the survey process. The aim of this paper is to describe algorithms of expert evaluation with estimation of opinion uncertainty, which were applied in practice. In particular, a new algorithm was proposed, where an expert evaluates criteria by probability distributions.


Introduction
Multiple criteria decision-making methods (MCDM) are often used in decision-aid systems. These methods help decision-makers to identify the best alternative among the ones available or to rank the alternatives in the order of their attractiveness with respect to the purpose of evaluation.
Multiple criteria decision-making (MCDM) methods designed for evaluation of attractiveness of available alternatives, whenever used in decision-aid systems, imply active participation of experts. Experts participate at all stages of evaluation: casting set of criteria, which should describe an evaluated process or an alternative; estimating the level of importance of each criterion; estimating values of some criteria and subcriteria; or even choosing the type of normalisation used.
There are several methods based on mathematical statistics, which help to determine level of importance (weights) of each criterion: ranking; direct and indirect evaluation; simple pairwise comparison; AHP (Analytic Hierarchy Process) by T. Saaty, etc. [3,5,7,9]  Social and economic processes are prone to laws of stochasticity. Consequently, they are well-described and could be forecasted using methods of the theory of probability. Weights of criteria, which reveal levels of their importance, could rarely be assessed with absolute level of precision. A number of experts are usually participating in a decision-making process. Each of them can have a distinct opinion and a point of view. The innate uncertainty can be well characterised by a probability distribution. That is, either a distribution or a density function has to be found for the purpose of replicating their opinion.
Statistical simulation method can be used for estimation of evaluation of weights and/or values of criteria by experts [6]. Alternatively, character of related uncertainty can be estimated by an expert himself during the survey process.
The aim of this paper is to estimate how implicit uncertainty of expert opinions can influence result of evaluation.

The discrete case
This case is often used in practice, because each expert evaluates importance of criteria in the simplest way by assigning a single number representing the level of importance. Such a discrete expert evaluation x ik (i = 1, 2, . . . , m; k = 1, 2, . . . , r; m -number of criteria; r -number of experts) does not reveal expert's opinion about importance of criteria precisely. Even dubbed evaluations by the same expert often differ. Nevertheless, in case if the number of experts is large, more than 40, then a theoretic probability distribution can be derived. The algorithm was applied at Vilnius Gediminas Technical University for selection of the most meaningful criteria representing efficiency of a professor.
Evaluations x 1 , x 2 , . . . , x r made by experts can be treated as values of a random variable X. A standard procedure for deriving the theoretic distribution can be applied [4].
Both the smallest and the largest values x min and x max are derived. The interval (x min , x max ), where the random variable belongs to, is partitioned to k parts (x i−1 , x i ), i = 1, 2, . . . , k each of length h; h = (x max − x min )/k. Frequencies n i of the random variable X are calculated, they reveal the number of experts, whose estimations belong to the interval (x i−1 , x i ), i = 1, 2, . . . , k. A histogram of the random variable X is drawn (Fig. 1). Depending on the shape of the histogram, a theoretic random distribution (normal, exponential, logonormal, or triangular) is chosen. In accordance to the data parameters of the distributions are obtained (mean, standard deviation, etc.). A distribution function F (x) or a density function f (x) is derived from the obtained parameters. Theoretic probabilities p i that the random variable X belongs to the interval ( Corresponding theoretic frequencies n * i = rp i are calculated. A hypothesis that the random variable X is distributed in accordance with the theoretic random distribution, is verified. For hypothesis verification Pearson, Kolmogorov or other criterion can be used.
In the case, if Pearson's criterion χ 2 is chosen, its value is calculated as follows: If the obtained value χ 2 sk of the criterion χ 2 is smaller than the critical value calculated for ν = k − q − 1 degrees of freedom (q is the number of parameters of the theoretic random distribution), for the chosen level of significance α, the hypothesis that the random variable X is distributed in accordance with the chosen random distribution, is not rejected.
Naturally, the shape of histogram should depend on choice of the random variable X, which in our case shows the number of criteria distinguished as important by the experts within the whole set of criteria. The histogram is created using the standard procedure of statistics by depicting frequencies n i of X on the axis of ordinates, which represent the number of different criteria which were checked as being important, the number of times represented by the corresponding interval on the axis of abscissas, as to be elicited from the experts.

Evaluation by experts into intervals
The stochasticity can be accounted in the ultimate evaluation if expert's opinion is expressed into an interval instead of a single point. An expert estimates an interval [a ik , b ik ] (k = 1, 2, . . . , r), where the i-th criterion varies (i = 1, 2, . . . , m), r is the number of experts (Table 1).
When all the smallest values x min of the left-hand boundaries, and the largest values x max of the right-hand boundaries are found the described algorithm is used.
Experts can be offered to evaluate not only boundaries of intervals, but in addition, the most probable value of the i-th criterion c ij (Table 2).
Evaluation to intervals with the requirement to experts to indicate the most probable values implies using the described algorithm or just applying average values c i Mentioned parameters can be parameters of the normal distribution.

Expert evaluation by a probability distribution
We develop the new approach, which is mentioned in [8]: every expert provides a probability distribution of the variable, which is to be evaluated. It is convenient to propose a triangular or normal distribution to each expert as a framework function for making the estimation. The normal distribution should be proposed to more skilled experts. In the latter case the expert has to estimate the mean and the standard deviation.
It is more convenient for experts to use the triangular distribution. It is defined by three following parameters: the mean, the smallest and the largest values. The density of such a distribution is defined as follows: ) , a x c, The graph of the triangular distribution is depicted in Fig. 2. Take actual parameters outlined in Table 2. Evaluation by experts of the i-th component into intervals can be shaped into the triangular density function with parameters a ik , b ik , c ik , where k is the number of an expert.
The final compound distribution elicited from all participating experts can be defined by the following parameters: or as the normal distribution with the mean c i (2) and the standard deviation S i obtained from (3):

Opportunities for using fuzzy sets for obtaining expert estimations
Assuming stochasticity in determination of weights and in comparison of components, the theory of fuzzy sets can be used. This theory is a synthesis of the classic theory of sets and the classic formal logic. It was proposed by Lofti Zadeh [10]. The triad of numbers M 1 = (l 1 , m 1 , u 1 ) is used by experts for making evaluations within this framework. In contrast, instead of the theory of probability the theory of fuzzy sets is used for estimation of weights of components [1,10]. Fuzzy sets are characterised by the membership function MF c , which is an analogue of probability (MF c (x) ∈ [0, 1]). The function MF c (x) reveals the grade of membership of fuzzy numbers x to the fuzzy set C. Several typical membership functions are used in practice. Three functions can be distinguished as most popular: triangle, trapezium, and Gaussian.
The triangular membership function is defined by the triad of numbers (l, m, u). Values of the function are defined by formula (6): The triangular fuzzy membership function is an analogue of the triangular density distribution function.
Results of estimations elicited from experts can be used in quantitative MCDM methods [2].

Conclusions
Social and economic processes are prone to laws of statistics. Consequently, they are well described and forecasted by the theory of probability. Experts actively participate in decision-making process. They each have their own distinct opinion or preference. Uncertainty of evaluation is characterised by a probability distribution. Consequently, a probability distribution or a density function is elicited from experts. In case a sufficiently large number of experts is participating, known non-parametric methods for validation of hypothesis can be used for deriving the random distribution. Evaluations by experts into intervals allow to find intervals, where values of criteria belong to, and a theoretic random distribution. The paper proposes a new approach, suggesting to elicit parameters of random distribution from experts. The triangle distribution can be used most often, but the normal distribution can be used too. It is demonstrated that for estimation of weights of criteria and their values the theory of fuzzy sets can be used.