Abstract

Paper aims

the paper aims to demonstrate the advantages of several modifications of combinatorial method of expert judgment aggregation in AHP. Modifications are based on 1) weighting of spanning trees; 2) sorting of spanning trees by graph diameter during aggregation.

Originality

Both the method and its modifications are developed and improved by the authors. We propose to 1) weight spanning trees, based on quality of respective expert estimates, and 2) sort them by diameter in order to reduce the impact of expert errors and the method’s computational complexity.

Research method

we focus on theoretical and empirical studies of several modifications of combinatorial method of aggregation of expert judgments within AHP.

Main findings

modified combinatorial method has several conceptual advantages over ordinary method. It is also less sensitive to perturbations of initial data. Additionally, selection of spanning trees with smaller diameters allows us to reduce computational complexity of the method and minimize the impact of expert errors.

Implications for theory and practice

Combinatorial method is a universal instrument of expert judgment aggregation, applicable to additive/multiplicative, complete/incomplete, individual/group pair-wise comparisons, provided in different scales. It is used in the original strategic planning technology, which has recently found several important applications.

Keywords
Pair-wise comparison; Consistency; Compatibility; Spanning tree graph diameter; Prüfer sequence

1. Introduction

This article represents an extended version of the respective paper (Kadenko & Tsyganok, 2020Kadenko, S., & Tsyganok, V. (2020). An update on combinatorial method for aggregation of expert judgments in AHP. In Proceedings of the International Symposium on the Analytic Hierarchy Process (ISAHP-2020). Pittsburgh: AHP. Retrieved in 2021, May 25, from http://www.isahp.org/uploads/035.pdf
http://www.isahp.org/uploads/035.pdf... ), presented at the International Symposium on the Analytic Hierarchy Process (ISAHP-2020). The key additions to the original conference paper include: elaborated description of several versions of combinatorial method, motivational examples, and results of the authors’ most recent research on the subject.

The best way to evaluate alternative decision options according to intangible criteria, that cannot be numerically described, is to compare them among themselves. This suggestion, made and empirically confirmed by Tom Saaty and his followers (Saaty, 1980Saaty, T. (1980). The analytic hierarchy process. New York: McGraw-Hill., 1996Saaty, T. (1996). Decision making with dependence and feedback: the analytic network process. Pittsburgh: RWS Publicaitons.), provided the basis for a whole family of AHP/ANP expert evaluation methods. These methods are widely used by decision-makers around the world. AHP/ANP application domains cover almost all spheres of human activity (as we can see from proceedings of multiple editions of ISAHP at isahp.org). Results of multiple new case studies of AHP usage in supply chain management (Oliveira Ramos et al., 2020Oliveira Ramos, M., Silva, E. M., & Lima-Júnior, F. R. (2020). A fuzzy AHP approach to select suppliers in the Brazilian food supply chain. Production, 30, e20200013. http://dx.doi.org/10.1590/0103-6513.20200013.
http://dx.doi.org/10.1590/0103-6513.2020... ), healthcare/insurance (Wollmann et al., 2014Wollmann, D., Steiner, M. T. A., Vieira, G. E., & Steiner, P. A. (2014). Details of the analytic hierarchy process technique for the evaluation of health insurance companies. Production, 24(3), 583-593. http://dx.doi.org/10.1590/S0103-65132013005000070.
http://dx.doi.org/10.1590/S0103-65132013... ; Ferrari Neto et al., 2020Ferrari Neto, G., Lapasini Leal, G. C., Cardoza Galdamez, E. V., & de Souza, R. C. T. (2020). Prioritization of occupational health and safety indicators using the Fuzzy-AHP method. Production, 30, e20200054. http://dx.doi.org/10.1590/0103-6513.20200054.
http://dx.doi.org/10.1590/0103-6513.2020... ), and other domains, as well as in theoretical studies (Lipovetsky, 2021Lipovetsky, S. (2021). AHP in nonlinear scaling: from two-envelope problem to modeling by predictors. Production, 31, e20210007. http://dx.doi.org/10.1590/0103-6513.20210007.
http://dx.doi.org/10.1590/0103-6513.2021... ) are being published.

Calculation of priorities (weights of alternatives being compared) based on individual and group expert judgments (pair-wise comparisons of alternatives) is an essential step of AHP/ANP algorithm (Saaty 1980Saaty, T. (1980). The analytic hierarchy process. New York: McGraw-Hill., 1996Saaty, T. (1996). Decision making with dependence and feedback: the analytic network process. Pittsburgh: RWS Publicaitons.). Numerous studies indicate that different methods of calculation of priorities based on judgments in AHP yield different results (Choo & Wedley, 2004Choo, E. U., & Wedley, W. C. (2004). A common framework for deriving priority vectors from pairwise comparison matrices. Computers & Operations Research, 31(6), 893-908. http://dx.doi.org/10.1016/S0305-0548(03)00042-X.
http://dx.doi.org/10.1016/S0305-0548(03)... ; Tsyganok, 2010Tsyganok, V. (2010). Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise comparison method. Mathematical and Computer Modelling, 52(3-4), 538-544. http://dx.doi.org/10.1016/j.mcm.2010.03.052.
http://dx.doi.org/10.1016/j.mcm.2010.03.... ; Lundy et al., 2017Lundy, M., Siraj, S., & Greco, S. (2017). The mathematical equivalence of the “spanning tree” and row geometric mean preference vectors and its implications for preference analysis. European Journal of Operational Research, 257(1), 197-208. http://dx.doi.org/10.1016/j.ejor.2016.07.042.
http://dx.doi.org/10.1016/j.ejor.2016.07... ). That is, priorities (relative alternative weights) calculated using different methods are, in the general case, different from each other. The question “which aggregation method works better for deriving priorities?” is still relevant and open to discussion. Eigenvector method (Saaty, 1980Saaty, T. (1980). The analytic hierarchy process. New York: McGraw-Hill., 1996Saaty, T. (1996). Decision making with dependence and feedback: the analytic network process. Pittsburgh: RWS Publicaitons.), geometric mean (GM) (Lundy et al., 2017Lundy, M., Siraj, S., & Greco, S. (2017). The mathematical equivalence of the “spanning tree” and row geometric mean preference vectors and its implications for preference analysis. European Journal of Operational Research, 257(1), 197-208. http://dx.doi.org/10.1016/j.ejor.2016.07.042.
http://dx.doi.org/10.1016/j.ejor.2016.07... ; Kułakowski, 2020Kułakowski, K. (2020). On the geometric mean method for incomplete pairwise comparisons. Mathematics, 8(11), 1873. http://dx.doi.org/10.3390/math8111873.
http://dx.doi.org/10.3390/math8111873... ), logarithmic least squares (Bozóki & Tsyganok, 2019Bozóki, S., & Tsyganok, V. (2019). The (logarithmic) least squares optimality of the arithmetic (geo-metric) mean of weight vectors calculated from all spanning trees for incomplete additive (multiplicative) pairwise comparison matrices. International Journal of General Systems, 48(4), 362-381. http://dx.doi.org/10.1080/03081079.2019.1585432.
http://dx.doi.org/10.1080/03081079.2019.... ), and combinatorial method are just a few examples of methods used for aggregation of data from a pair-wise comparison matrix (PCM).

Combinatorial method, that is the subject of this paper, is based on enumeration of all possible basic sets of pair-wise comparisons from a given PCM (not just columns or rows). This approach allows us to utilize redundancy of expert data most thoroughly and check consistency, compatibility (in the case when pair-wise comparisons are provided by a group of experts), and completeness of this data prior to aggregation. Moreover, the approach allows us to easily detect the most inconsistent and/or incompatible elements of initial PCM and, if necessary, request the respective expert to reconsider his/her respective judgment.

In some publications the method is addressed as “enumeration of all spanning trees” (Siraj et al., 2012Siraj, S., Mikhailov, L., & Keane, J. (2012). Enumerating all spanning trees for pairwise comparisons. Computers & Operations Research, 39(2), 191-199. http://dx.doi.org/10.1016/j.cor.2011.03.010.
http://dx.doi.org/10.1016/j.cor.2011.03.... ), as it is largely based on graph theory. Indeed, every basic set of pair-wise comparisons can be represented by a graph called a spanning tree. Vertices of this graph represent the objects being compared, while edges represent respective pair-wise comparisons of these objects.

In the current paper we will revisit and outline several conceptual versions of combinatorial method and empirically compare them, particularly, in terms of computational complexity and sensitivity to variations of input data.

2. Literature review

The theory of spanning trees was developed back in the second half of the 19^th century alongside the general graph theory (Cayley, 1889Cayley, A. (1889). A theorem on trees. The Quarterly Journal of Mathematics, 23, 376-378.). Before taking their place in the AHP paradigm, spanning tree-based approaches were being applied to logistics, chemistry (by Cayley himself), physics (Kirchhoff’s circuit laws), and other domains. The idea of combinatorial enumeration of all spanning trees, representing basic pair-wise comparison sets from a pair-wise comparison matrix (PCM) was independently suggested in Tsyganok (2000)Tsyganok, V. (2000). Combinatorial method of pair-wise comparisons with feedback (in Ukrainian). Data Recording. Baoxian Yu Jiagong, 2, 92-102. and Siraj et al. (2012)Siraj, S., Mikhailov, L., & Keane, J. (2012). Enumerating all spanning trees for pairwise comparisons. Computers & Operations Research, 39(2), 191-199. http://dx.doi.org/10.1016/j.cor.2011.03.010.
http://dx.doi.org/10.1016/j.cor.2011.03.... . Ongoing attention of several academic schools worldwide additionally confirms the relevance of research subject. Somewhat brief description of combinatorial algorithm of priority aggregation was presented in Tsyganok et al. (2015Tsyganok, V. V., Kadenko, S. V., & Andriichuk, O. V. (2015). Using different pair-wise comparison scales for developing industrial strategies. International Journal of Management and Decision Making, 14(3), 224-250. http://dx.doi.org/10.1504/IJMDM.2015.070760.
http://dx.doi.org/10.1504/IJMDM.2015.070... ), where the authors primarily focused on the multi-scale approach to expert estimation. In Lundy et al. (2017)Lundy, M., Siraj, S., & Greco, S. (2017). The mathematical equivalence of the “spanning tree” and row geometric mean preference vectors and its implications for preference analysis. European Journal of Operational Research, 257(1), 197-208. http://dx.doi.org/10.1016/j.ejor.2016.07.042.
http://dx.doi.org/10.1016/j.ejor.2016.07... and Tsyganok (2010)Tsyganok, V. (2010). Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise comparison method. Mathematical and Computer Modelling, 52(3-4), 538-544. http://dx.doi.org/10.1016/j.mcm.2010.03.052.
http://dx.doi.org/10.1016/j.mcm.2010.03.... the method was presented in comparison to other aggregation methods.

Tsyganok (2010)Tsyganok, V. (2010). Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise comparison method. Mathematical and Computer Modelling, 52(3-4), 538-544. http://dx.doi.org/10.1016/j.mcm.2010.03.052.
http://dx.doi.org/10.1016/j.mcm.2010.03.... empirically demonstrated the efficiency of combinatorial method. Lundy et al. (2017)Lundy, M., Siraj, S., & Greco, S. (2017). The mathematical equivalence of the “spanning tree” and row geometric mean preference vectors and its implications for preference analysis. European Journal of Operational Research, 257(1), 197-208. http://dx.doi.org/10.1016/j.ejor.2016.07.042.
http://dx.doi.org/10.1016/j.ejor.2016.07... proved the equivalence of combinatorial and row GM aggregation methods. In Bozóki & Tsyganok (2019)Bozóki, S., & Tsyganok, V. (2019). The (logarithmic) least squares optimality of the arithmetic (geo-metric) mean of weight vectors calculated from all spanning trees for incomplete additive (multiplicative) pairwise comparison matrices. International Journal of General Systems, 48(4), 362-381. http://dx.doi.org/10.1080/03081079.2019.1585432.
http://dx.doi.org/10.1080/03081079.2019.... the equivalence of logarithmic least squares and combinatorial method was proved.

In Lundy et al. (2017)Lundy, M., Siraj, S., & Greco, S. (2017). The mathematical equivalence of the “spanning tree” and row geometric mean preference vectors and its implications for preference analysis. European Journal of Operational Research, 257(1), 197-208. http://dx.doi.org/10.1016/j.ejor.2016.07.042.
http://dx.doi.org/10.1016/j.ejor.2016.07... and Siraj et al. (2012)Siraj, S., Mikhailov, L., & Keane, J. (2012). Enumerating all spanning trees for pairwise comparisons. Computers & Operations Research, 39(2), 191-199. http://dx.doi.org/10.1016/j.cor.2011.03.010.
http://dx.doi.org/10.1016/j.cor.2011.03.... the authors did not address incomplete PCM or suggest ideas for organizing feedback with experts if it was necessary for consistency and compatibility improvement (as it was not their purpose). At the same time, in the original publication (Tsyganok, 2000Tsyganok, V. (2000). Combinatorial method of pair-wise comparisons with feedback (in Ukrainian). Data Recording. Baoxian Yu Jiagong, 2, 92-102.) the idea of feedback was already present. Since then, consistency improvement in combinatorial method was one of the key directions of the method’s modification.

In Tsyganok et al. (2015)Tsyganok, V. V., Kadenko, S. V., & Andriichuk, O. V. (2015). Using different pair-wise comparison scales for developing industrial strategies. International Journal of Management and Decision Making, 14(3), 224-250. http://dx.doi.org/10.1504/IJMDM.2015.070760.
http://dx.doi.org/10.1504/IJMDM.2015.070... the group method was described, however only briefly, as part of a strategic planning technology. Later the technology, including the method, was also described in Tsyganok et al. (2017)Tsyganok, V., Kadenko, S., Andriychuk, O., & Roik, P. (2017). Usage of multicriteria decision‐making support arsenal for strategic planning in environmental protection sphere. Journal of Multi-criteria Decision Analysis., 24(5-6), 227-238. http://dx.doi.org/10.1002/mcda.1616.
http://dx.doi.org/10.1002/mcda.1616... .

Siraj et al. (2012)Siraj, S., Mikhailov, L., & Keane, J. (2012). Enumerating all spanning trees for pairwise comparisons. Computers & Operations Research, 39(2), 191-199. http://dx.doi.org/10.1016/j.cor.2011.03.010.
http://dx.doi.org/10.1016/j.cor.2011.03.... , Lundy et al. (2017)Lundy, M., Siraj, S., & Greco, S. (2017). The mathematical equivalence of the “spanning tree” and row geometric mean preference vectors and its implications for preference analysis. European Journal of Operational Research, 257(1), 197-208. http://dx.doi.org/10.1016/j.ejor.2016.07.042.
http://dx.doi.org/10.1016/j.ejor.2016.07... and Bozóki & Tsyganok (2019)Bozóki, S., & Tsyganok, V. (2019). The (logarithmic) least squares optimality of the arithmetic (geo-metric) mean of weight vectors calculated from all spanning trees for incomplete additive (multiplicative) pairwise comparison matrices. International Journal of General Systems, 48(4), 362-381. http://dx.doi.org/10.1080/03081079.2019.1585432.
http://dx.doi.org/10.1080/03081079.2019.... focus on the ordinary individual method, where all spanning trees have the same weights. Modifications of the method, suggested in Tsyganok (2000Tsyganok, V. (2000). Combinatorial method of pair-wise comparisons with feedback (in Ukrainian). Data Recording. Baoxian Yu Jiagong, 2, 92-102., 2010Tsyganok, V. (2010). Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise comparison method. Mathematical and Computer Modelling, 52(3-4), 538-544. http://dx.doi.org/10.1016/j.mcm.2010.03.052.
http://dx.doi.org/10.1016/j.mcm.2010.03.... ) assume, that spanning trees (and respective basic pair-wise comparison sets) are assigned different weights, depending on consistency, compatibility, and completeness of initial expert data. These publications consider aggregation methods for both individual and group judgments. Besides that, weighting (rating) of basic pair-wise comparison sets allows us to request the experts to reconsider the most inconsistent and incompatible estimates.

Finally, ideas, suggested in Szádoczki et al. (2020)Szádoczki, Zs., Bozóki, S., & Tekile, A. H. (2020). Proposals for the set of pairwise comparisons. In Proceedings of the International Symposium on the Analytic Hierarchy Process (ISAHP-2020). Pittsburgh: AHP. Retrieved in 2021, May 25, from http://www.isahp.org/uploads/057_001.pdf
http://www.isahp.org/uploads/057_001.pdf... and Kadenko & Tsyganok (2020)Kadenko, S., & Tsyganok, V. (2020). An update on combinatorial method for aggregation of expert judgments in AHP. In Proceedings of the International Symposium on the Analytic Hierarchy Process (ISAHP-2020). Pittsburgh: AHP. Retrieved in 2021, May 25, from http://www.isahp.org/uploads/035.pdf
http://www.isahp.org/uploads/035.pdf... , outlined a new direction of combinatorial method development, based on such concepts from graph theory as graph diameter and regularity.

This brief review indicates that some aspects of combinatorial method development were given insufficient attention in academic publications, and need to be elaborated.

3. Research objectives

Based on the review, the main tasks of the current research are as follows.

1. 1

   We need to develop and describe:

• the procedure for application of combinatorial approach when a group of experts is providing estimates in different scales;

• the mechanism for taking the quality of expert data (i.e., consistency, compatibility, completeness, detail) into account during aggregation;

• an approach to improvement of expert data quality through feedback with experts when necessary.

These extensions will make the results of expert sessions, in which combinatorial method is used for aggregation of pair-wise comparisons, more credible.

1. 2

   We need to show the difference between ordinary and modified combinatorial method, where each spanning tree is assigned a different weight, and demonstrate qualitative and quantitative advantages of the modified method over the ordinary one.

2. 3

   In view of the latest research (Szádoczki et al., 2020Szádoczki, Zs., Bozóki, S., & Tekile, A. H. (2020). Proposals for the set of pairwise comparisons. In Proceedings of the International Symposium on the Analytic Hierarchy Process (ISAHP-2020). Pittsburgh: AHP. Retrieved in 2021, May 25, from http://www.isahp.org/uploads/057_001.pdf
   http://www.isahp.org/uploads/057_001.pdf... ; Kadenko & Tsyganok, 2020Kadenko, S., & Tsyganok, V. (2020). An update on combinatorial method for aggregation of expert judgments in AHP. In Proceedings of the International Symposium on the Analytic Hierarchy Process (ISAHP-2020). Pittsburgh: AHP. Retrieved in 2021, May 25, from http://www.isahp.org/uploads/035.pdf
   http://www.isahp.org/uploads/035.pdf... ), we need to check, whether computational complexity of the method and the impact of expert estimation errors can be reduced through sorting of spanning trees in the order of increasing diameter.

4. Research methodology

4.1. Problem statement

The problem solved by combinatorial method is a classical group AHP problem, addressed, for instance, in Forman & Peniwati (1998)Forman, E., & Peniwati, K. (1998). Aggregating individual judgments and priorities with the analytic hierarchy process. European Journal of Operational Research, 108(1), 131-145. http://dx.doi.org/10.1016/S0377-2217(97)00244-0.
http://dx.doi.org/10.1016/S0377-2217(97)... and Saaty & Peniwati (2007)Saaty, T. L., & Peniwati, K. (2007). Group decision-making: drawing out and reconciling differences. Pittsburgh: RWS Publications.. Let us assume that during a group AHP session $m$ experts are providing pair-wise comparisons of $n$ objects. As a result of the expert session, we obtain a set of PCMs $\left\{A^{\left(k\right)};k=\mathrm{1..}m\right\}=\left\{a_{ij}^k:k=\mathrm{1..}m;i,j=\mathrm{1..}n\right\}$. PCMs have the following properties: 1) matrices are reciprocally symmetrical (i.e., $\forall i,j:a_{ij}=1/a_{ji}$, so we can analyze only the upper triangle where $i<j$); 2) matrices are multiplicative (i.e. $\forall i,j\in \left\{\mathrm{1..}n\right\}:a_{ij}=w_i/w_j;a_{ij}=1/a_{ji};$ and in the ideally consistent case $\forall k\in \left\{\mathrm{1..}n\right\}:a_{ij}=a_{ik}\times a_{kj}$) (although, in the general case, they can be additive as well (i.e. $\forall i,j\in \left\{\mathrm{1..}n\right\}:a_{ij}=\left(w_i-w_j\right);a_{ij}=-a_{ji};$ and in the ideally consistent case $\forall k\in \left\{\mathrm{1..}n\right\}:a_{ij}=a_{ik}-a_{kj}$)); 3) in the general case, matrices can be incomplete (i.e., some elements can be missing); 4) every single element of a PCM is obtained in some scale, which is assigned a weight coefficient $s_h$ where $h$ is the scale number. $c_l:l=\left[\mathrm{1..}m\right]$ are coefficients of relative competence of experts in the group. We should note that it is not critical for expert competence coefficients to add up to 1. What matters is their relative significance and the number of experts in the group (Tsyganok et al., 2011Tsyganok, V. V., Kadenko, S. V., & Andriichuk, O. V. (2011). Simulation of expert judgements for testing the methods of information processing in decision-making support systems. Journal of Automation and Information Sciences, 43(12), 21-32. http://dx.doi.org/10.1615/JAutomatInfScien.v43.i12.30.
http://dx.doi.org/10.1615/JAutomatInfSci... ).

We should find: The resulting object weight (priority) vector $\left\{w_k:k=\left[\mathrm{1..}n\right];{\displaystyle \sum }_{k=1}^n{w}_k=1\right\}$.

4.2. Solution process: the idea and key steps

Before describing the mathematical details of the proposed method, we would like to emphasize that the application of the technique is, indeed, simple, as calculations are performed by the respective DSS software, while the experts only need to provide a connected set of pair-wise comparisons.

In essence, combinatorial method of deriving priorities from judgments is a combination of analysis and synthesis. In order to calculate priorities (i.e., relative wights of compared objects), it is sufficient to have just (n – 1) connected comparison and not the complete PCM. For example, we can calculate a priority vector based on just one line or column of the matrix. Combinatorial method, however, is based on enumeration of all such basic pair-wise comparison sets and aggregation of priorities across all of them.

The main steps of the problem solution, as we see it, are as follows.

1. 1

   Bringing pair-wise comparisons to the unified (i.e., the most detailed) scale;

2. 2

   Enumeration of all basic pair-wise comparison sets and construction of respective ideally-consistent pair-wise comparison matrices (ICPCM). By ICPCM we mean a PCM, in which the elements of a selected basic pair-wise comparison set are taken from the initial PCM, while all other elements are reconstructed according to transitivity rules. An example of an ICPCM construction is provided below (in the example from section 4). The basic pair-wise comparison set elements are shown in bold. They are taken from the PCM, built by the 2^nd expert, while other elements are calculated based on multiplicative transitivity rules ($\forall k\in \left\{\mathrm{1..}n\right\}:a_{ij}=a_{ik}\times a_{kj}$).

If some element is missing from the original PCM, then the respective trees cannot be built.

1. 3

   Calculation of priority vectors based on each ICPCM;

2. 4

   Verifying agreement of priorities built based on ICPCMs;

3. 5

   If necessary and/or possible – improvement of the agreement level of estimates and return to the previous step 4. Otherwise – proceeding to step 6;

4. 6

   If agreement level is sufficient (or if it cannot be improved) – aggregation of priorities calculated based on ICPCMs and “exit”. The obtained aggregate priority vector is considered the final problem solution.

Let us briefly outline the key stages process and explain the differences of the method from existing ones.

4.3. Aggregation of estimates

Prior to aggregation, the estimates should be brought to a unified scale, as described in Tsyganok et al. (2015)Tsyganok, V. V., Kadenko, S. V., & Andriichuk, O. V. (2015). Using different pair-wise comparison scales for developing industrial strategies. International Journal of Management and Decision Making, 14(3), 224-250. http://dx.doi.org/10.1504/IJMDM.2015.070760.
http://dx.doi.org/10.1504/IJMDM.2015.070... . This is the scale with the largest number of grades. Judgments, provided in less detailed scales, are approximated by respective grades from the more detailed scale, as shown in Tsyganok et al. (2015)Tsyganok, V. V., Kadenko, S. V., & Andriichuk, O. V. (2015). Using different pair-wise comparison scales for developing industrial strategies. International Journal of Management and Decision Making, 14(3), 224-250. http://dx.doi.org/10.1504/IJMDM.2015.070760.
http://dx.doi.org/10.1504/IJMDM.2015.070... , and assigned scale weights, given in the problem statement above. We assume that if an expert provides an estimate in the scale with the larger number of grades, this estimate is more informative, and the expert is more confident in his or her knowledge regarding the respective pair-wise comparison.

Based on the classical Cayley’s formula (Cayley, 1889Cayley, A. (1889). A theorem on trees. The Quarterly Journal of Mathematics, 23, 376-378.), the number of basic pair-wise comparison sets (spanning trees) we can obtain is $T\le m{n}^{n-2}$ (equality holds when all PCM are complete). Every basic pair-wise comparison set number $q$ allows us to reconstruct an ideally consistent PCM (ICPCM) and a vector of priorities $\left\{w_j^q;j=\mathrm{1..}n;q=\mathrm{1..}T\right\}$. Under the “ordinary” approach, addressed in Siraj et al. (2012)Siraj, S., Mikhailov, L., & Keane, J. (2012). Enumerating all spanning trees for pairwise comparisons. Computers & Operations Research, 39(2), 191-199. http://dx.doi.org/10.1016/j.cor.2011.03.010.
http://dx.doi.org/10.1016/j.cor.2011.03.... , Lundy et al. (2017)Lundy, M., Siraj, S., & Greco, S. (2017). The mathematical equivalence of the “spanning tree” and row geometric mean preference vectors and its implications for preference analysis. European Journal of Operational Research, 257(1), 197-208. http://dx.doi.org/10.1016/j.ejor.2016.07.042.
http://dx.doi.org/10.1016/j.ejor.2016.07... and Bozóki & Tsyganok (2019)Bozóki, S., & Tsyganok, V. (2019). The (logarithmic) least squares optimality of the arithmetic (geo-metric) mean of weight vectors calculated from all spanning trees for incomplete additive (multiplicative) pairwise comparison matrices. International Journal of General Systems, 48(4), 362-381. http://dx.doi.org/10.1080/03081079.2019.1585432.
http://dx.doi.org/10.1080/03081079.2019.... , the formula for aggregate priority calculation is simple geometric (or arithmetic) mean:

Modified combinatorial method, that we suggest, uses not simple but weighted GM. The weights (or ratings) of basic pair-wise comparison sets reflect completeness, degree of detail, consistency, and compatibility of the respective expert judgments, as well as the relative competence of the expert, who provided the respective pair-wise comparisons. The modified priority calculation formula we suggest using looks as follows:

In order for the rating of a specific basic pair-wise comparison set (spanning tree) to incorporate consistency and compatibility of the estimates, we need to introduce some measure of difference between the elements of a specific ICPCM, reconstructed based on this particular pair-wise comparison set, and the initial expert PCM. Consistency and compatibility levels should be inversely proportionate to this difference. That is why we suggest introducing the product ${\displaystyle \prod }_{u,v}\max (\frac{a_{uv}^{kq}}{a_{uv}^l};\frac{a_{uv}^l}{a_{uv}^{kq}})$ into denominator. In order to avoid very sharp differences in ratings (i.e. to ensure that all ratings fall within one order of magnitude, and none can be neglected), we suggest using logarithm function. Completeness of expert data is reflected by the fact, that if some comparison is missing from the initial PCM (because the expert could not provide it), then the respective spanning trees are not built and the number of multipliers in product (2) decreases. As for degree of detail of expert estimation scales, it is reflected by the respective coefficient $s$, based on Hartley’s formula, as explained below (see (5) and (6)). Based on these considerations, multiplicative form of basic pair-wise comparison set (spanning tree) rating that we suggest is as follows:

where $k,l$ are the numbers of experts ($k,l=\mathrm{1..}m$), whose PCM are being compared with each other (in order to check the compatibility of estimates); $c_k,c_l$ are a-priori values of relative expert competence; $k$ and $l$ can be equal or different; $q$ is the number of a particular ICPCM $q=\mathrm{1..}m{T}_k$.

$s^{kq}$ is the relative average weight of scales in which basic pair-wise comparison set elements are input. $s^l$ in is the average weight of scales, in which elements of the respective individual PCM of expert number l are input. $N$is the number of grades in the scale, in which the respective pair-wise comparison is provided. The weight of estimation scale should be proportional to the quantity of information, which can be conveyed by means of the respective scale. In order to calculate this weight, we propose to use Hartley’s formula for amount of information in a signal, transmitted using an alphabet of $N$ symbols (Hartley, 1928Hartley, R. (1928). Transmission of information. The Bell System Technical Journal, 7(3), 535-563. http://dx.doi.org/10.1002/j.1538-7305.1928.tb01236.x.
http://dx.doi.org/10.1002/j.1538-7305.19... ). We suggest using a similar approach to calculate the amount of information in an estimation scale of $N$ grades: $S_N=I={\log }_{2}N$. So, if we are dealing with a specific set of pair-wise comparisons (spanning tree) number q, reconstructed based on the PCM, provided by expert number k, then we suggest calculating the weight coefficient of this set of pair-wise comparisons as GM of the quantities of information, expressed by the scales, in which all pair-wise comparisons from this basic set were provided (GM of weights of all spanning tree elements):

$N_u^{\left(kq\right)}$is the number of grades in the scale, in which basic pair-wise comparison number $u$ is provided.

If we are dealing with an individual PCM, we, similarly, suggest calculating its weight in terms of the quantity of information as GM of the weights of PCM elements. As PCM are reciprocally symmetrical, we can analyze only $\frac{n\left(n-1\right)}{2}$ elements, lying above the principal diagonal of the PCM. So, weight of an individual PCM (in terms of the quantity of information), provided by expert number l, can be calculated as:

$N_{uv}^{\left(l\right)}$is the number of grades in the scale, in which the respective element of the individual PCM $a_{uv}^{\left(l\right)}$is provided.

Similarly, for additive PCM case we suggest using an additive formula of a spanning tree rating:

Logarithm operator in denominator of (3), (6) is used in order to ensure that denominator (and, consequently, spanning tree rating) values belong to the same order of magnitude.

When aggregate priorities are calculated, we can either accept their values as the final result and the problem solution, or improve the level of estimates’ agreement.

4.4. Checking and improving completeness and agreement of the estimates

Before aggregating the weights obtained from ICPCMs, we should check completeness and agreement of initial individual PCMs.

Procedure of completeness check is described in Tsyganok et al. (2015)Tsyganok, V. V., Kadenko, S. V., & Andriichuk, O. V. (2015). Using different pair-wise comparison scales for developing industrial strategies. International Journal of Management and Decision Making, 14(3), 224-250. http://dx.doi.org/10.1504/IJMDM.2015.070760.
http://dx.doi.org/10.1504/IJMDM.2015.070... . If some basic set of pair-wise comparisons is not complete, the priority vector cannot be calculated. In such case, the expert should be re-addressed with a request to provide the missing basic pair-wise comparisons, and, thus, ensure completeness of the set. Here, we are going to consider cases when expert information is redundant and individual PCM, provided by every expert, allow us to build spanning trees: $\forall k\in \left\{\mathrm{1..}m\right\}:T_k\ne 0$, where $k$ is the number of a particular expert and $T_k$ is the number of spanning trees, which can be built based on this expert’s individual PCM.

Combinatorial algorithm allows us to check and improve both consistency (i.e., “inner” consistency of the elements of a PCM, provided by an individual expert) and compatibility (i.e., “mutual” agreement of the respective elements of PCMs, provided by different experts) of pair-wise comparisons. In contrast to methods of aggregation of individual judgments (AIJ) and aggregation of individual priorities (AIP) (Forman & Peniwati, 1998Forman, E., & Peniwati, K. (1998). Aggregating individual judgments and priorities with the analytic hierarchy process. European Journal of Operational Research, 108(1), 131-145. http://dx.doi.org/10.1016/S0377-2217(97)00244-0.
http://dx.doi.org/10.1016/S0377-2217(97)... ; Saaty & Peniwati, 2007Saaty, T. L., & Peniwati, K. (2007). Group decision-making: drawing out and reconciling differences. Pittsburgh: RWS Publications.), it is a one-phase method, allowing us to organize a step-by-step consistency/compatibility improvement procedure.

As an agreement measure, we suggest using the spectral double entropy index (Olenko & Tsyganok, 2016Olenko, A., & Tsyganok, V. (2016). Double entropy inter-rater agreement indices. Applied Psychological Measurement, 40(1), 37-55. http://dx.doi.org/10.1177/0146621615592718. PMid:29881035.
http://dx.doi.org/10.1177/01466216155927... ). It is, in a way, the extension of spectral consistency coefficient suggested by Totsenko (1996)Totsenko, V. G. (1996). The agreement degree of estimations set with regard for experts’ competence. In Proceedings of the IV International Symposium on the AHP (ISAHP ’96) (pp. 229-241). Burnaby: Simon Fraser University.. The key advantage of spectral coefficients over other consistency measures (for instance, listed in Brunelli & Fedrizzi, 2011Brunelli, M., & Fedrizzi, M. (2011). Characterizing properties for inconsistency indices in AHP. In Proceedings of the XI International Symposium for the Analytic Hierarchy Process (ISAHP-2011). Retrieved in 2021, May 25, from http://www.isahp.org/uploads/55_064_fedrizzi.pdf
http://www.isahp.org/uploads/55_064_fedr... ) is that they provide specific clues for organizing feedback with experts (show, which expert should revise his judgment, which pair-wise comparison should be changed, and in what direction). C.R. and C.I. used as consistency measures in classical AHP (Saaty, 1980Saaty, T. (1980). The analytic hierarchy process. New York: McGraw-Hill., 1996Saaty, T. (1996). Decision making with dependence and feedback: the analytic network process. Pittsburgh: RWS Publicaitons.) do not provide any clues for consistency improvement if it is necessary. If C.R. and C.I. exceed the threshold values, the original recommendation is to “reconsider the problem and revise the judgments”. While some research on the subject has been done (Iida, 2009Iida, Y. (2009). Ordinality consistency test about items and notation of a pairwise comparison matrix in AHP. In Proceedings of the X International Symposium for the Analytic Hierarchy Process (ISAHP-2009). Pittsburgh: AHP. Retrieved in 2021, May 25, from http://isahp.org/2009Proceedings/Final_Papers/32_Iida_Youichi_ConsistencyTest_in_Japan_REV_FIN.pdf
http://isahp.org/2009Proceedings/Final_P... ; Mikhailov & Siraj, 2011Mikhailov, L., & Siraj, S. (2011). Improving the ordinal consistency of pairwise comparison matrices. In Proceedings of the XI International Symposium for the Analytic Hierarchy Process (ISAHP-2011). Pittsburgh: AHP. http://dx.doi.org/10.13033/isahp.y2011.128.
http://dx.doi.org/10.13033/isahp.y2011.1... ), targeted step-by-step feedback procedures based solely on C.R. and C.I are still underdeveloped.

In Tsyganok et al. (2015)Tsyganok, V. V., Kadenko, S. V., & Andriichuk, O. V. (2015). Using different pair-wise comparison scales for developing industrial strategies. International Journal of Management and Decision Making, 14(3), 224-250. http://dx.doi.org/10.1504/IJMDM.2015.070760.
http://dx.doi.org/10.1504/IJMDM.2015.070... it was recommended to use spectral consistency coefficient $K_c$, suggested in Totsenko, 1996Totsenko, V. G. (1996). The agreement degree of estimations set with regard for experts’ competence. In Proceedings of the IV International Symposium on the AHP (ISAHP ’96) (pp. 229-241). Burnaby: Simon Fraser University. as agreement measure. Olenko & Tsyganok (2016)Olenko, A., & Tsyganok, V. (2016). Double entropy inter-rater agreement indices. Applied Psychological Measurement, 40(1), 37-55. http://dx.doi.org/10.1177/0146621615592718. PMid:29881035.
http://dx.doi.org/10.1177/01466216155927... demonstrated this coefficient’s drawbacks. For example, originally $K_c$ was intended to lie within the range between 0 and 1, while in fact it could assume negative values (Olenko & Tsyganok, 2016Olenko, A., & Tsyganok, V. (2016). Double entropy inter-rater agreement indices. Applied Psychological Measurement, 40(1), 37-55. http://dx.doi.org/10.1177/0146621615592718. PMid:29881035.
http://dx.doi.org/10.1177/01466216155927... ). Additionally, introduction of $K_c$ was an attempt to unite two independent agreement measures: entropy and dispersion. This resulted in non-monotonous behavior of the coefficient. I.e., sometimes, after the pair-wise comparisons, lying on the largest distance from the average, were moved towards this average, $K_c$ decreased. It is to eliminate the listed flaws of $K_c$ that Olenko & Tsyganok (2016)Olenko, A., & Tsyganok, V. (2016). Double entropy inter-rater agreement indices. Applied Psychological Measurement, 40(1), 37-55. http://dx.doi.org/10.1177/0146621615592718. PMid:29881035.
http://dx.doi.org/10.1177/01466216155927... introduced the double-entropy agreement index.

Consistency and compatibility of expert data should be improved through changes of the most inconsistent (“outlying”) pair-wise comparisons, similarly to Delphi method. If double entropy index lies below pre-defined threshold value, then the expert who provided the estimate that differs the most from the respective element of the aggregate ICPCM across all trees and experts, is requested to change it. For example, the element, that should be changed in order to improve the double entropy index for priority number $l$ is

A step-by-step description of feedback procedure using double entropy coefficient is provided in Tsyganok et al. (2018)Tsyganok, V., Kadenko, S., Andriichuk, O., & Roik, P. (2018). Combinatorial method for aggregation of incomplete group judgments. In Proceedings of 2018 IEEE 1st International Conference on System Analysis & Intelligent Computing (SAIC) (pp. 25-30). New York: IEEE. http://dx.doi.org/10.1109/SAIC.2018.8516768.
http://dx.doi.org/10.1109/SAIC.2018.8516... .

If all experts whose estimates should be changed in order to improve consistency of the PCMs refuse to introduce the necessary changes, then credibility of expert examination results becomes questionable. However, in the general case, PCMs can be built based not only on expert data, but on data, retrieved from open sources. In this latter case feedback becomes totally impossible (for example, you cannot change the data in some online or printed document, that is already published). However, we should remind, that ICPCM ratings, featured in Formula 2, reflect the quality of information contained in the respective matrices. So, during aggregation, “lower-quality” (inconsistent, incomplete, less detailed) data will be assigned smaller weights anyway. Consequently, introduction of ICPCM ratings allows us to make expert session results more credible even in the cases when feedback cannot be organized.

5. Experimental research of the modified method

In this section we will compare the modified combinatorial method with other priority calculation methods used in AHP. Let us start with a numerical example.

5.1. Example of modified combinatorial method’s work

3 equally competent experts $E_1,E_2,E_3$ ($c_1=c_2=c_3=1$) are providing multiplicative comparisons of 4 objects ($A_1,A_2,A_3,A_4$) in integer scales. That is, in our problem statement from subsection 4.1 ($m=3;n=4$). We need to calculate relative object weights (priorities) based on PCMs, provided by the experts. Total numbers of grades in the scales, selected by experts, are given in Table 1. Table 2 provides particular grade numbers, selected by the experts. Table 3 provides the PCM, brought to the unified scale (step 1 of the algorithm from subsection 4.2). For experimental purposes, after scale unification we allow the PCM elements to assume fraction values and exceed maximum value 9. That is, we do not confine ourselves to the fundamental preference scale used in classical AHP.

48 ideally consistent PCM (ICPCM) ($m{n}^{n-2}=3\times 4^2=48$) are built based on initial 3 PCM. Each ICPCM is constructed from the respective basic pair-wise comparison set (spanning tree). For instance, the basic set of pair-wise comparisons of objects $\left(A_1,A_2\right),$ $\left(A_1,A_3\right)$, and $\left(A_2,A_4\right)$ corresponds to the spanning tree, shown on Figure 1. From the respective elements of the PCM provided by the expert $E_2$ we reconstruct the ICPCM, shown in Table 4 (basic pair-wise comparison values are highlighted in bold, while other elements are reconstructed based on transitivity rule).

Similarly, ICPCM are reconstructed from all basic pair-wise comparison sets, provided by each of the 3 experts. In this process, in order to verify consistency and compatibility of the initial PCM, ICPCM are compared with these initial PCM provided by the experts. For this purpose, 3 copies of each ICPCM are built ($m=3$). So, the total number of ICPCM to be analyzed is $T=m^2{n}^{n-2}=144$. Each ICPCM is assigned a rating, calculated according to (3). For, instance, when ICPCM, shown in Table 4, is compared to the PCM of the 1^st expert $E_1$, non-normalized value of the respective rating equals 1.191. For this particular rating, in (3) $k=2;l=1;c_2=c_k=c_l=c_1=1.$ $q=2$ (the spanning tree shown on Figure 1 is assigned number 2 among 16 spanning trees ($n^{n-2}=4^2=16$)); so, based on (4), $s^{kq}=s^{22}={\left({\log }_{2}3\times {\log }_{2}4\times {\log }_{2}7\right)}^{\frac{1}{3}}\approx 2.072$ (numbers of grades in the respective scales (3, 4, 7), come from Table 1).

Based on Formula 5, $s^l=s^1={({\log }_{2}9\times {\log }_{2}8\times {\log }_{2}7\times {\log }_{2}6\times {\log }_{2}5\times {\log }_{2}4)}^{\frac{1}{6}}\approx \mathrm{2.615.}$

Denominator of the fraction from (3) equals $\ln (\left(7\frac{1}{2}/2\right)\times \left(8\frac{1}{6}/4\frac{1}{3}\right)\times \left(26\frac{1}{4}/8\frac{5}{6}\right)\times \left(2\frac{2}{7}/1\frac{1}{7}\right)\times \left(6\frac{1}{2}/3\frac{1}{2}\right)\times \left(3\frac{1}{5}/2\frac{5}{6}\right)+e-1)\approx \ln (92.951+e-1)\approx \mathrm{4.550.}$This value reflects the difference between ICPCM from Table 4 and initial PCM of the 1^st expert from Table 3.

So, indeed, according to (3), $R_{221}\approx \left(2.072\times 2.615\right)/4.550\approx \mathrm{1.191.}$

As the method is based on combinatorial enumeration of pair-wise comparison sets, its computational complexity is rather large. So, providing all calculation details here would be impossible as it would require too much space.

When all 144 ICPCM ratings are calculated, they are normalized by sum (see power index in (2)). From each ICPCM copy number $q=\mathrm{1..144}$ we reconstruct a vector of relative object weights $w_1^q,w_2^q,w_3^q,w_4^q$ (ideal consistency of the matrix allows us to use any basic set of pair-wise comparisons as priority vector; for instance, the 1^st row). For instance, the vector reconstructed based on ICPCM from Table 4 is (4/105; 2/7; 1/3; 1) (last line of the ICPCM in Table 4).

Finally, the aggregate priority (object weight) vector is calculated according to (2).

Normalized priorities $w_1,w_2,w_3,w_4$, calculated using the modified combinatorial method (2), based on the example data equal (0.563734299; 0.263382041; 0.120820159; 0.052063501). Values of priorities, calculated using ordinary combinatorial method (1) equal (0.590174795; 0.243658012; 0.114086692; 0.052080501).

As we have mentioned above, ordinary combinatorial priority aggregation method (1) is equivalent to row GM (Lundy et al., 2017Lundy, M., Siraj, S., & Greco, S. (2017). The mathematical equivalence of the “spanning tree” and row geometric mean preference vectors and its implications for preference analysis. European Journal of Operational Research, 257(1), 197-208. http://dx.doi.org/10.1016/j.ejor.2016.07.042.
http://dx.doi.org/10.1016/j.ejor.2016.07... ) and LLSM (Bozóki & Tsyganok, 2019Bozóki, S., & Tsyganok, V. (2019). The (logarithmic) least squares optimality of the arithmetic (geo-metric) mean of weight vectors calculated from all spanning trees for incomplete additive (multiplicative) pairwise comparison matrices. International Journal of General Systems, 48(4), 362-381. http://dx.doi.org/10.1080/03081079.2019.1585432.
http://dx.doi.org/10.1080/03081079.2019.... ). However, the example shows, that results produced by ordinary (1) and modified method (2) are different, so these two methods are not equivalent.

Both row GM and LLSM have lower computational complexity than combinatorial method, and this is their advantage. At the same time, the key advantage of modified combinatorial method is that it allows us to consider the quality of expert data prior to its aggregation and organize feedback for consistency/compatibility improvement when necessary. Consequently, results of its work more adequately reflect the level of expert competence in the issue under consideration.

5.2. Advantages of modified combinatorial method in terms of sensitivity to expert errors

In previous sections we have mentioned the key conceptual advantages of the modified combinatorial method over ordinary one. We also demonstrated that ordinary and modified combinatorial methods yielded different results. In order to further justify the method’s relevance, we need to empirically demonstrate its advantage over other methods used for priority calculation in AHP (eigenvector method originally suggested by Saaty (1980)Saaty, T. (1980). The analytic hierarchy process. New York: McGraw-Hill., GM, ordinary combinatorial method and others).

In order to compare the priority calculation methods among themselves, we need to devise some quantitative criterion of a method’s efficiency.

Accuracy/precision of estimates cannot be used as such a criterion, because in real expert estimation there are no benchmarks or measurement units. Moreover, in the process of estimation real experts tend to make their own assumptions concerning the ratios between weights of objects they compare. Based on these assumptions, experts can, so to say, “make consistent mistakes”, even if we ask them to estimate some model objects (like figure squares from Saaty’s famous example (Saaty, 1980Saaty, T. (1980). The analytic hierarchy process. New York: McGraw-Hill.)), whose weights are known or can be easily measured.

Since it is problematic to compare priority methods in terms of accuracy/precision, we suggest comparing them in terms of their sensitivity to perturbation of initial data. For this purpose, we suggest simulating the whole process of expert estimation, including model priority values, expert estimates, and expert errors (Tsyganok, 2010Tsyganok, V. (2010). Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise comparison method. Mathematical and Computer Modelling, 52(3-4), 538-544. http://dx.doi.org/10.1016/j.mcm.2010.03.052.
http://dx.doi.org/10.1016/j.mcm.2010.03.... ; Tsyganok et al., 2011Tsyganok, V. V., Kadenko, S. V., & Andriichuk, O. V. (2011). Simulation of expert judgements for testing the methods of information processing in decision-making support systems. Journal of Automation and Information Sciences, 43(12), 21-32. http://dx.doi.org/10.1615/JAutomatInfScien.v43.i12.30.
http://dx.doi.org/10.1615/JAutomatInfSci... ). The experiment is as follows.

1. 1

   Set true model object weights;

2. 2

   Build an ICPCM $A$ (based on the rule $a_{ij}=w_i/w_j$ for multiplicative or $a_{ij}=w_i-w_j$ for additive comparisons, where $a_{ij}$is the element of $A$);

3. 3

   After that, add a certain “noise” to matrix $A$ so that each element (except diagonal ones) changes as follows: $a_{ij}^{\text{'}}=a_{ij}\pm a_{ij}\cdot \delta /100\%$, where $\delta >0$ is a value set in advance, which defines maximum relative deviation of pair-wise comparisons provided by the expert (i.e., elements of $A$) as percentage of true values. In this case$\delta$ simulates potential relative error made by the expert during pair-wise comparison session;

4. 4

   “Perturbed” PCM $A^{\prime }$ is used as input data for one of aggregation methods, which produces aggregate object weights $w_i^{\text{'}}$ (priority vector). We propose to define the efficiency of expert data aggregation method based on maximum possible relative deviation of calculated object weight from true value of the same weight. The method producing smaller deviations should be considered more efficient.

   $\Delta =\underset{i}{max}\left|\frac{w_i^{\text{'}}-w_i}{w_i}\right|\times 100\%$(8)

Calculated values of $\Delta$ will depend on both $\delta$ and true relative weight values, set by experiment organizer. That is why the values of efficiency indicator for each method are presented in the form of function $\Delta \left(\delta \right)$ for each priority vector value.

Dependence $\Delta \left(\delta \right)$ is defined for each method on an interval $\delta \in \left]0;100\right[$ (we assume that relative pair-wise comparison error made by an expert should not exceed 100%, although in the general case the function $\Delta (\delta )$ is defined in a wider range $\delta \in \left]0;\infty \right[$). We propose to find maximum possible deviation $\Delta$ for every $\delta$ using the genetic algorithm (GA) (Holland, 1975Holland, J. (1975). Adaptation in natural and artificial systems. Ann Arbor: University of Michigan Press.).

To conduct the experiment, we used an original software module. The module allows us to generate and perturb the initial PCM and launch the GA (Figure 2).

In terms of the GA, the “individuals” are the perturbed PCM with given relative error value $\delta$. “Fitness function” is the maximum relative deviation of resulting priorities from the true value of object weights $\Delta \left(\delta \right)$. The GA works as follows.

1. 1

   For given true values of object weights and $\delta$ a “population” of individuals (perturbed PCM) is generated. (Cardinality of complete enumeration of matrices of dimensionality $n\times n$, whose elements differ from true values by $\delta$, is very large ($2^{\frac{n\left(n-1\right)}{2}}$, to be exact), so a population includes only a part of PCM from the complete set. However, for $n\le 5$ complete enumeration is possible);

2. 2

   Individuals with maximum fitness function value are selected from the population. That is, we are selecting PCM, which produce maximum deviation of priority vector from true value after aggregation;

3. 3

   Individuals from the selected subset are “interbred” through weighted summation (convex combination) and mutation. As a result, we get a new generation of individuals;

4. 4

   If for the new generation the value of fitness function $\Delta$ is larger than for the previous one, we should move to step 2. If during a fixed number of generations $\Delta$ does not grow, then the algorithm stops and terminates its work. As a result, we get maximum $\Delta$ for a given $\delta$.

In essence, we are looking for a maximum of a function of many variables $f\left(a_{ij}^{\text{'}}\right);i,j=\left(\mathrm{1,}n\right)$. Its arguments are elements of PCM $A^{\prime }$. Values of $\Delta \left(\delta \right)$ for each aggregation method also depend on specific values of initially set true values of object weights $w_i,i=\left(\mathrm{1,}n\right)$. Examples of functions $\Delta \left(\delta \right)$ for given true model values of object weights are shown on Figure 3. Variants of model weight vectors are set in a way that illustrates different ratios between object weights (equal, equal in pairs, arithmetic progression, geometric progression, extreme values of scale range etc.).

In Tsyganok (2010)Tsyganok, V. (2010). Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise comparison method. Mathematical and Computer Modelling, 52(3-4), 538-544. http://dx.doi.org/10.1016/j.mcm.2010.03.052.
http://dx.doi.org/10.1016/j.mcm.2010.03.... individual “weighted” combinatorial method was compared to several other individual pair-wise comparison aggregation methods (Figure 3). In the context of the present paper, we should remind that row GM (one of the methods, studied in Tsyganok (2010)Tsyganok, V. (2010). Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise comparison method. Mathematical and Computer Modelling, 52(3-4), 538-544. http://dx.doi.org/10.1016/j.mcm.2010.03.052.
http://dx.doi.org/10.1016/j.mcm.2010.03.... is equivalent to ordinary combinatorial method (Lundy et al., 2017Lundy, M., Siraj, S., & Greco, S. (2017). The mathematical equivalence of the “spanning tree” and row geometric mean preference vectors and its implications for preference analysis. European Journal of Operational Research, 257(1), 197-208. http://dx.doi.org/10.1016/j.ejor.2016.07.042.
http://dx.doi.org/10.1016/j.ejor.2016.07... ). Consequently, the advantage of modified (weighted) combinatorial method over row GM entails its advantage over the ordinary method (Tsyganok, 2010Tsyganok, V. (2010). Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise comparison method. Mathematical and Computer Modelling, 52(3-4), 538-544. http://dx.doi.org/10.1016/j.mcm.2010.03.052.
http://dx.doi.org/10.1016/j.mcm.2010.03.... ) and LLSM (that is also equivalent to ordinary method (Bozóki & Tsyganok, 2019Bozóki, S., & Tsyganok, V. (2019). The (logarithmic) least squares optimality of the arithmetic (geo-metric) mean of weight vectors calculated from all spanning trees for incomplete additive (multiplicative) pairwise comparison matrices. International Journal of General Systems, 48(4), 362-381. http://dx.doi.org/10.1080/03081079.2019.1585432.
http://dx.doi.org/10.1080/03081079.2019.... )).

So, available empirical comparison data of several pair-wise comparison aggregation methods (obtained through simulation of expert estimation of model object weights) confirm the advantage of modified combinatorial method over other methods. The efficiency indicator is its stability to perturbations of the initial PCM (i.e., to experts’ errors).

6. Modification of combinatorial method based on graph theory

6.1. Motivation

In order to analytically (and not only empirically) compare the sensitivity of different aggregation methods to perturbations of initial expert data, we would have to find an analytical expression for the fitness function $\Delta \left(\delta \right)$ from the previous section. Presently, finding an analytical expression for this function is problematic. However, we can see that it directly depends on the diameters of spanning tree graphs, used during priority aggregation (as shown in Kadenko & Tsyganok, 2020Kadenko, S., & Tsyganok, V. (2020). An update on combinatorial method for aggregation of expert judgments in AHP. In Proceedings of the International Symposium on the Analytic Hierarchy Process (ISAHP-2020). Pittsburgh: AHP. Retrieved in 2021, May 25, from http://www.isahp.org/uploads/035.pdf
http://www.isahp.org/uploads/035.pdf... and Szádoczki et al., 2020Szádoczki, Zs., Bozóki, S., & Tekile, A. H. (2020). Proposals for the set of pairwise comparisons. In Proceedings of the International Symposium on the Analytic Hierarchy Process (ISAHP-2020). Pittsburgh: AHP. Retrieved in 2021, May 25, from http://www.isahp.org/uploads/057_001.pdf
http://www.isahp.org/uploads/057_001.pdf... ). In the context of our problem, graph diameter is the length of longest path between its two vertices (nodes).

For instance, if one object is compared to all the others, the basic pair-wise comparison set is just a line or a column of a matrix, and the spanning tree is a star-type graph. Diameter of such a graph equals 2 edges. If every object is compared to just one other object (the 1^st to the 2^nd, the 2^nd with the 3^rd, etc.), then the basic comparison set includes all the elements above the principal diagonal of the matrix, and the spanning tree is a path-type graph. Its diameter equals $\left(n-1\right)$ edges. Then, there are plenty of non-isomorphic spanning trees with diameters ranging from 2 to $\left(n-1\right)$ (Wu & Chao, 2004Wu, B. Y., & Chao, K.-M. (2004). Spanning trees and optimization problems. USA: Chapman & Hall/CRC Press. http://dx.doi.org/10.1201/9780203497289.
http://dx.doi.org/10.1201/9780203497289... ). Examples of spanning tree graphs are shown on Figure 4.

If an expert makes a maximum error $\delta$ in each pair-wise comparison, then each new edge in the spanning tree diameter will increase the error. As a result, an ICPCM, reconstructed based on a spanning tree with larger diameter will have larger deviation from the initial one.

6.2. Illustrative examples: specific and general case

An expert compares 5 alternatives and makes maximum error $\delta$ during estimation. One of the path-type spanning trees that should be enumerated is shown on Figure 5.

The respective basic set of pair-wise comparisons is ($a_{12},a_{15},a_{24},a_{34}$). All other elements of the respective ICPCM should be reconstructed from the listed basic elements according to transitivity rule. For example,

Now, if comparisons $a_{15},a_{34}$ are increased by $\delta$ and comparisons $a_{12},a_{24}$ – decreased by $\delta$ due to expert errors, then we get the following expression for perturbed element $a_{35}^{*}$:

As we are considering cases when $\delta <1$ (otherwise experts’ errors exceed 100%), maximum possible deviation for element $a_{35}$ exceeds $4\delta$.

Similarly, in the general case, under such “worst-case scenarios” we might be getting accumulated errors (deviations) exceeding $\left(n-1\right)\delta$, where $n$ is the number of compared objects, while $\left(n-1\right)$ is also the diameter of the path-type spanning tree graph. And if we have a spanning tree with diameter $k$, then the deviation of ICPCM elements we might get will look as follows:

where $k_1+k_2=k$.

6.3. Sorting of spanning trees: algorithmic aspect

As we can see, spanning tree diameters significantly influence expert errors, accumulated during aggregation of priorities across spanning trees. Larger deviations of ICPCM elements (caused by larger spanning tree diameters) will result in larger errors in priorities, derived from them. So, it makes sense to study the possibility of further modifying the combinatorial method from the standpoint of spanning tree diameters. Particularly, we need to compare the priority aggregation results, obtained based on spanning trees with smaller and larger diameters. For this purpose, we should be able to sort the spanning trees in the order of increasing diameters.

For sorting of spanning trees, we suggest using the so-called Prüfer sequences (Prüfer, 1918Prüfer, H. (1918). Neuer Beweis eines Satzes über Permutationen. Archiv der Mathematik und Physik, 27, 742-744.). Prüfer managed to prove the Cayley’s theorem on trees by inventing a bijective mapping from a set of indices of a $\left(n-2\right)$-dimensional hypercubic array with the length of the side $n$ to a set of $n^{n-2}$ spanning trees on $n$ edges.

For instance, in the examples shown on Figures 4 and 5, we are dealing with comparisons of 5 objects. According to Cayley’s theorem, if all pairs of objects are compared, we can construct up to 125 spanning trees. 125 is also the volume of a cube with side length 5, and the number of elements in the respective 3-dimensional array. The element of this array with indices (1, 1, 1) is mapped into a star-type spanning tree graph of diameter 2 with the 1^st vertex in the center (Figure 4A). This graph represents a situation when all objects are compared to the 1^st one. Degree of the 1^st vertex (i.e., the number of edges that enter it) equals 4. Degrees of all other vertices equal 1.

By construction, Prüfer sequences have several properties which we can use for spanning tree sorting. First, spanning tree diameter exceeds the number of different indices, encountered in the respective Prüfer sequence, by 1. Second, the degree of a vertex with a given number in the spanning tree exceeds the number of times this number is found in the respective Prüfer sequence. For example, the spanning tree shown on Figure 4B is mapped into Prüfer sequence (1, 1, 2). “1” is encountered in the sequence 2 times. Degree of the respective vertex is 3. And the diameter of the graph also equals 3 (as the sequence is comprised of 2 different indices – “1” and “2”). Similarly, the tree shown on Figure 4C is mapped into Prüfer sequence (5, 1, 2). “3” and “4” are not present in the sequence, so degrees of each of the respective vertices equal 1. The sequence is comprised of 3 different indices (“5”, “1”, and “2”), so the diameter of the respective spanning tree equals 4. The listed properties significantly simplify the process of spanning tree sorting in the order of increasing diameter.

6.4. Experimental research: motivational examples

Empirical study we have conducted so far, is, in a way, similar to the research on comparisons of methods from Section 5. However, we have introduced some new steps in order to illustrate the behavior of aggregate priorities depending on spanning tree diameter. The key conceptual steps of the experimental study are as follows.

1. 1

   Set some model weight (priority) vector $W=\left\{w_i;i=\mathrm{1..}n\right\}$;

2. 2

   Construct an ICPCM based on model weights $A=\{a_{ij};a_{ij}=\frac{w_i}{w_j};i,j=\mathrm{1..}n$};

3. 3

   Perturb the ICPCM elements with a certain noise $\pm \delta$. $a_{ij}^{\text{'}}=a_{ij}\pm a_{ij}\cdot \delta /100\%$;

4. 4

   Calculate priorities $W^{\prime }$ based on perturbed ICPCM, as the GM across all spanning trees (keeping track of the weight behavior as more trees are added to the aggregation formula).

It is important to stress that in our experiment we are adding trees to the aggregation formula one by one. Thus, we can see, how weights change as more trees (with different diameters) are taken into account during aggregation.

We compared 4 modifications of combinatorial method using different aggregation formulas: 1) weighted GM (Formula 2) taking the ordering of trees into account (trees were sorted from smallest to largest diameter); 2) weighted GM without sorting of trees by diameter; 3) ordinary GM (1) without sorting of trees; 4) ordinary GM with sorting of trees.

Figure 6 demonstrates the behavior of the weight of one of 6 alternatives if it is calculated as GM across spanning trees, analyzed one by one (in one order or another). The total number of spanning trees is $6^4=1296$. We can use either ordinary or weighted GM, and either sort spanning trees by diameter or not. Thus, we arrive at 4 above-listed modifications of the method.

For different ratios between model weight values (arithmetic progression, geometric progression, equal/close values, extreme values of scale range, others) behavior of weights is similar. We are currently at the starting point of the research, however, experimental results obtained so far allow us to draw some interim conclusions.

1. 1

   Just like in experiments, described in Section 5, if we aggregate the priorities using weighted GM (2), pair-comparison aggregation results are less sensitive to expert errors, then in the cases when ordinary GM Formula 1 is used;

2. 2

   Experiments empirically confirm the mathematical equivalence of row GM and ordinary GM across all spanning trees, proved in (Lundy et al., 2017Lundy, M., Siraj, S., & Greco, S. (2017). The mathematical equivalence of the “spanning tree” and row geometric mean preference vectors and its implications for preference analysis. European Journal of Operational Research, 257(1), 197-208. http://dx.doi.org/10.1016/j.ejor.2016.07.042.
   http://dx.doi.org/10.1016/j.ejor.2016.07... ). That is, if we use ordinary GM formula, we can enumerate only $n$ star-type graphs of diameter 2 instead of enumerating all $n^{n-2}$ spanning trees;

3. 3

   If we enumerate the trees in the order or increasing diameters, we can obtain the result, close to the one obtained through complete enumeration, if we confine ourselves to enumeration of trees with smaller diameters (2 and 3).

As the method is based on combinatorial enumeration of spanning trees, its computational complexity may become a burden, especially under larger $n$. For example, in (Szádoczki et al., 2020Szádoczki, Zs., Bozóki, S., & Tekile, A. H. (2020). Proposals for the set of pairwise comparisons. In Proceedings of the International Symposium on the Analytic Hierarchy Process (ISAHP-2020). Pittsburgh: AHP. Retrieved in 2021, May 25, from http://www.isahp.org/uploads/057_001.pdf
http://www.isahp.org/uploads/057_001.pdf... ) the largest dimensionality analyzed is $n=24$. Complete enumeration of all spanning trees would take considerable time or even lie beyond the computational power of any machine. Even with PCM completion ratio of about 0.2, the enumeration cardinality would be very large. So, reduction of the number of trees to be enumerated without significant impact upon aggregation results would be a valuable improvement of the method.

6.5. Application of graph (quasi)-regularity concept to enumeration of spanning trees

Smaller-diameter spanning tree graphs accumulate smaller expert errors. Another critical component of stability and credibility of pair-wise comparisons, addressed in (Szádoczki et al., 2020Szádoczki, Zs., Bozóki, S., & Tekile, A. H. (2020). Proposals for the set of pairwise comparisons. In Proceedings of the International Symposium on the Analytic Hierarchy Process (ISAHP-2020). Pittsburgh: AHP. Retrieved in 2021, May 25, from http://www.isahp.org/uploads/057_001.pdf
http://www.isahp.org/uploads/057_001.pdf... ) (besides graph diameter), is graph regularity. A graph is considered regular if degrees of all its vertices are the same, or quasi-regular, if the degree of exactly one vertex exceeds the degrees of all other vertices by 1 exactly.

While we cannot apply the regularity concept to a single spanning tree, we should, definitely, keep the stability of the whole pattern of comparisons in mind. That is why, if we decide to include the trees of diameter $k$ into enumeration process, it is important that all isomorphic graphs of this diameter are considered. Thus, the whole pattern of comparisons will remain stable and balanced. That is, elimination or absence of some pivotal comparison from the set, would not result in the loss of connectivity of the overall comparison pattern.

7. Applications of combinatorial method

Just like any priority calculation method, combinatorial method is applicable to any weakly-structured subject domain, where classical AHP is applied.

During the recent years, the method has been used as the key priority calculation instrument within the original strategic planning technology.

The technology includes the following conceptual steps:

1. 1

   Strategic goal formulation;

2. 2

   Selection of experts;

3. 3

   Building of criteria hierarchy;

4. 4

   Expert estimation of weights of criteria within the hierarchy;

5. 5

   Selection of the optimal strategy. The strategy is the optimal distribution of limited resources among projects that influence the achievement of strategic goal. This resource allocation pattern allows us to achieve the strategic goal with maximum efficiency (at a given moment in time). In this step-by-step procedure combinatorial algorithm is just a sub-step of step 4). It is utilized to aggregate pair-wise comparisons, provided by a group of experts, and, if necessary, to improve consistency and compatibility of estimates, given by different experts.

During the years of its development, combinatorial method was applied in several subject domains, including the following ones.

1. 1

   Evaluation of space projects (Tsyganok et al., 2015Tsyganok, V. V., Kadenko, S. V., & Andriichuk, O. V. (2015). Using different pair-wise comparison scales for developing industrial strategies. International Journal of Management and Decision Making, 14(3), 224-250. http://dx.doi.org/10.1504/IJMDM.2015.070760.
   http://dx.doi.org/10.1504/IJMDM.2015.070... ). The task was to develop a methodology and a system of indicators, which could be used to evaluate the efficiency of implementation of the National Space Programme of Ukraine. The above-mentioned strategic planning technology was used and combinatorial approach was selected as the primary priority calculation tool in the context of the specific subject domain;

2. 2

   Urban-level environmental protection project evaluation (Tsyganok et al., 2017Tsyganok, V., Kadenko, S., Andriychuk, O., & Roik, P. (2017). Usage of multicriteria decision‐making support arsenal for strategic planning in environmental protection sphere. Journal of Multi-criteria Decision Analysis., 24(5-6), 227-238. http://dx.doi.org/10.1002/mcda.1616.
   http://dx.doi.org/10.1002/mcda.1616... ). Allocation of limited resources is an ever-relevant urban-level problem. The authors devised the mechanism for application of the described strategic planning technology (including evaluation of criterion weights using combinatorial method) in environmental protection sphere;

3. 3

   Information operation counteraction (Tsyganok et al., 2020Tsyganok, V., Kadenko, S., & Andriychuk, O. (2020). Hybrid decision support methodology based on objective and expert data. In 2020 IEEE 11th International Conference on Dependable Systems, Services and Technologies (DESSERT) (pp. 265-271). New York: IEEE. http://dx.doi.org/10.1109/DESSERT50317.2020.9125022.
   http://dx.doi.org/10.1109/DESSERT50317.2... ). The authors developed a hybrid approach, utilizing both expert and open data for recognizing and counteracting information operations. Combinatorial approach was used to calculate the relative weights of critical information operation target components.

8. Limitations

Both the method and its applications have their limits. The limitations are as follows.

1. 1

   Extensive results of empirical comparisons of the two versions of the method still do not allow us to state that the modified method yields more accurate results of priority calculations on all possible sets of expert judgments;

2. 2

   GA we use to calculate the deviations of priorities might, by definition, “skip” significant results;

3. 3

   Finding an analytical expression for fitness function $\Delta$ (subsection 5.2) is problematic, as it depends on multiple parameters. That is why analytical comparison of different modifications of combinatorial method is still a challenge;

4. 4

   In broader sense, when the method is applied to resource allocation problems (section 7), constraints on resource volumes required for project implementation impose additional limitations upon the whole decision support methodology, within which the method is used.

9. Conclusions

Several modifications of combinatorial method for aggregation of group incomplete pair-wise comparisons using scales with different numbers of grades have been suggested. The main conceptual advantages of the method are as follows. The method

1. 1

   utilizes redundancy of expert information most thoroughly;

2. 2

   allows us to aggregate pair-wise comparisons, obtained from an individual expert or a group of experts;

3. 3

   is suitable for both complete and incomplete PCMs;

4. 4

   allows us to organize step-by-step feedback with individual experts in order to improve both consistency and compatibility of individual PCMs;

   1. 6

      works for both additive and multiplicative PCMs;

   2. 7

      can use both GM and arithmetic mean as the principal aggregation rule;

   3. 8

      allows experts to use different scales for each pair-wise comparison, making estimation results more accurate in the eyes of both experts and decision-makers;

   4. 9

      allows us to take all the aspects of expert competence (a-priori competence values as well as completeness, detail, and agreement of obtained expert data) into consideration during aggregation of expert judgments.

While ordinary combinatorial method using simple GM aggregation formula is equivalent to LLSM and row GM, modified method yields different results.

Experimental comparisons of the method with other priority calculation methods according to sensitivity of expert session results to perturbations of initial data speak in favor of the modified combinatorial approach, using weighted GM formula for calculation of alternative weights.

Finally, the key directions of further improvement of the method have been set. The improvements are based on sorting of spanning tree graphs by diameter during enumeration. This last modification of the method allows us to minimize expert errors, accumulated on spanning trees of larger diameter, and, if necessary, reduce the computational complexity of the method through reduction of enumeration cardinality. The method with sorting of spanning trees according to diameter should prove useful for cases of large dimensionalities of PCMs as well as for incomplete PCMs.

Further research will be directed, primarily, at more extensive experimental studies of combinatorial method modifications, as well as at finding analytical expression for the fitness function in the GA, which would allow us to analytically compare the combinatorial method with other priority calculation methods.

References

Publication Dates

History