Performance Evaluation of Basketball Referees with an Integrated MCDM Approach

Basketball referees have to make decisions quickly and accurately under pressure and stress within a limited time. Their decisions affect the results of the match. Therefore, it is necessary to evaluate the performances of the referees and give them feedback. This study aims to propose an MCDM (Multi-Criteria Decision Making) approach to evaluate the performances of basketball referees for the observers and instructors who evaluate officials’ performance. AHP (Analytic Hierarchy Process) and WASPAS (Weighted Aggregated Sum Product Assessment) methods are integrated to evaluate the performance of basketball referees. Criteria weights are considered as equal in the current performance evaluation system of the basketball referees. In this study, the weights of the performance evaluation criteria were determined with the help of the AHP (Analytic Hierarchy Process) method, taking into consideration the fact that the criteria will have different importance degrees. Then, the ranking of the basketball referees according to their performances was obtained using the WASPAS method. The obtained rankings of referees by the help of the proposed integrated approach is different from the ranking obtained with the existing basketball referee evaluation system. This is done because, in the proposed approach, weights of different importance are given to the criteria with the AHP method; also, for ranking the referees, the WASPAS method is used. The proposed approach will improve the referee evaluation process and quality. In this manner, the referees’ performance can be discussed more objectively. Objective evaluation criteria will also improve the performance of referees by strengthening the trust of the referees in the system.


Introduction
Referees have different places and roles in sport, working diligently in the field to apply the rules. They have different roles including being a leader, manager, rule maker, or mediator. Quick decision making, impartiality, control of the game, good communication skills, applying the rules, and using physical and mental abilities are other essential features for referees.
In addition to these, another important feature of the referees is problem-solving. While Heppner and Krauskopf (1987) defines problem-solving skills as complex internal and external demands and desires for the integration of cognitive and be-havioural processes, Bingham (1998) describe it as the process of reaching a specific goal to eliminate difficulties that require a series of efforts. Making a quick decision and solving the problem in the right way is a complex procedure under stress. As we know, officiating is a stressful job; it is imperative to understand their role and give to the referees' feedback to have a good result. Stress in referees occurs when players, coaches, or spectators misperceive the referee's whistle, improperly applying the rules or partiality (Mark, Bryant, & Lehman, 1983). If the referees cannot effectively deal with stress and stressful events, this will adversely affect the performance of both the referee and the athlete. The result of weak, ineffective coping is PERFORMANCE EVALUATION OF BASKETBALL REFEREES | Y. A. D. EKMEKÇI ET AL. slower information processing, less accurate decision-making, improper performance mechanics, burnout and, eventually, quitting (Anshel, 2012;Anshel, Sutarso, Ekmekçi, & Sarasati, 2014). Referees received their feedback the context of various criteria's for performing coping with stressful moment on the court.
In the literature, many studies examine the factors that influence the decision-making process of referees because of their complex decision-making process (MacMahon, Starkes, & Deakin, 2007). These factors include live broadcast pressures (Solomon, Paik, Alhauli, & Pan, 2011); being observed by hundreds or thousands of fans in each match/game, and many other people, such as athletes, coaches, managers, and fans, trying to influence referee's decision (Guillén & Feltz, 2011), and sports involving combat contact, and game change within seconds. Even the relationship of the crowd with the number of yellow cards (Downward & Jones, 2007) referees' decision communication skills (Mellick, Fleming, Bull, & Laugharne, 2005), and the effects of nutritional interventions on cognitive performance for referees (Reilly & Gregson, 2006) were examined. Especially in sports with physical contact, referees are focusing on foul or no foul situations (Plessner, Schweizer, Brand, & O'Hare, 2009). Johnson (2006) suggests the use of sequential sampling models over other approaches to decision-making. Their gaze behaviours are examined, and although referees do not differ in such behaviours, higher-level referees have superior decision-making accuracy and decision sensitivity than lower-level referees do (Hancock & Ste-Marie, 2013). After giving the decisions, feedbacks on their accuracy can be useful for correcting intuitive decisions (Schweizer & Plessner, 2011). To explain biases in referees' judgments a combination of social and embodied cognition is needed, because referees' motor experience influenced perceptual judgments and interacted with contextual factors (Dosseville, Laborde, & Raab, 2011).
All of these studies focused on the psychological, biological, physical and environmental factors needed for referees to be more effective in making decisions. However, although all these elements have been improved, the evaluation criteria of the observer/evaluator should be ideal. This study addresses this issue to enable the referees to perform more effectively as a whole.
The evaluation system of the referees' performance in the world is quite similar. An observer for the organization makes a report with some numerical results according to evaluation criteria, but none of them is including weighted criteria process. Referees complain about this kind of evaluation frequently, because of its subjective nature and incorrect results. According to the assumption of this study, there should be priority differences between the evaluation criteria. The first of the actors who can define the weight of the evaluation criteria may be the referees themselves. Thus, awareness can be cre-ated about the values and points of view given by the referees according to these evaluation criteria.
In addition, the primary purpose of the referees' evaluation process is to direct the referees to manage better competitions. Therefore, the referees need to know the weighted priorities of these evaluation criteria. In this manner, the referees' self-development will have a positive impact on the quality of the competition. From this point of view, the opinions of the referees were consulted in determining the priorities of the evaluation criteria in this study and a new integrated approach is proposed based on the AHP and WASPAS methods for referee performance evaluation.
In the first section of this study, a brief introduction is given. In the second section, the AHP method is introduced and described. In the third section, WASPAS method is explained, a literature review is given, and the method is summarized. In the fourth section, the application of the performance evaluation of basketball referees with the integrated method is given. Finally, in the fifth section, the results of the application are discussed, and recommendations for future studies are given.

AHP method
The Analytic Hierarchy Process (AHP) was first proposed by Saaty (1980). Later, it was widely used as an efficient MCDM method for determining the weights of the criteria and ranking the alternatives. The AHP method allows decision makers to model the complex problems in a hierarchical structure that shows the relationship between criteria, sub-criteria and alternatives. The most important feature of the AHP method is that the decision-makers can include both objective and subjective judgements in the decision process, and they can also capably handle both qualitative and quantitative data. However, the use of AHP requires cumbersome calculations due to pairwise comparisons and consistency checking.
In the literature, the AHP method has been widely used in the studies both in determining the criteria weights and ranking the alternatives. It has been applied in various fields, such as selecting the best alternative, planning, optimization, resolving conflict, resource allocations, among others (Vaidya & Kumar, 2006). A detailed literature review for AHP method has been done by Vaidya and Kumar (2006) and Russo and Camanho (2015).
The steps of the AHP method can be summarized as: Step 1. Decision criteria and alternatives to the problem are determined by the decision-makers. A hierarchical structure of the decision problem is constructed. The goal of the decision problem takes place at the highest level of the hierarchy, and the alternatives are at the lowest level. Between goal and alternatives, criteria and sub-criteria are placed (Wang, Chu, & Wu, 2007).
Step 2. Pairwise comparisons for n criteria are made by the decision-makers using Saaty's 1-9 scale. Based on these pairwise comparisons, matrix A is obtained, as given in Eq. 1.
In this matrix, a ij >0, a ji =1/a ij , a ii =1 and a ij is the decision (1) maker's rating of the relative importance of criterion i respect to criterion j. If criteria i and j have equal relative importance for the decision-maker, then a ij = a ji = 1 (Caputo, Pelagagge, & Salini, 2013).
Step 3. From the pairwise comparison matrices, local weights and priorities of elements in the same level are calculated, and weight vector is obtained as in Eq. 2.
This vector is the normalized principal eigenvector of matrix A. The elements of the weight vector are calculated from the normalized pairwise comparison matrix A by taking the average value of the rows as given in Eq. 3, Step 4. Pairwise comparison matrices of alternatives under each criterion are built by following the procedure in Step 2. Then, normalized relative rating b ij is computed for each ith alternative respect to any judgment criterion Cj, in comparison with the other alternatives.
Step 5. In this step, firstly the consistency indices (CI) of the pairwise comparison matrices are checked by using the Eq. 4, then the consistency ratio (CR) is calculated via Eq. 5. λ max seen in Eq. 4, is the largest eigenvalue of A. In Eq. 5, RI is the average value of CI one would obtain were the entries in A chosen at random, subject that all diagonal entries must equal 1.
For different n values, RI values can be found in the article by Saaty (2013).
In the case of CR <0.10, the decision matrix is said to be consistent. In the case of CR> 0.10, the comparison matrix is reviewed, and necessary changes are made by the decision-makers to bring the matrix into a consistent form.
Step 6. In the last step, global priorities, including global weights and global scores, are obtained by aggregating all local priorities with the application of a simple weighted sum. A ranking score R i is calculated for the ith alternative as seen in Equation (6); In the end, the final ranking of the alternatives is determined based on these global priorities.

WASPAS method
The WASPAS (Weighted Aggregated Sum Product Assessment) method was developed by Zavadskas, Turskis, Antucheviciene, and Zakarevicius (2012); it is an MCDM method that combines the results of Weighted Sum Model (WSM) and Weighted Product Model (WPM). According to the joint generalized criterion of weighted aggregation of additive and multiplicative methods, the ranking of the alternatives is determined. This method can control the consistency of the alternative rankings by conducting sensitivity analysis within its process (Chakraborty & Zavadskas, 2014).
The literature shows that WASPAS method has been applied in different fields. Zavadskas et al. (2012) assessed building design alternatives with WASPAS and MULTIMOORA methods. Chakraborty and Zavadskas (2014), proposed using it for solving manufacturing decision-making problems. They concluded that the WASPAS method could accurately rank the alternatives in such problems. Chakraborty, Bhattacharyya, Zavadskas and Antucheviciene (2015) applied the WASPAS method to parametric optimization of five non-traditional machining processes, concluding that it can be used as a useful tool for both single response and multi-response optimization of the non-traditional machining processes. Karabašević, Stanujkić, Urošević, and Maksimović (2016) proposed using an approach based on SWARA and WASPAS methods for personnel selection. Urosevic, Karabasevic, Stanujkic, and Maksimovic (2017) applied the SWARA and WASPAS methods to personnel selection in the tourism industry. Stojic, Stevic, Antucheviciene, Pamucar, and Vasiljevic (2018), proposed a new rough WASPAS approach to select a supplier in a PVC carpentry product manufacturing company.
In this paper, WASPAS method is used to evaluate the performances of basketball referees. While determining the criteria weights, the AHP method is used, and the ranking of the referees are determined with the help of WASPAS method.
Step 2. Weights of the criteria are determined with the help of an MCDM method such as AHP, SWARA and similar or decision-makers can determine these weights by intuition.
Step 3. After determining the weights of the criteria, a decision matrix is constructed.
Step 4. The decision matrix is normalized via Eq. 8 or 9 according to the type of the criterion.
For benefit criteria; For cost criteria; Step 5. For each alternative, the total relative importance Q i (1) is calculated based on the Weighted Sum Model (WSM) with the help of Eq. 10.
Here; indicates the weight of the j th -criterion.
Step 6. Then, for each alternative, the total relative importance Q i (2) is calculated based on the Weighted Product Model (4) (2) PERFORMANCE EVALUATION OF BASKETBALL REFEREES | Y. A. D. EKMEKÇI ET AL.
Step 7. Finally, the total relative importance of i th alternative is calculated via Eq. 12. This more generalized equation is developed to increase the ranking accuracy and effectiveness of the decision-making process (Chakraborty et al., 2015).
Here; = coefficient of joint optimality and ∈ [0,1]. If WSM and WPM have the same importance degree, then the coefficient of combined optimality is taken as =0.5. When =0 WASPAS method becomes WPM, and when =1 it transformed into WSM (28).
Step 8. Alternatives are ranked according to their values. The best alternative would be that one having the highest value.

Application
In this part, performances of the basketball referees are evaluated with an integrated approach based on AHP and WASPAS methods. In the current performance evaluation system of the basketball referees, criteria weights are considered as equal. Because the criteria will have different importance degrees in the evaluation process, the AHP method is used for determining the weights of the criteria. Then, basketball referees are ranked according to their performances with the help of WASPAS method.
In the proposed integrated approach, the AHP method is applied to determine the weights of the criteria. In the first step, a committee of decision-maker is formed from four decision-maker DM1, DM2, DM3, DM4. The first decision-makers is an FIBA instructor; the second is a Euroleague referee; the third one is a Euroleague delegate and referee observer; and the fourth decision-makers is a high-level basketball coach. Then, these decision-makers evaluated 10 decision criteria used in the current evaluation system. These evaluation criteria are; Competition and self-esteem (C1), Physical state (C2), Foul assessment (C3), Violation assessment (C4), Mechanical (C5), Standard (C6), Teamwork (C7), Game control (C8), Disciplinary implementation (C9), Overall performance (C10).
Pairwise comparisons of criteria are made firstly, and then pairwise comparisons are made for alternatives under each criterion. The consistency of each comparison matrix is checked; they were found to be consistent. Later, these four matrices are reduced into one matrix by the help of geometric mean. This obtained matrix is shown in Table 1. Then, this comparison matrix is normalized by using Eq. 3. Finally, the weights of the criteria are calculated by taking the averages of each row of the normalized matrix. The weights for each criterion is obtained as; Standard (C6) 0.23, Competition and self-esteem (C1) 0.22, Teamwork (C7) 0.13, Game control (C8) 0.12, Foul assessment (C3) 0.09, Physical state (C2) 0.06, Disciplinary implementation (C9) 0.05, Overall performance (C10) 0.04.
After determining the criteria weights with the AHP method, the steps of WASPAS method are followed to determine the ranking of the basketball referees. A decision matrix is formed, as given in Table 2. This matrix includes the performance values of basketball referees. The decision matrix is normalized by using Eq. 8 as all the criteria are benefit criteria. Then, the normalized decision matrix is formed. Later, the total relative importance Q i (1) is calculated with the help of Eq. 10 for all referees. In this equation, the weight values obtained with the AHP method are considered. These total relative importance values for the referees are given in Table 3.
(continued on next page) Referees C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 10 Table 3. Weighed normalized decision matrix and total relative importance Q i (1) Referees C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 10 Q i ( Later, the total relative importance Q i (2) for each alternative are calculated by using Eq. 11 and given in Table 4. Finally, the total relative importance of the referees is calculated with the help of Eq. 12. Here the coefficient of combined optimality is taken as =0.5. The total relative importance val-ues (Q i ) of referees are obtained. In the end, basketball referees are ranked according to their Q i values in descending order, as seen in Table 5. The rankings obtained of referees in Table 5 by the help of the proposed integrated approach is different from the ranking obtained with the existing basketball referee evaluation system. The reason for this is, in the proposed approach's different importance weights given to the criteria with the AHP method and because the WASPAS method is used for ranking the referees.

Conclusion
The starting point of this study was that the importance of the criteria of referee evaluation was not determined in the current evaluation system. First to eliminate this deficiency, by taking the opinions of the referees, the weights of the evaluation criteria were determined with the help of AHP method and the ranking of the criteria according to their weight is obtained as follows: Standard, Competition and self-esteem, Teamwork, Game control, Foul assessment, Physical state, Disciplinary implementation, Overall performance, Mechanical, Violation assessment.
In addition, in the current evaluation system, a ranking is obtained by considering the total number of points received by the referees based on the criteria. In this study, the WASPAS method, which is an MCDM method that combines the results of Weighted Sum Model (WSM) and Weighted Product Model (WPM), is proposed for determining the ranking of basketball referees. In this manner, a more precise and accurate ranking is obtained. This approach guides the decision-makers in the basketball referees' performance evaluation process. In future studies, criteria can be evaluated by different decision-makers in the field. Also, other MCDM can be applied both in determining the weights of the criteria and ranking of the basketball referees. In the end the obtained results can be compared.
Referees are considered one of the most critical factor for evaluating athletes' performance; therefore, their performance on the court or field is essential. Classical methods of evaluating referees' performance are sometimes not objective and suitable, because the evaluating criteria are not examined detailed. The model presented by this study will improve referee evaluation process and quality. Therefore, if the quality of evaluation improves, the referees' performance can be discussed more objectively. In addition, objective evaluation criteria will improve the performance of referees by strengthening the trust of the referees in the system. The referee's role specificity may be taken into consideration while evaluating the evaluation criteria of the performance of the referees in further studies.