A ROBUST-STOCHASTIC DATA ENVELOPMENT ANALYSIS MODEL FOR SUPPLIER PERFORMANCE EVALUATION OF THE TELECOMMUNICATION INDUSTRY UNDER UNCERTAINTY

. The primary activities of any organization rely on the procurement of the required goods and services at the shortest time and highest quality possible. On this basis, the problem of supplier evaluation, ranking, and selection is considered critically important. Data envelopment analysis is a well-known and successful approach in this field. In this study, we propose a robust-stochastic data envelopment analysis model to measure the efficiency of decision-making units under uncertainty. We measure efficiency through a standard and an inverted model in terms of resilience and agility. In order to demonstrate the practical potential of the proposed model, we apply the model to a case study of the Iranian telecom industry with 90 decision-making units. Numerical results reveal that human resources and cash assets are the most important input criteria. Also, the output indicators, including adaptability, reliability, visibility, and coordination, have high importance in measuring the efficiency of decision-making units. It should be noted that employing the robust-stochastic optimization approach leads to controlling the fluctuations of uncertain parameters and maintaining a desirable optimal level of efficiency for decision-making units under different scenarios. The results suggest that the model is sufficiently valid and reliable for evaluating the performance of suppliers in the telecom industry, may be employed under uncertain conditions, and can incorporate decision-makers’ varying preferences. The managerial insights derived from this research indicate that, in the short term, uncertainty throughout the evaluation process of suppliers often leads to reduced efficiency among the decision-making units. However, operating under uncertainty is associated with several advantages in the long term, such as increased decision-making consistency and improved vital ability to cope with uncertainty.


Introduction
Providing high-quality services and products to customers at affordable prices and in a quick way is key to the success of organizations [1].Today, the extreme competitiveness of domestic and international markets leaves policymakers with no choice but to introduce new solutions for reducing costs and increasing service speed to ensure customer satisfaction.Suppliers often have a crucial role in this regard; thereby, proper selection and evaluation of suppliers are considered strategic factors in the success of organizations [2].Efficient suppliers could help consolidate the above capabilities and increase effectiveness, sustainability, and competitiveness [3].
Suppliers experience a variety of risks and problems, such as storage limitations, insufficient human resources, ineffective sales strategies, and so forth.In the face of such complications, improper procurement management processes could seriously damage suppliers' performance by preventing the fulfillment of customers' orders and causing dissatisfaction with the organization [4].Therefore, maintaining suppliers' acceptable level of performance under different conditions, especially under risks, is enormously important.To be more specific, the selected suppliers should be agile and resilient enough to take the proper measures in a crisis and keep supplying the materials required by organizations with minimum cost and maximum quality [5].As such, evaluation, ranking, and selection of resilient and agile suppliers gain increased importance as suppliers' performance directly affects organizations' ability to satisfy customers and win their loyalty.Agile suppliers seek to respond quickly and efficiently to unforeseen market changes and fluctuations [6].Resilience in suppliers reduces the likelihood of failures, minimizes the consequences of potential failures, and helps restore the organization's processes to normal status.Resilience enables organizations to manage supply chain (SC) failures and deliver their products and services to customers [7].Consequently, identifying suppliers with an adequate level of agility and resilience has become an appealing area of research in SC management under risks.
In this study, a hybrid robust-stochastic data envelopment analysis (DEA) model is proposed to evaluate, rank, and select suppliers based on agility and resilience performances under risk.The model is designed to have multiple inputs and outputs and is executed over two stages under uncertainty to bridge the research gaps identified through our literature review.In the standard DEA method, calculating the efficiency of each decisionmaking unit (DMU) extremely depends on the total number of DMUs and the number of input and output criteria, which is considered one of the most significant weaknesses of DEA.This article uses a hybrid approach for calculating the efficiency of DMUs based on both efficient and inefficient boundaries, simultaneously, to cover the mentioned deficiency.In other words, the performance of the candidate suppliers is measured in terms of agility and resilience by adopting a hybrid approach in which efficiency is determined based on the longest distance from the efficient frontier and the shortest distance from the inefficient frontier; a solution rarely adopted in the literature thus far.The decisions made in this research include weighting each input and output criteria, the expected performance of each DMU based on their distances from the efficient and inefficient frontiers, and their combined performance.In addition, the uncertainty in estimating some required parameters in DEA models leads to destructive effects on the optimal efficiency of DMUs.Furthermore, we apply a scenario-based robust optimization approach to deal with fluctuations of parameters.Lastly, to demonstrate the real-world applicability of the proposed approach, it is applied to a case study of the Iranian telecom industry, from which some numerical results and managerial implications are derived.
In this research, some questions are raised, including: what are the main criteria to assess the resilience and agility of suppliers for procurement management?How the agility and resilience of suppliers can be evaluated, simultaneously?How qualified resilient and agile suppliers can be identified?What is an effective procedure to evaluate and rank suppliers to select qualified suppliers in terms of agility and resilience?How operational risks and fluctuations of the business environment should be addressed in assessing the agility and resilience of suppliers?This study aims to provide appropriate answers to the questions raised by proposing a novel framework to evaluate, rank and select qualified suppliers in terms of agility and resilience.
The remainder of this paper is structured as follows.Section 2 contains a review of the literature on the subject.Section 3 states the problem and presents our DEA-based mathematical model.Section 4 discusses the proposed robust-stochastic DEA approach.Section 5 describes the case study and outlines the numerical results and managerial implications.Finally, Section 6 gives a brief conclusion and offers several recommendations for future research.

Literature review
Selecting the supplier(s) in a SC is a strategic decision that has drawn researchers' attention recently.Given the competitiveness of all markets and industries, it is impossible to manufacture quality products or provide satisfactory services at a low cost without having suitable suppliers.Many different and conflicting criteria should be considered throughout the decision-making process, making selecting suppliers an even more complex issue.In today's global economy, accurate evaluation, ranking, and selection of suppliers are increasingly important and decisive factors.The following subsections review some of the most effective and applicable methods for these purposes.

Supplier evaluation and ranking
There are several methods for evaluating and ranking candidate suppliers; however, multi-criteria decisionmaking (MCDM) and DEA models have generally proven more popular than others.In order to make sure that maximum performance is maintained, organizations evaluate and rank their suppliers at each periodic.This approach guarantees consistent performance assessment and creates competition in procurement and meeting the basic needs of organizations.This process is carried out through four primary stages: identifying the candidate suppliers, determining the factors affecting suppliers' performance, developing a suitable model to rank the candidate suppliers, and selecting the top-ranked suppliers [8].
MCDM models themselves require methods of weighting each decision criterion [9].From another perspective, MCDM techniques are categorized into two broad groups of compensatory and non-compensatory models, as explained below.Non-compensatory models examine each criterion in the decision-making process independently and come in three types: when there is no information on the importance of criteria (e.g., dominance, maximin, and maximax methods); when the importance of the criteria is specified sequentially (e.g., lexicographic, permutation, and elimination methods) when there is information on the standard and acceptable range of each criterion (e.g., conjunctive and disjunctive satisfying methods) [10].Compensatory models are classified into three types: scoring, compromising, and outranking methods.In scoring methods, the preferred choice has the highest score.In adaptive methods, the preferred choice is the closest and most similar to the optimal choice.Moreover, in non-ranking methods, the preferred choice has the best status in terms of a specific attribute [11].
Numerous studies have been conducted on ways of ranking the suppliers using MCDM methods, including but not limited to Guo et al. [4], Kaviani et al. [12], Wang et al. [5], and Zakeri et al. [13].Guo et al. [4] evaluated and selected green suppliers using a fuzzy axiomatic design decision-making method in the apparel and clothing industry.According to the results, the integrated approach was highly efficient at selecting green suppliers.Kaviani et al. [12] presented an integrated MCDM model to evaluate and select the suppliers of a petrochemical company under uncertainty.The authors used the Delphi method and the Grey-Shannon entropy approach to incorporate the impact of operational risks on the decision-making process.The results pointed to the success of the integrated approach in evaluating, ranking, and selecting the suppliers in the petroleum industry.Wang et al. [5] employed the SC operations reference (SCOR) metrics, analytic hierarchy process (AHP), and the technique for order of preference by similarity to ideal solution (TOPSIS) to evaluate, rank, and select the suppliers of a petroleum company.The results confirmed the effectiveness of the cited approaches.Zakeri et al. [13] developed a new ranking-based MCDM method to evaluate and select qualified supplier in dairy industry.They also proposed a novel weighting technique called win-loss-draw to assess the importance of supplier assessment criteria.The results indicate that accurate performance of the presented approach for selecting elite suppliers in different industries.
DEA is a non-parametric method that computes the efficiency scores of DMUs through multiple inputs and outputs.Farrell [14] was the first to develop a model to measure and evaluate efficiency.Charnes et al. [15] expanded Farrell's model and introduced DEA.The authors called each of the institutes under evaluation a DMU whose function is to transform the input into output.This approach does not require the criteria to be scaled or weighted and is based on linear programming [16].After solving the mathematical model, the importance weights of the input and output criteria and the efficiency of each DMU are obtainable.DEA relies on identifying a production frontier which is a range within which the DMUs operate efficiently.The scores of inefficient DMUs are computed relatively, i.e., through comparison with efficient DMUs.Obtaining the score of each DMU solely based on comparison with the efficient DMUs is considered a defect, and some researchers, such as Shen et al. [17], have attempted to compute the efficiencies of DMUs based on the score they obtain relative to the efficient and inefficient frontiers.Various studies have also been published on supplier evaluation, ranking, and selection using DEA models, including but not limited to Zarbakhshnia and Jaghdani [18], Ghoushchi et al. [19], Tavana et al. [20], and Nasri et al. [21].Zarbakhshnia and Jaghdani [18] evaluated and selected the suppliers in the plastic industry through a two-stage DEA approach with inputs and outputs considered uncontrollable and undesirable, respectively.Ghoushchi et al. [19] evaluated, ranked, and selected suppliers in the rice industry using goal-programming DEA (GP-DEA) under uncertainty.Tavana et al. [20] evaluated and selected sustainable suppliers in a reverse SC using a fuzzy DEA approach.Nasri et al. [21] proposed an integrated approach to assess, rank and select elite suppliers in a petroleum industry using DEA, ANP and decision-making trial and evaluation laboratory (DEMATEL).They considered sustainability criteria to score suppliers and identify qualified cases.The results indicate that the aforementioned hybrid approach can be employed as a successful framework for evaluating suppliers in similar industries, considering criteria based on sustainable development.

Resilience and agility of procurement
Since customers' needs and requests change constantly, suppliers should be flexible and adaptive to respond effectively to the market's demand and fluctuations [22].Procurement agility generally depends on seven parameters: customer satisfaction, quality improvement, cost minimization, delivery velocity, introducing new products, service level enhancement, and reducing waiting times [4].In agile procurement, the central goal is to minimize delivery times; consequently, the emphasis is on adaptability, flexibility, responsiveness, and effectiveness in meeting the ever-changing market demand [23].In this regard, Dursun and Ogunclu [24], Kumar et al. [25], and Abdollahi et al. [26] are considered some of the most influential studies on agile supplier evaluation and ranking in recent years.Dursun and Ogunclu [24] evaluated, ranked, and selected agile suppliers in a travel agency using the TOPSIS method, making it possible to combine many factors in a multi-level hierarchical structure.The results indicated that selecting agile suppliers could be highly profitable in the long term.Kumar et al. [25] adopted a fuzzy DEMATEL approach to evaluate and rank agile suppliers under uncertainty.The results showed that using fuzzy logic and the DEMATEL technique results in better-performing suppliers and gives them a competitive advantage in global markets.Abdollahi et al. [26] presented an integrated approach to evaluate, rank, and select a portfolio of agile and lean suppliers using a mix of the analytic network process (ANP) and DEA.The findings indicated that considering the connections and interdependencies among supplier evaluation criteria using the ANP and DEA would lead to remarkably more accurate evaluation, ranking and selection of the required suppliers.
Today's markets go through fluctuations, and organizations constantly face growing business risks; consequently, suppliers are also exposed to various disruptions.While the main objective of supplier evaluation, ranking, and selection practices was to minimize costs or optimize services, the emphasis in recent has shifted toward ways of building resilience [27,28].The concept of resilience refers to a system's ability to reduce the likelihood of failure and minimize the longer-lasting consequences of failure during the efforts to restore its performance level to normal [29].Analyzing and managing supplier's resilience leads to preventing undesirable conditions, i.e., situations that may lead to failure.Building resilience can be beneficial for any SC member as it helps identify the SC's potential risks and offer possible recovery strategies, thereby improving the collaboration between different levels and members of the SC [30].Various resilience metrics and strategies have been proposed in the literature, including supply incentives, dynamic assortment planning, accurate demand forecasting, SC viability, demand-based management, flexible sourcing, flexible supply base, flexible transportation, strategic stock, SC visibility, revenue management, emergency stock, gradual decline and elimination of the product from the production cycle in case of inefficiency under disruption, the decision on whether to build or purchase, and postponement to minimize the likelihood of manufacturing products with low market elasticity [31].Some of the most considerable studies on this subject are Pamucar et al. [30], Davoudabadi et al. [32], Sureeyatanapas et al. [31], and Abedian et al. [33].Pamucar et al. [30] ranked and selected resilient suppliers using a neural network-based fuzzy decision-making approach to identify the connections and interdependencies between various factors.Davoudabadi et al. [32] employed a fuzzy interval-valued decision-making framework to rank and select suppliers under uncertainty.Sureeyatanapas et al. [31] adopted the TOPSIS decision-making method to evaluate and rank the resilience of suppliers.Abedian et al. [33] employed fuzzy DEA to assess the resilience and sustainability of candidate suppliers in the electronic industry.They aimed to select qualified suppliers for employing in procurement management system.The results show that considering the indicators of resilience and sustainability simultaneously in evaluating suppliers leads to selecting cases with high effectiveness and efficiency.

Procurement and supply management under uncertainty
Uncertainty in sourcing and procuring the customers' demanded products is often caused by constant changes in the business environment and fluctuation in the relevant parameters [34].According to Mula et al. [35], there are three broad types of uncertainties: random (which originates in the randomness of parameters), epistemic (caused by lack of knowledge or historical data on some parameters), and fuzzy (caused by the flexible or fuzzy constraints).On the other hand, Thunnissen [36] posits that uncertainty is either environmental (caused by external factors such as customer demand, the capacity of facilities, unit costs, etc.) or systemic (caused by intra-system factors such as production line failures or machinery breakdown).Many different approaches have been proposed in the literature to cope with uncertainty, such as fuzzy programming, robust optimization, and stochastic programming.Some of the essential recent studies on supplier evaluation, ranking, and selection include Sahebjamnia et al. [37], Vahabzadeh et al. [38], and Solgi et al. [39].Sahebjamnia et al. [37] attempted to evaluate, rank, and select resilient suppliers and allocate orders under uncertainty using a mathematical model in addition to fuzzy DEMATEL and ANP decision-making approaches.This study adopted the interval programming technique to cope with uncertainty.Vahabzadeh et al. [38] presented a sustainable approach to evaluate and select flexible suppliers in the Iranian metal casting industry under uncertainty while considering competition as a factor.To this end, the best-worst and the DEMATEL approaches were employed to investigate the connections and interdependencies among the critical factors in supplier selection.Moreover, the evidence theory was used to cope with the uncertainty of the problem.Table 1 presents a summary of reviewed studies, which can be employed to extract research gaps and related contributions.

Research gaps
Based on the reviewed studies, some of the current literature gaps are outlined in this subsection.Many supplier evaluations and ranking studies have been done using MCDM models and mathematical programming methods such as DEA.However, in most cases, the efficiency of suppliers has been measured based on the minimum distance from the efficient frontier, with little attention given to the maximum distance of DMUs from the inefficient frontier.Moreover, few studies have computed the efficiency of suppliers based on distances from both the efficient and inefficient frontiers.The parameters involved in supplier evaluation and ranking are inherently uncertain; however, few researchers have tried to cope with this uncertainty by integrating the two-stage stochastic programming approach and robust optimization.It is also essential to focus on suppliers' efficiency, agility, and resilience under disruption to ensure rapid, accurate, and effective provision of the organization's requirements.
In the present research, we evaluate the efficiency and rank suppliers by a novel integrated DEA approach in which both the efficient and inefficient frontiers are considered under uncertainty and disruption.We use Mulvey's robust-stochastic optimization (RSO) method to cope with uncertainty.The DEA model evaluates suppliers in terms of resilience and agility separately to ensure their preparedness for disruption incidence.
Next, the two efficiencies obtained are combined to form a final efficiency score.Lastly, to verify the validity and reliability of the proposed model, we apply it to a case study in the Iranian telecom industry and extract several managerial implications and insights.

Hybrid framework to evaluate agile and resilient performance
Evaluating SC members' agility and resilience to disruptions is a reliable way of ensuring that appropriate adaptive and/or preventive strategies and collaborative approaches are in place in case of minor disruptions or full-scale crises.In particular, performing the evaluations above often results in more resilient, agile, responsive, and adaptive suppliers in the SC.Also, it improves the SC's cost efficiency and overall performance in case of risks.DEA is one of the most efficient techniques for evaluating the efficiency of DMUs without valuable insights into the weights of assessment criteria [40].DEA is a non-parametric method with proven success in a performance appraisal that evaluates the efficiency of DMUs by considering multiple inputs and outputs.In this method, selecting and determining the input and output variables for all system DMUs is crucial to each unit's accurate performance evaluation.DEA also manages the complex relationships between input and output variables [41].

Problem description
DEA is a technique based on linear programming (LP) that utilizes LP problem-solving methods and duality theorems to identify the source and level of inefficiency for each input and output.The fundamental concept of this method is based on outlining the production frontier, which refers to the range within which DMUs operate efficiently.The scores of inefficient units are then computed relatively by being compared to efficient units [42].According to relevant studies [43,44], DEA can determine the importance weight of input and output criteria.In this context, weights are considered decision variables, and their respective values are determined in such a way as to maximize efficiency, helping identify inefficient DMUs in the process.However, given that a large number of units are located on the efficient frontier in most cases, the DEA's ability to detect inefficient DMUs significantly decreases.Calculating the efficiency of DMUs based on the degree of their deviation from the efficient frontier is regarded as a defect.
Consequently, researchers (e.g., [17]) have tried to measure the efficiency of DMUs based on the score they obtain relative to both the efficient and anti-efficient frontiers via a hybrid DEA model.The hybrid model combines the standard and inverted DEA formulations developed initially by Charnes et al. [15], and Yamada [45].Standard and inverted DEA models outline the efficient and anti-efficient frontiers, respectively.The graph in Figure 1 presents a geometric outline of the mentioned frontiers.As can be seen, the efficient frontier is formed by obtaining the efficiency of the DMUs with the best performance, i.e., DMUs A, F, E, and D in the standard model.Moreover, the anti-efficient frontier is formed by obtaining the efficiency of DMUs with the worst performance, i.e., DMUs A, B, C, and D in the inverted model.
Integrating the calculated efficiencies based on the efficient and anti-efficient frontier enhances the performance measurement capability of the proposed evaluation model for each DMU.In this research, any particular potential supplier is considered a DMU, and the aforementioned hybrid DEA is applied to evaluate potential suppliers' agile performance and resilience capabilities in case of disruptions.Since the two indicators, including agile and resilient performance, should be evaluated simultaneously for each potential supplier in case of disruptions, the aggregated efficiency of each supplier is computed by combining the scores obtained for both agile and resilient efficiencies through the weighted average method for each supplier.If a potential supplier does not meet minimum aggregated agility and resilience efficiency requirements under disruptions, it will be disregarded in the SC design process.On the other hand, as the performance of an accepted supplier edges closer to meeting the minimum requirements, its total cost increases, and its available capacity decreases simultaneously.
Figure 2 illustrates the flow chart of the considered hybrid approach designed to evaluate, rank, and select agile-resilient suppliers in a disruption situation.In this context, first, according to the organization's priorities and experts' opinions, a set of critical commodities requested by the organization are identified.Second, a set of primary candidate suppliers are extracted by examining the results of the initial evaluation of suppliers by the organization.Third, input and output criteria for assessing candidate suppliers' agile/resilient performance are determined by reviewing relevant literature and experts' opinions.Forth, the quantity of each agility/resilience input and output criterion under disruption scenarios are estimated to formulate the mathematical model for hybrid DEA.Fifth, the hybrid DEA model is executed and solved to obtain both agile/resilient efficiency for each candidate supplier.Sixth, two thresholds for agile/resilient efficiency of suppliers under disruption situation is determined by experts.Accordingly, if a supplier's agile efficiency meets the corresponding agility threshold, then the selected supplier and its efficiency based on agility will be recorded; otherwise, it will be failed and be excluded from the list of candidate suppliers.It should be noted that a similar policy is taken to eliminate non-compliant suppliers based on resilient performance.Seventh, the remaining suppliers' resilient and agile efficiency are aggregated in agile-resilient performance.Finally, the agile-resilient suppliers are ranked based on the aggregated efficiency and will be employed to procure key required commodities based on the organization's needs and available financial resources.

Mathematical formulation for hybrid stochastic DEA
The indices, parameters, and variables employed in the mathematical formulation of the ahead study, which is common to both the standard and inverted DEA models, are as follows.Weight of output criteria  Equations ( 1)-( 5) describe a non-linear and fractional scenario-based standard DEA model that is separately solved for each candidate supplier to independently determine both expected agile/resilient efficiency variables in case of disruptions.Note that herein ℎ ∈  represents the ℎth DMU among potential considered suppliers in this study.
Equations ( 6)-( 10) illustrate the non-linear and fractional scenario-based inverted DEA model in which the expected inefficiency is optimized for each candidate supplier separately to calculate both expected agile/resilient efficiencies under disruption for any particular DMU.
Since equations ( 1)-( 5) describe a non-linear and fractional scenario-based standard DEA model, Charnes et al. [15] suggested an equivalent linear formulation for the model as follows.
Due to the inherent uncertainty of the parameters caused by operational risks and business environment fluctuations, a tailored mechanism should be employed to tackle such risks.This study adopts a RSO approach inspired by Mulvey et al. [47] to enable the proposed deterministic model to cope with uncertainties.The formulation is thoroughly described in Section 4.
After solving both the above-mentioned robust models for the DMU, i.e., supplier ℎ, the efficiency of the ℎth DMU is determined based on the efficient and anti-efficient frontiers that are represented by Φ  ℎ and Φ  ℎ , respectively.Suppose that Φ ℎ is the aggregated efficiency of DMU ℎ.In order to obtain a final efficiency score for each DMU and integrate the results of both models, Φ  ℎ and Φ  ℎ are aggregated into Φ ℎ by calculating the following indicator for each DMU: Since Φ  ℎ and Φ  ℎ are respectively set in intervals (0, 1] and [1, +∞), the conversion factor 2 ; and if it is located on both the efficient and anti-efficient frontiers, simultaneously, Φ  ℎ = 1, Φ  ℎ = 1, and Φ ℎ = 1 2 .If DMU ℎ is located on the efficient frontier, the Φ ℎ will be larger than 1  2 that has a higher efficiency score on both frontiers.Given the developed performance evaluation approach, each supplier's agility and resilience under disruption should be evaluated independently.Hence, both the standard and inverted DEA models should be run twice to obtain the accurate values of the mentioned indicators for each DMU.AP  ℎ and AP  ℎ represent the agile performance of DMU ℎ under disruption and are calculated based on the efficient and anti-efficient frontiers, respectively.RC  ℎ and RC  ℎ represent the resilient performance of DMU ℎ under disruption and are calculated based on the efficient and anti-efficient frontiers, respectively.AP ℎ and RC ℎ indicate the final aggregated agility and resilience of DMU ℎ, respectively.Lastly,  Agi and  Res represent the importance weight of the agility and resilience of DMU ℎ under disruption, respectively.The final agile-resilient efficiency of DMU ℎ, denoted by ∆ ℎ , is calculated by equation (24).

Input and output criteria of proposed hybrid DEA
In this study, all potential suppliers' efficiency is evaluated separately in disruption incidence in terms of agility and resilience.This approach requires that two independent performance evaluation problems be defined.The performance score derived from the two mentioned perspectives is aggregated through a suitable approach.In the end, the integrated resilient-agile efficiency score obtained for each supplier is considered as the indicator of the supplier's performance under disruption, which is directly employed to evaluate and rank the candidate suppliers for selecting qualified cases.Note that the input criteria for assessing the performance of DMUs in terms of agility and resilience are considered identical, but the output criteria are different.The input and output criteria are introduced below.

-Input criteria
In this study, the same input criteria are considered for assessing the efficiency of DMUs from the perspective of agility and resilience.Human resources, operational capacity, cash, and non-cash assets are the most crucial input criteria in supplier evaluation and ranking problems [48].Human resources refers to a group of people [ 53,54] Recovery quality SC's ability to return to its pre-disruption performance or a better state in disruption incidence Business continuity SC's capability to achieve following the recovery process under disruption Coordination SCs' components capability to take measures that optimize the overall performance Revenue and risk sharing SC's ability to efficiently and evenly distribute all incentives, including profit and losses, among all components under disruption.[55,56] who run an organization's operations and work to improve their performance at work.Operational capacity is the maximum capacity of an organization to provide services and products to its customers and has a lasting impact on customer satisfaction.Cash and non-cash assets are resources that have economic value, are owned by individuals, organizations, or governments, and are expected to generate profit in the future.Assets are acquired to increase the organization's value or benefit from the company's operational advantages [49].According to the opinion of the decision-makers (DMs), the suppliers' reduced scores in the three criteria mentioned above are optimal, and they are considered the input criteria for evaluating the performance of suppliers under disruption.

-Output criteria
The output criteria for agility and resilience measurement are pretty different.Agility consists of three capabilities: responsiveness, effectiveness, and flexibility.Responsiveness refers to the ability of a SC member to respond to any changes or deviations within an appropriate length of time and involves three enablers: velocity, visibility, and reactiveness.Effectiveness refers to the ability to do the right thing and consists of two capabilities: reliability and completeness.Lastly, flexibility can change oneself or respond to a situation quickly with minimum effort, cost, and skill.Three metrics can measure flexibility: process, delivery, and product [50,51].The criteria described in this paragraph are summarized in Table 2.
Resilience involves some output criteria, including adaptability, recovery quality, business continuity, coordination, affordability, collaboration, integration, revenue, and risk-sharing, as detailed in Table 3.
The input and output criteria for evaluating the performance of potential SC members under disruptions from the two perspectives of agility and resilience are evaluated separately.Due to the application of the hybrid DEA proposed by Shen et al. [17] to evaluate the efficiency of DMUs under disruption, considering numerous input and output criteria will not cause the efficiency of all DMUs to tend to be 100%.

Robust-stochastic DEA
In this section, an RSO approach is applied to capture operational risks in the concerned SC.RSO methods are efficient at countering risks caused by insufficient historical data or knowledge for estimating the probability distribution of uncertain parameters [57].Many researchers, such as Vahabzadeh-Najafi et al. [38], adopted RSO techniques for supplier evaluation, assessment, ranking, and selection problems to cope with fluctuations and uncertainties caused by the turbulent business environment.Mulvey et al. [47] proposed an RSO approach, which has been successfully adopted in the relevant literature.Their proposed methodology, i.e. (the Mulvey's approach), considers variance as an index for measuring the variability of objective functions.Mulvey et al. [47] proposed two measures, including solution robustness (SR) and model robustness (MR), to assess and control the performance of the robust model.SR seeks to achieve a very close to the optimal solution, and MR ensures the feasibility of the solution by considering an infeasibility penalty function.Suppose a scenario-based problem is compactly formulated as follows: s.t.
: free ∀ ∈  (31) illustrates the indices of probable scenarios. and  correspond the technical coefficients matrix for first stage variables.  and   define the technical coefficients matrix for second stage variables. and   denote the objective function coefficients for first and second stage variables  and   denote pre-event and post-event decision variables, respectively. demonstrates the expected objective function and   denotes the objective function under scenario .Accordingly, the compact formulation for applying the considered RSO method (i.e., [47]) to the above-mentioned scenario-based model is presented below. s.t.
,   ≥ 0 ∀ ∈  (37) ,  ′  : free ∀ ∈  (39) Note that   and  ′  denote the violation variable for the constraint with an uncertain right-hand side.In addition,  and  represent the variability weight and the risk aversion weight for the robust-stochastic model, respectively.Since the violation variable for an inequality constraint with non-deterministic right-hand side belongs to the set of positive real numbers, therefore, the square power of that variable can be ignored in the objective function (33), but this case does not hold for equality constraints.
The robust objective function (33) consists of three terms, the first and second of which are the mean and variance of the objective function under disruption, respectively.Increasing the variability weight reduces the sensitivity of the model to fluctuations of input parameters.The two terms guarantee the robustness of the obtained solution.The third term controls the model's infeasibility, which is the foundation of any robust formulation.The optimal robust solution should be located inside the problem's feasible region and close to optimality.Please refer to Rahimi et al. [28] and Almaraj and Trafalis [58] for more detailed information on the Mulvey's method.
Due to the presence of non-linear terms in the objective function (33), the complexity of the proposed formulation significantly increases, and, as a result, it cannot be applied to large-scale problems.Since   and  ′  respectively represent violation variables for non-deterministic unequality and equality constraints, so the terms  2  and  ′2  in the objective function (33) can be converted into   and | ′  |, but due to the presence of absolute term in the mentioned objective function, the model will be still non-linear.Accordingly, to avoid this non-linearity, Yu and Li [59] proposed an equivalent linear robust formulation for the above-mentioned non-linear model as follows.
Note that  +  ,  −  ,  +  and  −  are auxiliary variables employed in the linearization of the proposed robust formulation.
Accordingly, a suitable technique should be adopted to cope with the inherent uncertainty of the parameters.This study employs the Mulvey's RSO method to counter the hybrid DEA models' input and output data uncertainties.Based on the Mulvey's method [47], the robust formulation of the standard DEA model is as follows.
Note that    is a violation variable for an inequality constraint with non-deterministic right-hand side, and the square power of    will be ignored in the objective function (50).
The robust inverted DEA model based on the Mulvey's method [47] is as follows.
≥ 0 ∀ ∈ ,  ∈  (65) where  and  respectively represent variability weight and risk aversion weight.Besides,  ℎ is the violation variable for supplier ℎ under scenario .Variable vectors Φ  ℎ and Φ  ℎ denote the expected efficiency of supplier  calculated on the efficient and anti-efficient frontier, respectively.
Given that equations ( 50)-( 58) and ( 59)-(67) describe the robust non-linear formulation for standard and inverted DEA models caused by the variance term  ∑︀ ∈   (  − ∑︀  ′ ∈   ′   ′ ) 2 , the complexity of both the standard and inverted models significantly increases.Therefore, converting non-linear formulations to linear equivalents improves the accuracy and reduces the required time of the solution process.Herein, a simple method is adopted to convert the non-linear robust-stochastic model into an equivalent linear formulation suggested by Yu and Li [59].Accordingly, by considering  +  and  −  as the conversion variables for the quadratic term of variance in the robust objective function, the non-linear robust standard DEA model can be linearized as follows.
Given that one between  +  and  −  takes the value zero, equations (71) and (80) evaluate the expected deviation of efficiency from its expected value in precisely the same way as the variance.Thus, the non-linear variance  ∑︀ ∈   (  − ∑︀  ′ ∈   ′   ′ ) 2 can be replaced by the linear variance by equations ( 71) and (80) in both standard and inverted non-linear robust DEA models, respectively.

Case study
In this study, the performance level and applicability of the proposed model are tested through a case study of a telecom company in Iran.Today, the company provides coverage for more than 1000 cities.The company, as one of the main telecom organizations in Iran, needs and cooperates with many suppliers.Given the geographical dispersion of its operational bases, it requires integrated supplier management and evaluation system.Accordingly, one of the objectives of the present study is to propose an optimal framework to identify, evaluate, select, and manage a group of approved suppliers to ensure an effective, efficient procurement process at the company.Efficient suppliers in the telecom industry should be agile enough to respond to any sudden changes or unforeseen disruptions, to maintain their ability to meet the company's large-scale requirements as quickly and effectively as possible.Moreover, because most of the devices required in the telecom industry are considered strategic, and various disruptions may regularly occur during their procurement process, the suppliers need to be resilient and go through most disruption events relatively unscathed.On this basis, it can be concluded that only agile and resilient suppliers suit the telecom industry.
Based on the above, the objective of this study is to evaluate and rank the company's suppliers according to their level of agility and resilience.For this purpose, we analyzed the annual data of 90 candidate suppliers to provide the physical devices and/or specialized services required by the company.It should be said that, in the real world, various limitations such as the lack of sufficient data, suppliers' refusal to provide correct information, auditors' not being granted appropriate access to certain data, etc. forced the researchers to estimate some of the necessary data for supplier evaluation and prioritization.Therefore, despite our best efforts, there is an inevitable degree of uncertainty in the data.In order to minimize the impact of said uncertainty, three scenarios were defined as optimistic ( 1 ), most likely ( 2 ), and pessimistic ( 3 ) with probability rates of 0.25, 0.50, and 0.25, respectively.Table 4 indicate the input and output criteria for evaluating the candidate suppliers' agility and resilience.Herein, Table 5 provides the estimated values of each potential supplier's input and output parameters for resilient and agile perspectives under three possible scenarios to compute the efficiency.Note that the optimistic scenario has fewer inputs and more outputs than the pessimistic scenario.

Numerical results
This study applies the proposed approach to the company, as a case study.The mathematical model is solved by the CPLEX solver in the optimization software GAMS using a Windows 10 PC with a 10th generation Intel R ○Core TM i7 processor and 10 GB of RAM.The performances of candidate suppliers are evaluated once in terms of resilience and once in terms of agility using the hybrid DEA approach developed by Shen et al. [17].The two efficiencies are then combined by assigning importance weights to form a hybrid resilient-agile efficiency.The candidate suppliers' scores in this hybrid metric will be the basis for this research's supplier evaluation and ranking approach.In addition, Mulvey's RSO approach is adopted to cope with the uncertainty of the parameters.At the end of this subsection, several numerical results and managerial implications are presented, along with the analyses performed thereon.It must be noted that the optimal importance weights for agility and resilience and the variability and risk-aversion coefficients in Mulvey's model are  = 3.6,  = 2.4,  Res = 0.45, and  Agi = 0.55, respectively.
Table 6 details the results of solving the proposed DEA model for resilience and agility criteria separately.It also gives the aggregated scores of the two efficiencies to compute each DMU's resilient-agile efficiency and their final rank in terms of resilience, agility, and resilience-agility efficiencies.
It is worth noting that due to the application of the hybrid DEA proposed by Shen et al. [17] to evaluate the efficiency of DMUs or facilities under consideration in the incidence of disruption, the possibility of considering a large number of input and output criteria without tending the efficiency of all DMUs to 100% is provided.
Figure 3 illustrates the impact of changes in the variability coefficient () of Mulvey's robust model on the hybrid resilient-agile performance of the candidate suppliers.This work adjusts the value of the variability coefficient and helps maintain the model's relative consistency.
Table 5.The outputs and inputs of suppliers under all scenarios (optimistic, most likely, pessimistic).It can be seen that as the variability coefficient of the robust-stochastic DEA model increases, the values of the resilience and agility objective functions and, thereby, the hybrid resilience-agility efficiency decrease.This behavior is because as the variability coefficient (or variability penalty) in the standard and inverted DEA model increases, the variance value increases as well; consequently, the objective functions of the two models decrease.Given that the final efficiency score is obtained from combining the standard and inverted models, the final efficiency score in both the resilient and agile approaches decrease by increasing the variability coefficient.Based on our analyses, when  = 2.4, consistent robustness is observed in the objective functions, and the robustness cost is acceptable.It can thus be concluded that the most suitable value for the variability coefficient is 2.4.
Figure 4 depicts the impact of changes in the risk-aversion coefficient () of Mulvey's robust model on the hybrid resilience-agility performance of the candidate suppliers.
The graph shows that as the risk-aversion coefficient increases, the values of the resilience and agility objective functions and the hybrid resilience-agility efficiency decrease.This behavior is because increasing the riskaversion weight in the standard and inverted DEA model makes it harder to violate the constraints; consequently, Table 5. continued.the objective functions of the two models decrease.Since the final performance score is obtained from aggregating the results of solving standard and inverted models, the final efficiency score in both the resilient and agile approaches decrease.Based on our analyses, when  = 3.6, there is consistent robustness in the objective functions, and the robustness cost is also acceptable.Therefore, the most suitable value for the risk-aversion coefficient is 3.6.Table 7 analyzes the components of the objective function in the robust-stochastic DEA model for the suppliers.As previously mentioned, to ensure consistent robustness, the optimal importance weights of agility and resilience and the variability and risk-aversion coefficients are set at  = 3.6,  = 2.4,  Res = 0.45, and  Agi = 0.55, respectively.
As the table shows, the value of the hybrid resilience-agility score is at its largest under the optimistic scenario and its most minor in the pessimistic scenario.Moreover, the expected value of the objective function of Mulvey's robust model for each DMU is close to the optimal efficiency in the most likely scenario.Finally, the total efficiency value in the robust model for each DMU is always lower than the value of the objective function in the pessimistic scenario.

Managerial insights
In this section, some of the most significant managerial insights are provided, which arise from employing the proposed approach to evaluate, rank and select qualified suppliers in term of resilience and agility in the telecommunication industry.These insights can be a guide for other organizations to select suppliers with the appropriate agility and resilience.Some of the obtained managerial implications are as follows: (1) Simultaneous evaluation of the candidate supplier's agility and resilience leads to the selection of reliable, flexible, and capable suppliers for procurement management system under risks; (2) Decreasing input criteria and increasing output criteria in a resilient and agile supplier improves overall performance; (3) Suppliers with higher adaptability, reliability, visibility and coordination in procurement management have a significant chance for being selected in the telecommunication industry or similar companies; and (4) Considering operational risks or uncertainty in the evaluation, ranking, and selection of qualified suppliers lead to the preparation of the procurement management system for the worst scenarios ahead.

Conclusion
Activities and processes in organizations can only be carried out well if their needs and requirements are provided with the highest quality and in the shortest possible time.The performance of suppliers directly affects organizations' ability to satisfy their customers and earn their loyalty.Therefore, supplier evaluation, ranking, and selection are considered essential and taken seriously by organizations.Suppliers often face various risks in the business environment, which inevitably negatively impact the procurement process and may lead to customer dissatisfaction.The suppliers selected as part of the SC should be agile and resilient to deal with  This research proposes a hybrid robust-stochastic DEA method to evaluate, prioritize, and select eligible suppliers based on their resilience and agility under uncertainty and disruption.For this purpose, a two-stage disrupted mathematical model is developed to determine the resilience and agility of the candidate suppliers using a hybrid approach by considering the maximum distance from the inefficient frontier and the minimum distance from the efficient frontier simultaneously.Therefore, the decisions made in the research include the importance weights of each input and output criterion and the resilient-agile performance of each DMU based on the efficient and inefficient frontiers.Furthermore, Mulvey's RSO approach was adopted to counter the uncertainty of the problem.Lastly, to verify the validity and reliability of the proposed model, it was applied to a real-world case study in the Iranian telecommunication industry, and several numerical results and managerial insights were obtained.Results reveal the high importance of some criteria, including human resources, cash assets, adaptability, reliability, visibility, and coordination for assessing the resilience and agility of suppliers involved in procurement management of the telecommunication industry.Observations indicate that the simultaneous consideration of indicators related to resilience and agility in evaluating and ranking suppliers leads to the selection of suppliers that are flexible against risky business conditions and quick to respond to the changing needs of the communication industry.Also, it is important to use the RSO approach to control the fluctuations of uncertain parameters and maintain a desirable optimal level of efficiency for DMUs under different scenarios.
Although we did our best to present a practical solution and enrich the literature through this research, there are various ways to build upon, expand, or improve the proposed model.For instance, if there are meaningful

Figure 2 .
Figure 2. Flow chart of developed hybrid approach to evaluate, rank, and select agile-resilient suppliers.

Figure 3 .
Figure 3. Impact of varying variability weight on the agile-resilient performance of DMUs for  = 3.6,  Res = 0.45.

Table 1 .
Categorization of reviewed studies on agile-resilient supplier evaluation, ranking and selection.
OU   Quantity of output  provided by supplier  in case of disruption    Occurrence probability of scenario Set of potential suppliers indexed by  ∈   Set of supplier's input criteria indexed by  ∈   Set of supplier's output criteria indexed by  ∈   Set of disruption scenarios indexed by  ∈  Parameters IN   Quantity of input  utilized by supplier  in case of disruption

Table 2 .
Output criteria for evaluating agility of potential facilities.

Table 3 .
Output criteria for assessing resilience performance of potential SC members.

Table 4 .
Output and input criteria for assessing resilience and agility of suppliers.