Performance Indicators Framework for Assessment of a Sanitary Sewer System Using the Analytic Hierarchy Process (AHP)

In this study, performance indicators (PIs) for assessing services of the sanitary sewer system in South Korea were evaluated based on general opinions collected from experts in the field. The analytic hierarchy process (AHP) was then carried out. The evaluated set of PIs consisted of five major criteria: management, operation and maintenance, service, environment, and finance. Using the experts’ survey incorporated into the AHP tool, the prioritization of the five criteria was performed, consisting of a total of 14 indicators and 34 checklists on three levels. Of the criteria groups, operation and maintenance was found to be the most important indicator, comprising 43% of all the scores. The AHP results showed that, of the 34 checklists, 13 indicators were explained as candidates of key PIs: on-the-job training and work role and responsibility from management, sewer condition inspection, pump, sewer maintenance, flow rate/water quality/odor monitoring from operation and maintenance, complaint resolution from service, inflow volume from environment, and operational cost in annual expenditures from finance. The PIs developed in this study are expected to be used by stakeholders involved in the provision of sewer services, such as undertaking companies, policy-making bodies, and financing agencies.


Introduction
Sanitary sewer systems (SSSs) are designed to collect and transport domestic, commercial, and industrial wastewater, and limited amounts of stormwater to a wastewater treatment plant. Sanitary sewers, also often called separate sewers, are different from combined sewer systems (CSSs), which transport domestic and industrial wastewater and stormwater runoff in the same sewer to treatment facilities. In contrast, in SSSs, wastewater and stormwater are collected and transported in separate pipes.
A standardized performance assessment for wastewater services was established by Matos et al. [1], based on a system of performance indicators (PIs) that cover all fundamental tasks of wastewater and sewerage services. Various international organizations, such as the International Water Association (IWA) [1], International Benchmarking Network for Water and Sanitation Utilities-World Bank (IBNET) [2], the Office of Water Services (OFWAT) [3], United States Environmental Protection Agency (US EPA) [4], and the American Water Works Association (AWWA) [5], have proposed different PIs covering all the fundamental tasks of wastewater systems. Other working groups in several countries have also proposed PIs that reflect the chief aspects of the management of a water and sewerage service [6][7][8][9]. In those systems, PIs to assess the performance of wastewater services were

Selection Criteria of Performance Indicators
As aforementioned in the introduction, in South Korea, a set of the existing PIs for sewer services is made up of four criteria: management, operation, maintenance, and quality of service. This set of PIs does not consider the effect of environmental pollution by sewer service failures or contamination accidents, such as sanitary sewer overflows. Indicators to evaluate financial sustainability are also included based on the literature survey, such as the IWA and AWWA frameworks and the pre-query by experts in undertaking companies in the field. Thus, the PIs for performance evaluation of SSSs have a hierarchical structure with four levels (i.e., main category, main group, sub-group, and calculation level) according to their thematic categories, as displayed in Figure 1. In this section, we only explain about three levels, and in the next section (Section 4) for KPIs.
Main category (level 1): this is the first layer of indicators and consists of the five main categories: management, operation and maintenance, service, environment, and finance. The composition of level 1 is defined as below (A1 to A5). Groups (level 2): this is the second layer of indicators that are distinguished by the major tasks in the upper level. In this study, a total of 14 indicators are separately contained in level 1, and each category in level 1 contains two to four working tasks groups.
Sub-groups (level 3): these contain 34 indicators to check the detail activities in terms of how they should be performed; each indicator assigns points to qualitatively and quantitatively evaluate the sewer service performance. Each group in level 2 contains one to four specific activities, resulting in 34 indicators in level 3.
Calculation level (level 4): this level includes the unit and measurement concept for each indicator, and provides a numeric expression for every PI.

Selection Criteria of Performance Indicators
As aforementioned in the introduction, in South Korea, a set of the existing PIs for sewer services is made up of four criteria: management, operation, maintenance, and quality of service. This set of PIs does not consider the effect of environmental pollution by sewer service failures or contamination accidents, such as sanitary sewer overflows. Indicators to evaluate financial sustainability are also included based on the literature survey, such as the IWA and AWWA frameworks and the pre-query by experts in undertaking companies in the field. Thus, the PIs for performance evaluation of SSSs have a hierarchical structure with four levels (i.e., main category, main group, sub-group, and calculation level) according to their thematic categories, as displayed in Figure 1. In this section, we only explain about three levels, and in the next section (Section 4) for KPIs.
Main category (level 1): this is the first layer of indicators and consists of the five main categories: management, operation and maintenance, service, environment, and finance. The composition of level 1 is defined as below (A1 to A5).

•
Management indicators (A1): evaluate the efficiency and effectiveness of personnel undertaking management activities for long-term sewer services; these indicators include functions, qualifications, and safety training. • Operation and maintenance indicators (A2): are used for assessment the performance of the undertaking related to the operation activities and maintenance programs for SSSs. • Service indicators (A3): determine the level of services provided to customers as well as customer satisfaction. Information on the sewer service and responses to complaints are measured to ensure quality of service. • Environmental indicators (A4): evaluate the performance of the sewer undertaking regarding environmental impacts, including sanitary sewer overflows and infiltration/inflow. The compliance with wastewater discharge standards and regulations is also assessed. • Financial indicators (A5): assess the financial status of the undertaking company, as well as the financial effort (such as reinvestment rate for a sewer system) to maintain the service life of the infrastructures.
Groups (level 2): this is the second layer of indicators that are distinguished by the major tasks in the upper level. In this study, a total of 14 indicators are separately contained in level 1, and each category in level 1 contains two to four working tasks groups.
Sub-groups (level 3): these contain 34 indicators to check the detail activities in terms of how they should be performed; each indicator assigns points to qualitatively and quantitatively evaluate the sewer service performance. Each group in level 2 contains one to four specific activities, resulting in 34 indicators in level 3.
Calculation level (level 4): this level includes the unit and measurement concept for each indicator, and provides a numeric expression for every PI.

Design of Survey Questionnaire for Experts' Opinion
Subsequent to reviewing the literature and cases of PIs developed by international groups, a draft framework of PIs was initially prepared and sent to sewer service experts in multi-stakeholder forums to request identification of any problems that may arise as well as revisions reflecting domestic situations that do not necessarily fit other PIs, such as IWA or US EPA. Based on their feedback, the criteria and indicators in the draft framework were finalized and an online survey form was prepared in Google that was then distributed to various experts in sewer service companies. The panel of experts, consisting of engineers and managers with more than seven years of work experience in the field, were asked to provide their knowledge-based judgement on the questionnaire. A total of 45 responses were obtained from the 100 surveys sent, and the collected input data were used for analytic hierarchy process (AHP) implementation.

AHP Procedures
The analytic hierarchy process (AHP), introduced by Thomas Saaty (1980), is one of the most popular multiple-criteria decision-making (MCDM) tools for formulating and analyzing decisions. It can assist the decision-maker to set priorities and make the best decisions [21]. Based on a pairwise comparison, AHP can be described based on three principles, namely, decomposition, comparative judgement, and synthesis of priorities [22]. The advantage of using AHP is its ability to integrate the quantitative and qualitative factors obtained through experts' opinions and categorize them into a multi-criteria ranking. The AHP is also flexible, has intuitive appeal to decision-makers, and the ability to judge inconsistencies. The AHP supports the capture of subjective and objective evaluation measures that are easy to use and are scalable. However, the AHP has several weaknesses, including problems due to the interdependence between the criteria and alternatives, inconsistencies between judgment and ranking criteria, and rank reversal. The application of AHP to a decision problem incorporates the following steps.

Step 1. Defining the Problem and Determining Its Goal
This step involves defining the decision problem and then determining the goal through AHP implementation. In the present study, the goal is to develop PIs for assessing a sanitary sewer system.

Step 2. Structuring the Decision Problem into the Hierarchy
In this step, the decision problem is decomposed from the top level to the next lower level (intermediate through to the lowest levels). The topmost level, i.e., the goal or the objective of the problem, is decomposed into the subsequent intermediate upper levels depending on the criteria. The lowest level usually contains the list of alternatives. In our study, five criteria (management, operation and maintenance, service, environment, and finance) have been chosen to evaluate the performance of the sanitary sewer system. Each of the five criteria is then further decomposed into decision alternatives (Table 1)

Step 3. Making Pairwise Comparisons and Obtaining the Judgmental Matrix
After arranging the problem and structuring it in hierarchical terms, the extent to which the different elements on one level influence those on the next higher level is determined. Thus, the elements of a particular level are compared with respect to a specific element, and the importance of each element in each level is assessed with respect to its upper (parent) level by pairwise comparisons between peer elements. In order to make the pairwise comparison, experts are asked to select one of two elements according to a scale that indicates the degree to which one element is more important, preferred, or dominant (Table S1). Saaty [23] stated that a nine-point scale could be sufficient to distinguish between two elements. Nine-point scales are explained in Table S1. Usually, an element that receives a higher rating is viewed as more important (or more critical) than an element that receives a lower rating. Each entry, a ij of the pairwise comparison matrix is governed by the three rules: a ij > 0, a ij = 1/a ji , and a ii = 1 for all i . Thus, the diagonal values of the matrix are always '1', and the values of a ij are reciprocals of a ji . Table 2 displays the pairwise comparison matrix for the selection of importance of elements in the uppermost level (i.e., criteria), and a total of 18 pairwise comparison matrices were obtained in this study since the hierarchy is structured with five criteria (level 1), 14 major groups (level 2), and 34 sub-groups (level 3). Each entry is the input data obtained from the geometric mean calculated from the responses of 45 experts. For example, in the preferential selection between A1 (management) and A2 (operation and maintenance) criteria, the value of 3.153 was entered in the position of a 21 . This indicates that the experts deemed that the activities in regard to 'operation and maintenance' were moderately more important than those of 'management' for the successful performance of sanitary sewer system services. Meanwhile, the reciprocal of the value in a 21 was input into the a 12 position. Therefore, only the upper triangle of the matrix is input and all judgments below the diagonal are the reciprocal of those above the diagonal.

Step 4. Computation of Local Weights and Consistency of Comparisons
In this step, the local weights of the elements are computed using the eigenvector method. The normalized eigenvector corresponding to the principal eigenvalue of the judgmental matrix provides the weights of the corresponding elements [22]. The local weights, which can be obtained by summing each row and dividing each by the total sum of all rows, are priorities of the elements and are normalized within a 1% or 100% scale. This represents the typical method of reporting results. Priorities are only meaningful if derived from consistent or near-consistent matrices. Consistency can be checked using the consistency index (CI), which is a natural measure of the consistency yielded by the eigenvalue method [24]. The CI is defined as: where λ max is the principal (maximal) eigenvalue of the matrix, and n is the size of each matrix. λ max is obtained by calculating the scalar product of the principal eigenvector and the vector of the column sums of the matrix [24]. Then, the obtained CI can be compared to an appropriate consistency index.
The consistency ratio (CR), which is the ratio of CI and RI (random index, Table S2) (where RI is calculated from the mean CI value) is given by [25]: The CR is basically a measure of how a given matrix compares to a purely random matrix in regard to its consistency indices [24,25]. The CR is acceptable if it does not exceed 0.1, and consistent matrices with CR < 0.1 are considered to be those in which the responses were consistent and the priorities obtained are reliable. For a consistent matrix CR = 0, and if the CR for a matrix is more than 0.1, judgements should be elicited again from the decision-maker until they give a more consistent judgment. In this study, of the 18 CRs, nine were found to be less than 0.1, and we considered the results to be consistent; the remaining eight CRs consisted of 2 × 2 matrices that were not checked for CI because 2 × 2 matrices are always consistent [26].

Step 5. Aggregation of Weights across Various Levels to Obtain the Final Weights of Alternatives
Once the local weights of elements of different levels are obtained, they are aggregated to obtain the final weights of the decision alternatives (elements at the lowest level). The final weight of alternative L 1 is computed using the following hierarchical aggregation rule: By definition, the weights of the alternatives and importance of criteria are normalized so that they sum to unity.

Analysis of the AHP Results
Based on the received 45% responses, results were analyzed in terms of local and global weights (scores) normalized to 1, and any checklist with higher weights is considered to be more prioritized. Figure 2 shows the order of the main criteria (level 1) depending on their weights (also shown in Table 2). As expected, the operation and maintenance indicators, comprising 0.430 of the entire scores, were ranked in the most important group for successful performance of sanitary sewer service. This indicated that experts in the field deemed that the tasks belonging to operation and maintenance are critical for effective and efficient performance assessment. The other four groups showed similar priorities, in the order of service (0.168) > environment (0.137) > finance (0.135) > management (0.131). Noticeably, environment and finance were selected to be more important indicators than management, implying that experts in the field agreed on the importance of sustainability in the sewer system. The consistency ratio (CR), which is the ratio of CI and RI (random index, Table S2) (where RI is calculated from the mean CI value) is given by [25]: The CR is basically a measure of how a given matrix compares to a purely random matrix in regard to its consistency indices [24,25]. The CR is acceptable if it does not exceed 0.1, and consistent matrices with CR < 0.1 are considered to be those in which the responses were consistent and the priorities obtained are reliable. For a consistent matrix CR = 0, and if the CR for a matrix is more than 0.1, judgements should be elicited again from the decision-maker until they give a more consistent judgment. In this study, of the 18 CRs, nine were found to be less than 0.1, and we considered the results to be consistent; the remaining eight CRs consisted of 2 × 2 matrices that were not checked for CI because 2 × 2 matrices are always consistent [26].

Step 5. Aggregation of Weights across Various Levels to Obtain the Final Weights of Alternatives
Once the local weights of elements of different levels are obtained, they are aggregated to obtain the final weights of the decision alternatives (elements at the lowest level). The final weight of alternative L1 is computed using the following hierarchical aggregation rule: By definition, the weights of the alternatives and importance of criteria are normalized so that they sum to unity.

Analysis of the AHP Results
Based on the received 45% responses, results were analyzed in terms of local and global weights (scores) normalized to 1, and any checklist with higher weights is considered to be more prioritized. Figure 2 shows the order of the main criteria (level 1) depending on their weights (also shown in Table 2). As expected, the operation and maintenance indicators, comprising 0.430 of the entire scores, were ranked in the most important group for successful performance of sanitary sewer service. This indicated that experts in the field deemed that the tasks belonging to operation and maintenance are critical for effective and efficient performance assessment. The other four groups showed similar priorities, in the order of service (0.168) > environment (0.137) > finance (0.135) > management (0.131). Noticeably, environment and finance were selected to be more important indicators than management, implying that experts in the field agreed on the importance of sustainability in the sewer system.  Of the operation and maintenance indicators, sewer system maintenance (OM2), followed by operational monitoring and inspection (OM1), were believed to be preferentially important, with 43.2% (local weight 0.186) in the same group. Tasks in the sewer system maintenance include drainage (C13), manhole (C14), sewer (C15), and pump (C16) maintenances. As presented in Table 3, the pairwise comparison matrix revealed that sewer and pump maintenances are the two most important indicators in the sewer system maintenance selected by the experts' group.  Figure 3 presents the local weights belonging to the major group with less than five criteria. Of the 14 indicators, the top two indicators (sewer system maintenance (OM2) and operational monitoring and inspection (OM1)) comprised 33% of the level. Complaint and response (S2) of service indictors was found to be the third most important in effective performance assessment. Of the environmental indicators, infiltration/inflow (E2) was considered to have a higher priority than sanitary sewer overflows (SSOs) (E1). This might be because infiltration/inflow have occurred more frequently than SSOs in Korea. Of the financial indicators, capital investment plan (F3), which includes tasks related to budget planning (C33), and medium-and long-term budget and business planning (C34), was the least important indicator, which might imply that the undertaking is reluctant to invest their expenditure in sewer system maintenance. Of the operation and maintenance indicators, sewer system maintenance (OM2), followed by operational monitoring and inspection (OM1), were believed to be preferentially important, with 43.2% (local weight 0.186) in the same group. Tasks in the sewer system maintenance include drainage (C13), manhole (C14), sewer (C15), and pump (C16) maintenances. As presented in Table 3, the pairwise comparison matrix revealed that sewer and pump maintenances are the two most important indicators in the sewer system maintenance selected by the experts' group.  Figure 3 presents the local weights belonging to the major group with less than five criteria. Of the 14 indicators, the top two indicators (sewer system maintenance (OM2) and operational monitoring and inspection (OM1)) comprised 33% of the level. Complaint and response (S2) of service indictors was found to be the third most important in effective performance assessment. Of the environmental indicators, infiltration/inflow (E2) was considered to have a higher priority than sanitary sewer overflows (SSOs) (E1). This might be because infiltration/inflow have occurred more frequently than SSOs in Korea. Of the financial indicators, capital investment plan (F3), which includes tasks related to budget planning (C33), and medium-and long-term budget and business planning (C34), was the least important indicator, which might imply that the undertaking is reluctant to invest their expenditure in sewer system maintenance. The distribution of the global weights obtained from the 34 checklists in level 3 is shown in Figure 4. As shown, the indicators of operation and maintenance (A2) have relatively high weights, which implies that tasks in this group are more prioritized. On the other hand, tasks in the management (A1) group showed overall less importance than those in other groups. Of the 10 checklists in the group, two indicators (on-the-job training (C8), and work role and responsibility (C5)) were positioned in relatively high weights, implying that these two indicators may be key PIs in this group. The overall prioritization of all 34 checklists is summarized according to their rankings as shown in Figure 5 and Table 4. The indicators ranked from 1 through 10 gained importance, reaching 55% of all indicators, and maintenance was not ranked in any of the 10 checklists. The indicators ranked from 11 to 34. Based on the overall weight results, sewer condition inspection (C12), pump (C16), sewer (C15) maintenance, and flow rate/water quality/odor monitoring (C11) belonging The distribution of the global weights obtained from the 34 checklists in level 3 is shown in Figure 4. As shown, the indicators of operation and maintenance (A2) have relatively high weights, which implies that tasks in this group are more prioritized. On the other hand, tasks in the management (A1) group showed overall less importance than those in other groups. Of the 10 checklists in the group, two indicators (on-the-job training (C8), and work role and responsibility (C5)) were positioned in relatively high weights, implying that these two indicators may be key PIs in this group. The overall prioritization of all 34 checklists is summarized according to their rankings as shown in Figure 5 and Table 4. The indicators ranked from 1 through 10 gained importance, reaching 55% of all indicators, and maintenance was not ranked in any of the 10 checklists. The indicators ranked from 11 to 34. Based on the overall weight results, sewer condition inspection (C12), pump (C16), sewer (C15) maintenance, and flow rate/water quality/odor monitoring (C11) belonging to A2 can be explained as key performance indicators. Of the service (A3) and environmental (A4) indicators, complaint resolution (C25) and inflow volume (C29) are the key PIs, respectively. Of the financial indicators (C5), the operational cost to annual expenditures (C31) would be a key PI. Of the 34 checklists, nine indicators could be possibly extracted as key performance indicators of the new PIs.

Proposed Set of KPIs
The new PI system includes 41 indicators that are categorized in accordance with five main criteria. The PI system is helpful in comparing the performance of wastewater utilities in a specific area; however, it cannot be used to compare the overall performance among utilities. On the other hand, the role and importance of each indicator in the PI system are different. Some indicators in the PI system will become the most important information source for improving performance, and for guiding wastewater utilities on a successful track. Thus, a set of key performance indicators (KPIs) that include the most important PIs from the new PI system is proposed for the SSSs in this study.
KPIs are measurable characteristics of products, services, processes, and operations, directly associated with the organization's strategy, and provide a good indication of the success (or failure) in determining factors that are critical for the execution of this strategy [27]. According to Parmenter [27], "KPIs indicate what should be accomplished to obtain a significant increase in performance, and represent a set of measures focused on those aspects most critical to organizational success today and tomorrow". KPIs are commonly defined in a way that is understandable, meaningful, measurable, and should follow the specific, measurable, attainable, relevant, and time-based criteria (SMART) ( Figure S3 of Supplementary Materials). The use of KPIs in the performance appraisal of companies and organizations has become more and more popular to ensure that all employees can comply with their responsibilities to make the performance appraisal become transparent, clear, specific, more effective, and easier to implement. Today, KPIs are applied successfully in other sectors, such as the construction, public sector, social media, healthcare industry, business, and water sectors .
A set of KPIs proposed for performance assessment in this study has the following functions: • Provides measurements to compare the overall performance among wastewater utilities • Provides key information needed to determine the efficiency and effectiveness of delivered services, and focuses management attention on what matters most • Monitors asset management processes and provides a common language for communication • Identifies potential problems and areas for improving the performance of systems • Provides snapshots of systems' performances focusing on goals and strategies • Evaluates the performance of systems in accordance with the classification categories of consistent, transparent, clear, specific, auditable, more effective, and easier to implement • Assigns responsibility and encourages accountability • Supports decision-making and provides a way to see if the strategic plan is working • Functions as a tool to drive desired behaviors. Figure 6 illustrates the process used to construct the set of KPIs. The work involves an identification of a set of KPIs together with their weights to obtain the overall score for the performance of SSSs. hand, the role and importance of each indicator in the PI system are different. Some indicators in the PI system will become the most important information source for improving performance, and for guiding wastewater utilities on a successful track. Thus, a set of key performance indicators (KPIs) that include the most important PIs from the new PI system is proposed for the SSSs in this study.
KPIs are measurable characteristics of products, services, processes, and operations, directly associated with the organization's strategy, and provide a good indication of the success (or failure) in determining factors that are critical for the execution of this strategy [27]. According to Parmenter [27], "KPIs indicate what should be accomplished to obtain a significant increase in performance, and represent a set of measures focused on those aspects most critical to organizational success today and tomorrow". KPIs are commonly defined in a way that is understandable, meaningful, measurable, and should follow the specific, measurable, attainable, relevant, and time-based criteria (SMART) ( Figure S3 of Supplementary Materials). The use of KPIs in the performance appraisal of companies and organizations has become more and more popular to ensure that all employees can comply with their responsibilities to make the performance appraisal become transparent, clear, specific, more effective, and easier to implement. Today, KPIs are applied successfully in other sectors, such as the construction, public sector, social media, healthcare industry, business, and water sectors [3,.
A set of KPIs proposed for performance assessment in this study has the following functions: • Provides measurements to compare the overall performance among wastewater utilities • Provides key information needed to determine the efficiency and effectiveness of delivered services, and focuses management attention on what matters most • Monitors asset management processes and provides a common language for communication • Identifies potential problems and areas for improving the performance of systems • Provides snapshots of systems' performances focusing on goals and strategies • Evaluates the performance of systems in accordance with the classification categories of consistent, transparent, clear, specific, auditable, more effective, and easier to implement • Assigns responsibility and encourages accountability • Supports decision-making and provides a way to see if the strategic plan is working • Functions as a tool to drive desired behaviors. Figure 6 illustrates the process used to construct the set of KPIs. The work involves an identification of a set of KPIs together with their weights to obtain the overall score for the performance of SSSs.  KPIs for sanitary sewer systems were selected from the new PI system that was established in Section 3. Only a few carefully selected indicators may be adequate to convey the performance information simplistically and effectively. The KPIs are selected based on the following criteria: 1.
The AHP result: the selected indicators must reflect the important indicators that were evaluated by experts (as expressed in Section 3) 2.
The set of KPIs must include all important aspects when judging an SSS 3.
KPIs should adhere to SMART criteria and other criteria to be directly relevant to objectives, should be as few as necessary, should be applicable to the options under consideration, should be comprehensive, meaningful, and relevant to all stakeholders, and applicable over time.
The selection criteria ensure that the indicators are useful and effective in terms of information to decision-makers. Based on the AHP results, the set of KPIs was devised under 10 indicators of rank 1 to 10 ( Figure 5), contributing to 55% of the overall weight. The set of KPIs (Figure 7) includes 13 indicators, which are selected from entire 41 indicators in the new PI system, covering five main aspects. The unit and measurement concepts of 13 KPIs are shown in KPIs for sanitary sewer systems were selected from the new PI system that was established in Section 3. Only a few carefully selected indicators may be adequate to convey the performance information simplistically and effectively. The KPIs are selected based on the following criteria: 1. The AHP result: the selected indicators must reflect the important indicators that were evaluated by experts (as expressed in Section 3) 2. The set of KPIs must include all important aspects when judging an SSS 3. KPIs should adhere to SMART criteria and other criteria to be directly relevant to objectives, should be as few as necessary, should be applicable to the options under consideration, should be comprehensive, meaningful, and relevant to all stakeholders, and applicable over time.
The selection criteria ensure that the indicators are useful and effective in terms of information to decision-makers. Based on the AHP results, the set of KPIs was devised under 10 indicators of rank 1 to 10 ( Figure 5), contributing to 55% of the overall weight. The set of KPIs (Figure 7) includes 13 indicators, which are selected from entire 41 indicators in the new PI system, covering five main aspects. The unit and measurement concepts of 13 KPIs are shown in

Performance Scores for Sanitary Sewer Systems
Comparison of each indicator individually is a relatively complicated task, and may not result in a useful evaluation to make further decisions. For example, a utility may perform poorly in terms of certain indicators, but may perform exceedingly well in terms of other indicators (e.g., poor financial status, but good operation and management practices). In such cases, evaluating, comparing, and rating the overall utility performances will not be easy and would require the integration of indicators to obtain an aggregated performance score (S). The aggregated performance score (S) is the sum of all weighted individual scores, and can be obtained as the following equation: where w i is relative weights obtained from the AHP results, and s i is individual scores that are normalized to a range of [0, 10], are assigned to all KPIs. Relative weights (w i ): The indicator weights, which can be expressed in either % or a fractional scale, are obtained from the AHP study and were implemented in Section 3. Table 6 presents the final weights assigned to the KPIs. Individual scores (s i ): Each KPI has different units and scales. Therefore, a normalization process is needed to transform all the KPIs to a common reference scale. The scaled scores of KPIs used in this study are explained in Table S2 (Supplementary Materials). Based on Table 6 and Table S2, the aggregated performance score (S = w i s i ) has maximum value (S max = 130) and minimum value (S min = 0). The evaluation of the performance of the system is provided as quartiles, as the following: Therefore, the performance scores (S = w i s i ) are used to compare the overall performance among wastewater utilities. Moreover, the set of KPIs integrates financial and technical measures to better evaluate the implementation of strategy for utilities. Thus, this section attempts to develop a spreadsheet-based scorecard that employs a few indicators to assess the performance of wastewater utilities. The developed scorecard can rate the relative performance in terms of different aspects, and also provide a single utility performance score. These results may be utilized by decision-makers and utility managers to define the areas of shortcomings and flaws in the operation of the utilities, and to convey the performance results in a comprehensive numeric form to non-specialists and the user community. The methodology is simple and flexible enough to incorporate any number of indicators that may be representative of utility performance. Also, the scores or weights obtained from each subcategory can be further used to assess the risk of failures in functionalities of key assets or to estimate the serviceability of the sewer systems [31,32]. The incorporation of performance scores with artificial neural networks or machine learning techniques (e.g., self-organizing map, support vector machines) enable it to classify or predict the conditions of infrastructures, such as pumps or sewers, so that preventive measures for the failures can be implemented in a timely manner [32,33].

Conclusions
A new set of PIs for the performance assessment of sanitary sewer systems was developed in this study. Using the experts' survey incorporated into the AHP tool, the prioritization of five criteria consisting of a total of 14 indicators, 34 checklists in level 3, and 41 indicators in level 4 (calculation level) was performed. Of the criteria groups, operation and maintenance was found to be the most important indicator for successful performance in sanitary sewer systems, with 43% of all scores. Indicators in environment and finance categories were considered as more important than management for the sustainability of sewer services.
Based on the AHP results and other criteria, a set of KPIs was proposed. The set of KPIs consisted of 13 indicators that were selected from 41 PIs in the new PI system, and the 13 KPIs cover 57% of the entire performance scores assessed from five major aspects of sewer services. The scores of each KPI with different units and scales were normalized into a common reference scale (i.e., the aggregated performance score, S) by incorporating the relative weight of indicators (i.e., w i ) with individual scores (i.e., s i ), thus enabling the application of KPIs to evaluate, compare, and rate the overall performances among service providers.
The advantages of this new PIs system are as follows. The performance scores of KPIs and PIs enable it to assess the performance of a single wastewater utility over time, and also to make direct comparisons or ratings among wastewater service utilities based on the performance results, so that decision-makers or utility managers can define the shortcomings and flaws in utilities operations. The comprehensive numeric form applied for PIs can be easily understood by non-specialists, such as customers or user communities, thus it is easy to convey the present status on service performances. Also, the evaluation scores or weights from criteria or categories in the PIs system can be practically used for risk analyses; specifically, the incorporation of PIs results into advanced data analyses, such as machine learning techniques, would enable it to predict and prevent failures in sewer services as well as to reliably manage infrastructure assets.
Of course, the new PIs system has limitations insofar as it would not fit into the performance evaluation of the combined sewer systems because the framework of PIs was developed for separate sewer systems. Also, the prioritization of PIs would be changed if variations due to climatic or cultural factors should be put into prior consideration, and in such cases, the PIs framework should be reconstructed in accordance with individual situations.
In conclusion, the authors encourage the set of PIs of this study to be actively used by stakeholders involved in the activity of sewer services, such as undertaking companies, policy-making bodies, and financing agencies, so that an assessment database would enable the comparison of performance at nation-wide or international levels, and, eventually, sewer services would be proceed into sustainable ways of operation.