Comparative analysis of standardized indicators for Smart sustainable cities: What indicators and standards to use and when?

City managers need indicators for target setting, performance assessment, monitoring, management and decision-making purposes. The choice of the most suitable indicator framework is crucial, but difficult, as it requires expert knowledge. To help cities in their choice, this paper compares seven recently published indicator standards for Smart sustainable cities. A taxonomy was developed to evaluate each of their 413 indicators against five conceptual urban focuses (types of urban sustainability and smartness), ten sectoral application domains (energy, transport, ICT, economy, etc.) and five indicator types (input, process, output, outcome, impact). The results clearly discriminate between indicator standards suited for evaluating the implementation of predominantly smart city approaches versus standards more focused on sustainability assessment. A further distinction is possible in standards almost fully oriented towards impacts reached, and standards that allow for progress evaluation according to steps in the implementation process. Some standards provide a narrow focus on output indicators evaluating the progress in implementing smart urban ICT solutions (e.g. number of smart meters installed). Cities are encouraged to complement such evaluations with impact indicators that demonstrate the effects of those solutions. This paper provides guidance for city managers and policy makers to select the indicators and standard that best correspond to their assessment need and goals, and align with their stage in Smart sustainable city implementation.


Introduction
Rapid urbanization puts cities in central position to solve urgent global issues such as climate change while maintaining the service level for the extended population with limited resources. Quick digitalization and technological development fuel smart city solutions that try to help cities in optimizing their efficiency and quality in service provision with help of ICT, new technologies and participatory approaches. Smart city solutions have been, however, heavily criticized of being often too techno-centric, driven by technology companies' own agendas while lacking proper attention to cities' needs and environmental sustainability (Colding & Barthel, 2017;Hollands, 2015;Marsal-Llacuna & Segal, 2017;Mora, Bolici, & Deakin, 2017;Yigitcanlar et al., 2019). Therefore, a currently active stream of academic discussion analyses how smart city solutions can ensure progress towards balanced sustainability, which has led to the emergence of the new concept "Smart RQ1. How do the analyzed standards differ in their balance of indicators addressing different types of urban focus, i.e. sustainable vs. smart city goals?

RQ2. How do the compared standards differ in their balance of indicators relevant for different city sectors?
RQ3. For which evaluation purposes are the compared standards useful (e.g. short-term progress evaluation or long-term impact assessment) based on the types of indicators included?
The results are expected to be useful for city managers and policy makers as they provide easily understandable information and guidance that can be used in selection of most appropriate indicators depending on city specific needs, situation and goals. Therefore, Section 2 introduces key concepts and the indicator standards for Smart sustainable cities included in this study. Section 3 develops the taxonomy for comparing indicators according to the research questions, and presents the methods used for analyzing and comparing indicators. Section 4 presents and analyses the differences between the standards across the dimensions of the taxonomy. Its three sub-sections each provide answers to one of the research questions. Sections 5 and 6 discuss the meaning and significance of the results.

Smart sustainable cities: combining urban sustainability and smartness
Among the variety of urban concepts (De Jong, Joss, Schraven, Zhan, & Weijnen, 2015;Khan & Zaman, 2018) sustainability (Castells, 2000;WCED, 1987) has the longest history and widest acceptance, but has been surpassed in popularity by the concept of "smart cities" during the past decade (Hollands, 2008;Mora et al., 2017). Despite the history and popularity of these concepts in scientific literature, neither of them has a uniformly agreed definition on their meaning which creates confusion among scientists, policy makers, municipalities, citizens and businesses. Additionally, the concept of "smart cities" has been widely criticized of its techno-centricity, lack of attention to cities' needs and questionable contribution to sustainable development (Colding & Barthel, 2017;Hollands, 2015;Mora et al., 2017;. On the other hand, the concept of sustainability, as originally introduced in 1987, with its three pillars of social, environmental and economic sustainability (WCED, 1987), could be criticized of being partly outdated as the needs of the highly digitalized society have quickly changed.
The concept of "Smart sustainable cities" has been recently proposed as a response to the previous criticism on urban sustainability and smartness (Ahvenniemi et al., 2017;Bibri & Krogstie, 2017;Höjer & Wangel, 2015;ITU, 2016a). Essentially, this new emerging concept combines urban sustainability and smartness emphasizing that both aspects should be considered simultaneously. Its emergence can be seen both as a) a response to the critics of such smart city solutions that are contradictory to sustainability, and b) as an attempt to address the needs of the currently highly digitalized cities more comprehensively than the traditional concept of sustainability.
This new city concept is wide in scope, which raises the interest in research question 1, that attempts to reveal differences in city evaluation approaches by international standardization bodies under this wide conceptual umbrella, more specifically whether the focus is more on smart technologies or sustainability. Answers to this question are expected to be important for the potential users of the related indicator standards to better understand the applicability of the standards.

City indicators
Indicators are by definition quantitative, qualitative or descriptive measures (ISO, 2018a) that enable information on a complex phenomenon, such as the dynamic urban environment, to be simplified into a form that is relatively easy to use and understand. The three main functions of indicators are quantification, simplification and communication (ISO, 2010). When periodically evaluated and monitored, they show trends and change in the measured phenomenon (Haapio, 2012). City indicators thus assist cities in setting targets and monitoring their performance over time (ISO, 2018a).
Consequently, cities regularly use defined sets of indicators to quantify their targets and systematically monitor the progress towards their goals (Munier, 2011). Cities typically report annually on strategically important indicators to internally keep track and externally communicate on progress (Dameri, 2017). With the exploding amount of urban data, a carefully selected and relatively small number of easily understandable Key Performance Indicators is useful for city managers to get a snapshot of the city's performance in different areas. Recently, the use of indicators in decision-making has become increasingly popular, as an exponent of the trend to informed decision-making (Kourtit & Nijkamp, 2018). Cities use indicators as support when considering different decision alternatives. Another important trend in the use of indicators in city management aims to increase transparency towards citizens through city dashboards (Dameri, 2017). Opposed to indicators used in annual reporting that are mostly based on statistics, city dashboards use real-time data and focus on visualizing indicators on aspects useful for citizens (Kitchin et al., 2015).
In summary, indicators are applied in cities for a variety of purposes. While the selection of the most suitable indicator framework is difficult for city managers, as it requires expert knowledge typically lacking in municipalities, the consequences of misuse can be significant. Therefore, research questions 2 and 3 try to reveal differences in applicability of existing international city indicator standards, to be able to provide currently lacking guidance for potential users.

Indicator standards for Smart sustainable cities
A large variety of indicator frameworks and tools exist to assess either urban sustainability or smartness (Albino et al., 2015;Anthopoulos, Janssen, & Weerakkody, 2016;Berardi, 2015;Moonen & Clark, 2013;Science for Environment Policy, 2018;Sharifi & Murayama, 2013). However, standardized frameworks of city indicators have been introduced only recently and related scientific literature is surprisingly scarce. Related international standardization work is carried out by three bodies, i.e. by ISO and ITU worldwide and by the coalition of the European standardization organizations CEN, CENELEC and ETSI in Europe. Currently, there are six international city indicator standards relevant for Smart sustainable city evaluation and reporting. Those are introduced in Table 1 (see Appendix A. Supplementary data for details).
The ISO standards on sustainable cities were developed by the working group "City indicators" of the committee "Sustainable cities and communities". ISO 37120 focuses on performance of city services and quality of life. It was first published in 2014 and a revised version was released in July 2018 with addition of 28 new indicators, removal of 24 old ones and slight modification to 10 indicators. ISO 37122 provides indicators for smart cities and was first publicly released in the form of draft international standard in June 2018. A third indicator standard ISO 37123 on resilient cities is under development. These standards have been developed with sustainability as a guiding principle and therefore can be used in conjunction to provide a holistic approach to urban sustainability. The World Council on City Data (WCCD) is involved in development of ISO indicators and certifies cities based on the amount of ISO 37120 indicators calculated and published at www.dataforcities.org (McCarney, 2015;WCCD, 2018). By February 2018, 52 cities worldwide had been certified using indicators of the 2014 version of ISO 37120 standard. They will now, however, need to update the revised indicators if they want to conform to the standard. ISO 37120 makes a distinction into three types of indicators: a) core indicators that must be calculated by all cities internationally to apply the standard, b) supporting indicators that are recommended, and c) profile indicators that provide context for the assessment. The present study considered both core and supporting indicators in the comparative analysis later introduced.
In Europe, the standardization activities on Smart sustainable cities are coordinated by a joint effort between CEN, CENELEC and ETSI on the Sector Forum on Smart and Sustainable Cities and Communities (SF-SSCC), created in January 2017, and following the work of a similar coordination group (CEN et al., 2015;. Until now, one set of Smart sustainable city indicators has been published by ETSI in the form of a technical specification on Key Performance Indicators for "Sustainable Digital Multiservice Cities" (ETSI, 2017a), supported by related group specification (ETSI, 2017b). These indicators were originally defined by the European CITYkeys initiative, together with European cities, based on analysis of 20 cities' needs, 43 existing indicator frameworks and feasibility testing by around 50 cities and other stakeholders .
The ITU focus group on Smart sustainable cities developed the standardized definition of Smart sustainable cities and coordinated the development of related city indicators. These three standards have the status of recommendations. Each of them provides indicators for Smart sustainable cities but with slightly different focuses: ITU 4901 focuses on the use of ICT, ITU 4902 on its sustainability impacts and ITU 4903 on assessing the SDGs. The ITU and UNECE led initiative "United 4 Smart sustainable cities" has prepared a KPI manual based on ITU 4903 and with links to UN SDGs (ITU and UNECE, 2017). This initiative is currently carrying out case studies in cities to get feedback on the feasibility of the indicators and to develop a global Smart sustainable city index (ITU, 2018).
The UN SDG 11 'Make cities and human settlements inclusive, safe, resilient and sustainable', and associated targets and indicators, represent an internationally accepted framework for the evaluation of sustainability at a global level (UNGA, 2017;UN-Habitat et al., 2016). A fundamental question is, however, how to apply the adage "think globally, act locally" (Geddes, 1915) for effective implementation and monitoring of city-level actions (Klopp & Petretta, 2017;Wendling, Huovila, zu Castell-Rüdenhausen, Hukkalainen, & Airaksinen, 2018). While the SDGs clearly focus on sustainability, they are relevant for our study as smart city solutions are expected to play an important role to support cities in the achievement of these goals by helping stakeholders to monitor the state and manage progress towards the achievement of the SDGs through universally accepted indicators (Corbett & Mellouli, 2017;UN, 2017).

Research methodology
The methodological approach of this study was to first identify indicator standards relevant for Smart sustainable cities in Section 2.3 (see summary in Table 1), secondly, development of a uniform taxonomy to compare these standards against the research questions (introduced in Section 3.1), and thirdly, application of the taxonomy in comparison of the indicators and standards (as explained in Section 3.2).
Seven sets of city indicators published by the major international standardization bodies and relevant for evaluation of Smart sustainable cities were identified in Section 2.3 and all of their 413 indicators were included in the analysis (see summary in Table 1 and details in Appendix A. Supplementary data). To uniformly compare each of the 413 indicators against the research questions (see end of the Introduction section), Section 3.1 develops a taxonomy (see Fig. 1) consisting of types of conceptual urban focuses (RQ1), city sectors (RQ2) and indicator types (RQ3).

Fig. 1. Taxonomy for indicator analysis.
A. Huovila et al. Cities 89 (2019) 141-153 indicators existed earlier, and as such was developed in this study, the Section 3.1.3 on development of this indicator typology is more detailed specifying also the background and methodological grounds for indicator analysis and comparison. The actual research process and methods applied in analysis and comparison of indicators against the taxonomy presented in Fig. 1 is introduced in Section 3.2 Analysis and scoring.

Classification of urban focuses: types of urban sustainability and smartness
The standard definition of "Smart sustainable cities" (ITU, 2016a), presented in the first paragraph of the Introduction, is used in this study as reference. Essentially, this new concept combines urban sustainability and smartness, and emphasizes that both aspects should be considered simultaneously.
Smart sustainable cities is a wide concept and when analyzing the types of indicators included in the analyzed standards we are interested in the focus of urban development that the indicators address. Therefore, a classification of urban goals was developed to compare indicators. This type of analysis is important for cities as each city has its own strategic goals and context and it is therefore important to use indicators that align with those goals.
The ITU definition of Smart sustainable cities consists of two parts: the beginning defines the smart characteristics of a city and the second part describes urban sustainability. Consequently, the smart characteristics of a city relate to innovations -using ICT and technology or citizen engagement -with the aim of improving quality of life, efficiency of urban operations and services, and competitiveness. The sustainable characteristics of a city, on the other hand, are those that ensure that the city meets the needs of present and future generations with respect to economic, social, environmental as well as cultural aspects.
The triple bottom line of sustainability -consisting of People (social sustainability), Planet (environmental sustainability) and Prosperity (economic sustainability) -is generally accepted in the development of indicator systems for urban sustainability (Hák, Moldan, & Dahl, 2007) and is used in our study to compare the types of sustainability indicators included in the standards.
In order to further analyze the types of smart city indicators, our study uses the division of urban smartness into two types, i.e. hard and soft, which is a well-established categorization in the smart city literature (Ahvenniemi et al., 2017;Albino et al., 2015;Angelidou, 2014;Caragliu, del Bo, & Nijkamp, 2011;Kummitha & Crutzen, 2017;Neirotti, De Marco, Cagliano, Mangano, & Scorrano, 2014). Hard smartness relates to tangible assets, such as ICT, technology and hard (physical) infrastructure, and soft smartness to intangible assets and people. Examples of hard infrastructure include ICT, transport, water, waste and energy while the soft indicators relate to social, cultural and human capital, well-being, knowledge, policy, governance, participation, innovation, economy, inclusion and equity. The resulting taxonomy for analyzing the conceptual urban focus of indicators is presented in Fig. 1.

Classification of city sectors
A classification of 10 city sectors was developed in Ahvenniemi et al. (2017) based on Neirotti et al. (2014) to compare smart vs. sustainable city indicators. As this classification fully fits also the purposes of this study, the same classification of city sectors (see Fig. 1) was adopted.

Classification of indicator types
City indicators have been previously categorized based on how they are developed (Reed, Fraser, & Dougill, 2006) or assessed (Garau & Pavan, 2018). However, a standard classification related to what they measure is missing (Carli, Dotoli, Pellegrino, & Ranieri, 2013) probably because they integrate indicators from many fields such as economy, environment and built environment that are used to their own conventions. For example, "Key Performance Indicators", a term originally used to measure the success of an organization in business (Jones, 2006), are often used in urban indictor sets to measure the performance of a city (Bosch et al., 2017). Environmental sciences typically use variations of a "Pressure-State-Response" indicator typology relating causes and effects (Munier, 2011;Smeets & Weterings, 1999;Tanguay, Rajaonson, Lefebvre, & Lanoie, 2010). Building indicators can be classified into "prescriptive" and "performance oriented" distinguishing between indicators guiding, but also limiting technical solutions, and indicators on final performance that leave room for innovative solutions in design (Gibson, 1982;Klobut, Mäkeläinen, Huovila, Hyvärinen, & Shemeikka, 2016).
On the other hand, the input-output model has been the basis for methods widely used in economics (Leontief, 1986) and later in environmental science (Hendrickson, Horvath, Joshi, & Lave, 1998). Recently, the input-process-output (IPO) model logic with a "systems view" (Chadwick, 2013;Fincher, 1972) has been also used to characterize smart city transformation with attempts to link desired outcomes (Gil-Garcia et al., 2016;Kumar, Singh, Gupta, & Madaan, 2018;. Several UN bodies use the input-processoutput-outcome-impact typology to measure the performance of their international programs, strategies and projects (e.g. UNISDR, 2015). While impact assessment is the most crucial part in urban policy evaluation, innovation policies are also measured with input, output and outcome indicators (Janger, Schubert, Andries, & Rammer, 2017).
The input-process-output structure is a good basis to classify Smart sustainable city indicators. It, however, neglects the crucial measures of true benefits and impacts of urban solutions and policies. As Smart sustainable city indicators often evaluate the extent to which certain innovations (ICT or other) have been deployed or the impacts they generate, the categories of outcomes and impacts are added to our indicator typology. The resulting typology thus consists of input, process, output, outcome and impact indicators (see Table 2 and list below for definitions).
This typology captures well different phases of innovations and has already been proposed for smart city indicator classification (Bosch et al., 2017). Another benefit of this typology is that it directly corresponds to the key stages in cities' transformation towards Smart sustainable cities as recently proposed by Ibrahim, El-Zaart, and Adams (2018). Although impact indicators are most relevant for the final What was achieved by the interventions? Long-term evaluation Reporting on real impacts or overall performance assessment of success, inclusion of combinations of indicators addressing inputs, process, outputs and outcomes helps to capture progress at different time scales and also better specifies cities' local perspectives (Lützkendorf & Balouktsi, 2017;Turcu, 2013). As impacts of interventions can often be observed only years later, additional use of output and outcome indicators is useful for cities to report on progress in the short term. Combined use of input and impact indicators helps to answer key questions such as: what benefits and value can a city achieve with its investments? And process indicators can help in diagnosis of why certain objectives were not reached. The categories of indicators in this typology are defined as follows (Bosch et al., 2017): • Input indicators refer to the resources needed for the implementation of interventions, measuring the quantity, quality, and timeliness of resources. Policies, human resources, materials, financial resources are examples of input indicators.
• Process indicators measure whether planned activities took place.
Examples include holding of meetings, conducting training courses, distribution of smart meters.
• Output indicators add more details in relation to the product ("output") of the activity, e.g. the number of smart meters distributed, the area of roof that has been isolated or the number of electric busses in the system.
• Outcome indicators measure intermediate results generated by outputs. Outcome indicators refer more specifically to the objectives of an intervention relating to the quantity and quality of the activities implemented. Often they are coverage indicators measuring the extent to which the target population has been reached, e.g. percentage of car owners using a parking app.
• Impact indicators measure the state with regard to a set city target (impact of policy), e.g. city's energy consumption, and can be used to evaluate for example the sustainability impacts of smart solutions.
When classifying indicators it is often difficult to make the distinction between output and outcome indicators based on the name alone. An indicator defined as X per 1000 inhabitants is often intended as an output indicator for evaluating the progress in delivering the product or service X, whereby the expression "per 1000 inhabitants" has been added for comparability reasons. However, if the focus was on reaching a target of maximizing the use of X, then the same indicator could be classified as an outcome indicator. For a correct interpretation, often, more information is needed. The premise of our study is that the proposed classification of indicators can be useful for cities as it can help them to understand for which types of evaluation purposes the indicators included in a standard are useful. This type of indicator typology is not yet in use for city indicators and the standard documents do not provide information to classify indicators by use purpose. The developed indicator typology is included in the taxonomy for analysis in Fig. 1.

Analysis and scoring
In order to analyze indicators with regard to relevant city sector, indicator type and urban focus, the taxonomy presented in Fig. 1 was developed in Section 3.1. Careful analysis of each of the 413 indicators, introduced in Table 1, was required for reliable comparison of the standards against research questions as the standard titles use nonuniform terminology and only refer to city concepts that can be misleading while indicators concretely define the measurable characteristics of a Smart sustainable city (Höjer & Wangel, 2015). Development of the taxonomy was necessary to answer the research questions as this type of information was mostly missing from the standards. While some of the standards refer to city sectors with varying logic, categorization into indicator types and smart or sustainable indicators was completely missing and had to be carefully analyzed for each indicator to reveal the differences between compared standards.
The indicator analysis method adopted follows the principles of Ahvenniemi et al. (2017) to analyze and score urban indicators with regard to different categories of the taxonomy presented in Fig. 1. It is a mixed-method approach (Creswell, 2003) combining qualitative content analysis of altogether 413 indicators and related technical documentation (step 1), and quantitative scoring of each indicator across the dimensions of the developed taxonomy. The analysis process is illustrated in Table 3. The scoring method is based on a distribution of points per indicator for each aspect considered. In step 2, first, two points were given for each indicator based on the city sectors of application. The consideration of two points was necessary for indicators such as "traffic accidents" for which one point was given to the sector "Transport" and another to "Safety". Second, three points were distributed based on whether the indicator predominantly assessed urban smartness or sustainability. Here again, distribution of points was often needed as many Smart sustainable city indicators measure the extent to which smart measures, aiming at lowering environmental burden, are implemented, thus integrating both concepts. As an example, "Application of city water monitoring through ICT" was scored with two points for smartness and one point for sustainability. Those three points were further distributed under the sub-categories of smartness and sustainability, i.e. hard or soft smartness and people, planet or prosperity of sustainability. Third, one point was given to the indicator type category. Finally, in step 3, the percentage of points distributed under different categories and sub-categories of the taxonomy were calculated for each of the analyzed standards. Additionally, in step 4, the coverage of different types of indicators (input, process, output, outcome, impact) in other categories of the taxonomy (city sector and types of urban sustainability and smartness) were analyzed by calculating the scalar products of the vectors consisting of points given for indicators in the compared categories of the taxonomy. Details of scoring and calculations are available in Appendix A. Supplementary data.

Results
The results are divided into three sub-sections that each provides answers to one of the research questions.

Balance between sustainability and smartness
Relative scores on the balance of indicators included in the compared standards with regard to urban sustainability and smartness are presented in percentages in Fig. 2. The results confirm the initial goal of the ISO 37120 and UN SDG 11+ indicator standards of measuring Table 3 Analysis process.

Qualitative analysis of indicator standards including technical documentation, guidelines and metadata of indicators
Scoring of individual indicators against categories and subcategories of the taxonomy presented in Fig. 1 Quantitative comparison of standards based on distribution of indicator scores in different categories of the taxonomy

Quantitative analysis of all indicators comparing the distribution of indicator types in other categories of the taxonomy based on scalar products of indicator scores in compared categories of the taxonomy
A. Huovila et al. Cities 89 (2019) 141-153 urban sustainability, as almost 90% of the indicators focus on sustainability. ITU 4901, on the other hand, predominantly (with a score of 78%) focuses on smartness. The remaining four standards are more balanced with regard to both concepts with ISO 37122 having a 64% focus on smartness and ETSI and the two remaining ITU indicator standards being by 63-73% more oriented towards sustainability. Thirty years after the Brundtland report (WCED, 1987) one would expect that the standards pay more or less equal attention to each of the People, Planet and Prosperity categories. However, all of the analyzed standards address prosperity with least amount of indicators, the maximum being 20% for ETSI and minimum 7% for ISO 37122 (see Fig. 3). Four of the standards (ISO 37120, ITU 4901 & 4903 and UN SDG 11+ indicators) predominantly focus on People category with a share between 52% and 66%. The results of the two ISO standards are surprising: while the "sustainability" framework (37120) has mostly social indicators (52%), the "smart city" framework (37122) clearly focuses on environmental sustainability (62%). ETSI and ITU 4902 provide the most balanced approach with both having slight dominance of indicators in Planet category, which can be explained by the variety of environmental issues and hence a variety of parameters dealt with.
With regard to the types of smartness, UN SDG 11+ stands out as all of its few smart indicators deal with soft aspects (Fig. 4). Among the larger amount of smart ETSI indicators 71% relate to soft smartness while the share is 76% for ISO 37120. On the other hand, ITU 4901 and ISO 37122 predominantly (73%) focus on hard smartness, which is not very surprising as their indicators are focused on the use of ICT. The remaining two ITU standards (4902 & 4903) are balanced between hard and soft smartness.

Sectoral distribution
The sectoral analysis (see Table 4) shows a glaring difference in focus on the ICT sector: ITU 4901 stands out with 54% of the points, followed by ISO 37122 (32%). Another group focuses more on Health, well-being and safety, Water and waste or Economy, including the UN SDG 11+, ISO 37120, ITU 4902 and ITU 4903 standards. ETSI provides a good balance with all sectors being covered by 4-16%.
The most underrepresented sector in the analyzed standards is Energy (average 5%). This is an interesting finding as it has together with Transport and ICT formed the sectoral cornerstones of the European smart city policy (European Commission, 2012). These results corroborate the findings of Ahvenniemi et al. (2017) who found that Energy was underrepresented in both smart city and sustainable city indicator frameworks. A possible reason for small number of Energy indicators could be that they are rather simple to aggregate in Joules and kWh whereas some of the social aspects, such a health, social inclusion and governance, are much more complex issues therefore needing a larger number of indicators. Additionally, Energy is still a A. Huovila et al. Cities 89 (2019) 141-153 highly centralized sector with few actors whereas the other sectors are more diverse.

Types of indicators
Altogether, the indicator sets have a relatively small amount of input and process indicators but there are significant differences in focus with regard to measuring outputs, outcomes or impacts. Ideally, one would expect that the indicator standards have a bias towards impact indicators, as it is the impact in society that counts in the end. As the indicators are directly related to the Sustainable Development Goals, the UN SDG 11+ indicator set has indeed 50% of impact indicators (see Fig. 5). This set has also the largest number of process indicators as they provide an overview of the local government's adoption of various strategies. The same pattern is also reflected in the ISO 37120 and ITU 4902 standards, which, however, instead of process indicators include more output indicators.
Not surprisingly, the ITU 4901 standard stands out with its focus on output indicators (58%). This standard monitors the application of specific ICT applications in various domains to great detail (availability  A. Huovila et al. Cities 89 (2019) 141-153 of smart meters, traffic monitoring, parking guidance, smart street lights etc.) but completely lacks impact indicators. ISO 37122 resembles ITU 4901 in the sense of attention for smart appliances, but in this standard these are phrased as outcome indicators (asking for percentages of the population covered). Finally, ETSI and ITU 4903 are more balanced between output, outcome and impact indicators. A further analysis clearly displays the distinctive and specific natures of the three ITU standards. Where 4901 consists of mainly output indicators for sectors in which ICT is applied, 4902 reports on impacts and outcomes (see Appendix A. Supplementary data). The natural environment, lacking from the 4901 set, is present in 4902 with indicators on air pollution, soil quality and green areas. For the education sector, we see the move from inputs and process indicators in 4901 to outputs in 4902. Consequently, 4901 is most suitable for short-term evaluation of efficiency in deployment of smart urban technologies. However, either 4902 or 4903 is also needed to evaluate the impacts achieved through the technologies where the former has a focus on the sustainability impacts of ICT and the latter in the progress of achieving the Sustainable Development Goals.
Similar analogy is noticed in the complementarity of the two ISO standards: the new 37122 set focuses on smart enabling technologies and policies, and should be used in conjunction with the 37120 indicators on sustainability assessment.
Considering the 413 analyzed indicators altogether, we notice a glaring difference in impact indicators: they are most typical for sustainability assessment (45%) but only rarely used in evaluation of smartness (6%) that typically uses output indicators (45%) (see Fig. 6). Input and process indicators are most common among indicators of soft smartness.
With regard to the distribution of indicator types across sectors, ICT stands out once again with almost no impact indicators and mostly output and outcome indicators (see Fig. 7). This means that ICT related indicators mainly measure the efficiency in implementation of smart urban solutions as well as the coverage they reach. In other words, ICT is an enabler and its impact can be seen, and is measured, in other sectors. Most of the Water and waste indicators are outcome indicators as they typically indicate the share of population covered by related services. Input indicators are relatively most common among economic indicators as they measure the resources needed for implementing urban solutions.

What indicators and standards to use and when?
The comparative analysis of seven standardized urban indicator sets resulted in clear differences regarding the distribution of indicators focusing on urban sustainability vs. smartness, different city sectors and types of indicators. These results are useful for comparing the applicability of the standards for different types of city evaluations. They, thus, provide guidance for city managers and policy makers to choose indicator sets that are most suitable for their needs. Each of the subsections in the Results section provides answers to one of the research questions and thus helps cities to choose between indicator standards according to the three dimensions of the taxonomy of Fig. 1 (urban  focus, city sector, indicator type).
The main theoretical novelty of this study was the introduction of the input-process-output-outcome-impact typology for Smart sustainable city indicators (see Section 3.1.3 on Indicator types). This classification is useful for cities as it directly corresponds to different stages in Smart sustainable city development and implementation (Ibrahim et al., 2018). Cities can thus select indicators according to their assessment need and stage, using the assigned type for each indicator (see Appendix A. Supplementary data for details) and following the categorizations on when and for which type of assessment to use an indicator type (see Table 2 in Section 3.1.3).
The definition of Smart sustainable cities combines traditional urban sustainability with the needs of modern cities (ICT and innovative participatory methods). Ideally, an integrated approach balancing across multiple criteria is beneficial for comprehensive city evaluation and wide usability and acceptance of an indicator standard. ETSI 103463 and ITU 4903 best provide such balance with the criteria considered in this study.
Based on the distribution of indicators, cities that focus predominantly on becoming a smart (sustainable) city should consider following ITU 4901 or ISO 37122 indicators. ITU 4901 is most suitable for a city that is in an early stage of development or interested in the stage of implementation of various smart technologies. ISO 37122 is slightly wider in scope including also some environmental and social impact indicators. ITU 4901 has a narrow scope measuring outputs of smart urban ICT solutions (e.g. availability of smart water meters) with complete lack of impact indicators. Such evaluation is not sufficient alone as it only gives information on the efficiency of implementing smart urban solutions but nothing on their positive (or negative) effect, thus possibly leading to wrong incentives (Niemann, Hoppe, & Coenen, 2017). However, combined use of such indicators with sustainability impact indicators (e.g. ITU 4902) can be useful. It should be noted that with the growing importance of the Sustainable Development Goals a narrow-minded focus on smartness will soon be less relevant.
At the other end of the spectrum, we find the standards characterized by a majority of sustainability indicators. Depending on the goal orientation of the city there is a choice: if there is a need to report real A. Huovila et al. Cities 89 (2019) 141-153 impacts, then suitable standards are UN SDG 11+ and ITU 4902. If it is considered important to report also on the immediate implementation of policy measures, then ISO 37120, ETSI 103463 or ITU 4903 may be more suited.
On a high level, the sustainability-smartness balance thus appears as a useful criterion to make a first choice among available standards. Next, the balance between input, output and impact indicators is a useful criterion.
It should be noted that the exact quantitative results may suffer from some subjectivity that cannot be completely avoided in the application of the scoring method. While the applied method has been consistent for all the analyzed indicators, benchmarking indicators against sustainability and smartness and their sub-categories is not always straightforward. However, the large number of indicators in each standard diminishes the weight of an individual indicator as the findings are calculated based on aggregated results. As the conclusions only report on significant differences in shares of large number of indicators, there should be no concern on their validity.

Concluding remarks
The present study is first of its kind as these city indicator standards have not been comparatively analyzed previously in scientific literature. The approach developed for comparison is also new and could help other researchers in their related works. Finally, a third novelty of the analysis consists of the inclusion of two internationally significant recently published ISO standards on sustainable and smart cities (ISO, 2018a(ISO, , 2018b, as no previous studies on these standards yet exist and most of the potential users are not yet familiar with the new indicators. Therefore, the usefulness of related results regarding the potential applicability for different evaluation needs is expected to be high.
The purpose of this study was to provide guidance for city managers and policy makers in the selection of the most suitable indicator standard. However, those applying these indicators have an important responsibility of correct and useful figures and thus should be aware of the following considerations that are not clearly presented in the standard documents.
The selection of most appropriate indicators depends on many factors: phase in city development (planning, operation), spatial scale (district, city, region, country), time scale of evaluation (real-time to annual) and purpose of assessment (target setting, monitoring, official reporting, self or cross-city benchmarking, marketing) (Deakin, Huovila, Rao, Sunikka, & Vreeker, 2002;Huovila, Tuominen, & Airaksinen, 2017). While standardization of indicators in principle ensures certain level of quality for indicator selection and calculation methods, standards are always a compromise for a large group of different cities with different agendas, contexts and needs. Selected indicators can be meaningless for a specific context due to the differences between cities (Borsekova, Koróny, Vaňová, & Vitálišová, 2018;Kitchin et al., 2015). This means that any standard indicator set should be considered as a starting point. Indicators and measurements should not become a goal in themselves but support the fulfilment of individual cities' needs (Kaika, 2017;Verma & Raghubanshi, 2018). Individual cities should thus always select and adapt indicators corresponding to their needs (Moreno Pires, Fidélis, & Ramos, 2014; Reed et al., 2006). Similarly, cross-city comparisons and benchmarks need to carefully select indicators linked and limited to the scope and purpose of the comparison and use transparent communication of results.
The current article contributes to the development of methods for screening assessment instruments on conformity with the needs and/or ambitions of stakeholders. Such methods and the screening prevent that technology in the wide sense (applied technologies in the cities, or data gathering technologies) becomes more important than the human needs behind the developments that are to be assessed. With the further proliferation of indicator frameworks and sets of KPIs for aspects of Smart sustainable cities, it will be necessary to further develop methods for evaluating their qualities. This will not only need to include the selection of indicators and the process of selecting indicators, but also the indicator definition, documentation and the proposed data collection procedures.
When using indicators, cities have the important responsibility to ensure the quality of the data underlying indicators, as it has a direct link to the quality of decisions made with those indicators (Kitchin et al., 2015;McArdle & Kitchin, 2016). Careless use of non-transparent or non-reliable data, indicators or indices presents a risk for city management as successfully documented by Kitchin et al. (2015). For example, the choice of statistical city boundaries can have dramatic effects on indicator results (Kitchin et al., 2015). The choice of indicator and data boundaries is crucial for example in the case of GHG emissions of a city having its airport located outside the municipality's boundaries. City indicator results always need to be analyzed and interpreted within their context as cities exploit resources and produce external impacts beyond their boundaries (McDonnell & MacGregor-Fors, 2016;Mori & Christodoulou, 2012;Ramaswami, Russell, Culligan, Rahul Sharma, & Kumar, 2016).
Both ISO and ETSI indicator documents provide detailed definitions, unique assessment methods, guidance on data sources/feasibility, and typical data availability is well-documented WCCD, 2018 A. Huovila et al. Cities 89 (2019) 141-153 while acknowledging that most of the indicators are not feasible yet (UN, 2018;UN-Habitat et al., 2016). All ITU standards, however, only include a short definition of the indicator. Therefore, they seem less easily applicable or might lead to different interpretations of indicators and their calculation methods, which has the risk to lead to different results (Huovila, Tuominen, & Airaksinen, 2017;Lützkendorf & Balouktsi, 2017). Without provision of sufficient metadata and guidance, they might have limited implementation in practice and lack reliable cross-city comparability. Among the analyzed ITU standards the adoption of 4903 indicators seems most probable as the interagency initiative "United 4 Smart sustainable cities" has produced an easily accessible indicator collection manual containing references to respective SDG sub-objectives and is preparing a global Smart sustainable city index based on cities' feedback (ITU, 2018;ITU and UNECE, 2017). Smart sustainable cities is a concept that seems to have established itself and related scientific literature is quickly increasing. It is an attractive policy concept for cities and the globally agreed urban Sustainable Development Goal (SDG 11) supports its popularity. The importance of the wider concept has made smart cities that neglect sustainability less relevant. On the other hand, it seems widely agreed that innovative ICT solutions are needed to achieve the SDGs and to collect the currently missing data (Corbett & Mellouli, 2017). UN bodies and statistical agencies are currently struggling to obtain the mostly lacking data needed to calculate the progress towards the SDGs. In consequence, they are actively working on using e.g. big data in statistics (UN, 2017). Another sign that the SDGs are being seriously addressed in Smart sustainable cities is that the indicators of three analyzed standards (ETSI 103463, ISO 37120, ITU 4903) have been already mapped against related SDGs (ITU et al., 2017;WCCD, 2017;Wendling et al., 2018). Alignment of existing frameworks with the proposed UN SDG 11+ indicators is beneficial as it helps in harmonization of indicators and methods, and thus minimizes duplicate data collection, calculation and reporting in cities (Wendling et al., 2018).
Future studies will need to provide empirical evidence on the feasibility of indicators in practice and of their use in actual decisionmaking. Another interesting research avenue consists of developing methods to analyze the systemic relations between indicators and indicator types. What are the relations between resources invested (i.e. inputs), processes used, technological outputs, and actual outcomes and impacts in a Smart sustainable city project? And what are the relations and systemic interdependences between social, economic and environmental indicators, e.g. actual effects of new technological innovations on people or environmental life cycle impacts of related sensors and data centers? The future development of indicators and evaluation methods could incorporate more dynamic and systemic approaches to better capture the complexities of the quickly changing society, as suggested by Nieminen and Hyytinen (2015). However, the developers of these methods need to keep in mind that simplicity and applicability should be guiding principles when approaching city managers. City managers are interested in the big picture and have limited time and monetary resources. They often only want to get a snapshot on how the city is performing in different areas and what are the benefits of investments, without interest or expertise in technical or methodological details. Therefore, in all these studies, the further development of indicator frameworks needs to be embedded in the analysis of cities' needs. Other interesting areas include the possibilities of artificial intelligence in city management while tackling the challenges of data quality and ethical issues.