http://orcid.org/0000-0002-5800-6654
Figures
Abstract
The digital divide in Europe has not yet been bridged and thus more contributions towards understanding the factors affecting the different dimensions involved are required. This research offers some insights into the topic by analyzing the e-Government adoption or practical use of e-Government across Europe (26 EU countries). Based on the data provided by the statistical office of the European Union (Eurostat), we defined two indexes, the E-Government Use Index (EGUI) and an extreme version of it taking into account only null or complete use (EGUI+), and characterized the use/non use of e-Government tools using supervised learning procedures in a selection of countries with different e-Government adoption levels. These procedures achieved an average accuracy of 73% and determined the main factors related to the practical use of e-Government in each of the countries, e.g. the frequency of buying goods over the Internet or the education level. In addition, we compared the proposed indexes to other indexes measuring the level of e-readiness of a country such as the E-Government Development Index (EGDI) its Online Service Index (OSI) component, the Networked Readiness Index (NRI) and its Government usage component (GU). The ranking comparison found that EGUI+ is correlated with the four indexes mentioned at 0.05 significance level, as the majority of countries were ranked in similar positions. The outcomes contribute to gaining understanding about the factors influencing the use of e-Government in Europe and the different adoption levels.
Citation: Yera A, Arbelaitz O, Jauregui O, Muguerza J (2020) Characterization of e-Government adoption in Europe. PLoS ONE 15(4): e0231585. https://doi.org/10.1371/journal.pone.0231585
Editor: Renuka Sane, National Institute of Public Finance and Policy, INDIA
Received: October 15, 2019; Accepted: March 27, 2020; Published: April 17, 2020
Copyright: © 2020 Yera et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data are third party data belonging to EUROSTAT. The access to microdata is restricted because the data refer to individual persons that can theoretically be identified. These are personal data. The legal restrictions in accessing CSIS microdata are referred to in Regulation 557/2013 which specifies conditions that must be fulfilled to be granted access. You can refer to Eurostat, Unit B1 “Methodology, Innovation in official statistics” as the entity providing access to microdata, contact: estat-microdata-access@ec.europa.eu (Aleksandra Bujnowska). Any replication of the results achieved by you in the course of the project will have to be done by the accredited research organisation on the microdata received following the standard access procedure: (https://ec.europa.eu/eurostat/web/microdata/overview).
Funding: The work has been funded by: (1) - (A. Yera) University of the Basque Country UPV/EHU (www.ehu.eus) (PIF15/143 grant), (2) (A. Yera, O. Arbelaitz, J. Muguerza) Department of Education, Universities and Research of the Basque Government (https://www.euskadi.eus/gobierno-vasco/universidades-investigacion/), (ADIAN group, grant IT980-16) (3) (O. Arbelaitz, J. Muguerza) Ministry of Economy and Competitiveness of the Spanish Government (http://www.mineco.gob.es/portal/site/mineco/?lang_choosen=en), co-founded by the ERDF (European Regional Development Fund) (Physcomp, TIN2017-85409-P) The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Despite the efforts made by European governments and administrations in recent decades, the digital divide in the old continent still persists. The Digital Economy and Society Index (DESI) [1], a composite index which has been published annually since 2014 by the European Commission to measure the progress of the 28 European Union (EU) countries towards a digital economy and society, provides an idea of the digital divide in Europe. In particular, DESI regroups 34 indicators in five principal policy areas weighted as follows: 25% connectivity, 25% human capital, 15% use of Internet services, 20% integration of digital technology and 15% digital public services. From 2014 to 2019 the highest digital divide between the 28 EU countries (max-min) was increased 2%, from 32% to 34%. Analyzing the DESI of 2019, we observe that only one country achieved high DESI values (≥70%), whereas 61% of the countries (17) scored medium values (∈[50%, 70%)) and 36% of the countries (10) achieved low values (<50%). In contrast, in 2014 the great majority of the European countries (24/28) scored low DESI values and just 14% of them (4/28) showed medium values. Although the average value of DESI in Europe was improved from 39% in 2014 to 52% in 2019, it is clear that more forceful actions are called for bridging the digital divide.
One of the aspects affected by the consequences of the digital divide and also analyzed within DESI [1] is the e-Government use or adoption. E-Government has several aspects, including social, technical, economic, political, and public administrative but most works define the mission of e-Government as systems that use Information and Communication Technology (ICT) to provide citizens with a better service [2][3]. E-Government has been defined as the use of digital technology, specially Web-based applications, to enhance access to and efficiently deliver government information and services [4]. Although it has featured a substantial growth, development and diffusion, citizens in all developed and developing countries may not be willing to adopt such services [5].
According to Shareef et al. [2] e-Government has several aspects, including social, technical, economic, political, and public administrative. Most dominating concepts of e-Government arise from the technical perspective and a combination of the socio-economic and public administrative perspectives but in the academic literature, the adoption models offered so far are mainly conceptual. For instance, inside DESI e-Government is represented by the E-Government Development Index (EGDI) a composite index which has been published biannually in the UN e-Government Survey since 2010 [6] and which considers three aspects: telecommunications infrastructure, human capital and online services. In 2018 the average value of the EGDI for the 28 EU countries (80%) was rated as Very-High (>75%) and in line with DESI, the average EGDI score for the period 2010-2018 improved by 14% [6–10]. Although some digital divide related themes are still mentioned in the 2018 UN e-Government Survey [10], e.g. access, affordability, age, bandwidth, content, disability, education, gender, migration, location, mobile, speed and useful usage, it seems that the e-Government situation in Europe is promising, at least from a theoretical point of view.
Another conceptual analysis is provided by the World Economic Forum’s Networked Readiness Index (NRI), also referred to as Technology Readiness, which measures the propensity of countries to exploit the opportunities offered by Information and Communications Technology (ICT). It is published in collaboration with INSEAD (a graduate business school with campuses in Europe, Asia, and the Middle East), as part of their annual Global Information Technology Report (GITR) [11]. The report is regarded as the most authoritative and comprehensive assessment of how ICT impacts the competitiveness and well-being of nations. The index is a composite of three components: the environment for ICT offered by a given country or community (market, political, regulatory, and infrastructure environment), the readiness of the country’s key stakeholders (individuals, businesses, and governments) to use ICT and the usage of ICT among these stakeholders.
Beyond the conceptual models, some limited empirical studies exist. For instance, the study carried out by Schwester [12] in US municipalities, concludes that e-Government adoption is a function of financial, technical, and human resources. Holding all other factors constant, municipalities with higher operating budgets, more full-time IT staff, and technical resources are more likely to implement a comprehensive e-Government platform. However, extensive empirical studies among the actual users to validate and generalize the models are absent [2].
In this context, we consider the Eurostat Community Statistics on Information Society [13] (Eurostat CSIS Surveys) an opportunity to carry out an extensive empirical study in European countries. They stated that in 2019 the average e-Government use (EGU) in the 28 EU countries reached just 55%. The EGU is computed as the percentage of individuals who used the Internet to interact with public authorities, for example by obtaining information from public websites and downloading or submitting official forms.
The aim of this research is to offer some insights into the empirical e-Government adoption across Europe (26 EU countries). Inspired on the e-Government adoption options described by different authors [14][15][16] and based on the Eurostat CSIS Surveys’ [13] e-Government use (EGU) question, which asks about the contact of respondents with public authorities or public services, we defined two indexes: E-Government Use Index (EGUI) which is computed based on the four answers available for the e-Government use question, and EGUI+ an extreme version of the previous index computed based only on the two answers showing extreme levels of e-Government use (complete and null). With regard to EGUI+ we defined four levels of e-Government use (very high, high, low and very low) and characterized the extreme levels of e-Government use (null and complete) by applying supervised learning procedures to the Eurostat data of two countries from each of the four EGUI+ levels. To carry out this step, we built a total of eight datasets containing 14 independent variables selected from the Eurostat CSIS Surveys [13], and we built classification trees, classifiers with explaining capacities, based on the Consolidated Tree Construction [17] algorithm. These procedures achieved an average accuracy of 73% and determined the main factors related to the practical use of e-Government in each of the countries, e.g. the frequency of buying goods over the Internet or the education level.
In addition, we compared one of the indexes calculated using empirical data given by Eurostat (EGUI+) to the following composite indexes measuring the level of e-readiness of a country: E-Government Development Index (EGDI) and its Online Service Index (OSI) component provided by United Nations [6–9] and the Networked Readiness Index (NRI) and its Government usage component (GU) published by The World Economic Forum [11, 18–22]. The Significance Tests for Kendall’s Tau () [23] applied to the rankings provided by these four indexes indicated that all of them were correlated to the one given by the empirically calculated index EGUI+.
The paper proceeds with the description of the Eurostat data on e-Government practical use (EGU) and the indexes we defined to quantify this concept (EGUI, EGUI+) in Section Eurostat CSIS Surveys. In the next section, Characterization of extreme values of e-Government use (EGU), we show the results of the supervised learning procedures applied to EGU for a selection of countries. Section Comparison between EGUI+ and other indexes+ describes four indexes that measure the e-readiness of countries and features related to e-Government, EGDI and OSI [9]/ NRI and GU [11], and compares them with EGUI+ in terms of ranking differences on average for the 2009-2015 (annual indexes) / 2010-2016 (biannual indexes) periods in 26 EU countries. In Section Discussion theoretical and empirical studies on e-Government are argued. Finally, in Section Conclusions we present the main findings of the study.
Eurostat CSIS Surveys
In this section we first describe the data on e-Government extracted from the Eurostat’s Community Statistics on Information Society (CSIS) 2009-2015 (Eurostat CSIS Surveys). Then we show the two indexes defined to characterize the e-Government adoption, EGUI and EGUI+, based on the information extracted from the Eurostat CSIS Surveys.
Description of the Eurostat CSIS Surveys
From 2002 to the present, Eurostat CSIS Surveys have been annually conducted in all Member States, two countries of the European Free Trade Association (EFTA), candidate and accession countries to the EU. The data collection is based on Regulation (EC) 808/2004 [24] of the European Parliament and the Council of the European Union and since 2011 the transmission of microdata to Eurostat is mandatory.
The Eurostat CSIS Surveys collect data on access and use of information and communication technologies (ICT) from households and individuals. Their sampling frames are established by countries and respect the quality requirements for European statistics and are in line with legal requirements. The survey covers households with at least one member aged between 16 and 74 and individuals of an age between 16 and 74. Information on access to ICT, e.g. connection to the Internet, is collected at household level while statistics on the use of ICT, mainly on the use of the Internet, is gathered for individuals. According to the Eurostat’s Methodological Manual for the statistics on the Information Society (survey year 2015) [25] in general, one individual in the household will answer the household related questions having the household perspective in mind. This one individual can for instance be the head of the household or the individual which has been selected for the individual questions. The survey distinguishes between annual core subjects, which are included in the survey every year, and episodic topics on various ICT phenomena, which change in different survey years. In particular, there are six annual core subjects: access to ICT, use of computers, use of the Internet, e-Government, e-Commerce and e-Skills. In order to analyze variables of access and use of ICT in relation to household or individual characteristics, a number of background variables are collected. These include composition, income and regional location of the household as well as the age, gender, educational attainment and employment situation of individuals.
We were given access to the annual micro-datasets for the period 2009-2015, which we used for the analysis on the practical use of e-Government. Some questions in the micro-datasets varied from year to year, thus, for the analysis we used just the seven questions (qi) common to all the years and the eight background variables (b.vi), which are shown in Table 1 below.
Among the seven questions selected, one is at a household level and related to Internet access (IACC) whereas the remaining six are at individual level, two related to computer use (CU / CFU) and four related to the use of Internet (IU, IFU, IBUY and EGU). In the last row of Table 1 we show in bold the question selected as dependent variable (d.vi) to measure e-Government practical use, EGU, which was obtained by coding a question about the activities related to interaction with public services or administrations through the Internet for private purposes, providing four possible values: 1 if none of the three possible activities was carried out, 2 if the obtaining information activity (OI) was carried out, 3 if the OI and the downloading official forms (DF) activities were completed and 4 if OI, DF and sending filled in forms activities were carried out. The last column of Table 1 (Mode) provides the overall mode of the variables used. A country by country analysis revealed that the overall modes of nine of these variables shown in the table remained stable in all the countries: number of children (HH_CHILD), employment situation (EMPST), ICT occupation (OCC_ICT), manual occupation (OCC_MAN), Internet access (IACC), computer use (CU), computer frequency of use (CFU), Internet use (IU) and Internet frequency of use (IFU). Oppositely, six of these variables showed fluctuations among the countries analyzed in terms of mode values (see S1 Appendix): income quartile (HH_IQ), age range (AGECLS), gender (SEX), education level (ISCED), buy goods over the Internet (IBUY) and e-Government use (EGU). In these cases, although the mode was not the same for all the countries analyzed, we also found common values for a majority of them: e.g., in the case of highest income quartile (HH_IQ = 4) for 77% of the countries (20/26) and upper secondary education level (ISCED = 2) for 77% of the countries (20/26). So we could conclude from this analysis that the average social background characteristics of the respondents are similar for the 26 countries included in the analysis. In addition, the penultimate column in Table 1, No. resp (%), provides for each of the questions or variables, the percentage of respondents for each possible value/answer available, giving thus, an idea of the distribution of the sample.
Analyzing the micro-datasets, we realized that six countries were missing data for the year 2008 and thus, we focused our analysis on the period 2009-2015. United Kingdom and Croatia were removed from our analysis because they were missing data for two of the years (2009 and 2010) of the period of time of our scope. Therefore our analysis comprises a total of 767,691 surveys from 26 different EU countries. Table 2 illustrates the ample variability in the number of surveys for the countries selected over the years analyzed.
As shown in the table, in the years analyzed on average, the number of CSIS Surveys is not proportional to the population of the countries (see columns 11 and 12 respectively): e.g. the biggest number of surveys which corresponds to Italy (19,100) is 25 times higher than the smallest number which corresponds to Malta (761), whereas the population of this second country (426,943) is 140 times lower than that of the first one (59,982,585). However, although the percentage of number of CSIS Surveys for the 26 countries regarding the population changes (see last column of Table 2), in every case, the number of responses used ensures a confidence level above 95% and an error margin below 5%. This would always be a pessimistic estimation since the target population of CSIS Surveys is in the age range 16-74 and thus, it is always smaller than the population shown in Table 2. In general, in countries with small populations, e.g., Malta and Luxembourg, this percentage was higher than in countries with big populations, e.g., France and Italy, (0.20% > 0.02% on average).
According to the literature review the information provided in the surveys seems to be promising for the empirical study proposed in this contribution. In the study conducted by Carter and Bélanger [5], perceived ease of use, compatibility and trustworthiness appear to be significant predictors of citizens’ intention to use an e-Government service. In an empirical study conducted by Shareef et al. [2], the authors observed that e-Government adoption behavior differs based on service maturity levels, i.e., when functional characteristics of organizational, technological, economical, and social perspectives of e-Government differ. A user will not arrive at an intention to use an e-Government system, which requires computer knowledge to get a competitive advantage, unless the user has competence from experience in the use of modern ICT. From technological, behavioral, economic, and organizational perspectives, it is anticipated that failing to get hands-on experience of technology will not create in the user an attitude favorable to adopting the system. Therefore, from organizational perspectives, computer self-efficacy is an important predictor of whether a user will adopt an e-Government system instead of using traditional government services. Bélanger and Carter [14] propose a model of e-Government trust composed of disposition to trust, trust of the Internet (TOI), trust of the government (TOG) and perceived risk. Results from a citizen survey (214 responses) indicate that disposition to trust positively affects TOI and TOG, which in turn affects intentions to use an e-Government service. According to Nam [15] the degree of e-Government use for a specific purpose is predicted by five sets of determinants: psychological factors of technology adoption, civic mindedness, information channels, trust in government, and socio-demographic and personal characteristics. Socio-demographic conditions influence usage level of various transactional services provided by e-Government. Perceived ease of use facilitates the acquisition of general information through e-Government.
E-Government use indexes: EGUI / EGUI+
The literature identifies different e-Government adoption levels. Bélanger et al. [14] differentiated the dependent variable “Adoption” into two sub-groups:
- Adoption 1: Decision to accept and use an e-Government system to view, collect information, and/or download forms for different government services as the user requires with the positive perception of receiving a competitive advantage. This would include the situations with information transference from the government to the user but not in the other sense. This adoption level is represented in Table 1 by the following answers available for the e-Government use question (EGU): EGU2 = obtain information (OI), and EGU3 = obtain information (OI) and download forms (DF).
- Adoption 2: Decision to accept and use an e-Government system to interact with, and seek government services, and/or search for queries for different government services as the user requires with the positive perception of receiving a competitive advantage. In this case, the user engagement is bigger and the communication done in the user to government sense, is also done using electronic facilities. This adoption level is represented in Table 1 by the answer to the e-Government use question (EGU) which includes the send filled forms action (SF): EGU4.
On the other hand, Nam [15] and Thompson et al. [16] identified three main purposes of e-Government use: information use, service use or engaging in electronic transactions with government and policy research or to participate in government decision making. The first two, could be equivalent to the Adoption 1 and 2 defined in [14].
Bearing these definitions in mind, in order to quantify e-Government adoption we defined two indexes, EGUI and EGUI+, which are computed as ratios between the number of answers (#) to the question on EGU (EGUi) that reveal some level of e-Government use (i ∈ {2,3,4}) and the ones that indicate no use (i = 1). Eq 1 specifies how the two defined e-Government Use Indexes are computed. As it can be observed, EGUI takes into account the Adoption 1 or information use idea and EGUI+ is an extreme version of EGUI that only involves the users engaged in electronic transactions, null use against complete use (#EGUi, i ∈ {1,4}).
(1)
In Table 3 we provide the list of countries ordered according to the EGUI+ ranking, the total number of possible answers gathered for the EGU question (#EGUi, i ∈{1, 2, 3, 4}) and the EGUI and EGUI+ values.
Based on the EGUI+ values shown in Table 3 we were able to rate the countries into four different e-Government use levels: very high (≥2.0), high (∈[1.0, 2.0)), low (∈[0.5, 1.0)) and very low (<0.5). As a result, two countries were rated as having a very high level (DK, NO), six as having a high level (FI, NL, SE, FR, IE, EE), 10 with a low level (AT, LU, ES, PT, SI, HU, LT, LV, BE, MT) and eight with a very low level (EL, CY, SK, IT, BG, CZ, PL, RO).
Characterization of extreme values of e-Government use (EGU)
Aiming to obtain a greater understanding of e-Government adoption, we characterized the factors involved in the EGUI+ index. As a preliminary study we computed the Pearson correlation for the 26 countries to get the correlation of the 14 independent variables with the two extreme values of the dependent variable e-Government use, (EGU): EGU1 (null) and EGU4 (complete). This provided us with a global picture of the factors which most influenced the extreme values of e-Government use in Europe. To facilitate the interpretation of the correlation results, the irrelevant answers (9 = no answer / don’t know) were removed for this analysis.
According to Pearson, a high frequency to buy goods over the Internet (IBUY) was the variable with the highest correlation coefficient (|r| = 0.43) with e-Government use (EGU), which according to Cohen [26] suggests a medium strength correlation (0.3 < |r|<0.5). In addition, a medium strength correlation (|r| = 0.34) was also found between education level (ISCED) and e-Government use. Finally, manual occupation (OCC_MAN) and Internet frequency of use (IFU) were found to be inversely and positively correlated with EGU respectively (|r| = 0.27), which are considered nearly medium strength correlations. In all the cases the p-value of the test was lower than the significance level alpha, 0.05 and thus, the correlations found are significant although the majority of the values are of small strength (0.1 < |r|<0.3).
To find more specific characteristics of e-Government use, we used the supervised learning algorithm Consolidated Tree Construction (CTC) [17], which beyond a specific discriminating capacity to distinguish between the two extreme levels of e-Government use (null and complete), provided a particular and stable description of the most influential variables for each level. For the analysis we selected two countries from each of the four EGUI+ levels defined, very high, high, low and very low.
In particular an experiment was run in Weka [27] with CTC for the eight countries selected, using the 14 independent variables and the dependent variable EGU with two possible values, null (EGU1) and complete (EGU4). A ten-fold cross-validation (10-fold CV) strategy was used for validation. Table 4 shows the characteristics of the datasets and the obtained classification rates. As can be observed the datasets are quite unbalanced in the majority of countries selected: columns #EGUi and #EGUi (%). Thus, in order to obtain a better characterization of the minority EGU class in each country, CTC was run using a distribution of the minority class of 50% and 2% of each dataset as the minimum number of instances per leaf, which limits the minimum size of any decision node to the specified value.
According to Table 4 the average results achieved by the CTC trees in terms of Precision (Pr), Recall (Re), F-measure (Fm) and Accuracy (Acc) were good with values over 0.71 in the three groups, except in Romania where Recall and Accuracy scored 0.67. This is not surprising since Romania has a very unbalanced dataset, with 94% of the surveys being of null e-Government use type (#EGU1), which reduces the Recall and Accuracy of the minority class.
The structures of the classification trees provide an explanation of the classification. In Figs 1, 2 and 3 CTC trees obtained for Denmark, Belgium and Poland are shown by way of example. In the CTC trees displayed, in the leaf nodes the first number (0/1) represents the class given to the leaf node, the null or complete e-Government use level (EGU1/EGU4), whereas inside the parenthesis, the values before and after the slash represent the number of examples involved and the number of misclassified examples respectively.
Globally analyzing these structures, we concluded that excluding the countries with a very low EGUI+ level, complete e-Government use (EGU4) was closely related to recent online shopping (IBUY = 3), whereas the same action carried out more long time ago (IBUY = 1) seemed to be connected to null e-Government use (EGU1). Table 5 summarizes the main rules provided by the CTC trees for each country and their descriptions are given in sections Countries with a very high EGUI+ level: Denmark and Norway, Countries with a high EGUI+ level: Ireland and Estonia, Countries with a low EGUI+ level: Latvia and Belgium and Countries with a low EGUI+ level: and Countries with a very low EGUI+ Poland and Romania.
Analyzing Table 5, the following variables were found to have a close connection with extreme e-Government use levels (null and complete), listed in a descending number of appearances in the main 29 rules presented for the eight countries: Buy goods over the Internet (IBUY) = 21/29, Education level (ISCED) = 14/29, Internet frequency of use (IFU) = 8/29, Computer frequency of use (CFU) = 6/29, Manual occupation (OCC_MAN) = 5/29, Employment situation (EMPST) = 2/29 and ICT occupation (OCC_ICT) = 1/29.
Countries with a very high EGUI+ level: Denmark and Norway
In Denmark the citizens who rarely bought goods over the Internet (IBUY = 1) and those who had done online shopping recently and were employees/self-employees or students (IBUY = 3 & EMPST = 1/3) did not use e-Government tools (EGU1). On the other hand, Danish citizens who did use e-Government tools (EGU4) had bought online quite recently and were not ICT professionals (IBUY = 3 & OCC_ICT = 0) or had done online shopping between three months and a year previously (IBUY = 2). These main CTC rules found for EGU in Denmark are also available in the most representative nodes of each class (0:EGU1/1:EGU4) in Fig 1.
In Norway the people who used the Internet almost everyday but had not bought goods over the Internet for a time (IFU = 3 & IBUY = 1) and the ones who did not daily use the Internet (IFU≠3) did not use e-Government tools. Conversely, Norwegian people who used the Internet almost everyday and had recently bought goods over the Internet (IFU = 3 & IBUY = 3) did use e-Government. In addition, Norwegians who used the Internet almost daily, had bought goods over the Internet quite recently and did not have a low education level (IFU = 3 & IBUY = 2 & ISCED≠1) also used e-Government tools.
Countries with a high EGUI+ level: Ireland and Estonia
In Ireland the citizens not using e-Government tools (EGU1) were those who hardly ever bought goods over the Internet and who did not use the computer daily (IBUY = 1 & CFU≠3) together with those who hardly ever did online shopping, used the computer almost daily but did not have a high education level (IBUY = 1 & CFU = 3 & ISCED≠3). On the other hand, the Irish citizens who recorded a complete use of e-Government (EGU4) were those who had bought goods over the Internet recently, used the Internet daily and had a high education level (IBUY = 3 & IFU = 3 & ISCED = 3). Additionally, Irish people who had shopped online recently, used the Internet almost daily, had a medium education level and were not in the age range of 16-24 years also used e-Government tools (IBUY = 3 & IFU = 3 & ISCED = 2 & AGECLS≠2).
In Estonia citizens who had not bought online for long time and had a manual occupation (IBUY = 1 & OCC_MAN≠0), as well as those who had not shopped online for a long time, but did not have a manual occupation and did not use the computer daily (IBUY = 1 & OCC_MAN = 0 & CFU≠3) did not use e-Government tools. However, Estonian citizens who had bought goods over the Internet recently and did not have a low education level (IBUY = 3 & ISCED≠1) or those who had bought online long time ago, did not have a manual occupation but used the computer almost every day (IBUY = 1 & OCC_MAN = 0 & CFU = 3), did use e-Government tools.
Countries with a low EGUI+ level: Latvia and Belgium
In Latvia, people with a low education level (ISCED = 1) and those with a medium education level who had bought goods over the Internet a long time ago (ISCED = 2 & IBUY = 1) had no inclination to use the e-Government tools (EGU1). On the other hand, Latvians with a high education level who were not retired (ISCED = 3 & EMPST ≠ 4) and those with a medium education level who had bought online in the previous 12 months (ISCED = 2 & IBUY≠1) did use such tools (EGU4).
In Belgium the null use of e-Government is related to citizens who did not use the Internet daily (IFU≠3) along with the ones who used it almost daily but had not bought goods over the Internet for a long time and did not have a high education level (IFU = 3 & IBUY = 1 & ISCED≠3). On the other hand, Belgian citizens making complete use of e-Government, used the Internet daily and had shopped online within the previous 12 months (IFU = 3 & IBUY≠1). The most representative nodes of each class (0:EGU1/1:EGU4) in the CTC tree shown in Fig 2 also exhibit the main CTC rules found for EGU in Belgium.
Countries with a very low EGUI+ level: Poland and Romania
In Poland citizens with a low education level (ISCED = 1) together with those with a medium education level who did not use the computer daily (ISCED = 2 & CFU≠3) showed a null trend towards the use of e-Government (EGU1). On the other hand, Poles who used e-Government tools (EGU4) had a high (ISCED = 3) or a medium education level, used the computer almost daily and had bought goods over the Internet quite recently (ISCED = 2 & CFU = 3 & IBUY = 3). The most representative nodes of each class (0:EGU1/1:EGU4) shown in Fig 3 also illustrate the main CTC rules found for EGU in Poland.
Romania was the only country where the two extreme e-Government use levels, null (EGU1) and complete (EGU4), were characterized by two rules that involved a single factor, manual occupation (OCC_MAN): manual workers (OCC_MAN≠0) did not use the e-Government tools, whereas non manual workers did use them.
Comparison between EGUI+ and other indexes
Aiming to analyze if the performance of the index we defined to measure the practical e-Government use is similar to other conceptual indexes broadly used as indicators of related features such as e-readiness of a country, we selected four indexes and compared them to EGUI+: E-Government Development Index (EGDI) and its Online Service Index (OSI) component, and the Networked Readiness Index (NRI) and its Government usage (GU) component. Next we describe the indexes mentioned and the comparison carried out.
Description of the indexes
From 2001 to the present, The United Nations Department of Economic and Social Affairs (UNDESA) has published the UN E-Government Survey [10]. In 2003 this survey began to provide an analysis of the progress in using e-Government via the E-Government Development Index (EGDI), a composite index based on the weighted average of three normalized (norm.) indices, assigning one third of weight to each of them (see Eq 2): the Telecommunications Infrastructure Index (TII), the Human Capital Index (HCI) and the Online Service Index (OSI). As a composite indicator, the EGDI is used to measure the readiness and capacity of national institutions to use ICTs to deliver public services [10]. Prior to the normalization of the three component indicators, the Z-score standardization procedure is implemented for each component indicator to ensure that the overall EGDI is decided equally by the three component indexes.
(2)
The TII index is based on data provided by the International Telecommunications Union (ITU) and is computed as the weighted average of the Z-score of five component indicators: Internet users, Fixed telephone subscribers, Mobile/Cellular telephone subscription, Active mobile broadband subscription and Fixed broadband. [10]
The HCI index is based on data supplied by the United Nations Educational, Scientific and Cultural Organization (UNESCO), and it is computed as the weighted arithmetic mean with one-third weight assigned to the adult literacy rate and two-ninths weight assigned to the gross enrollment ratio, estimated years of schooling and mean years of schooling. [10]
The OSI index, one of the three component indicators of the EGDI index described in Eq 2 that was selected for the analysis, is a composite normalized score based on an independent survey questionnaire, conducted by UNDESA, which assesses the national online presence of all 193 United Nations Member States. The survey questionnaire computes several features related to online service delivery, including whole-of-government approaches, open government data, e-participation, multi-channel service delivery, mobile services, usage up-take, digital divide as well as innovative partnerships through the use of ICTs. The majority of questions are of binary type (yes, no) and the total number of points scored by a country is normalized to a range of 0 to 1. The value for a given country is computed as the subtraction between the total number of points scored by that country and the lowest score for any country divided by the range of values for all countries in the survey; for instance, let 114 be the score of a country “x”, 0 the lowest score of any country and 153 the highest one, then the Online Service Index of “x” will be . [10]
The World Economic Forum has been annually publishing The Global Information Technology Report [11] since 2001, where the Networked Readiness Index (NRI) is provided. As shown in Eq 3, the NRI is a composite index computed as the weighted average of four main subindexes, being all the weights a quarter: Environment subindex, Readiness subindex, Usage subindex and Impact subindex.
(3)
The Environment subindex assesses the extent to which a country’s market conditions and regulatory framework support entrepreneurship, innovation, and ICT development [11]. This subindex is computed as the weighted average of two pillars (using weights of one half) which are based on nine indicators respectively: the Political and regulatory environment pillar (e.g., intellectual property rights protection and prevalence of software piracy) and the Business and innovation environment one (e.g., red tape and ease of starting a business).
The Readiness subindex measures the extent to which a country has in place the infrastructure and other factors to support the uptake of ICTs [11]. This subindex is computed as the weighted average of four pillars (using weights of one quarter). The Infrastructure pillar is based on four indicators, e.g., mobile network coverage, international Internet bandwidth, secure Internet servers etc. The Affordability pillar is based on three indicators, e.g., mobile telephony usage costs and broadband Internet subscription costs. The Skills pillar is based on four indicators, e.g., enrollment rate in secondary education and the overall quality of the education system.
The Usage subindex of NRI assesses the level of ICT adoption by a society’s main stakeholders: government, businesses and individuals [11]. In particular, the Usage subindex is computed as the weighted average of three pillars (using weights of one third). The Individual usage pillar is based on on seven indicators (e.g., mobile telephony penetration and Internet usage). The Business usage pillar is based on six indicators (e.g. firm-level technology absorption and capacity for innovation). The Government usage pillar (GU) used in this analysis, assesses the leadership and success of the government in developing and implementing strategies for ICT development, as well as in using ICTs, as measured by the availability and quality of government online services [11]. The Government usage pillar is computed as the average of three indicators: the importance of ICTs to government vision, the Government Online Service Index and the Government success in ICT promotion.
Finally, the Impact subindex measures the broad economic and social impacts accruing from ICTs [11]. This subindex is computed as the weighted average of two pillars (using weights of one half) which are based on four indicators respectively: the Economic impacts pillar (e.g., number of patent applications and impact of ICTs on business models) and the Social Impacts pillar (e.g., impact of ICTs on access to basic services and Internet access in schools).
In particular, about half of the 53 individual indicators used to compute the NRI are sourced from international organizations, mainly from the International Telecommunication Union (ITU), the World Bank, the United Nations Educational, Scientific and Cultural Organization (UNESCO) and other UN agencies [11]. The other half of the NRI indicators are derived from the World Economic Forum’s annual Survey which is administered annually to over 14,000 business executives in all the economies included in the NRI [28]. The Survey represents a unique source of insight into many critical aspects related to a country’s enabling environment, the preparedness of its population, ICT usage, and ICT impacts. When a country misses five or 10 percent of all indicators involved, its NRI is not computed.
In summary, three of the conceptual indexes/pillars used in the comparison, EGDI, NRI and its GU pillar, as well as the indexes we define are composite indexes computed based on a frequently used approach named equal-weight [29]. Although the number of indicators used in each of them is different, the indexes compared are among the most cited ones [30][29]. To this regard, a critical review on the concept of e-readiness made by Danish Dada [31] pointed out that indexes measuring e-readiness do not completely reflect the possibility of achieving development from ICTs in developing countries and suggests to consider the level of the individuals within the organization using the technology in order to obtain a more accurate measure. In this regard, the comparison shown in next sections contributes to enrich the idea of e-readiness provided by the four conceptual indexes selected, by adding information on the real use of e-Government services.
Ranking comparison
In order to study the relationship between the practical use of e-Government (EGUI+) and the level of e-Government readiness (EGDI), network readiness (NRI), national online presence (OSI) and ICT adoption by government (GU), we present the values for these indexes (see Table 6) and we compared their rankings for the 26 countries analyzed. For the comparison we tried to use similar time periods, 2009-2015 period for the annual indexes or indicators, EGUI+, NRI and GU [11, 18–22], and 2010-2016 period for the biannual indexes or indicators, EGDI and OSI [6–9].
Specifically we computed the number of positions won or lost (positive or negative value) by the countries from the EGDI, OSI, NRI and GU rankings to the EGUI+ ranking, which in general terms is low (see Table 6). As shown in Table 6, we grouped the countries into three different sets using ±5 positions as a threshold for the ranking differences appreciated (nearly 20% of the ranking) represented by the following codes: blue-bold if they drop more than five positions, green-roman if they drop or gain fewer than five positions (stable countries) and red-italic if they gain more than five positions.
According to Table 6, for a great majority of the countries involved in the analysis, 84.6% on average (green-roman ones), the practical e-Government adoption does match the features measured by the conceptual indexes, EGDI, OSI, NRI and GU. To this regard, we found sixteen countries (62%) appearing in all the groups with small ranking differences (stable countries): Austria (AT), Bulgaria (BG), Greece (EL), Finland (FI), France (FR), Lithuania (LT), Luxembourg (LU), Latvia (LV), Netherlands (NL), Norway (NO), Poland (PL), Portugal (PT), Romania (RO), Sweden (SE), Slovenia (SI) and Slovakia (SK). In addition, we observed that EGDI is the most similar index to EGUI+, since 92% of the countries (24/26) are of stable type. Thus, in countries where the quality of e-Services is low, their use is difficult and appear consequently classified as very low by EGUI+ whereas in those where the quality of e-Services is high, they tend to be more used, and appear consequently classified as high by EGUI+ in general. For instance, Romania (RO) and Bulgaria (BG) remain in low positions in all the rankings with average differences with the four indexes compared of -2 and 2 positions respectively whereas Netherlands (NE) and Norway (NO) and remain in high positions in all the rankings with average rank differences of -1 and 2 positions in the four comparisons.
On the other hand, only 10.6% of the countries (blue-bold ones) on average showed higher positions in the rankings provided by the rest of the indexes than for that of EGUI+ (< − 5 positions), i.e. their theoretical situation seems to be better than the practical one. Analyzing all the groups with high negative ranking differences with EGUI+ (blue-bold ones), we did not find any country common to all of them but Belgium (BE) could be considered as common since it is nearly in the blue-bold group of the GU index. In addition, we observed that Belgium (BE) also was the country with higher drops, falling from 10th, 11th and 10rd positions in the EGDI, OSI and NRI rankings to the 17th one in the EGUI+ ranking. Consequently, we could state that the real use of e-Government services is below the expectations we could have in Belgium.
Finally, 4.8% of the countries on average achieved lower positions (red-italic) in the rankings of the four conceptual indexes than in the ones provided by EGUI+ (>5 positions), although the EGDI one does not have any country with such ranking rises. In the rest of indexes, we found that there is not any common country to all the groups but Ireland (IE) could be considered as common as it is nearly in the red-italic group of NRI index. In addition, Ireland also was the country with the highest ranking rise for the index we defined, rising from 13nd and 15th positions in the OSI and GU rankings to 7th one in that of EGUI+. This example would be in the opposite case of Belgium, the real use of e-Government services is above the expectations we could have in Ireland.
Beyond that, we observed greater differences between the values given by the index defined (EGUI+) for the different countries, specially when compared to EGDI and OSI (see columns 2-6 in Table 6). In this sense, although countries are similarly ranked they are more clearly distinguished according to the use of the e-Government services than according to the theoretical quality of such services. Those countries ranked in the first (Denmark) and last (Romania) positions according to EGUI+ are the clearest examples, with index values almost duplicating the values achieved by second best and worst ranked countries respectively.
For a deeper analysis of the similarity between the performance of the four conceptual indexes and that of EGUI+ we computed four pairwise comparisons based on Kendall correlation [23] using the rankings provided by each index. Table 7 shows the results of Kendall pairwise tests between EGUI+ and EGDI, OSI, NRI and GU, in terms correlation values () and significance (p-values). The second and third columns of the table show the performance of the stable countries (green-roman ones in Table 6) suggesting that the four indexes are highly correlated with EGUI+ at 0.05 significance level, p-value<alpha, with correlation values (
) on average of 0.8. In addition, in the in the fourth and fifth columns of Table 7 we also show the results of the Kendall tests carried out analyzing the complete set of countries. In this case, the values of
decreased down to 0.7 on average, being EGDI the index which scores the highest correlation value (
) with the index we defined. In the global comparison, we observed slightly higher correlation values for the EGDI index and its OSI index than for the NRI index and its GU pillar.
Considering all the above, we can state that the empirical analysis carried out on e-Government adoption across Europe through EGUI+ index concur to a large extent with the theoretical studies which measure the level of e-readiness of European countries through different indexes (EGDI, OSI, NRI and GU).
Discussion
On the one hand the digital divide makes the task of providing universally accessible online government services challenging [12] and on the other hand, citizen confidence in the ability of an agency to provide online services is imperative for the widespread adoption of e-Government initiatives [14]. According to Shareef et al. [2], e-Government adoption behavior differs when functional characteristics of organizational, technological, economical, and social perspectives of e-Government differ. The first part of the empirical study carried out based on Eurostat CSIS Surveys, the classification of countries in different e-Government use levels (see Table 3) is in concordance with the statement since, first of all, not all countries have the same e-Government use level, and, although with exceptions, more developed and wealthier countries seem to have higher levels of e-Government use.
With regard to the factors affecting the e-Government use, buying goods in the Internet could be expected to be one of the factors directly related to e-Government use due to the similarities existing between e-commerce and e-Government. According to Schwester [12] the same way factors from Technology Acceptance, Diffusion of Innovation and trustworthiness models play a role in user acceptance of e-commerce, it is expected that they will also affect e-Government adoption. Accordingly, the outcome of our study shows that in countries with higher e-Government adoption according to Eursotat CSIS Surveys, IBUY, the variable related to e-commerce is most of the times correlated to the use or not use of e-Government services.
But, this is not always the case, there are differences between commercial businesses and government agencies [14]. E-commerce and e-Government differ in their reasons for existence (profit vs. service) and constituents served (target market vs. population at-large. Businesses can choose their customers; however, in e-Government, agencies are responsible for providing access to the entire eligible population, including individuals with lower incomes and disabilities [12]. Mandatory relationships exist only in e-Government. Citizens perceive businesses differently than government. In addition, the structure of businesses is different from the structure of agencies in the public sector. Decision-making authority is less centralized in government agencies than in businesses. This dispersion of authority impedes the development and implementation of new government services. The third difference is accountability. In a democratic government, public sector agencies are constrained by the requirement to allocate resources and provide services ‘in the best interest of the public’. The political nature of government agencies is also a feature that makes e-Government and e-commerce different. These factors could be related with the fact that in countries with lower adoption level (EGUI+ = low/very low), other factors such as education level and occupancy appear to be related to the e-Government adoption.
On the other hand, some authors as Afyonluoglu and Alkar [32] for instance, compared 16 international e-Government benchmarking studies completed between 2001—2016 by five active organizations including UN and WEF and identified the common points and the differences with respect to 22 different criteria including indexes such as EGDI and NRI. They pointed out that none of studies compared measures the “usage of e-services by citizens”, “governance model of e-Government”, “benefits of e-services” and “satisfaction”, suggesting that they should be considered for future e-Government framework improvements. Similarly, Jadi and Jie [33] use the EPI E-participating index, a supplementary indicator designed by the UN, as an output of government effort to evaluate the performance of e-Government systems. The authors state, that although the EGDI index is used as a benchmark to provide a numerical ranking of e-Government development, building websites, infrastructures and providing online services only shows how the readiness of the government to exploit the facilities is. However, in addition to those indicators the performance of e-Government systems can be analyzed by measuring to what extent citizens are using these facilities. The second part of this work is aligned with the previous lines since we compared the e-Government practical use and the e-readiness of 26 EU countries, based on the EGUI+ index, empirically computed from the 2009-2015 Eurostat CSIS Surveys, the EGDI index and its OSI component published in the 2010-2016 UN e-Government Surveys, and the NRI and its GU component provided by the 2010-2015 World Economic Forum’s Global Information Technology Reports. According to our analysis, it seems that in the majority of the countries the situation of the e-Government does not differ substantially despite using different calculation methods. To this regard we think that the compute of e-readiness of countries (EGDI, NRI) could be improved by including the real use of e-Services quantified in the index we define (EGUI+).
Finally, the outcome of the index comparisons points some countries where we could focus to analyze other types of variables influencing the quality and use of the services provided by the Governments. That is, the analysis of politics, infrastructures and other aspects in countries where the quality of the provided services and the e-Government services use level does not match, such as Belgium and Ireland, could probably give important clues about aspects affecting to the use of these services.
Conclusions
In this paper we analyzed e-Government adoption, the practical use of e-Services supplied by Governments, across Europe (for 26 EU countries) based on the empirical data provided by Eurostat [13]. The outcomes contribute to gaining insight on some of the factors influencing the e-Government adoption in Europe and can provide some guidelines to improve the interaction of citizens with web services and information offered by institutions depending on their e-Government use level.
The data was first used to quantify the adoption level by defining two indexes: the E-Government Use Index (EGUI) and an extreme version of it, taking only into account the highest e-Government adoption for calculation (EGUI+). The countries were then rated according to four adoption levels based on EGUI+ levels (very high, high, low and very low) and two countries from each level were selected to characterize the e-Government adoption using supervised learning procedures, CTC trees [17]. In particular, the system was able to differentiate individuals doing null e-Goverment use from individual doing complete use of it with an average accuracy of 73% in the eight countries selected: Denmark and Norway, Ireland and Estonia, Latvia and Belgium and Poland and Romania. This enabled us to identify the main factors affecting the practical use of e-Government tools for eight different countries, which can contribute to better understand of these values. Furthermore, the main rules provided by the eight trees built, were supported by the Pearson correlation computed for the 26 EU countries.
Specifically, Pearson revealed that European citizens who had bought goods quite recently over the Internet did use e-Government tools and, although according to CTC the complete use of e-Government tools is also determined by other specific factors, the same conclusion arouse from the CTC structures of all the countries selected except for Romania. On the other hand, Pearson’s analysis determined that European citizens with high education levels do usually a complete use of e-Government tools what was also observed in the CTC trees of six of the eight countries analyzed with very high (Norway), high (Ireland and Estonia), low (Latvia and Belgium) and very low (Poland) EGUI+ levels, combined with other types of factors.
Finally, for the 26 EU countries analyzed, we compared the rankings of the EGUI+ index to that of other conceptual indexes measuring the level of e-readiness of a country such as the E-Government Development Index (EGDI) its Online Service Index (OSI) component, the Networked Readiness Index (NRI) and its Government usage component (GU). As a result, EGUI+ was found to be correlated with the four indexes mentioned at 0.05 significance level showing that adoption levels extracted from the empirical analysis are in general aligned with more conceptual and theoretical values.
In summary, we think that our research results provide some key-aspects that could be considered for future strategic decisions on the improvement of e-Government adoption in different European countries, in terms of knowledge about the most influential factors on null and complete e-Government use and also in terms of proposing complementary indexes based on empirical data.
With regard to future work our study could be extended to include new Eurostat (2016-2020) and UN (2018-2020) data and involve other e-Government indicators suggested by various authors [33–35]. On the other hand, a new analysis could be carried out excluding the countries with no Eurostat data for any of the years, e.g. Belgium (2015), Latvia (2009). Additionally, the research could be enriched by extending the geographical area of interest or focusing more closely on a smaller area. Finally, the analysis of other variables affecting the use and infrastructures of e-Services could help understanding the differences found regarding the countries achieving the best values for the main indexes used (EGUI+, EGDI and NRI).
Acknowledgments
This work would not have been possible without the data provided by the statistical office of the European Union (Eurostat), thus, we truly want to thank the altruistic support of this agency.
References
- 1.
European Commission. “The Digital Economy and Society Index (DESI)”; 2019.
- 2. Shareef MA, Kumar V, Kumar U, Dwivedi YK. e-Government Adoption Model (GAM): Differing service maturity levels. Government Information Quarterly. 2011;28(1):17—35. https://doi.org/10.1016/j.giq.2010.05.006.
- 3. Layne K, Lee J. “Developing fully functional E-government: A four stage model”. Government Information Quarterly. 2001;18(2):122–136.
- 4.
Brown MM, Brudney JL. Achieving advanced electronic government services: An examination of obstacles and implications from an international perspective. In: National Public Management Research Conference, Bloomington, IN. vol. 2; 2001. p. 143–49.
- 5. Carter L, Bélanger F. The utilization of e-government services: citizen trust, innovation and acceptance factors*. Information Systems Journal. 2005;15(1):5–25.
- 6.
UN. “United Nations E-Government Survey 2010: Leveraging e-government at a time of financial and economic crisis”; 2010.
- 7.
UN. “United Nations E-Government Survey 2012: E-Government for the People”; 2012.
- 8.
UN. “United Nations E-Government Survey 2014: E-Government for the future we want”; 2014.
- 9.
UN. “United Nations E-Government Survey 2016:E-Government in Support of Sustainable Development”; 2016.
- 10.
UN. “United Nations E-Government Survey 2018: Gearing E-Government to Support Transformation Towards Sustainable and Resilient Societies”; 2018.
- 11.
Dutta S, Geiger T, Lanvin B. The global information technology report 2015. In: World Economic Forum. vol. 1. Citeseer; 2015. p. 1–357.
- 12. Schwester R. Examining the Barriers to e-Government Adoption. Electronic Journal of e-Government. 2009;7(1):113—122.
- 13.
Eurostat. “Individuals using the internet for interaction with public authorities”, Dataset code: tin00012; 2004.
- 14. Bélanger F, Carter L. Trust and risk in e-government adoption. The Journal of Strategic Information Systems. 2008;17(2):165—176. https://doi.org/10.1016/j.jsis.2007.12.002.
- 15. Nam T. Determining the type of e-government use. Government Information Quarterly. 2014;31(2):211—220. https://doi.org/10.1016/j.giq.2013.09.006.
- 16. Thomspson DV, Rust RT, Rhoda J. The business value of e-government for small firms. International Journal of Servide Industry MAnagement. 2005;16(4):385–407. https://doi.org/10.1108/09564230510614022.
- 17. Pérez JM, Muguerza J, Arbelaitz O, Gurrutxaga I, Martín JI. “Combining multiple class distribution modified subsamples in a single tree”. Pattern Recognition Letters. 2007;28(4):414—422.
- 18.
Dutta S, Mia I. The global information technology report 2009–2010. In: World Economic Forum and INSEAD, SRO-Kundig Geneva, Switzerland; 2010. p. 1–415.
- 19.
Dutta S, Mia I. The global information technology report 2010–2011. In: World Economic Forum and INSEAD, SRO-Kundig Geneva, Switzerland; 2011. p. 1–411.
- 20.
Dutta S, Bilbao-Osorio B. The global information technology report 2012. In: World Economic Forum and INSEAD, SRO-Kundig Geneva, Switzerland; 2012. p. 1–413.
- 21.
Bilbao-Osorio B, Dutta S, Lanvin B. The global information technology report 2013. In: World Economic Forum; 2013. p. 1–383.
- 22.
Bilbao-Osorio B, Dutta S, Lanvin B. The global information technology report 2014. In: World Economic Forum; 2014. p. 1–343.
- 23. Kendall MG. A new measure of rank correlation. Biometrika. 1938;30(1/2):81–93.
- 24.
European Parliament and Council of the European Union. “Regulation (EC) No 808/2004 of the European Parliament and of the Council”; 2004.
- 25.
Eurostat. Methodological Manual for Statistics on the Information Society - survey year 2015; 2015.
- 26.
Cohen J. “Statistical power analysis for the behavioral sciences”. 2nd ed. Hillsdale, N.J.: L. Erlbaum Associates; 1988.
- 27. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH. “The WEKA data mining software: an update”. ACM SIGKDD explorations newsletter. 2009;11(1):10–18.
- 28. Browne C, Di Battista A, Geiger T, Gutknecht T. The executive opinion survey: The voice of the business community. The Global Competitiveness Report 2015–2016. 2015; p. 75–85.
- 29. Jovanovic MJ, Brajovic B, Milenkovic D, Vukmirovic D, Jeremic V. Beyond the equal-weight framework of the Networked Readiness Index: a multilevel I-distance methodology. Information Development. 2016;32(4):1120–1136.
- 30. Máchová R, Lnĕnička M. Modelling E-Government Development through the Years Using Cluster Analysis. eJournal of eDemocracy and Open Government. 2016;8:62–83.
- 31. Dada D. E-Readiness for Developing Countries: Moving the focus from the environment to the users. EJISDC. 2006;27:1–14.
- 32.
Afyonluoglu M, Alkar AZ. Comparison and Evaluation of International e-government Benchmarking Studies. In: European Conference on e-Government. Academic Conferences International Limited; 2017. p. 283–293.
- 33. Jadi Y, Jie L. “An Efficiency Measurement of E-Government Performance for United Nation Ranking Index”. World Academy of Science, Engineering and Technology, International Journal of Social, Behavioral, Educational, Economic, Business and Industrial Engineering. 2017;11(1):279–282.
- 34. Seri P, Bianchi A, Matteucci P. “Diffusion and usage of public e-services in Europe: An assessment of country level indicators and drivers”. Telecommunications Policy. 2014;38(5):496—513.
- 35.
Kabbar E, Dell P. “Weaknesses of the E-Government Development Index”. Uesugi S, editor. Vienna: Springer Vienna; 2013.