Measuring the Impact of Information Systems in Malaysia

— Information Systems researchers have employed a diversity of sometimes inconsistent measures of IS success, seldom explicating the rationale, thereby complicating the choice for future researchers. In response to these and other issues, Gable, Sedera and Chan introduced the IS-Impact measurement model. This model represents “the stream of net benefits from the Information System (IS), to date and anticipated, as perceived by all key-user-groups”. Although the IS-Impact model was rigorously validated in previous research, there is a need to further generalise and validate it in different context. This paper reported the findings of the IS-Impact model revalidation study at four state governments in Malaysia with 232 users of a financial system that is currently being used at eleven state governments in Malaysia. Data was analysed following the guidelines for formative measurement validation using SmartPLS. Based on the PLS results, data supported the IS-Impact dimensions and measures thus confirming the validity of the IS-Impact model in Malaysia. This indicates that the IS-Impact model is robust and can be used across different context.


I. INTRODUCTION
Information Systems (IS) researchers have shown greatest interest in IS evaluation based on a large number of articles published in the late 1970s [1], [2].Most early attempt on IS evaluation have focused on system availability and performance [2].Since then, IS performance evaluation has been investigated from three perspectives that are IS effectiveness/success, IS function evaluation and IS service quality [3].In the area of IS effectiveness/success, IS researchers have come up with a variety of measures to measure the success of IS [1].Some have introduced frameworks or models to help organisations in evaluating the success of IS in their organisations systematically (e.g.IS Success model [1], ERP Benefits Framework [4] and IS-Impact Model [5]).
Many IS success researches place more attention to the causal relationships between IS success constructs (e.g.[6], [7]).Less attention is given in developing a standard measurement model, discuss the rationale for their selection of measures [1], [5] and testing the relationship between the measures and the constructs [8], [9].With inconsistent choice of IS success measures, it is difficult to compare findings between IS success studies [7].Furthermore, not many researchers focus on the external validity of a model, to investigate the extent to which a theory or model performs and can be generalized in different contexts [6], [10] - [13].
This paper reported and discussed on the outcome of a study that extend and validate IS-Impact model, a measurement model that was introduced by Gable, Sedera and Chan [5] in Malaysia.This effort is to address known limitations of the model to yield a robust and standardised measurement model that can be used across different contexts.A survey was conducted at four state governments in which an integrated custom-made financial system was chosen as the unit of analysis.Findings indicate the applicability and validity of the IS-Impact model for measuring the impact of IS in organisations in Malaysia.
This paper begins with an introduction of the IS-Impact model which is the theoretical foundation of this study.The research methodology in conducting the study is discussed next.This is follow by the findings from the data analysis.
The paper concludes with a discussion on the findings and the limitations of this study to promote future works.

II. THEORETICAL FRAMEWORK
Gable et al. [5] introduced the IS-Impact model to measure the impact of a contemporary IS.This model represents "the stream of net benefits from an Information System (IS), to date and anticipated, as perceived by all keyuser-groups" ( [5], p. 831).The model is a formative multidimensional index with 27 perceptual measures along four dimensions in two halves: impact and quality (Fig. 1).The two "impact" dimensions (Individual-Impact and Organizational-Impact) are an assessment of net benefits to date while the two "quality" dimensions (System-Quality and Information-Quality) act as proxies for potential future impacts from the system.According to Gable et al. [5], "Individual Impact (II)" is a measure of the extent to which (the IS) has influenced the capabilities and effectiveness, on behalf of the organisation, of key-users."Organizational Impact (OI)" is a measure of the extent to which (the IS) has promoted improvement in organisational results and capabilities."Information Quality (IQ)" is a measure of the quality of (the IS) outputsnamely, the quality of the information the system produces in reports and on-screen."System Quality (SQ)" is a measure of the performance of (the IS) from a technical and design perspective.
This study adopts IS-Impact model as the primary commencing theory-base.The IS-Impact model, by design, is intend to be robust and simple yet generalisable, yielding results that are highly comparable across time, stakeholders, different type of systems and system contexts.The model and approach employs perceptual measures, aiming to offer a common instrument answerable by all relevant stakeholder groups, thereby enabling the combining or comparison of stakeholder perspectives.Note that in this study, in attention to identify relevant new measure (that has not been identified from the previous work and due to possible context influence or current trend, to ensure model completeness) commenced with the full pool of 37 measures (the a-priori model developed by Gable et al. [5]).

III. RESEARCH METHODOLOGY
As mentioned in the introduction section, this paper report findings from a quantitative survey conducted to revalidate the IS-Impact model in Malaysia.Prior to this quantitative survey, a qualitative survey was conducted to find out whether the IS-Impact model is comprehensive for evaluating the impact of IS in the new context.The qualitative survey was conducted at a state government in Malaysia to identify relevant new measures to be included in the model from the perspectives of the users of a financial system in the state government.At the same time, this survey sought to address the content validity of the IS-Impact model.Due to paper length constraint, the findings from the survey are not discussed in this paper because it requires an elaborate discussion on how the content validity has been established.In summary, a new measure, "Security", was identified and is relevant for the new context.Moreover, a number of literatures support "Security" as an important aspect for an information system (i.e.[14][15][16][17][18]).With this strong argument, this measure was added in the model as one of the System Quality measure.
The original survey instrument was modified to suit the Malaysia context.One of the modifications made was translating the instrument to the national language of Malaysia, Bahasa Malaysia, for the benefits of users that are less conversant in English.The instrument was translated using both "back-translation" and "decentering" techniques following the suggestion of Brislin [19] and McGorry [20].The outcome of the translation processes resulted in a minor changes to the original instrument mainly on the structure of the sentences and different choice of words.These changes did not deviate from the original meaning of each of the items in the original instrument.The Bahasa Malaysia instrument was then pilot tested for face validity.
The questionnaire was divided in two main sections.The first section collected demography information from the respondents.The second section contained the 38 measures of the IS-Impact model and several dependent variables (for testing construct validity).Items in the questionnaire were measured using six-point LIKERT scale (with strongly agree and strongly disagree as the end values).Fig. 2 depicts the IS-Impact model with the 38 measures 1 .
Fig. 2 The IS-Impact Model with 38 measures All questions in the questionnaire (i.e. the descriptive, the IS-Impact measures and dependent variables), were made mandatory in the survey.The respondents were asked to complete all questions in the questionnaire and this requirement is stated at the introductory page of the questionnaire and at the start of the section in the questionnaire.
Data was collected at four state governments in Malaysia.The respondents were selected using a combination of cluster, convenience and snowball sampling methods.The targeted respondents were the organisation"s users of a financial system.Hardcopies questionnaire were distributed to the targeted respondents with the help of IT officers at each of the states governments involved in this study.
415 questionnaires were distributed targeting departments with high number of users and at the same time canvassing all employment cohorts (from strategic to technical users).310 questionnaires were returned with the response rate of 75% from 26 departments across four state governments.From the data cleaning process, 78 respondents were removed leaving 232 valid respondents to be used in the analysis.The profile of the respondents is summarised in Table I.

IV. FINDINGS
Several notable papers where referred to for guidelines in identifying, specifying and interpreting formative constructs and the index underlying the constructs.While Diamantopoulos and Winklhofer [21] and Petter, Straub and Rai [9] have provided a clear definition and understanding of formative construct and its difference with reflective construct, and provide guidelines to specify a formative construct, good papers with an exemplary interpretation of formative measurement results are scarce.More recently, Andreev, Heart, Maoz and Pliskin [22], Cenfetelli and Bassellier [23] and Henseler, Ringle and Sinkovics [24] have provided illustrative examples on formative construct validation and how to assess and estimate the construct using PLS softwares.One of the advantages of PLS is it allows for the use of both formative and reflective measures, which is not generally achievable with covariance-based SEM techniques such as LISREL or EQS [25].Generally, the validity of formative measurement model can be assessed in four steps as summarised in Table II.

A. Multicollinearity
The presence of collinearity can be observed from the "Collinearity Diagnostics" output from a regression test conducted for the items in a formative construct (the independent variables) with a dependent variable.The Tolerance and Variation Inflation Factor (VIF) value provided in the "Coefficients" table will indicate the present of multicollinearity.From the results, all 38 measures were below the common VIF cut-off point of 10 [21], [26]- [27] with the largest VIF reported is 5.618.
According to Diamantopolous and Winklhofer [21], one way to test the quality of the items is by observing the correlation of the items with another variable that is external to the index.Only items that have significant relationship with the variable should be retained.Following Diamantopolous and Winklhofer ( [21], p. 272) suggestion, four global items that "summarise the essence of the construct that the index purports to measure" are employed to examine the relationships between the items with the intended dependent variable (the global item) at each dimension.
Results indicated that the correlations between the independent and the dependent variables (DV) are range between 0.421 to 0.717 (at p < 0.05 or better).The correlation results also indicated three non-significant measures (with p > 0.05).One measure, although significant, but has very small correlation with the dependent variable (with r < 0.3) (based on Cohen [29] suggestion).Therefore the correlation results indicated four invalid measures; IQ7 Content Accuracy, SQ1 Data Accuracy, SQ3 Database Content and SQ6 Access.These measures were removed from further analysis.

B. Assessing the Validity of the IS-Impact Model Through Structural Relationship
Next, the IS-Impact model was tested through structural relationship by identifying the relationships between (i) latent variables and the observed or manifest variables (outer model) and between (ii) unobserved variables (inner model) or also known as Nomological (Net) Validity.These tests are carried out using SmartPLS, a software application for (graphical) path modelling with latent variables that used partial-least square (PLS) method for the latent variables analysis [29].Results from the structural relationship evaluation are presented in Fig. 3.

Fig. 3 The Structural Model
To assess the structural relationship of the measurement model (outer model), two reflective measures that summarised the "IS-Impact" are used.An adjusted R-square of 0.622 was reported from the analysis, indicating that 62.2% of the variance in the IS-Impact is explained by the II, OI, IQ and SQ.Following the recommendation from Chin [25], Henseler et al. [24] suggest R-square values of 0.67, 0.33 and 0.19 in PLS path models as substantial, moderate and weak.Therefore, this path analysis indicated that the IS-I model is almost substantial.All structural paths in the model are significant at p < 0.05 or better (estimated by bootstrapping procedure with 500 bootstrapping samples) with System Quality (SQ) provides the strongest contribution to IS-Impact.
The final approach to test the validity of the IS-Impact model is by linking the model with an antecedent or consequence construct that has been hypothesised to have significant and strong relationship with.When validating the IS-Impact model, Gable et al. [5] employed Satisfaction as the consequence of IS-Impact (IS-I) (refer to [5] for more detail).They hypothesised that "a higher level of IS-I yields a higher level of Satisfaction".From the analysis, they found a strong positive relationship between IS-I and Satisfaction with β = 0.854, and significance at the level α = 0.001.
Replicating the same approach, this study employed Satisfaction by including the same item used in Gable et al. [5] and adding two more measures identified from the literature.The PLS results (see Fig. 3) supported the hypothesis by depicting strong positive relationship between IS-I and Satisfaction, with β = 0.801 and significant at α = 0.001.Furthermore, the structural model indicated that IS-I explained 64.1% of the variance in the Satisfaction, thus demonstrating almost substantial model.With this result, the validity of IS-Impact model with 34 measures is established.

C. Explanatory Power of the Model
Following the PLS test, changed in R-Square was explored to investigate the impact of each dimension; the II, OI, IQ and SQ, on the overarching IS-Impact construct.This is done through repeated PLS estimates and calculate the effect size in which one dimension is excluded in each of the PLS runs.The effect size is calculated using the following formula: (1) The results (Table III) indicated that II, OI, IQ have medium effect on IS-I, with all > 0.15 (Cohen [28] suggests values of 0.02, 0.15 or 0.35 indicate small, medium and large effects respectively).Meanwhile, the effect of SQ on IS-I is large (with all > 0.35).The purpose of this analysis is to demonstrate the additivity of the four dimensions as a complete measurement model.Based on the result, it can be interpreted that combining all the dimensions in a model provided a strong contribution as depicted in the incremental change of the R-square.The findings discussed above demonstrated the validity of the IS-Impact model that consists of four dimensions with 34 items, which can be used to measure the impact of information system to date and at the same time predict its future impact to public organisations in Malaysia.This proposition is based on the VIF score, correlation analysis and the path analysis of the structural relationship between the constructs and their measures (measurement model assessment) and between the IS-Impact construct with other construct that is hypothesised to have significant and strong relationship with.Begins with 38 items, multicollinearity diagnostic test indicated no presence of collinearity among the items.However, four items were removed from the model due to low and non-significant correlations with the global items (dependent variables), violating the predictive validity assessment, thus these four measures are not valid as predictors.Moreover, the new added measure, "Security" is significant and a valid indicator for System Quality based on the path coefficient result.The adjusted R square indicated almost substantial model for IS-Impact model where the model explained 62.2% of the construct variance.All path weights between the construct to the higher order construct, IS-Impact, are significant.Furthermore, the path analysis indicated a strong positive relationship between IS-Impact and Satisfaction, and supports the hypothesis [5].This may suggest that the higher the impact of information system will lead to higher satisfaction of the users.
The PLS estimate results also demonstrate the path coefficients of the measures to the construct that the measures intended to measure (see Appendix).Chin [25] suggests that standardised paths should be at least 0.20 and ideally above 0.30 in order to be considered meaningful.A number of items in the model depicted smaller path coefficients (less than 0.2).This finding raised an issue regarding the significant of the items in the model, whether removing or modifying the model will result in better model fit.
According to Cenfetelli and Bassellier [23], the number of items has implications for the statistical significance and the magnitude of each item"s path coefficient.There is a probability that many of the items weights will be low in magnitude as well as statistically non-significant with a greater number of items, although these items had explicitly test for and exclude the possibility of multicollinearity.Formative measures essentially "compete" with one another to be explanatory of their targeted construct.It may be appropriate to remove non-significant measures to increase the likelihood of the remaining measures to explain the variance in the construct [21], however, researchers have be warned about the risk of changing the content of the model when removing measures [8].Thus, it is important to ensure that the construct is measuring the entire domain and content validity is preserved [9].Because of this reason, some researchers recommend retaining non-significant measures to retain content validity [30].
In order to observe the incremental change in R-square value when removing those measures with low path coefficients, several path estimate tests were conducted by excluding these measures one at a time.At the end of this test, we observed a decreased in the R-square value, thus leading to a conclusion that removing the measures has resulted in decreasing the explanatory power of the model.This observation indicates that some measures may not be a strong predictor to the construct, however, it is still significantly relevance with no indication of collinearity.Therefore, we argued that all 34 measures provide strong contribution to the IS-Impact construct and should be retained.
This study addressed the generalisability of the IS-Impact model when extending the model to a new context that is different in term of culture, language and type of system.The validity of the IS-Impact model in Malaysia context is confirmed.A new measure that was added in the original model is significant and relevant yielding a comprehensive model to evaluate impact of IS in Malaysia.Furthermore, with the validity of the IS-Impact model, this study presents validated instruments for both the English and Bahasa Malaysia versions.This indicates that the model is robust, and can be used across multiple contexts (package to custom, Australia to Malaysia, English to Bahasa Malaysia).
This study has several limitations but it should be noted that some of the limitations were control to minimize the differences between contexts.Sample was collected using several non-probability sampling techniques.Although this may introduce sampling bias, however, this study needs to identify the appropriate respondents to complete the questionnaire.Similar with the original work of IS-Impact model, the IS under study which is the unit of analysis, is a financial system.This system was chosen because it is commonly used type of system, has large number of users and being used by multiple level of cohort.Nevertheless, the study accomplished its purpose of testing the generalisability of the model and continuing the IS evaluation research by extending it in Malaysia context.Further research might investigate the validity of the model in other different context for example different type of organisations or different type of systems to present a more robust and standardised measurement model to evaluate the impact of information systems to organisation.

TABLE II Validity
Test for Formative Measurement Model