A meta-analysis of information system success: A reconsideration of its dimensionality
Introduction
Information quality, system quality (ease of use), perceived usefulness, and user satisfaction are criterion variables in information systems [IS] assessments. This paper calls them success surrogates. While much about success measurement eludes consensus, this belief prevails – each surrogate measures a unique concept. Expressed differently, success is multi-dimensional. To illustrate, software A is easier to use while software B provides more relevant information. The two goals are clearly distinguished suggesting separate aspects of success.
Nevertheless, this paper offers a different conclusion. Prior studies showed positive correlations between success surrogates – correlations that Cohen (1988) classifies as medium or large effects and considers exceptional. Here lies an inconsistency. Unique behavior produces anything but high positive correlation. Such correlation implies single dimensionality.
Given this paradox, parameter estimates from prior studies were gathered and analyzed in several ways. Previewing the results, single dimensionality wins and in fact wins easily. The paper concludes by discussing the consequences of single dimensionality and an avenue for further analysis.
Section snippets
Theory
Fig. 1 displays competing models. Fig. 1Aa and Ab exhibit multiple upstream-success-downstream paths while Fig. 1B possesses one causal path through a univariate construct.1 Fig. 1Aa is an example from the literature (Seddon, 1997) while Fig. 1Ab importantly possesses the
Meta-analysis
To compare models, a two-step process is typically followed: construct a sample matrix and fit models to that matrix. The matrix-construction step in this study was a meta-analysis which produced the correlation matrix in Table 1.4 It quantifies the pathways hypothesized in Fig. 1. This section describes that work with the next section describing the second step of analyzing
Justification
The key section of the report now commences. Three analyses are presented: a SEM comparison, an exploratory factor analysis of Table 1, and a data exposure to the individual correlations averaged in Table 1.
Implications
Does it matter? Under multi-dimensionality, one cannot speak of one scale being a better surrogate than another scale. Otherwise apples are compared with oranges. However with single dimensionality, comparison should occur. If surrogates measure the same construct, then they compete with one another. If they compete, then the ‘best’ surrogate becomes the lone criterion variable. It is proposed that correlation size (parameter or path coefficient) measure strength. If this is done, then past
Further research
If the disaggregated analysis is not persuasive, what next? One avenue, second-order factor models, is outlined here.10,11
Conclusion
If this study is persuasive, the conundrum is not resolved but only redefined. Past research was designed to produce multi-dimensional surrogates. Why are the surrogates single dimensional instead?
References (18)
- et al.
Structural equation modeling in practice: A review and recommended two-step approach
Psychol Bull
(1987) Total, direct, and indirect effects in structural equation models
Structural equations with latent variables
(1989)Statistical power analysis for the behavioral sciences
(1988)Analysis of correlation matrices using covariance structure models
Psychol Bull
(1989)Introduction to structural equation models
(1961)Methods for statistical data analysis of multivariate observations
(1977)- et al.
Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives
Struct Equ Model
(1999) - et al.
Perceived service quality and user satisfaction
Decis Sci J
(1994)
Cited by (8)
DeLone & McLean models of information system success: Critical meta-review and research directions
2020, International Journal of Information ManagementCitation Excerpt :Prior studies have striven to evaluate the dimensionality of ISS to some extent. Masoner, Lang, and Melcher (2011) examined multiple models representing the ISS dimensions and other variables that can influence or be influenced by ISS. Based on a meta-analytic comparison, Masoner et al. (2011) concluded that “only one common construct underlies” the ISS dimensions.
A further interpretation of the relational agency of information systems: A research note
2016, International Journal of Accounting Information SystemsCitation Excerpt :In the anthropocentric conceptualization of the agency, the technological components of IS are reduced to inert objectives that are not capable of action on their own (Orlikowski, 2007). Rather the technological components of IS only serve as tools that are manipulated by human beings to achieve desired goals (Arnold and Sutton, 2002; Masoner et al., 2011; Wheeler and Murthy, 2011; Azad and Faraj, 2013). Here, agency is defined in terms of use of the technology and human beings are those with the privilege to use technology to achieve AIS outcomes.
Impact of psychological traits on user performance in information systems delivering customer service: IS management perspective
2012, Decision Support SystemsCitation Excerpt :We developed a conceptual model (Fig. 1) that is grounded on IS-assisted customer services, organizational environments, and user psychological traits in the context of call centers. System satisfaction is one of the criterion variables [54], and most widely used measure in assessing the IS success [25,26]. Furthermore, it has been predominantly critical in the IS implementation as well [7].
Editorial: A dialectic on methodologies in AIS research
2011, International Journal of Accounting Information SystemsA model for measuring success of an organisational management system
2017, International Journal of Advanced Operations ManagementMeta-analysis of cultural correlation and product success
2016, Advanced Science Letters