Skip to main content
SpringerLink
Log in

Evaluating expert systems: a review of applicable approaches

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

This paper surveys various approaches for evaluating expert systems. More specifically, we categorize the applicable methods found in the literature as being qualitative, quantitative or a hybrid. The more seminal of these methods are described. The review is primarily intended for an audience of researchers and practitioners interested in understanding the scope and limitation of methods within each of the three approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Bailey, J. E. and Pearson, S. W. (1983). Development of a Tool for Measuring and Analysing Computer User Satisfaction.Management Science 29(5), 530–545.

    Google Scholar 

  • Baroudi, J. J. and Orlikowski, W. J. (1988). A Short-Form Measure of User Information Satisfaction: A Psychometric Evaluation and Notes on Use.Journal of Management Information Systems 4(4), 44–59.

    Google Scholar 

  • Boehm, B. W., Brown, J. R. and Lipow, M. (1976). Quantitative Evaluation of Software Quality.Proceedings of the Second International Conference on Software Engineering, pp. 592–605.

  • Buchanan, B. G. and Shortliffe, E. H. (Eds.) (1984).Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. Addison-Wesley, Reading, Massachusetts.

    Google Scholar 

  • Cohen, P. R. and Howe, A. E. (1988). How Evaluation Guides AI Research. COINS Technical Report 88-21, University of Massachusetts at Amherst.

  • Dehnad, K. (1990). Software Metrics from a User's Perspective.Journal of Systems Software 13(2), 111–115.

    Google Scholar 

  • Dickson, G. W., Senn, J. A. and Chervany, N. L. (1977). Research in Management Information Systems: The Minnesota Experiments.Management Science 23(9), 913–923.

    Google Scholar 

  • Doll, W. J. and Torkzadeh, G. (1988). The Measurement of End-User Computing Satisfaction.MIS Quarterly 12(2), 259–274.

    Google Scholar 

  • Ferrari, D. (1978).The Performance Evaluation of Computer Systems. Prentice-Hall, Englewood Cliff, New Jersey.

    Google Scholar 

  • Fox, M. S. (1990). AI and Expert System Myths, Legends, and Facts.IEEE Expert 5(1), 8–20.

    Google Scholar 

  • Galletta, D. F. and Lederer, A. L. (1989). Some Cautions on the Measurement of User Information Satisfaction.Decision Sciences 20(3), 419–438.

    Google Scholar 

  • Gaschnig, J., Klahr, P., Pople, H., Shortliffe, E. and Terry, A. (1983). Evaluation of Expert Systems: Issues and Case Studies. In Hayes-Roth, F., Waterman, D. A. and Lenat, D. B. (Eds.),Building Expert Systems. Addison-Wesley, Reading, Massachusetts, pp. 241–280.

    Google Scholar 

  • Ginsberg, A. (1988). Knowledge-base Reduction: A New Approach to Checking Knowledge Bases for Inconsistency and Redundancy.Proceedings of the Seventh National Conference on Artificial Intelligence (AAAI 88), v 2, pp. 585–589.

    Google Scholar 

  • Gutek, B. A. (1978). Strategies for Studying Client Satisfaction.Journal of Social Issues 34(4), 44–56.

    Google Scholar 

  • Hollnagel, E. (1989). Evaluation of Expert Systems. In Guida, G. and Tasso, C. (Eds.),Topics in Expert System Design Methodologies and Tools. Elsevier (North-Holland), Amsterdam, pp. 377–416.

    Google Scholar 

  • Ives, B., Olson, M. H. and Baroudi, J. J. (1983). The Measurement of User Information Satisfaction.Communications of the ACM 26(10), 785–793.

    Google Scholar 

  • Kumar, K. (1990). Post Implementation Evaluation of Computer-Based Information Systems: Current Practices.Communications of the ACM 33(2), 203–212.

    Google Scholar 

  • Lehner, P. E. (1989). Toward an Empirical Approach to Evaluating the Knowledge Base of an Expert System.IEEE Transactions on Systems, Man and Cybernetics 19(3), 658–662.

    Google Scholar 

  • Liebowitz, J. (1986). Useful Approach for Evaluating Expert Systems.Expert Systems 3(2), 86–96.

    Google Scholar 

  • Lindsay, R. K., Buchanan, B. G., Feigenbaum, E. A. and Lederberg, J. (1980).Applications of Artificial Intelligence for Organic Chemistry: The Dendral Project, McGraw-Hill, New York.

    Google Scholar 

  • Luger, G. F. and Stubblefield, W. A. (1989).Artificial Intelligence and the Design of Expert Systems. Benjamin/Cummings, Redwood City, California.

    Google Scholar 

  • Lyytinen, K. (1987). Different Perspectives on Information Systems: Problems and Solutions.ACM Computing Surveys 19(1), 5–46.

    Google Scholar 

  • McKerrow, P. (1988).Performance Measurement of Computer Systems. Addison-Wesley, Sydney.

    Google Scholar 

  • McLeod, R. Jr. and Bender, D. H. (1988). Perceptions of System Effectiveness as Viewed by Executives, Users, and Information Specialists.Proceedings of the Hawaii International Conference on Systems Sciences (3), Honolulu.

  • Moad, J. (1989). Asking Users To Judge IS.Datamation 35(21), 93–100.

    Google Scholar 

  • O'Keefe, R. M. (1989). The Evaluation of Decision-Aiding Systems: Guidelines and Methods.Information & Management 17(11), 217–226.

    Google Scholar 

  • O'Keefe, R. M., Balci, O. and Smith, E. P. (1987). Validating Expert System Performance.IEEE Expert 2(4), 81–90.

    Google Scholar 

  • O'Leary, T. J., Goul, M., Moffitt, K. E. and Radwan, A. E. (1990). Validating Expert Systems.IEEE Expert 5(3), 51–58.

    Google Scholar 

  • Pfleeger, S. L. (1987).Software Engineering: the Production of Quality Software. Macmillan, New York.

    Google Scholar 

  • Politakis, P. and Weiss, S. M. (1984). Using Empirical Evidence to Refine Expert System Knowledge Bases.Artificial Intelligence 22(1), 23–48.

    Google Scholar 

  • Rai, A. and Mendelow, A. (1989). Effectiveness of Information Systems Revisited.Proceedings of the 20th Annual Meeting of the Decision Sciences Institute, New Orleans, Florida, pp. 578–580.

  • Richer, M. H. (1988). An Evaluation of Expert System Development Tools. In Gupta, A. and Prasad, B. E. (Eds.),Principles of Expert Systems, IEEE Press, New York, pp. 405–421.

    Google Scholar 

  • Rothenberg, J., Paul, J., Kameny, I., Kipps, J. R. and Swenson, M. (1987). Evaluating Expert System Tools: A Framework and Methodology. DAPRA Report 3542, RAND Corporation.

  • Sackson, M. V., Varanelli, A. Jr. and Baugher, D. (1990). Expert System Validation.Proceedings of the Annual Meeting of the Decision Sciences Institute, v I, San Diego, California, pp. 367–375.

    Google Scholar 

  • Sassone, P. (1988). Cost Benefit Analysis of Information Systems: A Survey of Methodologies.ACM SIGOIS Bulletin 9(2 & 3), 126–133.

    Google Scholar 

  • Sharda, R., Barr, S. H. and McDonnell, J. C. (1988). Decision Support System Effectiveness: A Review and An Empirical Test.Management Science 34(2), 139–159.

    Google Scholar 

  • Sherwood-Smith, M. (1989). The Evaluation of Computer-Based Office Systems, Ph.D. Dissertation, Department of Computer Science, University College, Dublin.

    Google Scholar 

  • Srinivasan, A. (1985). Alternative Measures of Systems Effectiveness: Associations and Implications.MIS Quarterly 9(3), 243–253.

    Google Scholar 

  • Straub, D. W. (1989). Validating Instruments in MIS Research.MIS Quarterly 13(2), 147–170.

    Google Scholar 

  • Sviokla, J. J. (1989). PlanPower, XCON, and MUDMAN: An In-Depth Analysis of Three Commercial Expert Systems in Use. Working Paper 89-044, Harvard Business School, Boston.

    Google Scholar 

  • Swanson, E. B. (1982). Measuring User Attitudes in MIS Research: A Review.Omega 10(2), 157–165.

    Google Scholar 

  • Szewczak, E. J. (1990). A Research Framework for Evaluating MIS/DSS Effectiveness. Working Paper of the Wehle School of Business, Canisius College, Buffalo, New York.

    Google Scholar 

  • Turban, E. (1988). Review of Expert Systems Technology.IEEE Transactions on Engineering Management 35(2), 71–81.

    Google Scholar 

  • Turing, A. M. (1950). Computing Machinery and Intelligence.MIND — A Quarterly Review of Psychology and Philosophy LIX(236), 433–460.

    Google Scholar 

  • Waterman, D. A. (1986).A Guide to Expert Systems. Addison-Wesley, Reading, Massachusetts.

    Google Scholar 

  • Zmud, R. W. (1979). Individual Differences and MIS Success: A Review of the Empirical Literature.Management Sciences 25(10), 966–979.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Technology Division, School of Applied Science, Nanyang Technological University, Republic of Singapore

    Ravi S. Sharma

  2. Department of Management Sciences, Faculty of Engineering, University of Waterloo, Waterloo, Ontario, Canada

    David W. Conrath

Authors
  1. Ravi S. Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David W. Conrath
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sharma, R.S., Conrath, D.W. Evaluating expert systems: a review of applicable approaches. Artif Intell Rev 7, 77–91 (1993). https://doi.org/10.1007/BF00849078

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00849078

Key Words

Search

Navigation