Abstract
Over the past ten years there has been an increasing number of publications on lithe evaluation of medical expert systems parallel to the increase in theoretical papers on the evaluation methodology itself. A brief perusal of the most relevant European conference proceedings over the past two years is shown in Table 1. (Fox et al 1987, Serio et al 1987, Hansen et al 1988 and Rienhoff et a1 1988).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Books
I Barber. B., Cao Dexian, Quin Dulie, Wagner, G. (Eds) MEDINFO 89’. North Holland (1989).
II Fox, J., Fieschi, M., Engelbrecht, R. (Eds) AIME 87’. Lecture Notes in Medical Informatics, 33 (1987). Springer Verlag.
IIIHansen, R., Solheim, B., O’Moore, R.R., Roger, F.H. (Eds) MIE ‘88. Lecture Notes in Medical Informatics 35, Springer Verlag pp 1–764 (1988).
IVRienhoff, O., Piccolo, U., Schneider, B. (Eds) Expert Systems and decision support in medicine. The Peter Reichertz Memorial Conference. Lecture Notes in Medical Informatics, 36, pp 1–591 (1988). Springer Verlag.
V Serio, A., O’Moore, R.R., Tardini, A., Roger, F. (Eds) Proceedings MIE 87’. Vol I-III, EDI Press Rome, pp 1–1644 (1987).
General
Arborelius, E., Timpka, T Study of the Practitioner’s knowledge need and use during Health Care consultations. Part 2: The Dilemma Spectrum of the GP. In I. 101–105 (1989).
Beck, J.R. Laboratory Decision Science applied to Chemometrics: Strategic testing for thyroid function. Clin. Chem. 32, 1707–1713 (1986).
Brannigan, V. The regulation of medical computer software as a ‘device’ under the food, drug and cosmetic act. Computer Meth. Prog. in Biomed. 25, 219–229 (1987).
Campbell, J.A. The expert computer and professional negligence: Who is liable? Ed. Yazdani, M., Narayanan, A. In Artificial Intelligence: Human Effects. Pub. Ellis, Horwood, Chichester U.K. (1984).
Cannataci, J.A. Liability and responsibility for expert systems. Complex No. 588. Norwegian Research Centre for Computers and Law. Univ. Oslo 2, Norway (1988).
Gremy, E Persons and computers in medicine and health. Meth. Inform. Med. 27, 3–9 (1988).
Kilian, W. Liability for deficient medical expert systems-keynote address. Expert systems and decision support in Medicine. 33rd Annual Meeting of G.M.D.S. Hannover (1988) ( Available G.M.D.S.).
Melhorn, J.M. Current attitudes of medical personnel towards computers. Comp. Biomed. Res 12, 327–334 (1979).
Nolan, J., Brosnan, P., Murnane, L., Boran, G., Breslin, A., Grimson, J., Cullen, M., O’Moore, R.R. A PC based decision support/patient management system for thyroid disease. In AIME 89’ Lecture Notes in Medical Informatics. 38, Ed. Hunter, J. Cookson, J., Wyatt, J. 189–198 (1989).
O’Moore, R.R. The effectiveness of decision support systems in clinical chemistry. In Progress in Biological Function Analysis. Ed. Van Bemmel, J., Michel, J. & Willems, J. 269–276 (1988a). North Holland.
O’Moore, R.R. Decision support based on laboratory data. Meth. Inform. Med. 27, 187–190 (1988b).
Spackman, K.A. & Connolly, D.P. Knowledge based systems in laboratory medicine and pathology. Archiv. Pathol. Lab. Med. 111, 116–119 (1987).
Teach, R.L., Shortliffe, E.H. An analysis of physician attitudes regarding computer based clinical consultation systems. Comp Biomed Res 14, 542–558 (1981).
Tusch, G., Bernauer, J., Gubennatis, G., Rading, M. Knowledge acquisition using syntactic time patterns. In AIME 89’ Lecture Notes in Medical Informatics, 38. Ed. Hunter J., Cookson, J., Wyatt, J. 315–324, (1989)
Van Bemmel, J. Systems evaluation for the health of all. In III. 27–34 (1988a).
Van Bemmel, J.H. Decision support in Medicine: Comparative methodology and impact on the medical curriculum. In IV 3–19 (1988b).
Young, F.E. Validation of medical software: Present policy of the food and drug administration. Annals of Internal Medicine 106, 663–667 (1987).
Evaluations
Adams, I.D., Chan, M., Clifford, P.C., Cooke, W.M., Dallos, V, De Dombal F.T., Edwards, M.H., Hancock, D.M., Hewett, D.J., McIntyre, N.M., Somerville, P.G., Spiegelhalter, D.J., Wellwood, J., Wilson, D.H. Computer aided diagnosis of acute abdominal pain: A multi centre study B.M.J. 293, 800–804 (1986).
Adlassnig, K.P., Koarz, G., Scheithauer, W Present state of medical expert system–CADIAG.2 Meth. Inform Med 24, 13–20 (1985).
Adlassnig, K.P. The application of ROC curves to the evaluation of medical experts systems. In V, 951–957 (1987).
Aitkins, J.S., Kunz, J.C., Shortliffe, E.H., Fallat, R.J. PUFF: An expert system for interpretation of pulmonary function data: Comp. Biomed. Res. 16, 199–208 (1983).
Akhavan-Hedari, M., Adlassnig, K.P. Preliminary results on Cadiag-2/Gall: A diagnostic consultation system for Gall Bladder and biliary tract disease. In III, 662–666 (1988).
Barnett, G.O., Cimino, J.J., Hupp, J.A., Hoffer, E.P. DXPLAIN–An evolving diagnostic decision support system. JAMA 258, 67–74 (1987).
Berner, E.S., Brooks, C.M. Needs assessment for computer based medical decision support systems in Proceedings XII SCAMC Washington IEEE p 232–236.
Botti, G., Michel, C., Proudhon, D., Fieschi, D., Joubert, M., Fieschi, M. Feasibility study of the expert system Sphinx–In V, 957–966 (1987).
Botti, G., Joubert, M., Fieschi, D., Proundhon, H., Fieschi, M. Experimental use of the medical expert system SPHINX by General Practitioners: Results and analysis. In I, 67–71 (1989).
Bowen, T, Payling, L. Expert systems for performance review. J. Opl. Res. Soc. 38, 929–934 (1987).
De Bliek, R., Friedman, C.P., Blaschke, T.F., France, C.L., Speedie, S.M. Practitioner preferences and receptivity for patient specific advice from a therapeutic monitoring system. In proceedings of XII SCAMC Meeting, Washington, IEEE p 225–228 (1988).
Diamond, G.A., Staniloff, H., Forrester, J. Computer assisted diagnosis in the non invasive evaluation of patients with suspected coronary heart disease. J. Am. Coll. Cardiol. 1, 444–455 (1983).
Engelbrecht, R., Potthof, P., Schwefel, D. Expert systems in medicine: Results from a technology assessment study. In DIAC-87 Directions and Implications–Computer professionals for social responsibility, 125–133 (1987).
Engelbrecht, R., Schaaf, R., Lewis, M. Assistance in medical treatment and treatment analysis with an interactive drug consultation system. In I, 253–256 (1989).
Fox, J., Myers, C.D., Greaves, M.F., Pegram, S. Knowledge acquisition for expert systems: Experience in Leukaemia diagnosis. Meth Inform Med 24, 65–72 (1985).
Gorry, G.A., Sliverman, H., Pauker, S.G. Capturing clinical expertise–a computer program that considers clincial response to digitalis. Am. J. Med. 64, 452–460 (1978).
Habbema, J.D., Hilden, J., Bjerregaard, B. The measurement of performance in probabilistic, diagnosis V general recommendations, IV. Utility considerations. Meth. Inform. Med. 20, 80–96 and 97–100 (1981).
Hammersley, J.R., Cooney, K. Evaluating the utility of available differential diagnosis systems in Proceedings XII SCAMC meeting (1988), Washington, D.C. IEEE, p 220–224.
Hickham, D.H., Shortliffe, E.H., Bischoff, M.B., Scott, A.C., Jacobs, C.D. The treatment advice of a computer based cancer chemotherapy Protocol Advisor–Ann. Int. Med. 103, 928–936 (1985).
Kendall, R.I., Ulinski, D.E., Richardson, L.D., Bradley, C.A., Parl, F.F. Computer assisted interpretative reporting with trend analysis of creatinine kinase and lactate dehydrogenase isoenzymes. Am. J. Clin. Path. 79, 217–222 (1984).
Kingsland, L.C. Evaluation of medical expert systems: Experience with the AI/RHEUM knowledge based consultant system in Rheumatology. In selected topics in medical artificial intelligence. Ed. P.L. Miller pp 212–221. Springer Verlag (1988).
Kingsland, L., Sharp, G., Capps, J., Benge, D., Kay, D., Reese, P. Hazelwood, S., Lindberg, D. Testing of a criteria based consultant system in Rheumatology. In MEDINFO 86 Ed. Van Bemmel J., Ball, M.J., Wigertz, 0.514–157 (1986). North Holland.
Lavril, M., Chatellier, G., Degoulet, P., Jeunemaitre, X., Menard, J., Rovani, C. ARTEL: An expert system in hypertension for the general practitioner. In IV, p 314–321 (1988).
Lemonnier, P., Adlassnig, K.P., Horak, W, Hay, U. Hepatitis serology findings. In III, 636–670 (1988).
Michel, C., Botti, Fieschi, M., Joubert, M., SanMarco, J., Casanova, P. Validation of a knowledge base intended for general practitioners to assist in treatment of diabetes. In Medinfo 86 Ed Salamon, R. Blum, B., Jorgensen, M. North Holland 122–127 (1986).
Miller, R.A., Pople, H.E., Myers, J.D., INTERNIST-I. An experimental computer based diagnostic consultant for general internal medicine. New Eng. J. Med. 307, 468–476 (1982).
Myers, J.D. The computer as a diagnostic consultant, with emphasis on laboratory data. Clin. Chem. 32, 1714–1718 (1986).
McDonald, C.J., Hui, S.L., Smith, D.M., Tierney, W.M., Cohen, S.J., WeinbergerM., McCabe G.P. Reminders to physicians from an introspective computer medical record: A two year randomized trial Ann Int. Med, 100, 130–138 (1984).
Nakache, J.P., Gueguen, A., Dougados, M., Nguyen, M. Evaluation and Validation of a Functional Index in Ankylosing Spondylitis. In II, 229–238 (1987).
Nelson, S.J., Blois, M.S. et al. Evaluating reconsider–a computer program for diagnostic prompting J. Med. Systems 9, 379–389 (1985).
Potthof, P., Schwefel, D., Rothemund, M., Engelbrecht, R., van Eimeren, W, Expert Systems in Medicine. Int. J. Technology Assessment, 4, 121–133 (1988).
Pryor, T.A., Gardner, R.M., Clayton, P.D., Warner, H.R. The HELP system. J. Med. Sys. 7, 87–102 (1983).
Quaglini, S., Stefanelli, M., Barosi, G., Berzuini, A. Evaluating the performance of anaemia. In II,229–238 (1987).
Reggia, J.A. Evaluation of medical expert systems: Case study in performance assessment in proceedings IX SCAMC Baltimore IEEE 287–291 (1985).
Richardson, TThe effect of computer generated comments on the distribution of anticonvulsant concentrations. Ann. Clin. Biochem. 21, 184–187 (1984).
Rovani, C., Jeunmaitre, X., Degoulet, P., Sauguet, D., Aime, F, Lavril, M., Devries, C., Chatellier, G., Plovin, P.F., Corvol, P. Worst situation evaluation of an expert system in hypertension management. In V, 967–973 (1987).
Schewe, S., Scherrman, W, Gierl, L. Evaluation and measurement of benefit of an expert system for differential diagnosis in Rheumatology. In IV,36, 351–354. (1988).
Schneider, J., Renner, R., Engelbrecht, R., Piwernetz K. DIAMON: First Evaluation Results. In I, 222–225 (1989).
Shamsolmaali, A., Collinson, R, Gray, T.G., Carson, E.R., Cramp, D.G. Implementation and evaluation of a knowledge based system for the interpretation of laboratory data. In AIME 89’ Lecture notes in Medical Informatics. Ed. Hunter, J., Cookson, J., Wyatt, J. 38, 167–176 (1989).
Soula, G., Thirion, X., San Marco, J.L., Vialettes, B., Guliana, J., Navez, I. A multicentred validation of the fuzzy expert system Protis. In III, 647–651 (1988).
Spitzer, R.L. & Endicott, J. DIAGNO II: Further development of a computer program for psychiatric diagnosis. Am. J. Psychiat. 125, 12–21 (1969).
Tierney, W.M., McDonald, C.J., Hui, S.L., Martin, D.K. Computer prediction of abnormal test results. JAMA 259, 1194–1198 (1988).
Tusch, G., Bernauer, J., Gubernatis, G., Rading, M. A Knowledge-Based Decision Support Tool for Liver Transplanted Patients. In I, 131–135 (1989).
Weiss, S.M., Kulikowski, C.A., Galen, R.S. Representing expertise in a computer program: The serum protein diagnosis program. J. Clin. Lab. Auto 3, 383–387 (1983).
Wyatt, J. The evaluation of clinical decision support systems: a discussion of the methodology used in the Acorn project. In II, 229–238 (1987).
Wyatt, J. Lessons Learnt from the Field Trial of ACORN, an Expert System to Advise on Chest Pain. In I, 111–115 (1989).
Yu, V.L., Fagan, L.M., Wraith, S.M., Clancey, W.M., Scott, A.C., Hannigan, J., Blum, R.L., Buchanan, B.G., Cohen, S.N. Antimicrobial selection by computer a blinded evaluation by infectious disease experts. JAMA 242, 1279–1282 (1979).
Yu, V.L. Evaluating the performance of a computer based consultant. Comp. Prog. Biomed., 9, 95–102, (1979).
Theoretical papers on evaluations
Bonnet, A., Haton, J.P., Truong-Ngoc, J.M., Howlett, J. Validation of expert systems. In expert systems principles and practice pp 168–183, Prentice-Hall (1988).
Brender, J., McNair, P Watch the system. An opinion on user validation of computer based decision support systems in Clinical Medicine. In I, 275–279 (1989).
Cohen & Howe. How evaluation guides AI research. A.I. Magazine Winter: 35–43 (1988).
De Dombal, F.T. Towards a more objective evaluation of computer aided decision support systems–in MEDINFO 83’. Ed Van Bemmel, J., Ball, M.J., Wigertz, O. 436–439 (1983).
Fieschi, M., Joubert, M. Some reflections of the evaluation of Expert Systems in Medicine. Meth. Inform. Med. 25, 15–21 (1986).
Gashnig, J., Klahr, P., Pople, H., Shortliffe, E., Terry, A. Evaluation of expert systems. In Building expert systems. Ed. Hayes Roth et al. Pub. Addison Wesley pp 241–280 (1983).
Gjorup, T The KAPPA coefficient and the prevalence of a diagnosis. Meth. Inform. Med. 27, 184–196 (1988).
Gottinger, H.W. Technology assessment and forecasting of medical expert systems (MEST) Meth Inf. Med. 27, 56–66 (1988).
Grant, A., Parker Jones, C., White, R., Cramp, D., Barreiro, A., Mira, P, Artal, A., Montero, J. Evaluation of knowledge based systems from the user perspective. This volume, 312–324 (1991).
Kulikowski, C. Medical expert systems: Issues of validation, evaluation and judgement in policy issues. In Information and communication technologies in medical applications IEEE, 45–56 (1988).
Liebowitz, J. Useful approach for evaluating expert systems. Expert systems 3, 86–96 (1986).
Lundsgaarde, H.P. Evaluation medical expert systems Social Sc. Med. 24, 805–819 (1987).
Miller, P.L. Evaluation of artificial intelligence systems in medicine. Proceedings IX SCAMC, Washington, D.C. IEEE 281–286 (1985).
Miller, P.L. Evaluation of artificial intelligence systems in medicine. Comp. Prog. Method Biomed, 22, 5–11 (1986).
Miller, P.L. Goal directed critiquing by computer: Ventilation measurement. Comp. Biomed. Res. 18, 422–3 (1985).
Miller, P.L. Expert critiquing systems. Practice based medical consultation by computer (1986). Springer Verlag.
Nykanen, P (Ed.). Issues in evaluation of computer-based support to clinical decision making - Sydpol Working Group 5. The Norwegian Computing Centre, Oslo.
O’Keefe, R.M. Balci, O., Smith, E.P. Validating expert systems performance. IEEE Expert Winter 81–89 (1987).
Pryor, D.B., Barnett, O., Gardner, R.M., McDonald, C., Stead, W.W. Measuring the value of information systems in proceedings VIII SCAMC 1984, IEEE 26–28 (1984).
Rossi-Mori, A., Ricci, F.L. On the assessment of medical expert systems. In expert systems and decision support in medicine. In IV 292–297 (1988).
Rossi-Mori, A., Pisanelli, D.M., Ricci, F.L. The Role of Knowledge Based Systems in Medicine, Epistemological and Evaluation issues. This Volume, 291–303 (1991).
Rothschild, M.A., Miller, P.L., Fisher, P.R., Weltin, G.G., Sweet, H.A. Confronting subjective criteria in the evaluation of computer based critiquing advice. In proceedings XII SCAMC, Washington, IEEE, 220–224 (1988).
Shortliffe, E.H., Clancey, W.J. Anticipating the second decade. In readings in medical artificial intelligence. Eds. Clancey, W.J. & Shortliffe, E.H. p 469 (1984). Addison Wesley.
Shortliffe, E.H. Testing reality - the introduction of decision support technologies for physicians. Meth. Inform. Med 28, 1–5 (1989).
Sorgaard, P Evaluating expert systems prototypes. In 9th Scandinavian Seminar on development of expert systems, Bastad (1986).
Spiegelhalter, D. Evaluation of clinical decision aids, with an application to a system for dyspepsia. Statistics in Med., 207–216 (1983).
Whitebeck, C., Brook, R. Criteria for evaluating a computer aid to clinical reasoning. J. Med. Philos. 8, 51–65 (1983).
Wyatt, J., Spiegelhalter, D. Evaluating medical decision aids: What to test, and how. This volume 274–290 (1991).
Validation of knowledge base
Bahill, A.T., Jafer, M., Moller, R.F. Tools for extracting knowledge and validating expert systems. IEE 857–862 (1987).
Barachini, F., Adlassnig, K.P. CONSED: Medical knowledge base consistency checking–In V, 974–980 (1987).
Butler, K.A. Application of correlation measures for validating structured selectors. In proceedings of III conference on A.I. applications. Washington IEEE 327–330 (1987).
Fontaine, D., Le Beux, P, Strauss, A., Morizet, P An approach for maintaining the coherence in a medical knowledge base. In I, 44–48 (1989).
Gissi, P Logical checks in a rule based medical decision support system. In V, 981–985 (1987).
Green, C.J., Keyes, M.M. Verification and validation of expert systems. In Proceedings Western Conference on expert systems Anaheim. IEEE Computer Society 38–43 (1987).
Indurkhya N., Weiss, S.M. Models for measuring performance of medical expert systems. Artificial Intel. Med. 1, 61–70 (1989).
Mars, N.J., Miller, P.L. Knowledge acquisition and verification tools for medical expert systems. Medical Decision Making 7: 6–11 (1987).
Shwe, M.A., Tu, S.W., Fagan, L.M. Validating the knowledge base on a therapy planning system. Meth. Inform. Med., 28, 36–50 (1989).
Wigertz, O.B., Clayton, P.D., Huag, P.J., Pryor, T.A. Design of knowledge based systems for multiple use truth maintenance and knowledge transfer. In V, 987–991 (1987).
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1991 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
O’Moore, R., Engelbrecht, R. (1991). The Evaluation of Medical Decision Support and Expert Systems: Reflections on the Literature. In: Talmon, J.L., Fox, J. (eds) Knowledge Based Systems in Medicine: Methods, Applications and Evaluation. Lecture Notes in Medical Informatics, vol 47. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-08131-0_21
Download citation
DOI: https://doi.org/10.1007/978-3-662-08131-0_21
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-55011-2
Online ISBN: 978-3-662-08131-0
eBook Packages: Springer Book Archive