Guideline for good evaluation practice in health informatics (GEP-HI)
Highlights
► The GPE-HI guideline supports systematic design, execution and management of evaluation studies. ► The GEP-HI guideline is applicable to different types of health informatics evaluation studies. ► The guideline has been developed through consensus seeking process and by collecting feedback. ► The GEP-HI serves as a basis for development of specific guidelines, e.g for usability studies.
Introduction
Development and use of health informatics applications offer tremendous opportunities to improve health care, its delivery and outcomes. There is an increasing political urge to implement available information technology (IT) solutions, as discussed by Refs. [1], [2], [3], [4]. However, there are also hazards and problems related to the use of IT in health care, for example IT may be inappropriately specified, unreliable, user-unfriendly or the organization may not be properly prepared to adopt IT within the clinical work flows and processes, as shown by Ref. [5]. Health IT may also be ill-functioning, for example inducing medical errors by presenting faulty displays of the electronic health records, see Ref. [6], or having negative impacts on the outcome of care in a specialised care unit, as documented by Ref. [7]. The effectiveness of health IT is still limited and inconsistent, across a wide range of fields, as reviewed by Ref. [8]. In the primary care the only benefits reliably documented on health information exchange were those on efficiency, including improved access to test results and other data from outside the practice, and decreased staff time for handling referrals and claims processing. Barriers included cost, privacy and liability concerns, organizational characteristics, and technical barriers, see Ref. [9]. This situation calls for evaluation to provide robust evidence about the impacts and actual efficiency, quality, and safety gains of health IT that can be achieved.
Evaluation is the means to assess the quality, value, effects and impacts of IT in the health care environment, as stated by Refs. [10], [11]. Evaluation is defined as the “act of measuring or exploring properties of a health information system (in planning, development, implementation, or operation), the result of which informs a decision to be made concerning that system in a specific context” [[12], p. 480]. The white paper [13] suggests extensive and systematic pre-implementation evaluation studies to inform difficult decisions prior to starting any programme initiative as well as post-implementation evaluation studies. The present contribution provides a guideline for conducting health IT evaluations. It goes beyond pre- and post-impact evaluation to support evaluation studies in general, formative as well as summative studies. Formative evaluation is performed throughout the systems lifecycle and it provides information for improving the system under development. Summative evaluation is focused on assessing the effect or outcome of the evaluation object at a certain point of time after implementation [14].
The exploratory workshop HIS-EVAL by European Science Foundation (ESF) was instrumental in identifying the state of affairs with respect to evaluation of health IT applications and in defining the necessary steps to further evaluations [12]. An important result from the workshop was the Declaration of Innsbruck [15] summarizing the importance of evaluation as:
“Health information systems are intended to improve the functioning of health professionals and organizations in managing health and delivering healthcare. Given the significance of this type of intervention, and the intended beneficial effect on patients and professionals, it is morally imperative to ensure that the optimum results are achieved and any unanticipated outcomes identified. The necessary process is evaluation and this should be considered an essential adjunct to design and implementation of information systems”.
Reflective deliberations at the HIS-EVAL workshop led to the conclusion of a need for two guidelines: One for structured reporting of evaluation studies (STARE-HI1) and one for good practice for planning and execution of evaluation studies (GEP-HI). The STARE-HI statement is finalized and published in Ref. [16]. It is endorsed by the board of the European Federation of Medical Informatics (EFMI) and adopted as an official document by the International Medical Informatics Association (IMIA), and is cited by the Equator network for good reporting of health informatics studies [17]. The present paper describes the GEP-HI guideline as well as its development process.
The current state of evaluation studies was explored in a review of problems and pitfalls [[11], pp. 243–323] resulting in a listing of several biases one should be aware of when designing and executing an evaluation study. These biases may result in incorrect results, wrong interpretations and under- or overestimation of the findings. In the review [11] many examples were found for most of the identified biases while it was difficult to find publications on high quality studies that properly addressed all of these potential biases. The conclusion is that evaluation of health IT is more than the deployment of a collection of methods, it is both a discipline and a specialty, and requires an overarching objectives-led design and operational management. Evaluation can be compared with health technology assessment (HTA), but HTA is based on a pre-defined formal framework and usually has a summative nature using general evaluation methods and approaches, often quantitative in nature, see Ref. [18]. Reviews of the health IT evaluations commissioned for the IMIA Yearbook showed the importance of evaluation, the challenges, and the currently very limited response, both in recognising integrated methodologies and in endorsing the importance of robust studies, as argued by Refs. [19], [20].
The need to develop and publish a guideline was identified in the health IT evaluation domain concurrent with the GEP-HI guideline development. In a literature review on discourses, dimensions and methods of health IT evaluation the conclusion was that the identified evaluation frameworks differed in terms of generality, specificity, timing related to system development phases and theoretical underpinning, see Ref. [21]. GEP-HI adds to this knowledge by presenting a guideline for design and execution of an evaluation study.
A Model for ASsessment of Telemedicine applications (MAST) was developed in an EU-project Metho-Telemed to give advice to users on what to consider before an evaluation study. The MAST-model lists aspects of telemedicine evaluation within seven domains of outcomes: Health problem and characteristics of the application, safety, clinical effectiveness, patient perspectives, economic aspects, organizational aspects and socio-cultural, ethical and legal aspects, see Ref. [22]. It is planned as a toolkit, a checklist of issues that need to be considered in evaluation. MAST is based on a HTA approach and focuses on telemedicine systems, not on health IT generally. In contrast, this GEP-HI guideline is wider and applicable to evaluation of health IT in general. Additionally, MAST does not consider the execution and management of an evaluation project.
Cusack et al. developed for the US Agency for Healthcare Research and Quality (AHRQ) a toolkit to provide step-by-step guidance for developing evaluation plans for health IT projects [23]. The toolkit assists evaluators in defining the goals for evaluation, what is important to stakeholders, what needs to be measured to satisfy stakeholders, what is realistic and feasible to measure, and how to measure these items. Examples are presented with suggested evaluation methods for each item. The toolkit is very useful from the methodological point of view and it can be applied within the GEP-HI guideline. The toolkit does not, however, give guidance on the evaluation project itself, how to manage it, how to carry out the project, or how to complete and report the study. Thus it details only part of the GEP-HI guideline.
Life cycle frameworks for evaluation have been proposed for example by Catwell and Sheikh [24] and Clarke et al. [25]. These are focused on how to evaluate health IT interventions while being designed, developed and deployed. These schematic models are formative and relate evaluation to the phases of the system development. The frameworks propose various evaluation measures and checkpoints during the system phases. The lifecycle evaluation frameworks are valuable tools to monitor the development process and the deployment of a new system. The frameworks do not, however, give guidance for the planning and execution of an evaluation project, and do not support evaluation as an independent activity.
Additional reports on various approaches and frameworks on evaluation are published, for example in Refs. [26], [27], [28], [29], [30], [31]. These frameworks can be applied together with the over-arching GEP-HI guideline as they provide support for methodical choices but do not deal with the overall design and execution of an evaluation study as do the GEP-HI guideline.
The objective of the GEP-HI guideline is to give advice on how to design and carry out evaluation studies in various health IT contexts. The guideline aims to be general and practical, and to provide evaluators with a set of structured principles for good evaluation practice. The principles are inspired by the best evaluation practices as implicitly and explicitly described in the literature. In the guideline we point at issues and list recommendations on how to design evaluation studies, how to make methodological choices, how to conduct studies, and how to define evaluation criteria at specific phases of the health IT applications’ life cycle.
Section snippets
Method
The method applied to develop GEP-HI guideline was a consensus-seeking process within the community of health IT evaluation experts. The primary authors of GEP-HI, all were participants of the ESF HIS-EVAL Workshop (see Ref. [15]) and active in the EFMI and IMIA working groups dealing with evaluation of health IT applications and HTA; and they all have long experience in planning and conducting evaluation studies [see for example Refs. [5], [11], [12], [16], [19], [20], [25], [30], [32], [33],
GEP-HI guideline for evaluation STUDIES
This GEP-HI guideline presents the essential aspects and activities to be taken into account in the design and execution of an evaluation study, including the study's management. The specific methods to be used are not explicitly mentioned. It is up to the users of the guideline to identify which method is applicable in their specific concrete situation, for instance by means of the latest edition of handbooks and textbooks like Refs. [11], [35], [49], [50], [51], [52], [53], [54], and other
Discussion
This guideline for good evaluation practice in health informatics was developed to support evaluators, health care professionals, decision makers and other health IT stakeholders in design, planning and execution of an evaluation study. The guideline gives support and advice on what to do and on aspects needing attention at each evaluation phase. Existing methodological textbooks will give the details of which methods to apply and how.
The guideline has been developed through an informal
Conclusions
A comprehensive lifecycle of phases has been developed, strengthened by world-wide iteration, to plan and execute health IT evaluations effectively. The phases contain some 60 detailed items, which are presented in relation to the evaluation study phases. When designers and executers of evaluation studies address these items, the plan, structure, objectives and results of the studies will become more robust and consequently the studies contribute an important step towards evidence-based health
Authors’ contributions
This GEP-HI work has been a shared activity of EFMI's Working Group EVAL and IMIA's Working Group on Technology Assessment and Quality Development. The idea of this standard/guideline contribution crystallised during the ESF working conference 2003 in Innsbruck, organized by E Ammenwerth and attended – among others – by J Brender, P Nykänen, M Rigby, and J Talmon. P Nykänen and J Brender drafted a first list of issues, while in later versions P Nykänen and J Brender together as the responsible
Conflict of interests
P Nykänen and J Brender are co-chairs of the EFMI working group on Evaluation (EVAL) that has E Ammenwerth as the chairperson. N de Keizer is the chairperson of IMIA's working group on Technology Assessment and Quality Development. J Talmon has been chair of this IMIA working group till May 2008, and J Brender was his co-chair until the end of 2007. Besides this all authors have positions as university employed researchers with no commercial interests and no conflicting interests financially or
References (63)
- et al.
Effectiveness of telemedicine: a systematic review of reviews
Int. J. Med. Inform.
(2010) Evaluation of health information systems: beyond efficiency and effectiveness
Comput. Biol. Med.
(2002)- et al.
Visions and strategies to improve evaluation of health information systems. Reflections and lessons based on the HIS-EVAL workshop in Innsbruck
Int. J. Med. Inform.
(2004) - et al.
A framework for mandatory impact evaluation to ensure well informed public policy decisions
Lancet
(2010) - et al.
STARE-HI—statement on reporting of evaluation studies in health informatics
Int. J. Med. Inform.
(2009) - et al.
Beyond effectiveness: the evaluation of information systems using a comprehensive health technology assessment framework
Comput. Biol. Med.
(2002) - et al.
Investigating evaluation frameworks for health information systems
Int. J. Med. Inform.
(2008) - et al.
A methodology for evaluation of knowledge-based systems in medicine
Artif. Intell. Med.
(1994) Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism
Int. J. Med. Inform.
(2001)CHEATS: a generic information communication technology (ICT) evaluation framework
Comput. Biol. Med.
(2002)
An evaluation framework for health information systems: human, organization and technology-fit factors (HOT-fit)
Int. J. Med. Inform.
The VATAM guidelines
Int. J. Med. Inform.
Inventory of validation approaches in selected health telematics projects
Int. J. Med. Inform.
Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory
J. Am. Med. Inform. Assoc.
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses
Lancet
ICT for Health & i2010—Transforming the European Healthcare Landscape—Towards a Strategy for ICT for Health, June 2006
American Recovery and Reinvestment Act of 2009
Safe electronic health record use requires a comprehensive monitoring and evaluation framework
JAMA
Bad health informatics can kill—is evaluation the answer. Editorial
Methods Inf. Med.
Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system
Pediatrics
Systematic review of health information exchange in primary care practices
J. Am. Board Fam. Med.
Handbook of Evaluation Methods for Health Informatics
Evaluation Thesaurus
Evaluation—the Cinderella science of ICT in health
Evaluation and implementation: a call for action
Health information technology evaluation toolkit: 2009 Update (Prepared for the AHRQ National Resource Center for Health Information Technology under Contract No. 290-04-0016), AHRQ Publication No. 09-0083-EF
Cited by (83)
Identifying and addressing digital health risks associated with emergency pandemic response: Problem identification, scoping review, and directions toward evidence-based evaluation
2022, International Journal of Medical InformaticsCitation Excerpt :Furthermore, there is also an opportunity for such checklists to be created and validated by professional or academic bodies. Where a specific situation or deeper innovation requires it, a more comprehensive study on impact and outcomes may be needed, and guidance is available, including [113-117]. As addressed in the Results section, systematic collaboration among different digital health stakeholders during an emergency is difficult, but can be highly beneficial and impactful.
Evaluating health information technologies: A systematic review of framework recommendations
2020, International Journal of Medical InformaticsThe implementation of an integrated e-leprosy framework in a leprosy control program at primary health care centers in Indonesia
2020, International Journal of Medical InformaticsTask-technology fit model: Modelling and assessing the nurses’ satisfaction with health information system using AI prediction models
2024, International Journal of Healthcare Management