Quality metrics for detailed clinical models

https://doi.org/10.1016/j.ijmedinf.2012.09.006Get rights and content

Abstract

Objective

To develop quality metrics for detailed clinical models (DCMs) and test their validity.

Methods

Based on existing quality criteria which did not include formal metrics, we developed quality metrics by applying the ISO/IEC 9126 software quality evaluation model. The face and content validity of the initial quality metrics were assessed by 9 international experts. Content validity was defined as agreement by over 70% of the panelists. For eliciting opinions and achieving consensus of the panelists, a two round Delphi survey was conducted. Valid quality metrics were considered reliable if agreement between two evaluators’ assessments of two example DCMs was over 0.60 in terms of the kappa coefficient. After reliability and validity were tested, the final DCM quality metrics were selected.

Results

According to the results of the reliability test, the degree of agreement was high (a kappa coefficient of 0.73). Based on the results of the reliability test, 8 quality evaluation domains and 29 quality metrics were finalized as DCM quality metrics.

Conclusion

Quality metrics were validated by a panel of international DCM experts. Therefore, we expect that the metrics, which constitute essential qualitative and quantitative quality requirements for DCMs, can be used to support rational decision-making by DCM developers and clinical users.

Highlights

► A set of quality metrics for DCMs have been developed and validated. ► Using these DCM quality metrics, any DCMs may be compared and evaluated in a systematic and objective manner. ► The quality metrics for DCM also provide a qualitative and quantitative requirement to DCM developers, users and evaluators.

Introduction

To exchange and interpret clinical information consistently between electronic health record (EHR) systems, the data structures used to represent distinct items of information need to be standardized [1], [2], [3], [4], [5]. These data structure definitions, known as detailed clinical models (DCMs), specify how a particular kind of entry within an EHR has to be organized, for example, which clinical terms apply, what value range is valid, which measurement units may be used, which data items within the structure require a value [6], [7]. The clinical element model by Intermountain Healthcare [8], [9], templates defined by HL7 International [10], [11], archetypes defined by the openEHR Foundation [12], [13] and ISO 13606, the Clinical Information Model in the Netherlands [14], and the Clinical Content Model by the Center for Interoperable EHR in South Korea [15] are examples of published DCM formalisms that can be used to organize the clinical content of an EHR interoperability message and an EHR repository, share decision logic, and build a data capture form of a clinical application. Each instance of a DCM dictates how a corresponding generic EHR representation is to be used to represent particular types of clinical information. Examples of DCM instances might include representations for documenting a pain symptom, heart sounds, liver function tests, a prescribed drug, or a chest X-ray report.

Whereas DCMs have the potential to improve clinical decision support and clinical documentation in EHR system, the critical challenge is to identify the qualitative and quantitative requirements of DCMs. A few studies have suggested some quality requirements for DCMs (Table 1). For the flexibility and scalability of DCMs, general requirements for the system in which clinical data models are implemented demand the following [16], [17]: (1) the addition of elements and attributes to the clinical model without the necessity of changing the underlying software or database schema; (2) use an existing formalism/syntax for the representation of the model; (3) tight binding of model attributes to standard terminology systems; and (4) the existence of a mechanism for stating ‘negation’. General principles of good modeling include: (1) adoption of standard terminologies for use in the models; (2) representing the models in standard modeling languages; (3) sharing and approving the DCMs with a community of clinical experts; (4) define decision modules that reference the models. DCM quality criteria [18] have also been proposed where the following qualities were identified as being important requirements of a good DCM: usefulness, desirability, the degree of use/acceptance in clinical services, reusability, the quality of clinical content, the degree of clinician introduction/validation, the use of vocabulary, mapping to information models, applicability, application to other technologies, and maintenance.

Principles for the development of DCMs [19] can be classified as principles pertaining to the structure of the DCM, principles for creating the DCM content, and principles for the DCM development process. The principles that pertain to the structure of DCMs contains information about the language formalism, description of binding of attributes to standard terminologies, a strategy for supporting semantic links among DCM instances, the definition of standard data types, and the description of standard units of measure. The principles for DCM content creation emphasize the granularity, reusability, correctness, and comprehensiveness of the models. Principles for the DCM development process emphasize evidence based model development, the need for proper use cases, use of meta data to track changes, and compliance to the syntax of the modeling language.

Archetype representation requirements [20] published in ISO 13606-2 are focused more on the technological aspects of models. They are divided into requirements for archetype definition, archetype node constraints, and data value constraints. EuroRec, an organization that certifies EHRs in the European Union, led an EC funded research project to develop criteria for the quality classification of EHR systems [21]. The research resulted in a set of archetype quality criteria, which covered administrative, clinical, technological, information management, and repository operation requirements. Furthermore, they emphasized the use of standard terms and modeling language, the construction of repositories for DCM sharing, and the importance of metadata. Among the clinical requirements, the requirement for clinical use suggests the listing of accurate use patterns of clinical concepts, specification of whether the corresponding archetype is used in a specific workflow, description of subject population groups, as well as expert groups using an archetype, etc.

These published requirements and criteria are valuable sources as a starting place to define the good quality characteristics of DCMs (Table 1). However, these existing quality criteria have different levels of detail. Furthermore, these criteria have not been specified to the level of precision that is needed to undertake formal and objective evaluation of the quality of DCMs. Current DCM criteria cannot compare the quality of DCMs due to the absence of validated and reliable measurable characteristics. There is, therefore, still a need to develop quality metrics that can be used to objectively quantify quality criteria of DCMs.

Section snippets

Objective

This study was conducted to develop objective, reliable, and reproducible quality metrics for DCMs drawing on the published quality criteria referenced above, and to test their validity.

Materials and methods

The study was conducted in four phases. Phase 1 developed quality metrics; phase 2 tested the validity of the quality metrics; phase 3 tested the reliability of the quality metrics; and phase 4 finalized the quality metrics. Fig. 1 shows the applied methods of the study (Fig. 1).

Structure definition

Among the 6 domains used in AGREE, we included the following 4: the scope and purpose, stakeholder involvement, rigor of development, and clarity and presentation. We excluded the following 2 domains: applicability and editorial independence since these domains are not relevant in terms of guideline specificity. An attempt was made to map all of the archetype requirements in the published documents listed earlier to the four chosen AGREE domains, but some of the archetype requirements did not

Discussion

This study has developed DCM quality metrics with high validity and reliability for the first time with the collaboration of an international expert panel. This study started with criteria found in the published literature, but the criteria were consolidated and converted into a coherent set of metrics that could be objectively evaluated, which distinguishes this study from previous work. This set of criteria can now serve as a basis for practical use and implementation and for further research

Conclusion

A set of quality metrics for DCMs were developed and validated using existing published quality criteria and an international panel of experts. Given that the metrics were validated by a panel of DCM experts, we expect them to be used to support rational decision-making by DCM developers who will now have the essential qualitative and quantitative quality requirements to use as guidelines as they create new content. Clinical users can then use the quantitative assessments as they select models

Author's contributions

Sun-Ju Ahn contributed in the design and conduct of this research and the writing of this manuscript. Stan Huff contributed as a panel member and provided valuable comments to the study. Yoon Kim is an adviser to this paper. He provided valuable comments to the study. Dipak Kalra contributed as a panel member and provided valuable comments to the study.

Competing interests

None of the authors have any competing interests to declare.

Summary points

What was already known on the topic before the study?

  • Detailed clinical model supports health information exchange and reusability.

  • In order for DCMs to adequately support the EHR documentation needs of clinical practice, to be endorsed by clinical professional bodies and health services, and to be adopted by vendors, these models have to be of good quality, trusted and, in the future, certified.

  • To implement DCM in clinical

References (33)

  • S.M. Huff et al.

    Integrating detailed clinical models into application development tools

    Stud. Health Technol. Inform.

    (2004)
  • W. Goossen et al.

    Detailed clinical models: a review

    Healthcare Inf. Res.

    (2010)
  • S.M. Huff et al.

    An event model of medical information representation

    JAMIA

    (1995)
  • C.G. Parker et al.

    Detailed clinical models for sharable, executable guidelines

    Stud. Health Technol. Inform.

    (2004)
  • S. Garde et al.

    Archetype-based knowledge management for semantic interoperability of electronic health records

    Stud. Health Technol. Inform.

    (2009)
  • Cited by (11)

    • A framework for understanding process interoperability and health information technology

      2016, Health Policy and Technology
      Citation Excerpt :

      The lesson is we cannot implement automated processes to support care delivery without ensuring the processes themselves have sufficient maturity to benefit from the automation. This research is complementary to existing work on clinical and data models for interoperability such as [18,19] as well as frameworks such as AIDA [20]. While robust data models are the backbone of interoperable systems, this study illustrated that data issues are often superseded by people and process issues.

    • Requirements for clinical information modelling tools

      2015, International Journal of Medical Informatics
      Citation Excerpt :

      Two studies of the best practices internationally in the human and organisational processes for defining CIMs were previously performed by the authors based either on a systematic review of published experiences [11] and a qualitative analysis of the experiences from a set of 20 international experts [12]. These results are complementary to previous experiences detailing requirements for CIMP [4] and those studies focused on the definition of metrics for CIMs [13]. Clinical information modelling tools (CIMT) are software platforms and applications designed to support the processes associated with the definition of CIMs, the implementation of EHR communications and systems based on CIMs, as well as establishing governance for the multiple CIMs applicable within an infrastructure or domain.

    • Automatic generation of computable implementation guides from clinical information models

      2015, Journal of Biomedical Informatics
      Citation Excerpt :

      The quality of implementation guides and reference materials highly depends on the quality of source information models. Several metrics and requirements have been proposed to deal with clinical information model quality [37,38]. If clinical information models follow this metrics and requirements, we can ensure the quality of end systems [39].

    • Project Successful Deployment A Method for Evaluating the Success of Digitalization Projects

      2022, International journal of online and biomedical engineering
    • Measuring software product quality using ISO 9126: A systematic review

      2019, International Journal of Advanced Trends in Computer Science and Engineering
    View all citing articles on Scopus
    View full text