The NQRN Registry Maturational Framework: Evaluating the Capability and Use of Clinical Registries

Clinical registries are increasingly used as national performance measurement platforms. In 2018, nearly 70 percent of the more than 50 specialty society registries in the United States were used by the Centers for Medicare & Medicaid Services (CMS) to measure the quality of clinical care. Private payers and evaluating organizations also use or desire to use registry information to inform quality improvement programs and value-based payment models. The requirements for an entity to become a CMS Qualified Clinical Data Registry (QCDR) constitute a minimum set of standards for the purpose of reporting to the CMS Quality Payment Program. Models and frameworks exist that can help classify registries by purpose and use, and maturity models are available for evaluating health IT systems generally. However, there is currently no framework that describes the capability that should be expected from a registry at different phases of its development and maturity. In response, the National Quality Registry Network (NQRN) has developed a registry maturational framework. The framework models early, intermediate and mature development phases, the capabilities anticipated during these phases and 17 domains across which registry programs support those capabilities. The framework was developed and refined by NQRN registry stewards, users and other stakeholders between 2013–2018. It is intended to be used as a developmental guide or for registry evaluation. The successful use of registry information to execute value-based payment models is a critical need in U.S. health care. The NQRN framework can help ensure that our national system of registries is rising to the occasion.


Introduction
A clinical registry is an organized system for the collection, analysis and reporting of clinical data about patient care for the improvement of health outcomes [1]. With the increased use of registries for research, performance evaluation, quality improvement and payment, there is corresponding interest in new registry development and increasing the functionality of established registries. The National Quality Registry Network (NQRN) is a community of organizations interested in clinical registries. NQRN is a program of the PCPI Foundation, a national convener of medical specialty and health care professional societies and associations [2]. To accelerate these trends and guide registry development and use, NQRN has developed a maturational framework for clinical data registries.
Clinical registries have made many contributions to the health care knowledge system [3][4][5][6][7][8][9]. As registry data collection, analysis and reporting capabilities have improved, their use in improving health system effectiveness has expanded (Figure 1). Today's patchwork system of registries is moving toward a coordinated and transparent system, with improved interoperability envisioned between registries and other health information systems to support better efficiency and outcome evaluations. Additionally, registries have a need to be able to demonstrate the high quality of their data through transparency of performance measures and quality control techniques.
Although the United States leads the world in national health care expenditures, robust data to support quality improvement, accountability and consumer engagement are lacking [10]. As clinically rich data have become more widely available, there is interest in leveraging these data to improve the overall quality and efficiency of the U.S. health care system [11].
A clinical registry is utilized for many purposes including assessing performance; determining clinical, cost or comparative effectiveness; and measuring or improving the quality, safety or efficiency of care. Specific applications include surveillance, clinical practice guideline development, appropriateness determination, performance evaluation and improvement monitoring, research and public reporting.
All data sources -administrative claims, clinical information systems, electronic health records (EHRs) and registries -have advantages and limitations. Although originally developed for billing and other administrative functions, claims data have subsequently been utilized for a wide variety of purposes including performance evaluation due to their availability, low cost, standardized coding systems such as the International Classification of Diseases (ICD) and Current Procedural Terminology (CPT), and the large volume of available data they represent. However, administrative data often lack clinical granularity and specificity, may be difficult to aggregate and analyze across payers, and have been viewed skeptically by the provider community when used for performance evaluation.
Clinical information systems and EHRs also have advantages and limitations compared with registries. Their data models have been designed to support direct patient care and operations, with a secondary focus on performance evaluation. Roski et al. estimate that, "…less than 15 percent of health data in EHRs might be entered in structured data fields that allow those data to be analyzed using traditional retrieval and analysis methods [12]." A significant proportion of the data captured in these systems lack standardized data definitions, thus limiting their usefulness for the purposes articulated above.
Registries are designed by clinical experts and contain data that are clinically relevant, structured, precisely-specified, understandable, and acceptable to health care providers. Registries typically include data from real-world populations and multiple payers, distinguishing them from many administrative data sources. For research, registry data may offer an advantage over traditional randomized clinical trials as rigid study population inclusion and exclusion criteria in the latter may limit the generalizability of their results. Finally, registry data are collected using processes that ensure data validity and good inter-rater reliability, with data quality audits ensuring ongoing accuracy.
Registries also have limitations. Currently, most registries focus on short-term outcomes making it difficult to assess care over time. They commonly lack information on readmissions, re-interventions, medication adherence, patient functional status and cumulative costs. This information is becoming increasingly important as responsibility for outcomes and costs extends well beyond a single episode of care.
Given these considerations, one approach may be to combine the positive attributes of each data source, while mitigating their respective weaknesses [13]. To accomplish this, commonly used data elements will need to be standardized to facilitate data linkages [14]. For example, using the registry as the foundation, linkages could be established to EHRs for demographics, to EHRs or clinical information systems for laboratory and diagnostic results, and to administrative claims and other data sources for repeat hospitalizations, procedures, professional services, cost information and long-term survival [15,16].

The Need for a Registry Maturational Framework
There are over 60 national clinical registry programs in the United States, many of which are recognized for the quality of their data and their contributions to improved patient care [17]. While they are typically developed by professional organizations, the value of registries is increasingly recognized by payers. For example, the Medicare Access and  Registries are also being utilized by the U.S. Food and Drug Administration for post-marketing surveillance of medical devices [18].

Prior Work on Clinical Registry Evaluation
There is a growing body of work and increasing interest in registry description, classification and evaluation. In 2012 the NQRN conducted a registry landscape assessment which identified multiple unmet needs of registries, as well as variation in governance and operational processes. The survey was conducted a second time in 2015 [19]. In addition to these efforts, others have offered models and frameworks to help define and classify registries. In 2008 Drolet et al published a classification system for registries that identified five characteristics of a registry, using the term in its broadest sense: mergeable, standardized longitudinal data from multiple users, including outcomes, collected using a rules-based approach [20]. Klaiman et al examined registry use in 2013 and found that "effective registries were successful in 1 or more of 6 key areas: data standardization, transparency, accuracy/completeness of data, participation by providers, financial sustainability, and/or providing feedback to providers [21]." The Australian Commission on Safety and Quality in Health Care has published a framework for quality registries that describes desired registry attributes across domains including data collection, risk adjustment, data security, data quality, governance, ethics & privacy, reporting and funding [22]. A 2016 report by Crowley et al offered a Public Health Information Technology Maturity Index. Although not specific to registries, the index offered a maturity dimension in addition to domains including scale and scope, quality, human capital, policy, resources and infrastructure [23]. At the time of publication there was no national standard for evaluating the maturity or performance of registries in general. The QCDR requirements could be considered a baseline at least for reporting performance measure results to evaluating organizations for the purpose of carrying out value-based payment programs. Health Level Seven International (HL7), a standards developing organization, publishes domain analysis models on subjects of interest to the health care data standards and informatics communities. A domain analysis model contains an abstract representation of a specific subject area, oriented to the needs of health IT implementers and users [24]. As part of its Common Clinical Registry Framework project, in 2017 HL7 published a model that "…describes the function, organization, structure and major workflows of a general clinical registry" [25]. As a set these models and frameworks provide useful tools for describing and classifying registries, however none include a maturity model developed specifically for evaluating registry capability through multiple phases of maturity.
In response to the variations in registry development identified in its registry landscape surveys and given the lack of a targeted maturity model, the NQRN developed a clinical registry maturational framework (Framework) to guide the development of registries over time.

Methods
Development of the Framework began with the identification of important registry domains such as registry use, scope, data quality and performance measurement. Early, intermediate and mature developmental stages were then described for each domain, and the resulting intersecting cells were populated to demonstrate how a registry might evolve over time. The Framework was further refined with broader stakeholder input from NQRN participants in November 2013, as well as further updates by NQRN staff in 2016 and 2018.

Description and Use
The Framework describes the domains in which a registry might mature over time. The relative importance of each domain is based on the use(s) of the registry. The Framework is a roadmap for achieving the highest level of value from a registry and a guide to identifying the underlying capabilities and infrastructure that will be necessary to achieve that value.
The Framework has been developed according to a set of general principles: (1) it is intended to be a guide to registry development, not a standard; (2) content is organized by topical domain and phase of maturity; (3) content is generally cumulative as maturity increases within a given domain; (4) at any given time, a registry may be at different stages of maturity across the domains; and (5) the desired final stage of maturity in a domain may differ depending on the mission of the registry.
Within the Framework, registry activities are generally cumulative as maturity increases within each domain. However, not all registries will mature along the same trajectory, with differences driven by the needs of its participants and end users. Finally, the Framework reflects the current state of registry development which is expected to evolve over time as medical science, payment models and information technologies advance.

The Topical Layer: Registry Domains
Registries can be described by applying a model comprised of different domains -starting with registry use, followed by additional domains describing underlying capabilities at each phase of maturity ( Table 1).

The Structure for Development: Phases of Maturity
For a registry to expand its usefulness and overall maturity, increased supporting capabilities are needed. The Framework takes a developmental staged approach to describe what to expect of a registry at different phases of development, along with the capabilities needed. In the early phase of development, a registry typically begins to provide value to participants who use registry data for local quality improvement or performance assessment and reporting. A registry in the intermediate phase has achieved a critical mass of participation to support performance measurement and other activities with increased scope and rigor. Mature registries provide data to support health system decision making at a national or international level as well as for research and health care policy. Registries may achieve different maturity phases across the various domains at any given period of time, but some congruence is anticipated due to the cumulative advancement of underlying capabilities needed to support particular uses in the next higher phase. Examined together, the topical domains and the development structure allow a registry to be assessed holistically. (Figure 2).
A table containing the complete Framework matrix is in Appendix A.

Suggested Uses
The Framework is intended to guide registry development; it is not a set of standards or requirements. It is hoped that its guidance will facilitate the development of registries in a logical fashion and support the adoption of leading practices and national standards at an accelerated pace. The Framework has many potential uses. It can guide registry stewards along a consistent path to maturity with a shared understanding of what the different stages of maturation entail. It can be used as a general guide to understand what might be needed for a registry to qualify for use by a private or public payer such as CMS through its QPP. The Framework can help users of registry data e.g., health systems, researchers, evaluators and payers assess a registry's capabilities. It can also be used by registry stewards to communicate with consumers about how information was collected, analyzed and reported.
The Framework may be a useful assessment tool for registry stewards, those planning a new registry, and users of registry information. The Framework can be used to define the goals of the registry and the developmental pathways to achieve these goals. It is a valuable resource to identify a registry's strengths, areas where additional focus and resources may be needed, and actions required to expand a registry's purposes.
With its strong emphasis on descriptiveness combined with flexibility that allows each registry to fulfill its stated purposes, the Framework can provide valuable insight to accelerate the pace and quality of registry development. An existing registry can

Registry networks
The degree to which the registry participates in professional and data-sharing networks evaluate its maturity level and gain insight that can help prioritize further development efforts. A new registry can better understand the capabilities that will support its mission. Users of registry data for research, performance improvement and payment can better understand the level of rigor and scope they can expect from the data. There are many other potential uses ( Table 2).

Limitations
Although the Framework is comprehensive and offers guidance to potentially any registry developer, it has limitations. The information it provides, while important and valuable, is necessary but ultimately not sufficient for a sustainable and credible registry to emerge and evolve. The Framework as currently bounded does not address several relevant dimensions such as a) governance and ownership of registries and registry data, b) registry business models and access/use requirements, c) a range of technical issues such as execution of data use agreements to whether data is distributed, consolidated in one place, or some hybrid, and d) approaches to privacy protection. As specialty societies and other registry stewards continue their work in developing and refining sustainable business models, the registry community through the NQRN and other stakeholders has an opportunity to continue the development of this Framework to address these limitations.   An organization is being asked to submit data that support the execution of valuebased payment models Understand the capabilities needed in a registry that participates in payment programs Outlines these capabilities and provides a path to achieving them

Conclusion and Next Steps
It is envisioned that the uses of the Framework will expand in the future. For example, it could serve as an assessment tool for those considering participation in a registry i.e., health systems, practitioners, or health plans, consumers and others interested in using registry data for improvement and related purposes. Supported by the NQRN and the Framework, as well as additional NQRN resources, registries are expected to continue to mature as an important data source to drive improvement in health care delivery [26]. With strategic investments in data standardization, linkages among registries and other data sources, and alignment of financial incentives, the U.S. health care system will evolve from being "data poor" to "knowledge rich."