http://orcid.org/0000-0002-5559-1149
Figures
Abstract
Objective
To develop a set of structure and process indicators to evaluate tertiary hospitals’ performance in the Healthcare Improvement Initiative, a national program with a goal to improve quality of patient-centered care.
Methods
A modified Delphi technique, including literature review, multidisciplinary panel meeting and anonymous rating, was used to generate a set of indicators. A practice test involving both general and special hospitals was conducted to ensure the feasibility of data collection for these indicators.
Results
62 indicators were generated by literature review. The panel review procedure involving 39 panelists with diverse backgrounds resulted in a total of 59 indicators, which included 40 qualitative indicators and 19 quantitative indicators. In the practice test, six quantitative indicators were found unfeasible. According to the suggestion of the experts in the hospital evaluation committee, three of those indicators were kept by adjusting their data collection methods, while other three ones were discarded.
Discussion
A set of 56 structure and process indicators was developed to evaluate hospitals’ performance in the implementation of the Healthcare Improvement Initiative, which could be used in both general and special tertiary hospitals. Results of the indicator measurement could present a panorama of the quality of patient-centered care in tertiary hospitals nation-wide, and inform health administrators of the ways to attain the goal of the Initiative.
Citation: Zhou H, Bai G, Gao J, Zhou Y, Ma E, Hu L, et al. (2018) The development of indicator measure for monitoring the quality of patient-centered care in China’s tertiary hospitals. PLoS ONE 13(10): e0205489. https://doi.org/10.1371/journal.pone.0205489
Editor: Sandra C. Buttigieg, University of Malta Faculty of Health Sciences, MALTA
Received: September 8, 2017; Accepted: September 26, 2018; Published: October 11, 2018
Copyright: © 2018 Zhou et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: This study was funded by the National Health and Family Planning Commission of People’s Republic of China (http://www.nhfpc.gov.cn/) to LL. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Patient-centered care (PCC) can be commonly understood as healthcare which can cater to patient needs [1]. The elements which constitute PCC may vary among different professional groups, while the core components of PCC can be identified as (a) patient participation and involvement (e.g., customized care plan, addressing patient’ emotional needs as well as physical needs), (b) patient- doctor relationship (e.g., open communication of information, personal qualities of health professionals, such as appropriate skills, good etiquette, respectful and welcoming attitudes), and (c) the setting where healthcare is delivered (e.g., appropriate treatment time, adequate staff and friendly facilities) [2, 3].
As PCC has been increasingly recognized by health administrators across the world [4–6], the China’s government has become interested in improving quality of PCC especially in the tertiary public hospitals. Tertiary public hospitals account for 7.66% of the all-type health facilities in China, yet provide 42.50% of the inpatient care and 48.70% of the outpatient services [7]. Some patient surveys find room for service improvement in these overcrowded tertiary hospitals. For instance, the fifth national health survey indicates that 34.4% of patients are unsatisfied with outpatient environment [8]. Specifically, windows for registration and payment are concentrated in the hall of outpatient building, and online systems for registration, payment and inquiry are underdeveloped in most tertiary hospitals, which lead to long waiting time and overcrowding of patients in peak visiting periods [9]. Patients averagely spend 2.5 to 8 hours in the hospital for a consulting lasting 15–20 minutes [10]. Moreover, surveys among tertiary hospitals in -Beijing and Shanghai indicate that more than 70% of patients are dissatisfied with toilet hygiene because of smelly odor and filth in urinals[11]. More than 30% of patients are not satisfied with inpatient care mainly because of process inefficiency, concerns about quality and safety and attitudes of medical staff [8].
To target concerns of patient, the Ministry of Health (MoH, renamed as National Health and Family Planning Commission in 2014) came up with a new vision for improving healthcare, and implementing the Healthcare Improvement Initiative (Initiative for short) in January of 2015 [12]. The overall goal of this program is to make healthcare more patient-focused and improve patients’ experience and satisfaction. The Initiative encompasses 29 actions falling under nine objectives, which include, within a three-year time frame, (1) optimize the layout of the facilities and build a friendly service environment, (2) promote utilization of medical appointment services and guide patient flow, (3) improve service efficiency by rational allocation of resources, (4) take advantage of information technology to improve patient’ experience, (5) improve process reengineering and accommodation in inpatient department, (6) continuously improve quality of nursing care and enhance nursing workforce, (7) ensure patient safety by adoption of standard operating procedures, (8) strengthen humanistic care and provide social work services, (9) harmonize doctor-patient relationship and reduce medical disputes [12]. The objectives and actions of the Initiative were set by the MoH by canvassing the views of health administrators and public opinions in social media [13].
In order to assess the performance of tertiary public hospitals in the Initiative, the MoH required our research team to develop a set of indicators to measure whether those actions have been implemented. Although indicators for accreditation of tertiary hospitals has been constructed by the MoH since 2010[14], and some studies have developed indicator measurement from specific aspects (such as safety culture in radiological department, safety of nursing care, and quality of impatient care in general hospitals) [15–17], a set of indicators to comprehensively measure the quality of PCC in different types of tertiary hospitals is in demand. The development of indicators for the Initiative is supposed to focus on the quality of PCC, and evaluate the performance of both general and special tertiary hospitals in this national program in a more specific way.
According to the commonly used Donabedian Model, quality of healthcare can be measured by three types of indicators: structure, process and outcome indicators [18]. According to the objectives of the Initiative, structure indicators measure the settings of hospitals in program implementation, for instance, whether the hospital has regulations and facilities to support quality improvement of patient-centered care; process indicators assess the degree to which actions are being effectively implemented; outcome measurement, such as patient experience or satisfaction, can be taken as ultimate validator of hospitals’ performance, which could be measured by patient survey about waiting time, utilization of online services and their subjective feeling of medical environment, process as well as doctor-patient relationships [2].
This paper describes the development and practice test of structure and process indicators from the perspective of health organizations [19]. The outcome measurement from the perspective of patient, as per MOH’s requirement, will be developed by other studies in the form of questionnaire survey among patients.
Methods
A modified Delphi technique (RAND/UCLA Appropriateness Method), comprised of literature review, multidisciplinary panel meeting and two rounds of anonymous rating [20], was used in our study. This method has been commonly used to develop quality indicators in developed countries [19, 21–24]. Feasibility of proposed indicators was verified by practice test in pilot hospitals.
When evaluating the appropriateness of candidate indicators, criteria as follows were considered by research team and panelists [21, 25, 26]:
- Validity. An adequate indicator must have empirical rationale for its relevance to quality of patient-centered care.
- Utility. The structure and process indicators can be improved through changes in hospital administration or the way of service delivery, according to the measures of the Initiative.
- Feasibility. The data for indicator assessment can be extracted from hospital information systems, or can be explicitly observed in survey.
Development of structure and process indicators by the modified Delphi technique
Literature review.
Literature review was conducted to generate candidate indicators and their description for panel review. We searched articles in China National Knowledge Infrastructure database and PubMed database in June 2015, with search terms ‘Healthcare Improvement Initiative’, ‘service environment’, ‘medical appointment’, ‘allocation of resources’, ‘day surgery’, ‘emergency priority’, ‘health information technology’, ‘inpatient care’, ‘patient safety’, ‘humanistic care’, ‘social work’, ‘doctor-patient relationships’, ‘quality indicators’ and ‘hospital accreditation’. Those search terms were developed according to the objectives and actions of the Initiative. A total of 656 articles were initially identified, after article screening 314 articles of full text addressing healthcare quality control were finally selected. Fig 1 illustrates the inclusion and exclusion of the articles in literature review. We not only developed candidate indicators by article review, but also extracted some evidence from articles describing the process, results and problems in delivering patient-centered care. The evidence could support the validity and utility of some indicators.
In addition, we also reviewed the indicators and standards in three quality measurement systems: the China Tertiary Hospitals Accreditation (THA), the US Joint Commission International (JCI), and the German Cooperation for Transparency and Quality in Healthcare (KTQ) [14, 27–29]. To our knowledge, these systems included some structure and process indicators related to patient-centered care, which could be measured at organizational level. According to the content of the Initiative policy, we could extract some candidate indicators from these systems to meet our research aim.
GB and HZ extracted indicators and evidence, integrated information, and finally formed a document for candidate indicators, which included the definition, algorithm for quantitative indicator, standards for qualitative indicators, and data collection methods (such as survey, administrative data and medical records) for each of the indicators [25]. The origin of candidate indicators (from article review or indicator systems of THA, JCI and KTQ) were also marked. Candidate indicators were categorized into 29 specific actions falling under nine objectives of the Initiative. We coded the candidate indicators in a hierarchical format. For instance, ‘1.2.3’ indicates that this indicator is the third one assessing the implementation of the second action, which belongs to the first objective of the Initiative.
Multidisciplinary panel.
We selected the panelists considering their experiences in hospital management or clinical practice as well as their geographical diversity. Six universities, five research institutes and six tertiary hospitals located in different areas of China, were required to nominate panelists to participate in the panel review. Ultimately, 39 panelists with multidisciplinary backgrounds (16 professors major in health management and policy, 10 physicians and 13 hospital managers) consented to participate in two rounds of anonymous rating by Email and a face-to-face panel meeting between two rating procedures in Shanghai.
First round rating.
Documents including the information of candidate indicators and rating instruction were sent to the panelists by Email. According to the RAND/UCLA Appropriateness Method [20], the panelists were asked to rate each indicators by circling a number from 1 (not at all appropriate to assess the implementation of the Initiative) to 9 (very appropriate) in a questionnaire, and they could add new indicators in the questionnaire. Indicators were selected if the median scores were 7, 8 or 9 without disagreement (1/3 or more ratings in both 1–3 or 7–9 regions). Indicators with the median scores of 1–3 without disagreement were discarded. Indicators with the median scores of 4–6 or with disagreement were involved in the agenda of next step panel meeting.
Panel meeting.
In the face-to-face panel meeting, the panelists expressed their opinions of the indicators on which they did not reach consensus, rephrased the description of some indicators and discussed the new added ones.
Second round rating.
According to the opinions of panelists, we compiled all remained indicators with their description (rephrased if necessary), and sent them to the panelists by Email. The panelists rated these indicators in a questionnaire with the same format as the first-round rating. Indicators were accepted if the median rating scores were 7, 8 or 9 without disagreement. Indicators with the median scores of 1–6 were discarded.
Practice test
Settings.
Given these indicators will come into use in both general and special tertiary hospitals, we conveniently sampled four tertiary hospitals with different types in Jiangsu Province (a general hospital, a cancer hospital, a maternity hospital and a hospital of Traditional Chinese Medicine) to verify the feasibility of proposed indicators, and test the data resources and data collection methods for these indicators [30].
Data collection.
In order to efficiently and precisely evaluate tertiary hospitals’ performance, we invited nine experts to form an evaluation committee, and divided these experts into three groups depending on their backgrounds to share the work of data collection. The first group comprised of three health administrators from Health Authority of the MoH, and was responsible for collecting information for the indicators corresponding to the objectives 1, 8 and 9, which were mainly related to clinical environment, humanistic care and doctor-patient relationships. The second group including three chief physicians from a tertiary hospital not involved in the practice test, took charge of data collection for indicators falling under the objectives 3, 6 and 7, which were related to allocation of medical resources, quality of nursing care and patient safety. The third group involving three hospital managers from the same hospital, took responsibility of collecting data for indicators corresponding to the objectives 2, 4 and 5, which were related to medical appointment, information technology and logistics of impatient care.
Each expert was trained with data collection methods (e.g., to observe hospital environment, review hospital documents in survey and abstract information from administrative data or medical records). We asked experts in each group to supervise each other and reach consensus on result of each indicator. Each group was required to note results of qualitative indicators and original data of quantitative indicators in a questionnaire.
Feasibility assessment.
A specific indicator was considered ‘unfeasible’, if the information for its measurement could not be obtained by its data collection method [25], which resulted in missing data in the questionnaire. We discarded unfeasible indicators or adjusted their data collection methods according to the suggestions of experts.
Results
Fig 2 illustrates the inclusion and exclusion of indicators in each step of the modified Delphi technique and practice test.
Structure and process indicators for the Initiative implementation
Selection of indicators from literature.
We developed 14 indicators by article review. Quality indicator systems review yielded 48 indicators. Specifically, 35 indicators were derived from China Criteria for Accreditation of Tertiary Hospitals, 11 indicators were derived from JCI and 2 indicators were derived from KTQ. In total, 62 indicators were involved in the panel review procedure. Moreover, we also extracted some evidence by article review to support the validity and utility of some indicators. For instance, Ji’ research demonstrates that patients spend most of their time on waiting for consulting, especially for patients with common ailments, thereby suggests that tertiary hospitals should distribute consulting rooms according to patients’ need and arrange more rooms for department treating common diseases [9]. Depending on evidence like that, indicator “1.1.1 Improve the layout of consulting rooms” could be considered valid, and to arrange consulting rooms according to outpatient flow analysis was supposed to be an effective way to reduce waiting time for patients.
First round rating.
All 39 panelists responded the rating questionnaire by Email in four weeks. The panel reached consensus on 46 indicators (the median scores were 7, 8, 9 without disagreement), rejected six indicators (the median scores were 1, 2, 3 without disagreement) and suggested three new indicators. Ten indicators got median scores of 4–6 without disagreement, and required panelists’ opinions on their inclusion, adjustment or exclusion in the panel meeting. Table 1 presents these indicators with the corresponding objectives and actions, as well as their origins (article review, indicator systems of THA, JCI and KTQ, added by panelists), types (structure or process indicators) and median scores. The details of rating which were used to calculate median score and disagreement could be found in S1 Dataset.
Panel meeting.
Thirty-seven panelists attended the panel meeting and other two panelists participated by video conferencing. They gave their opinions on the new added indicators and the ones with median scores of 4–6, rephrased the description of 16 indicators to make these indicators more patient-focused and adapted for China’s context, resulting in a total of 59 indicators.
Second round rating.
All 39 panelists responded the rating questionnaire for the 59 indicators by Email, and approved all of them (the median scores were 7, 8, 9 without disagreement). Table 2 presents these indicators with their types and median scores. The details of rating which were used to calculate median score and disagreement could be found in S1 Dataset.
Practice test
All 40 qualitative indicators were proven feasible, because their data could be collected by survey. Among 19 quantitative indicators, data of 13 indicators could be abstracted from hospital administrative data, while the other 6 were found unfeasible in the practice test.
Specifically, indicators 2.1.3, 2.1.4 and 2.1.5 (presented in Table 1), which were used to measure the volume of appointment service in subsequent visit, were discarded, because hospital information systems could not discriminate appointment for subsequent visit from total appointment records.
Other three unfeasible indicators were kept by adjusting the data collection methods. Indicators 3.1.2 and 3.1.3 were aimed at measuring the efficiency of examination reporting in emergency department, and required the experts to collect information of time interval from sampling to reporting the results of examinations. However, those data were not captured by information systems. In consideration of the workload in evaluation, the experts suggested to choose first ten examination reports in the morning, note their time interval in the questionnaire as the original data, and calculate an average. In the practice test, the experts also found that they failed to abstract data to calculate the percentage of discharges who have received follow-up service for indicator 5.3.1, because the medical record systems were incapable to search and sort records by follow-up advices. Therefore, they suggested to conveniently sample five discharge medical records with follow-up advices for the past three months and check whether the discharges have received follow-up service. The experts in future evaluation will be asked to note the results or numerical value of these three revised indicators in the questionnaire, and the patient information will not be exposed.
The final version of these indicators with their description is available in S1 Table.
Discussion
In this study, we developed 56 indicators to evaluate the implementation of the Initiative, a nation-wide program aimed at improving patient experience in public hospitals. A modified Delphi technique and a practice test including both general and special tertiary hospitals were used to generate a set of indicators with face validity, utility and feasibility.
Our study has several strengths. Firstly, different from the traditional input-based structure indicator measurement (e.g. the financial investment and the number of beds or physicians), and the clinical outcome indicator measurement (e.g. mortality and complications) for China’s public hospitals [31, 32], our study generated a set of structure and process indicators heavily relevant to hospitals’ performance in improving patient experience. Some structure indicators measuring the institutional backgrounds for humanistic care and doctor-patient relationships, layout and barrier-free facilities for patient convenience were developed; some process indicators reflecting the efficiency of medical services, such as the percentage of daytime surgeries, percentage of time slots reservation were newly generated by this study. This set of indicators can comprehensively measure the quality of PCC, and will be used in the evaluation of tertiary hospitals nation-wide. These indicators enrich the quality indicator systems for monitoring public hospitals and demonstrate that the China’s government not only makes efforts to improve quality of clinical treatment, but also puts emphasis on promoting hospitals to provide health services in a more patient-focused way.
Second, our study used a modified Delphi technique (the RAND/UCLA appropriateness method) as the consensus method [20], which has been seldom used by Chinese researchers to develop quality indicators [33–35]. Seriously following the manual of the appropriateness method, we selected panelists with multidisciplinary backgrounds and geographical diversity, which could improve the applicability of this set of indicators among diverse hospitals and areas. In comparison with other expert consultation methods, such as expert interview and focus group discussion used in similar studies [15–17], the anonymous rating by Email in our study gave panelists adequate time to independently evaluate candidate indicators, and the structural panel meeting allowed panelists to express their opinions with minimum impact from talkative or authoritative panelists, which could come down to more reliable results [36].
Third, this set of indicators is feasible in both general and special tertiary hospitals when assessing the quality of PCC. This study indicates that the practice test is necessary to ensure the feasibility of indicators especially when the indicators are supposed to apply in a relative complex context [21].
This study has a number of limitations. First, we did not develop outcome indicators in this study. Ideally, proper organizational structure and appropriate process of service delivery will cause a higher probability of achieving positive outcomes. Panel review in the Delphi technique ensured the face validity of these indicators, while the construct validity of these indicators needs to be demonstrated by correlation relationship between outcome indicators and structure or process indicators [18, 37]. Therefore, indicator measurement for the Initiative implementation could be completed by the development of outcome indicators, and the construct validity of these indicators could be tested in further studies [19].
Second, limited by time and energy, we only selected indicators from three quality measurement systems which could meet the aim of the research and were familiar to us. The further study could review other indicator systems in different countries to refine the set of indicators developed in this study.
Third, this set of indicators cannot be used to demonstrate performance of individual hospital and hospital ranking. Positive or negative results of qualitative indictors, and numerical value of quantitative indicators will be obtained in the evaluation. Overall performance of those tertiary hospitals in each indicator will be calculated and reported to the MoH, which presents a panorama of the Initiative implementation. Subsequent studies could determine weighing and scoring rules for each indicator, depending on the baseline data collected in the first-year evaluation. The refined indicator system would allow for comparison of performance among different hospitals.
Fourth, data collection for some indicators are expensive or limited by current information systems. Data collection for 37 indicators depends on survey. This method is useful to obtain specific and detailed information for some indicators, and conveniently sampling is also a common method used by international hospital accreditation organizations such as JCI and KTQ [25, 26]. However, it is costly to conduct and cause financial burden in health administrations, which makes continuous monitoring of hospital performance less likely. Moreover, data for some indicator measurement cannot not captured in hospital information system. Considering the cost of data collection, experts suggested to reject some indicators, or abstract indicator data by convenient sampling with a small sample size, which may influence the validity of evaluation results [38, 39]. These limitations could be incrementally surmounted in along with the refinement of indicator system or the development of information systems, which may provide a more effective way of data collection for some indicators [40].
In conclusion, we developed a set of 56 structure and process indicators to evaluate tertiary hospitals’ performance in the implementation of the Healthcare Improvement Initiative, a national program aimed to improve quality of PCC and patient experience. The modified Delphi technique ensured the face validity of these indicators. The practice test played an important role in verifying the feasibility of data resource and data collection methods for these indicators. This set of indicators will be used in the evaluation of tertiary hospitals nation-wide, present a panorama of the quality of PCC in both general and special hospitals and inform health administrators of the ways to attain the goal of the Initiative. Further studies could be conducted to complete and refine this indicator system, determine the weighing and score for each indicator according to the baseline evaluation data, which allows for comparison of performance among different hospitals.
Supporting information
S1 Dataset. Results of two rounds of rating in panel review.
https://doi.org/10.1371/journal.pone.0205489.s001
(XLSX)
S1 Table. Final version of structure and process indicators for monitoring the implementation of the Healthcare Improvement Initiative in tertiary hospitals.
https://doi.org/10.1371/journal.pone.0205489.s002
(PDF)
Acknowledgments
The authors thank the panelists for reviewing quality indicators, and the experts and hospitals participating in the practice test.
References
- 1. Epstein RM, Fiscella K, Lesser CS, Stange KC. Why the nation needs a policy push on patient-centered health care. Health affairs. 2010;29(8):1489–95 pmid:20679652
- 2. Stewart M. Towards a global definition of patient centred care. Bmj British Medical Journal. 2001;322(7284):444 pmid:11222407
- 3. Kitson A, Marshall A, Bassett K, Zeitz K. What are the core elements of patient-centred care? A narrative review and synthesis of the literature from health policy, medicine and nursing. Journal of advanced nursing. 2013;69(1):4–15 pmid:22709336
- 4.
Services UDoHaH. Personalized Health Care: Pioneers, Partnerships, Progress. Washington, DC: US Department of Health and Human Services; 2008.
- 5.
Service NH. Creating a Patient-Led NHS: Delivering the NHS Improvement Plan. London: Department of Health; 2005.
- 6.
Healthcare ACoSaQi. Patient-Centred Care: Improving Quality and Safety by Focusing Care on Patients and Consumers Discussion Paper. Sydney: Department of Health and Ageing; 2010.
- 7.
MoH. China Health Statistical Yearbook. Beijing, China: Press of Peking Union Medical College; 2017.
- 8. Xu L, Meng Q. Outcomes of the fifth national health services survey. Chinese Journal of Health Informatics and Management. 2014;11(2):104–5
- 9. Ji H. Status and its optimization measures on hospital outpatient process. Chinese Hospitals. 2010;14(7):69–72
- 10. Chen S. Hospital information systems and improvement of outpatient process Chongqing Medicine. 2007;36(23):2390–1
- 11. Wang Y. How to improve toilets in outpatient department. China health industry. 2017;14(22)
- 12.
MOH. The notification of the implementation of the Healthcare Improvement Initiative 2015 [cited 2015 Jan 28]. http://www.nhfpc.gov.cn/yzygj/s3593g/201501/5584853cfa254d1aa4e38de0700891fa.shtml. (in Chinese).
- 13. Zhang Z. Understanding the Healthcare Improvement Initiative (in Chinese). Chin J Hosp Admin. 2016;32(6):401–3
- 14.
MOH. Criteria for accreditation of tertiary hospitals 2010 [cited 2010 May 25]. http://www.moh.gov.cn/wsb/s7987/201005/47518. (in Chinese).
- 15.
Zhang W. Study on the Synthetical Evaluation System and Empirical Research of Medical Institutions' Radiation Safety Culture Level: Shandong University; 2013.
- 16.
Wu Z. Research on Evaluation Index System and Comprehensive Evaluation of Service Quality for Level Three Hospitals in Hunan Province: Zhongnan University; 2009.
- 17. Chen L, Cheng S, Chen Y, She X, Li Z, He J. Indicator measurement development for safety of nursing care in general hospitals. Modern Clinical Nursing. 2011;10(10):57–8
- 18. Donabedian A. Evaluating the quality of medical care (Reprinted from The Milbank Memorial Fund Quarterly, vol 44, pg 166–203, 1966). Milbank Q. 2005;83(4):691–729 pmid:16279964
- 19.
Institute BM. Quality Indicator Measure Development, Implementation, Maintenance, and Retirement. Rockville, MD: Agency for Healthcare Research and Quality; 2011.
- 20.
Fitch K, Bernstein SJ, Aguilar MD. The RAND/UCLA Appropriateness Method User's Manual 2001. http://www.rand.org/pubs/monograph_reports/MR1269.html.
- 21. Hommel I, van Gurp PJ, Tack CJ, Wollersheim H, Hulscher MEJL. Perioperative diabetes care: development and validation of quality indicators throughout the entire hospital care pathway. Bmj Qual Saf. 2016 Jul;25(7):525–34 pmid:26384710
- 22. Adriaenssens N, Coenen S, Tonkin-Crine S, Verheij TJ, Little P, Goossens H, et al. European Surveillance of Antimicrobial Consumption (ESAC): disease-specific quality indicators for outpatient antibiotic prescribing. Bmj Qual Saf. 2011 Sep;20(9):764–72 pmid:21441602
- 23. Coleman P, Nicholl J. Consensus methods to identify a set of potential performance indicators for systems of emergency and urgent care. Journal of health services research & policy. 2010 Apr;15 Suppl 2:12–8 pmid:20354114
- 24. Ohtera S, Kanazawa N, Ozasa N, Ueshima K, Nakayama T. Proposal of quality indicators for cardiac rehabilitation after acute coronary syndrome in Japan: a modified Delphi method and practice test. BMJ open. 2017 Jan 27;7(1):e013036 pmid:28132004
- 25. Rubin HR, Pronovost P, Diette GB. From a process of care to a measure: the development and testing of a quality indicator. International journal for quality in health care: journal of the International Society for Quality in Health Care. 2001 Dec;13(6):489–96
- 26. Davies SM, Geppert J, Mcclellan M, Mcdonald KM, Romano PS, Shojania KG. Refinement of the HCUP Quality Indicators. Agency for Healthcare Research & Quality. 2001;33(3):775–8
- 27.
JCI. Joint Commission International Accreditation Standards for Hospitals 2014. https://www.jointcommissioninternational.org/assets/3/7/Hospital-5E-Standards-Only-Mar2014.pdf.
- 28.
International JC. Joint Commission International Accreditation Hospital Survey Process Guide 4th ed2010.
- 29.
Thomas SA. KTQ Guidline and Catalogue. Berlin: Grim Publishing Company; 2009.
- 30. Wang CC, Wang Y, Zhang K, Fang J, Liu W, Luo S, et al. Reproductive health indicators for China's rural areas. Social science & medicine. 2003 Jul;57(2):217–25
- 31. Xu Y, Liu YL, Shu T, Yang W, Liang MH. Variations in the Quality of Care at Large Public Hospitals in Beijing, China: A Condition-Based Outcome Approach. PloS one. 2015 Oct 2;10(10)ARTN e0138948 pmid:26430748
- 32. Ma XM, Chen XH, Wang JS, Lyman GH, Qu Z, Ma W, et al. Evolving Healthcare Quality in Top Tertiary General Hospitals in China during the China Healthcare Reform (2010–2012) from the Perspective of Inpatient Mortality. PloS one. 2015;10(12):e0140568 pmid:26624005
- 33. Yang K, Xu JF, Zhang JF, Li W, He J, Liang S, et al. Establishing and applying a schistosomiasis early warning index (SEWI) in the lower Yangtze River Region of Jiangsu Province, China. PloS one. 2014;9(4):e94012 pmid:24705352
- 34. Wang Y, Fu R, Wang Z, Bao H, Chen Y, Yang F, et al. Assessing the Quality of Care for Patients With Acute Myocardial Infarction in China. Clinical cardiology. 2015 Jun;38(6):327–32 pmid:26074496
- 35. Chen L, Huang LH, Xing MY, Feng ZX, Shao LW, Zhang MY, et al. Using the Delphi method to develop nursing-sensitive quality indicators for the NICU. Journal of clinical nursing. 2017 Feb;26(3–4):502–13 pmid:27404730
- 36.
Dalkey NC. The Delphi Method: An Experimental Study of Group Opinion1969.
- 37. Brook RH, Mcglynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care. New England Journal of Medicine. 1996;335(13):966 pmid:8782507
- 38. Hilborne LH, Leape LL, Bernstein SJ, Park RE, Fiske ME, Kamberg CJ, et al. The appropriateness of use of percutaneous transluminal coronary angioplasty in New York State. Jama the Journal of the American Medical Association. 1993;269(6):761 pmid:8423657
- 39. Kahn KL, Rogers WH, Rubenstein LV, Sherwood MJ, Reinisch EJ, Keeler EB, et al. Measuring quality of care with explicit process criteria before and after implementation of the DRG-based prospective payment system. Jama the Journal of the American Medical Association. 1990;264(15):1969 pmid:2120476
- 40. Schouten JA, Hulscher MEJL, Wollersheim H, Braspennning J, Kullberg BJ, Meer JWMVD, et al. Quality of Antibiotic Use for Lower Respiratory Tract Infections at Hospitals: (How) Can We Measure It? Clinical Infectious Diseases. 2005;41(4):450–60 pmid:16028151