Is it time to use checklists in mental health care auditing?

A key strategy for improving the quality of mental health care is the design and implementation of a mechanism for on-site inspection and clinical auditing. We discuss the use of checklists in auditing providing an objective, comprehensive system for recording and analyzing multi-disciplinary, clinical auditing in mental health services. We believe such an approach can identify potential risks and allow for better decision making.


Is it time to use checklists in mental health care auditing?
The use of checklists as a tool for auditing mental health care is increasing rapidly. Once employed only in the aviation and construction industries, the checklist has now hit the shores of clinical medicine. Though still not a tsunami, through the work of Gawande, who led the way, and others, this wave has made a splash in medicine and surgery, and has attracted public attention. [1][2][3] Psychiatry has always been wary of any kind of categorization in connection with mental health. It was only in 1980, years after defending itself from anti-psychiatry charges of capricious and unreliable diagnostic labeling, that the APA finally published its revolutionary DSM-III. This new disciplined approach caused a transformation in the practice of psychiatric medicine. Now the question is whether the time has come to use a similar, well-ordered approach for quality control of services in the field of mental health.
Mental health services in Israel have evolved and changed in recent years. The Ministry of Health has always depended primarily on on-site clinical inspection of inpatient and outpatient psychiatric services to ensure compliance, but mental health services have changed in recent years. Recent legislation (Community-Based Rehabilitation of the Mentally Disabled Bill of 2000) spurred the introduction of a plethora of services to meet the unique needs of long-term rehabilitation for the mentally ill. 4 As a result, the number of hostels and supported housing facilities has grown from eight hostels in 1998 to 140 in 2009, and the number of clients using supportive housing increased from 1000 to 7000. 5 Moreover, housing services and sheltered workshops as well as supported education and social activity services have all been privatized.
Whereas at one time it was possible to conduct on-site clinical inspections, write individual reports, send the results to the appropriate institutions, and monitor the results, the dramatic increase in the number of sites to audit and follow up created a pressing need for a new regulatory design. Moreover, the classic on-site inspection of a nurse and psychiatrist, sufficient for inpatient psychiatric units was replaced by a multi-disciplinary committee of mental health clinicians (psychiatrists, psychologists, psychiatric social workers, registered nurses) and additional professionals (nutritionists, occupational therapists) as needed. Many of the privatized services require the expertise of an accountant to oversee their financial records and prevent fraud. Different relevant disciplines have different modes of inspection and different ways of looking at the auditing process. In the course of examining individual incidents of subpar care, we often encountered excellent auditing reports that were entirely misleading (e.g., one hostel with consistently high marks for general rehabilitative care was found to have tenants suffering from frequent, life-threatening hypoglycemia because the staff was improperly trained to care for diabetics). It thus became very clear that an integrated, standardized system was needed. As a small country, Israel has the advantage of being able to coordinate a unified system of clinical auditing using universal criteria for the entire network of patient care. This article examines the issues we considered in the design of an integrated checklist system.

What parameters should be inspected?
In his recent book, The Checklist Manifesto: How to Get Things Right, Atol Gawande (2009) asserts that many errors in the areas of health care, government, law, and finance are the result of not properly using what we know because the sheer volume of information is overwhelming. 3 Thus, he says, we can try to simplify things by constructing manageable checklists of items in clear, precise terms. The items we use in these checklists are taken directly from consensus standards and are accessible on-line. A typical checklist may include questions concerning the cleanliness of the floor of the facility's day room, the docu-mentation of known allergies in a patient's file, and whether there has been a fire drill at the facility within the last 12 months. Each item has a rating scale, an explanation as to the procedure by which the item should be scored (e.g., a random audit of 50% of patient files, an audit of the 20 most recent patient files, and so on), the maximum number of days allotted to submit a correction plan depending on the observed deficiency, the source of the information required (e.g., patient records, observation, interview with staff or family members, etc.), and a link to the site where the relevant standard is posted on the Internet. In this way, an item such as whether the patient receives the minimum recommended daily caloric intake in his diet can be listed on the inspection checklist for both a hostel setting and a psychiatric ward, and the scores can be compared with fairness and accuracy.

Should inspections be geared specifically to mental health?
In the past, it might have been possible to think of mental health primarily in terms of acute psychiatric treatment delivered in a onedimensional, hospital-type venue. Now, of course, a wide spectrum of inpatient, outpatient, and rehabilitative services, all staffed by highly-trained personnel, is available to serve the public. Should a sheltered living rehabilitation setting in the community be defined as a psychiatric facility and be inspected as such? Is it more similar to a psychiatric ward in a hospital, or should it be evaluated in a way not unlike that used for nursing/residential homes in the community? Similarly, should a closed psychiatric ward be inspected no differently than a medical intensive care unit? In both settings, the primary interest is the delivery of high-quality, focused, and dedicated care for the patient. To simplify the auditing process and create clear and consistent standards, we decided to use a generic approach. Consensus standards of care are listed as items in an inspection checklist. Every sub-unit within a larger facility is inspected as a separate entity. For example, a large mental health center would be broken down into its individual units (e.g., laboratory, pharmacy, dental clinic, outpatient clinics, etc.). A generic approach allows for the multi-disciplinary team inspecting mental health facilities to use the same checklists and comparable scoring systems, so a comprehensive report can be generated that summarizes the findings of such a multidisciplinary audit. Moreover, if the dietary care, for example, was sub-standard, a sheltered living setting would not pass inspection even if the rehabilitative care was superb.

What is the hierarchy of importance of the individual items inspected?
All the items to be audited in all the clinical settings fall into one of three general categories: (i) Items involving patient safety. (ii) Items involving proper procedure and documentation. (iii) Other items (e.g., those requiring structural changes or site relocation). These categories are easily defined and cut across a wide range of clinical settings. Safety, for instance, is the first of six aims for high-quality care listed by the Institute of Medicine Committee on Quality of Health in America. 6 Thus, hazards in an inpatient setting would include, for example, an incompletely equipped resuscitation cart; 7 expired vials of depot anti-psychotic medication in an ambulatory setting; or a defective or unsafe elevator in a sheltered housing setting in the community. Hazards such as these must be corrected immediately, and the auditor would revisit the site the following week.
Proper procedure and necessary clinical documentation can be similarly defined in various clinical settings; we stipulate that a procedure or protocol demanded at the inspection must be presented within 21 days of the audit. Long-term items, such as plans to correct structural changes (e.g., constructing a ramp for the handicapped) must be shown for inspection within 90 days.

When does a facility fail an inspection?
We defined three ways in which an inspection can be failed; once again, these are rules that can be applied across the board in virtually any type of mental health setting: (i) The facility scores less than 60 overall (arbitrarily set for a trial period). (ii) The score on a specific topic/area of inspection (e.g., patients' rights, maintenance and cleanliness of the facility) is below 60. (iii) The failure to pass the criteria on a critical item. In every checklist, a few critical items are defined by a consensus of experts. If the criteria are not met on these specific items, the facility fails the entire inspection. These items are usually reserved for major lapses in patient safety or evidence of gross negligence (e.g., the mismanagement of a client's money or valuables). By determining the criteria for failure, facility personnel become familiar with the expectations, and fairness and consistency become apparent to caregivers and clients alike.

Are checklists practical in mental health settings?
Different ways of assessing mental health facilities have been tried over the years. Elements used to assess quality have included evidence-based practice guideline development and assessment, monitoring performance measures such as patterns of the utilization of services, and the use of consumer sat-isfaction questionnaires. [8][9][10] In proposing a generic checklist platform, we advocate a flexible system that can be used over a range of clinical settings and allow for changes in standards and checklists over time. Using a common method for recording inspections also simplifies integration of clinical and administrative databases (e.g., medical records and financial accounting). Similarly, effective checklists have to be short enough not to be ignored and still hit the salient points of the audit. The system we have developed has been running for more than two years. A total of 2236 facilities are listed in the database, of which 1841 are identified as units requiring inspection under the auspices of the Ministry of Health. The remainder, although they house clients treated by mental health professionals, are supervised by other agencies.
Questionnaires have been developed for clinical auditing of inpatient, ambulatory, and rehabilitation services taking into account such matters as daily routine and medical treatment, clinical nutrition, patients' rights, general upkeep of the facility, among others. It is also possible to use the questionnaires in survey form, that is, to send forms by e-mail to all relevant units, have them fill out and grade the items themselves, and then compile the results (Table 1). These results can be extracted from the system and used to plan specific on-site inspections, or they can be correlated with other data. On-site evaluations can be compared with the survey results to verify validity and complete the audit cycle.
A facility's data can be compared on a timeline to look for improvement or deterioration in the quality of care. For example, Table 2 Review shows the results of an audit and a re-audit conducted in 2008 and 2009, respectively, for sample items at three inpatient wards in a major mental health center. Some items show no change, whereas others demonstrate improvement (the addition of two-way intercom devices in all clinical wards, in this case). A longitudinal analysis of specific items examined in consecutive audits can thus become a powerful tool for improvement. It is difficult to find common denominators in nomenclature (e.g., patient vs. client) and in criteria for passing inspections (e.g., excellence vs. adequacy) among the different mental health services. Moreover, potential conflicts between the perspectives of the various mental health professionals have to be negotiated and reconciled. We have found that using regulations and standards of care as the basis for the items on the checklist enables us to bypass the thorny issue of evidence-based or clinically effective quality assurance. For example, we may not know of any randomized trial positively correlating the improvement of the psychiatric prognosis with the wearing of name tags by the personnel in a psychiatric ward (a regulation that is checked and customarily scored during on-site audits). Yet it seems that it is important in terms of patients' rights and the general accountability of staff. It is entirely possible that staff may feel that name tags are more appropriate for flight attendants and are inconsistent with a personalized approach to the client. Even so, it would appear that common sense and face validity are in total agreement in this illustration, and similarly in connection with many items that are routinely examined.
It is common practice for managers of facilities that fail an audit to seek litigation. Naturally, it becomes easier to defend a potentially controversial decision to close a shelter or hospital ward, for instance, when checklists are standardized; when the times allotted for correction are similar for items of comparable importance; and when the criteria for failure are consistent, transparent, and have been agreed upon by a range of professionals.
An added advantage to the system described here is that it lends itself to being taught and integrated into a curriculum for training new inspectors. Each item is checked according to instructions that are laid out explicitly. Thus, inspection to inspection reliability is increased as is the integrity of the entire system. Moreover, cooperation with hostel managers and department and hospital heads is enhanced as the inspection process becomes transparent, on-line, and thus de-mystified.
Some would argue that defining external standards (as we suggest) may cause a top down approach to assessment that can actually be more detrimental than helpful in improving services. External standards may undermine staff responsibility and create an atmosphere of finding a guilty party to assign blame. We do not claim that we have proof that a checklist system definitely improves the quality of mental health services, but rather we contend that this type of auditing sense seems to be in harmony with common sense and practice and that it seems to be working well in other medical settings . 11 We suggest a system for recording and analyzing multi-disciplinary, clinically-oriented independent auditing in mental health services. Just as the DSM was a long time in coming, checklists should eventually prove to be worthwhile in regulating and upgrading mental health services.