Quality assessment of primary care for common mental disorders in isolated communities: Taking advantage of health records

Introduction: This article is part of a research study on the organization of primary health care (PHC) for mental health in two of Quebec’s remote regions. It introduces a methodological approach based on information found in health records, for assessing the quality of PHC offered to people suffering from depression or anxiety disorders. Methods: Quality indicators were identified from evidence and case studies were reconstructed using data collected in health records over a 2-year observation period. Data collection was developed using a three-step iterative process: (1) feasibility analysis, (2) development of a data collection tool, and (3) application of the data collection method. The adaptation of quality-of-care indicators to remote regions was appraised according to their relevance, measurability and construct validity in this context. Results: As a result of this process, 18 quality indicators were shown to be relevant, measurable and valid for establishing a critical quality appraisal of four recommended dimensions of PHC clinical processes: recognition, assessment, treatment and follow-up. Conclusions: There is not only an interest in the use of health records to assess the quality of PHC for mental health in remote regions but also a scientific value for the rigorous and meticulous methodological approach developed in this study. From the


Introduction
Depression and anxiety disorders can have a negative effect on an individual's professional, academic, social and family life, with significant consequences for loved ones 1,2 .The high prevalence of these disorders is pushing health systems to increase the roles of primary health care (PHC) in order to recognize, assess, treat and follow up the majority of those affected by these conditions 3 .For individuals and families, PHC represents the first level of contact with the system of care and services (SCS) 4 .These services are regarded as less costly, more accessible and often more acceptable than specialized mental health services, which should be reserved for individuals with more complex conditions and needs 3,5 .The WHO recommends evaluation organizational methods and clinical practices to ensure the effectiveness and quality of PHC for people with common mental disorders 6 .
In the province of Quebec, Canada, little information is available on the nature and quality of PHC offered to people with mental health problems living in isolated rural communities.In addition, few assessment tools are available to decision-makers regarding quality of care adapted to their context.Researchers worked in collaboration with decisionmakers involved in two isolated regions of Quebec to adapt procedures and research tools so as to provide them with a way to assess the quality of PHC offered within their populations to those with various mental health-related needs.This article focuses mainly on the methodology developed for assessing PHC quality associated with common mental disorders, such as depression and anxiety disorders.

Primary health care in isolated rural communities in Quebec
Quebec province is a vast area of 1 667 441 km 2 .The province is sparsely populated: the majority of its eight million residents live in the southern part and occupy approximately one-fifth of the territory.The official language is French.Approximately 1% of Quebec's population live in isolated areas and 50% of this segment of the population belong to one of the 11 indigenous groups in the province 7 .From the health services perspective, isolated rural communities are defined as those situated more than 3.5 hours of travel time from a major urban center 8 .Health services are generally offered by a multidisciplinary team within the healthcare providers (nurses, social workers and community workers).To meet the health and social needs of the population an expanded clinical practice is adopted 9- 14 .Most health centers in isolated rural communities have no doctor or psychologist on site, but remote support is available to the PHC team and medical visits are organized on a regular basis.However, the provision of comprehensive and effective care for people with mental health needs can pose challenges, given the scarcity of local mental health resources, geographical distance from specialized resources, inadequacy of proposed guidelines, and the lack of support for initial clinical management.This situation is further complicated by a high employee turnover, because a large number of PHC providers come from elsewhere [9][10][11][15][16][17][18] . In man cases, problems could remain undiagnosed 18 .

Measuring the quality of care and services
Quality of care is about 'delivering the best possible care and achieving the best possible outcomes for people every time they deal with the healthcare system or use its services.Essentially, it means doing the best possible job with the resources available' 19 .Accordingly, quality care is effective, person-centered, accessible, timely, efficient, safe and fair 20,21 .Quality is typically measured based on indicators used as reference points.Quality indicators can relate to healthcare structures, to technical (prevention, detection, access, assessment, treatment/adherence, coordination, continuity, safety) and interpersonal (communication, decision-making, interpersonal style) clinical processes, as well as to outcomes for patients or healthcare systems 22 .Indicators can be developed through systematic or nonsystematic methods 23 .In systematic methods, indicators are first identified using evidence-based data from empirical studies or practice guidelines.Then, the indicators are reviewed by experts in the field to retain the most relevant 23 .In non-systematic methods, indicators are identified through available data and critical incidents documented from case studies 23 .

Taking advantage of health records in remote regions
Three data sources -patient-reported data, administrative records and health records -are frequently used to assess quality indicators in PHC settings 20,24,25 .Patient self-reported data are collected using questionnaires or by conducting interviews.These can provide information on technical and interpersonal aspects of care as well as their impact on patient experiences of care 25 .Such an approach has been used in mental health research in isolated populations 18,26 .However, concerns were expressed by decision-makers involved in the present project about repeated solicitation from the research community and its potentially negative impact on the population under investigation.The advantage of medical administrative databases is that information can be systematically collected for all service users across a given clinical setting or system.In mental health, this is based on the Diagnostic and statistical manual of mental disorders (DSM) 27 and the International classification of diseases (ICD) 28 , standardized diagnostic coding systems recording hospitalizations, outpatient visits and prescribed medications 25 .However, the databases currently available in Quebec do not consider the particularities of PHC in isolated rural communities, such as the expanded role of healthcare providers [9][10][11][12][13][14] .
This study aimed to better support local and regional stakeholders in their efforts to improve their population's access to quality mental health services by addressing the research question, 'Are health records a valuable source for assessing quality of PHC offered to individuals with common mental disorders who live in Quebec's isolated rural communities?'The goal was to identify useful, measurable and valid indicators 12,24,25,29 adapted to the context of health services in isolated rural communities and that could be measured from information found in health records.
The focus was primarily on developing indicators for technical-type clinical processes that reflect the four steps for managing service needs for those with common mental disorders: (1) recognition, (2) assessment, (3) treatment and (4) follow-up.Recognition involves early detection so that common mental disorders can be treated early to minimize functional impairment 30,31 .Information is collected during the assessment to identify the best treatment for the individual 32- 34 .This information includes current medical history, signs and symptoms, risk of suicide or hetero-aggressiveness, health, illness and treatment history, lifestyle, diagnoses, functional impairment intensity and the individual's personal preferences.Stepped care models are recommended in several practice guidelines for common mental disorders 33,35,36 .This approach suggests varying treatment types and levels based on the individual's needs and preferences 37,38 .The PHC providers have also to take into consideration their own clinical experience/skills, current legislative policies and available resources 38 .Consequently, patients with mild functional impairment are typically offered low-intensity interventions (such as bibliotherapy, supportive self-management, lifestyle interventions, light therapy, brief psychological treatment, all of which are readily available to all care providers).Conversely, high-intensity interventions (pharmacology, psychotherapy, combination therapies, electroconvulsive therapy, hospitalizations) are available to individuals with more severe symptoms or who have relapsed 33,36,39 .They require specialized medical care or mental health resources.Finally, the effectiveness and efficiency of care relies on the progression of treatment intensity based on an individual's health status and the implementation of sustained personalized follow-up 37,38 .

Methods
Using a systematic method to create quality indicators could result in assessment criteria that are too rigid and unsuitable for remote isolated regions.On the other hand, a nonsystematic method based on case studies could limit the potential for comparison of various remote isolated regions and identication of promising ways for improvement.Therefore, indicators were developed using a hybrid approach that integrates evidence, opinions from key informers and case studies derived from health records.

Locate evidence
A preliminary list of quality-of-care indicators adapted to isolated rural communities was created using scientific evidence sources, institutional records and case studies.Evidence was identified on the basis of recognized practice guidelines for depression and anxiety disorders 40,41 and indicators from the National Inventory of Mental Health Quality Measures (NINHQM) 25 .Institutional records refer to two qualitative studies on mental health services done in isolated regions of Quebec 10,11 and that were consulted in order to identify contextualized aspects deemed to be good quality indicators.Case studies were reconstructed using data collected from the health records of people with common mental disorders in the two isolated rural regions being studied.Means undertaken to gather information from health records are presented later in this section.

Ensure adaptation of quality indicators to the context of rural isolated communities
The adaptation of quality-of-care indicators to rural isolated communities was appraised according to their relevance, measurability and construct validity in this context.
Relevance was examined by analysing each indicator on the preliminary list of quality-of-care indicators.This was done together with three members of the mental health team (one psychologist, a nurse and a social worker) from two participating health centers.The aim was to retain the relevant indicators and make adjustments as necessary.Each indicator was presented and discussed in face-to-face meetings.
Measurability involves verifying the possibility of reconstructing indicators deemed relevant using the data collected from health records and establishing a measurability score for each indicator, if applicable.The allotment of the measurability scores aims to promote use of the indicators by research partners.Figure 1 presents the measurability chart where the score (1=easy, 2=moderate, 3=complex) took into account the nature of the denominator and the type of variables.Denominators could refer to the entire observation period, specific events or care episodes, ie a consolidation of related events that could extend over time.Variables can be primary variables stemming directly from the collected data (Fig2) or secondary variables created from primary variables or narrative text.
The construct validity ultimately relies on the known-groups method, which considers an indicator's construct to be valid when it is possible to demonstrate statistically significant differences between groups known to be theoretically different 42,43 .For this, the results of relevant and measurable indicators are subjected to a series of bivariate analyses based on individual characteristics (sex, age group, presence of chronic physical illness, presence of substance abuse disorders, presence of social problems, predominance of anxiety symptoms), organizational characteristics (presence of a local permanent physician) or contextual characteristics (regions) likely to cause measurement differences and documented from health records.When the denominator referred to the observation period, bivariate analyses were conducted using the χ 2 test.When the denominator referred to events or episodes of care, the generalized estimating equation (GEE) was used to account for multicollinearity data, given the possibility for a single individual to accumulate many events/episodes 44 .

Gather data from health records
A primary focus was on individuals aged 14 years or more presenting depression or anxiety disorder and using PHC in isolated rural communities throughout Quebec.The choice of this minimum age satisfies the need of research partners to check if there are breaks in the continuum of care between youth and adult services.The data collection in health chart was deployed in an iterative three-step process: (1) feasibility analysis, (2) development of the data collection process and tool and (3) the actual data collection.
1. Feasibility analysis: In this first step, health records of patients with a common mental disorder were examined to verify the type of sociodemographic and clinical information they contained, to assess their relevance, clarity and format 45,46 , and to identify elements, such as biases or potential errors, that might affect their validity 45,47 .This step was conducted over a 5-day period in 2007 in an isolated rural community health center in Quebec.Since no registry exists to identify patients with common mental disorders, a systematic review of the archives was performed to identify patients presenting a mental disorder.In health center archives, health records are organized by file number, in ascending order.These numbers are assigned chronologically by the regional health service: at birth for all those born in a community, and at first consultation for those from outside the community.
Notes were reviewed for all selected individuals aged 14 years or more on 1 January 2006, to identify those who presented with mental disorders.In the absence of a relevant medical diagnosis, a set of psychological counselling, medication(s), clinician impressions (ie differential diagnosis), or signs and symptoms associated with mental disorders, was used to identify potential cases.During the first day of the search, health records for 305 people were reviewed.Of these, 29 (9.5%) had a confirmed or suspected diagnosis of mental health disorders (severe or common).During the remaining four days, these 29 records were examined in detail to document each step of the clinical management process: recognition, assessment, treatment and follow-up.Data collected included sociodemographic characteristics, nature of the consultations, services consulted, reason(s) for initial consultation, current medical history, clinical signs and symptoms.Information retrieved based on documentation practices varied between providers, especially when it came to assessment of comorbidity, physical/mental/social history, substance abuse disorders and the individual's resources.Level of functional impairment was rarely documented; however, this could often be established retrospectively from clinical notes (ie symptoms and illness trajectory).More importantly, the record review showed that a 12-month observation period was not sufficient to assess a longitudinal clinical process such as follow-up.Confounding issues were frequent and included use of acronyms, illegible notes, and dual language documentation (eg dates are written differently in French and in English: YYY-DD-MM in English and YYYY-MM-DD in French).
2. Developing a data collection process and tool: Observations made in step 1 influenced how health records were selected, the length of the observation period, the type of variables required, and also the competencies of those who would be assigned to data collection.The secondary purpose of step 2 was to develop data collection tools.Given the absence of a registry for individuals with common mental disorders and the imprecision of certain clinical diagnoses, a tracking method based on experiences from step 1 was developed for locating health records.Records meeting the following criteria were selected: 1. Individual was aged 14 years or over on 1 January of the reference year.The second criterion was observed starting from the reference year end (December to January).A 2-year observation period was considered sufficient to obtain valid care trajectories.This period of observation was applied by creating a window of 12 months before and after the first observation of the second selection criterion.
The need to record a large amount of information over a long time period justified the use of a computerized tool.The data entry form was set up using the freeware program Epi Info v3.5.1 (Center for Disease Control and Prevention; http://wwwn.cdc.gov/epiinfo) that automatically generates an Access database.This database consists of two tables linked by a numerical code assigned to each record.Each row in the first table corresponds to a person and each row of the second table documents a note to the health record.
A first pre-test was done to ensure the feasibility and reliability of the data collection process and tools.In the pre-test, the average time to identify and process a record was set at 180 minutes.An inter-rater reliability test was carried out afterward in an isolated rural community by two bilingual (French, English) research nurses with experience working in remote regions.Using six records, an inter-rater agreement was measured on 15 statements corresponding to types of events (n=6), assessment aspects (n=4), treatment (n=4) and follow-up (n=2) from 30 randomly determined events.Adjustments were made to all statements generating an inter-rater agreement (kappa) less than 0.80.Whenever research nurses had trouble classifying certain information because of wording differences between health notes and the data entry form, some check boxes were replaced by text boxes.Figure 2 presents the fields from the final version of the data entry form.Based on the findings in step 1 (ie prevalence of 9.5%), the target population was estimated at 1090 records.Given the processing time (180 minutes), the estimated sample that could be obtained in the allowable time (918 hours) was established at 360 records.Records were selected using a randomly drawn number and reviewing all records that ended with that particular number.The selection process lasted until the pre-established number of records for each community in proportion to its population was reached.
Each day, the information gathered was submitted to a member of the research team in order to correct any input errors before the research nurse moved on to another community.The database included an electronic communication platform for sharing observations on processed records, sites visited and notes on the data collection process.
Health records from 3669 people aged 14 years or more were reviewed for all clinics and 290 records (7.9%) met the selection criteria.For these records all care or service events associated with mental health were documented.The collected data were anonymized and an alphanumeric code assigned to each health record.
All diagnoses rendered during the observation period were reviewed to identify cases of depressive or anxiety disorders.
If no diagnosis was available, research diagnoses were established using signs and symptoms documented during the 2-year observation period.These signs and symptoms were included as free text in the database (in narrative text format) and had to be classified beforehand based on diagnostic criteria from DSM-IV-TR (4th edition, text revision) for a major depressive disorder and the most common anxiety disorders 27 .Records where there were at least two signs and symptoms corresponding to diagnostic criteria for depression or anxiety disorder unexplained by other conditions were retained.The final sample for common mental disorders included 218 records (5.9%).

Ethics approval
This research received administrative approval from participating health centers for the access to health records and ethics approval from the Research Ethics Committees of the Agence de la Santé et des Services Sociaux de Montréal (Project 162) and Université Laval.Researchers and nurses who accessed the health records for this research signed a confidentiality agreement beforehand.

Selection of indicators
A total of 36 quality indicators for clinical procedures that could be adapted to isolated rural communities were initially selected after reviewing relevant practice guidelines for depression and anxiety disorders 40,41 and using indicators from the NINHQM 20 .Eleven new indicators were added following the analysis of the mental health services qualitative studies 10,11 and the 10 case studies from health records.Finally, a total of 47 quality indicators were set at the start of the study process.

Relevance of study
Members of the mental health team rejected three of the 47 indicators as irrelevant: rapid access to services (treatment), assessment of psychotic features in depression (assessment) and involuntary detention (treatment).First, rapid access to care in an emergency was not deemed an issue in this context since every person has immediate access to a nurse 24 hours a day and 7 days a week.Access to a second level of care can vary significantly from one context to another and establishment of a standard is difficult.Second, the assessment of psychotic features in depression was not deemed a priority as indicators for assessing suicide risk or concomitant disorders were preferred.Third, involuntary detention was considered descriptive data rather than a quality-of-care indicator.

Measurability and construct validity studies
Of the 44 indicators remaining, 10 could not be measured using the health record data.Their nature and the reasons for rejection are specified in Table 1.The definitions, origins, nature of the denominators, primary and secondary variables needed to create them, measurability scores as well as measures of the 34 indicators considered relevant and measurable are illustrated in Table 2. Of these, 11 indicators could not be subjected to bivariate analyses because a denominator was less than 30 (n=5), there was an insufficient number of occurrences per compared group (n=3), or the services were not available at the time of the research (n=3).
The remaining 23 indicators were cross-referenced for various individual, organizational and contextual characteristics documented from health records.Eighteen indicators were significant (p≤0.05) for at least one comparison group (ie construct validity) (Table 2).These indicators were mainly related to the assessment (n=5) and follow-up (n=8) processes.
Table 3 shows that scientific resources (empirical studies and practice guidelines) helped identify 12 relevant, measurable and valid indicators.This represents a third of the 36 indicators initially identified using this source.Consultation of institutional records would have led to the selection of 9 indicators, 41% of the 22 identified using this source.As for case studies, 10 indicators would have been identified, 55% of the 17 identified using this source.This table also shows that all the indicators identified using institutional records were identified with the two other sources as well.These indicators were all derived from the practice guidelines/NINHQM and case studies.

Discussion
This article addresses the question of whether health records are valuable sources for assessing the quality of care offered to individuals with common mental disorders who live in isolated rural communities.Making publicly available the tools developed and the selected indicators necessitates a discussion on the strengths and challenges associated with data sources, the representativeness of the sample, the method used to identify quality-of-care indicators and the usefulness of developed indicators.Health records as a data source In the field of mental health, health records have been described as underexploited data sources 47 .They provide an abundance of clinical information while keeping participants anonymous, which is critical for conducting research in small communities.In addition, the use of health records is facilitated by the simplicity of clinical structures in isolated rural communities.Isolated areas have an advantage over non-isolated areas where this type of exercise is complicated by the multiple access points to healthcare systems.However, health records also raise methodological challenges and limitations to be considered.First, records are not systematic and documentation varies widely.The quality of notes is an important factor/variable that is not controlled for.Moreover, attempting to decipher why certain information is missing is arbitrary.Did the PHC provider forget to write it down or does it reflect the inconsistency of clinical processes?Therefore, results are based on the feeble assumption that PHC providers, at least, noted information on what was reported to them, what they observed and what they did.Accordingly, a major challenge in using records relates to the complexity of data collection 45,47,49 .The credibility of collecting data from health charts is based largely on the documentation of biases and potential entry errors during previous steps, which helped develop ways to control them.Recording data in a free/narrative text format significantly increased data collection time and complicated data preparation for analysis.Staff motivation, isolation and fatigue were also a challenge in the data collection process and, as a result, time spent in the field was reduced from 6 weeks to 4 weeks.In short, health records are valuable for assessing the quality of care offered to individuals who have common mental disorders and live in remote regions, but the data collection process is demanding.Also, additional data collection for monitoring quality of care in participating regions could be simplified.A review of primary and secondary variables needed to create indicators (Table 2) revealed that all the categories of collected data (Fig2) were useful.However, documentation of non-mental healthrelated events could be trimmed (eg dates, PHC providers involved and reasons for generic consultation).Check boxes could also be reintroduced since the most frequently used terms to describe signs and symptoms, as well as assessment and treatment components, were documented during data collection.These terms could be specified in the reference manual.

Representativeness of the sample
The method used for identifying records involving individuals with common mental disorders circumvents the underdiagnosis of these conditions in primary care 18,32 .The rate of 5.9% of records (n=218) obtained for people 14 years or more appears to be consistent with national prevalence rates, which show that about 12% of Canadians undergo a high level of psychological distress each year, indicating a possible major depressive disorder or other mental problems 50 .Of these individuals, 45% will consult healthcare services for these matters 51 .The selection of records based on signs and symptoms associated with common mental disorders identified 22% of the cases.These selection criteria for cases go beyond the medical diagnosis to assess the recognition function and expand the quality-of-care study to include those individuals who had not met all the diagnostic criteria for common mental disorders, but who had symptoms that could develop into a disorder if nothing was done.However, these 'expanded' criteria promote sensitivity (selecting real cases) at the expense of specificity (rejecting false cases).Use of clinical and research diagnostics revealed an efficient way to identify and reject any such false cases.
Studying a large enough sample to translate results to the general population poses scientific and logistic challenges when research is carried out in remote settings, largely due to limited populations and expenses.The fact that certain indicators might be measured from events or care episodes enhanced statistical power.Also, while statistical significance was not reached in the retrospective study of 10 records, the review produced clinically useful results 45,47,52,53 .Considering these limitations, any quality-of-care assessment in remote regions would benefit from a mixed research design to triangulate results, either statistically or clinically, including qualitative data and input/interpretation by local stakeholders.

Method used to identify quality-of-care indicators
In the present study, the hybrid method of reviewing scientific evidence and case studies from health records would have been appropriate to identify relevant, measurable and valid quality indicators.This hybrid selection method underscores the importance of contextualizing indicators because certain indicators initially deemed measurable theoretically (eg mention of respect for the person's preferences 25 ) were not in fact measurable.Therefore, a strength of the approach used in this study was to compare data with the opinion of key informers.

Usefulness of developed quality-of-care indicators
Current mental health reforms advocate integration of resources and between service levels 3,39 .Yet, service integration experiences producing the best results are those that place an added value in clinical practices rather than in structures 54 .Assessing care is therefore a useful way to target clinical practices in need of improvement 55 .Also, data collection tools and databases using accessible software (Access and Epi Info) as well as measurability scores 20,56 are designed to promote transferability to research partners from participating regions.With measurability scores (Fig2), research partners can easily identify indicators for which data are easier to collect and analyze, and include them in an audit process.The transferability of developed indicators and deployed method to other settings with similar processes of care and contexts need to be studied.
It is also evident that more work is needed to better document the relationship between clinical procedures and desired outcomes in the context of research 55 , to ensure the sensitivity of the indicators to change 23 and to adjust them over time 20 .Several indicators are used to measure follow-up and assessment functions.However, few of these measure treatment aspects, including adequacy of care.To achieve this, research partners from isolated regions must work to adapt guidelines to their context (culture, available resources, etc.).While the approach in this study uses data collected a few years ago, the indicators developed are still relevant.Measurements from the 2006-2008 data can be used as a benchmark for assessing progress.Descriptive information can be drawn from the data collected in health records in support of this study.Most work on the adequacy of treatment for a common mental disorder is derived from a medical diagnosis 24 .Use of the functional impairment level, which can be established after the fact from narrative text and the Global Assessment of Functioning (GAF) scale 27 , is an avenue to explore for assessing future treatments.The GAF scale allows PHC providers or researchers to rate subjectively the social, occupational and psychological functioning of adults from their health history on a hypothetical continuum of mental health-illness (1 through 100).Such an approach appears well suited to multidisciplinary primary care especially when no diagnosis is available, as is often the case in isolated rural communities 18,26 .

Conclusions
Depression and anxiety disorders affect society's most productive age groups and represent a burden in small isolated rural communities [57][58][59][60] .To target areas for improvement in care, and for providers to offer solutions adapted for affected individuals and their families 54,61,62 , they must be relevant, measurable and valid quality indicators that are contextualized to local challenges.The approach presented in this paper has identified 18 quality indicators that may be useful for better understanding the PHC offered in rural isolated communities.These indicators are a suitable monitoring tool for those involved in improvement of quality of care in these contexts.

Figure 2 :
Figure 2: Fields for the data entry form

3 .
Data collection: This step involved collecting data in 21 isolated rural community clinics that are associated with three health centers in two of Quebec's health regions.The population of 21 communities contained 11 500 individuals aged 14 years or more 48 .Data collection was carried out between October 2009 and August 2010.Reference year was set as the year 2007 and therefore the 24-month observation period was between January 2006 and December 2008.

Table 1 :
Non-measurable quality-of-care indicators for common mental disorders (n=10)

Table 3 :
Number of quality indicators selected according to function and from data sources