Factors Affecting the Validity of Self-Reported Data on Health Services from the Community Health Survey in Korea

Self-administered questionnaires are among the approaches used to examine the current status of public health, including health behaviors, healthcare utilization, disease prevalence, and so on. Despite the ease with which this method acquires data, especially in situations in which more objective records are inaccessible or unavailable,1 this approach is limited by its dependence on the subjectivity of responses, which may detract from the accuracy of information collected. Indeed, the accuracy of data obtained via self-report questionnaires is relatively low compared with that gathered from either medical records or insurance claims.2-4

Community health surveys (CHSs) have been conducted in Korea since 2008. They have illuminated the current status of public health in 253 local jurisdictions by querying 220,000 citizens and by consuming more than 12 billion won every year. Now, CHSs' scale such as subjects, and cost etc. is bigger than the Korean National Health and Nutrition Examination Survey (KNHANES). In order that the results of CHSs are used as evidence for establishing and evaluating diverse local public health projects, it is necessary for CHSs to achieve a certain level of accuracy and to identify what factors affect the validity.

To-date, the only study of the validity of CHSs was performed by Rim, et al.5 Though this research revealed that CHSs have a certain level of sensitivity and specificity for healthcare utilization, such as rates of hospitalization and outpatient visits, it didn't show what factors are related with the sensitivity and the specificity. The sensitivity and specificity of questionnaires on healthcare utilization depend on subjects' knowledge and understanding of the relevant information, their ability to recall this information, and their willingness to report it.6 As a follow-up for the validity study of CHSs with the same data, we evaluated the factors affecting the accuracy of CHSs by investigating subjects' characteristics.

The CHS that addresses demographic characteristics, rates of disease and healthcare utilization, vaccination and health behavior, quality of life, and socio-physical environment is now becoming the main data source on the health of 253 jurisdictions of public health center. This point is the distinctive advantage of CHSs compared with KNHANES which has only national health indices. As a result, information obtained by the CHS from respondents and data on time-series changes can be used to establish agendas for public health programs or to evaluate the results of such programs. However, none of the questions, with the exception of those about rates of diseases and healthcare utilization, produces information that can be compared with objective data. Although rates of disease and healthcare utilization can be examined by reference to data (e.g., insurance claims, medical records, etc.) that are relatively more objective than those obtained via self-report surveys, these issues have been investigated via questionnaires due to the easier access to data, efficiency, and lower cost associated with this method. Thus, research on the degree to which data obtained via self-report surveys are accurate and on the factors associated with accuracy will enhance the usefulness of information regarding rates of diseases and healthcare utilization gathered by such methods.

This study that is the follow-up study for the validity of CHS was to identify subject characteristics related to sensitivity and specificity for purposes of evaluating the validity of the CHS. In this study, the sensitivity of hospitalization and outpatient visit questions was highest among married individuals living without their spouse, those with chronic diseases, those with poor health behavior, those who recently received medical checkups, and those with less favorable assessments of their health status. In contrast, specificity was highest in single individuals, those with more education, those without chronic diseases, those who had not received a medical checkup recently, and those with a more favorable assessment of their health status. In other words, those with poor health, whether judged objectively or subjectively, have higher sensitivity and lower specificity. Conversely, those with good health, judged either objectively or subjectively, have lower sensitivity and higher specificity.

This study tried to identify factors related to validity of some variables, but existing relevant literature suggests that kappa values represent the extent of the fit between obtained and true values and have presented findings concerning reliability in terms of comparisons with objective data. That is, kappa is an index of reproducibility or reliability, and can exclude the degree of agreement of two data by chance. If kappa is more than 0.75, then reliability between two data is interpreted as excellent; if kappa is more than 0.40 and under 0.75, it is fair or good; and if kappa is under 0.40, then it is poor.7 One of them showed the concordance between subjective and objective reports on healthcare utilization (hospitalization, outpatient visits), and was higher among those with frequent healthcare utilization than among those with infrequent utilization.8 Some studies suggest that age, gender, educational background, and ethnicity are associated with the accuracy of self-reports.9, 10 In this study, specificity was higher in single individuals and in those with more education, whereas sensitivity was higher among married individuals living without their spouse. Additionally, some studies have reported that older age is the only demographic factor significantly associated with inaccurate and under-reported healthcare utilization.11, 12 Indeed, other studies have reported that the accuracy of self-reports was not related to demographic factors such as education, gender, health status, socioeconomic status, and so on.3, 13

Moreover, we found that sensitivity and specificity differed as a function of whether subjects had been diagnosed with chronic diseases. Sensitivity for both hospitalization and outpatient visits was higher in those with chronic diseases, whereas specificity was higher in those without chronic diseases. According to a comparative study reviewing self-reports and doctors' claim records for hypertension, the kappa value reflecting the concordance rate in those with diabetes was 0.42, which was significantly lower than that (0.56) among those without diabetes.14 In this study, however, the kappa value reflecting agreement between self-report and insurance claims data was higher in those with hypertension (0.58) than in those without this condition (0.55) with respect to hospitalization, and higher in those without hypertension (0.50) than in those with hypertension (0.25) in terms of outpatient visits. This difference between the concordance rates for hospitalization and outpatient visits can be explained as follows. Because hospitalization does not occur often, those with hypertension who were highly interested in health would be expected to remember with greater accuracy whether they had been hospitalized. On the other hand, because outpatient visits are common among those who suffer from hypertension, it is possible that patients with this condition would provide inaccurate information about their visits to healthcare institutions during the past 2 weeks, due to failure to remember if their last visit occurred within that time frame. In fact, specificity for outpatient visits was high among those without hypertension (88.5%), whereas it was low among those with high blood pressure (66.8%).

Studies comparing medical records and self-reports with respect to eye examinations and diabetes found the following kappa values for different levels of subjective assessments of health status: 0.15 among those rating their health as very good, 0.23 among those rating it as good, 0.25 among those rating it as fair, 0.25 among those rating it as bad, and 0.24 as those rating it as very bad, confirming that self-rated assessments were not related to objective evaluations of health status.15 This study found that sensitivity for hospitalization and outpatient visits was higher among those who assessed their health to be poor than among those who assessed it to be good. On the other hand, specificity was higher among those who evaluated their health to be good than among those who assessed their health to be poor. The kappa values for the concordance rates for hospitalization by subjective ratings of health were 0.46 for those rating their health as very good or good, 0.55 for those rating their health as fair, and 0.60 for those rating their health as very bad or bad, whereas those for outpatient visits were 0.33, 0.36, and 0.33, respectively. Among those who did not receive necessary medical services and those experiencing depression or sadness, sensitivity was high, whereas the specificity was low with respect to outpatient visits.

However, this study has the following limitations: first, we did not examine factors other than demographic characteristics of respondents that may influence validity. In particular, because more than 70 interviewers in 13 areas participated in this survey, it is possible that some of the results were due to differences in interviewing dynamics. In addition, we didn't fully consider respondents' medical condition. We only used major chronic diseases, not severe disease such as severe cardiac insufficiency, severe chronic kidney disease (mainly end stage renal disease), peripheral artery disease, cancer and so on. It was because this study used the secondary data. So, these issues should be evaluated in future research. Second, because the criteria for validity were limited to comparisons with health insurance data, it could be classified into 'those without healthcare utilization' in this study, which may in turn decrease specificity even though other types of insurance such as workers' compensation and car insurance are available. This kind of problem can be overcome by using other comparison data as the true values. Third, because data from only rural areas (13 public health centers) were used, the results of this study may not be generalized to large cities such as Seoul. Moreover, because the questions used for the validity assessment were limited to those on hospitalization and outpatient visits, the results are not applicable to the utilization patterns of other healthcare services, such as visits to emergency medical centers or dentists. This problem could be overcome by examining factors associated with the sensitivity and specificity of the entire CHS or by increasing the number of comparison variables.

In summary, this study revealed the subject-related factors associated with the validity of the CHS. Efforts to improve the sensitivity and the specificity from self-reported questionnaires should consider how the characteristics of subjects may affect their responses. One of them is to educate interviewers about how to elicit more accurate answers to questions about healthcare utilization before they begin to conduct CHSs.