Hand, foot and mouth disease in China: evaluating an automated system for the detection of outbreaks.

OBJECTIVE
To evaluate the performance of China's infectious disease automated alert and response system in the detection of outbreaks of hand, foot and mouth (HFM) disease.


METHODS
We estimated size, duration and delay in reporting HFM disease outbreaks from cases notified between 1 May 2008 and 30 April 2010 and between 1 May 2010 and 30 April 2012, before and after automatic alert and response included HFM disease. Sensitivity, specificity and timeliness of detection of aberrations in the incidence of HFM disease outbreaks were estimated by comparing automated detections to observations of public health staff.


FINDINGS
The alert and response system recorded 106,005 aberrations in the incidence of HFM disease between 1 May 2010 and 30 April 2012 - a mean of 5.6 aberrations per 100 days in each county that reported HFM disease. The response system had a sensitivity of 92.7% and a specificity of 95.0%. The mean delay between the reporting of the first case of an outbreak and detection of that outbreak by the response system was 2.1 days. Between the first and second study periods, the mean size of an HFM disease outbreak decreased from 19.4 to 15.8 cases and the mean interval between the onset and initial reporting of such an outbreak to the public health emergency reporting system decreased from 10.0 to 9.1 days.


CONCLUSION
The automated alert and response system shows good sensitivity in the detection of HFM disease outbreaks and appears to be relatively rapid. Continued use of this system should allow more effective prevention and limitation of such outbreaks in China.


Introduction
To improve control of infectious disease outbreaks, it is critical to establish early detection and warning systems. In recent decades, technological advances in computing and communication and mathematical aberrancy-detection algorithms have been applied to high-volume data sets, to generate alerts and draw the attention of epidemiologists to statistical anomalies that may indicate a localized outbreak or the elevated risk of such an outbreak. [1][2][3] Several national public health agencies have successfully developed and operated automated early warning systems for the prompt detection of disease outbreaks. [4][5][6][7][8] Some epidemiologists have simulated outbreaks to evaluate the performance of such systems and the associated outbreak-detection algorithms. 9,10 However, there have been few prospective evaluations of the performance of early warning systems in operational settings. 11,12 In April 2008, a web-based automated system for the early detection of -and rapid response to -outbreaks of infectious disease was implemented across China. 13 This system -the China infectious disease automated alert and response system (hereafter referred to as the response system) -was developed by the Chinese Centre for Disease Control and Prevention, with the support of the Chinese Ministry of Health and the World Health Organization. The response system was based on surveillance data on dozens of notifiable diseases and on several statistical algorithms for the automated and routine detection of aberrations in such data, at county level, that might indicate the early stages of potential outbreaks.
Although hand, foot and mouth (HFM) disease can be caused by serotypes of several enteroviruses, it is most frequently caused by coxsackie virus A16 and human enterovirus 71. Most affected people develop only mild symptoms but some cases may result in serious and even fatal complications. [14][15][16] In China, HFM disease is frequently detected in children aged less than five years 17 and there have been over a million cases of the disease, including hundreds of fatal cases, reported annually over recent years. 18,19 In this study, we aimed to evaluate the performance of the response system by analysing the sensitivity, specificity and timeliness in the detection of HFM disease outbreaks. We also wished to evaluate the response system's effectiveness by comparing the size and duration of HFM disease outbreaks -and the post-onset delay in reporting such outbreaks -before and after HFM disease was included in the response system.
Objective To evaluate the performance of China's infectious disease automated alert and response system in the detection of outbreaks of hand, foot and mouth (HFM) disease. Methods We estimated size, duration and delay in reporting HFM disease outbreaks from cases notified between 1 May 2008 and 30 April 2010 and between 1 May 2010 and 30 April 2012, before and after automatic alert and response included HFM disease. Sensitivity, specificity and timeliness of detection of aberrations in the incidence of HFM disease outbreaks were estimated by comparing automated detections to observations of public health staff. Findings The alert and response system recorded 106 005 aberrations in the incidence of HFM disease between 1 May 2010 and 30 April 2012 -a mean of 5.6 aberrations per 100 days in each county that reported HFM disease. The response system had a sensitivity of 92.7% and a specificity of 95.0%. The mean delay between the reporting of the first case of an outbreak and detection of that outbreak by the response system was 2.1 days. Between the first and second study periods, the mean size of an HFM disease outbreak decreased from 19.4 to 15.8 cases and the mean interval between the onset and initial reporting of such an outbreak to the public health emergency reporting system decreased from 10.0 to 9.1 days. Conclusion The automated alert and response system shows good sensitivity in the detection of HFM disease outbreaks and appears to be relatively rapid. Continued use of this system should allow more effective prevention and limitation of such outbreaks in China.

Research
Detecting outbreaks of hand, foot and mouth disease in China Zhongjie Li et al.

Case reporting system
All HFM disease cases that occurred in China after May 2008 -when HFM disease became a notifiable disease in China 20 -should have been reported, by attending clinicians, via the nationwide notifiable infectious diseases reporting information system (hereafter referred to as the case reporting system). This system enables health-care institutes across China to report information on each case of a notifiable infectious disease rapidly, via the Internet, to the Chinese Centre for Disease Control and Prevention. For our study, we used the information on each laboratoryconfirmed or clinically diagnosed case of HFM disease that was reported to the case reporting system between 1 May 2008 and 30 April 2012.

Automated detection of outbreaks
Currently, the automated alert and response system searches the data collected in the case reporting system for aberrations in the incidence of HFM disease and another 29 notifiable infectious diseases. 13 HFM disease has only been included in the response system since 1 May 2010. In the response system, an aberration in incidence at county level leads to the automated generation of a warning signal and that signal's dissemination to the relevant county-level Centre for Disease Control and Prevention. Each signal is then investigated further by epidemiologists in the specific county ( Fig. 1).

Aberration detection
Aberration detection of HFM disease outbreak in the response system is based on the C3 algorithm of the early aberration reporting system developed by the United States Centers for Disease Control and Prevention. 2,9,[21][22][23][24] C3 compares the count of cases in the current day -day 0 -with the corresponding mean count and standard deviation for seven earlier days -days −9 to −3. If the calculated value of C3 surpasses a preset threshold, a warning signal is generated. Following the advice of senior epidemiologists and statisticians in the response system's research group, the preset threshold was given a value of 1.3 for HFM disease in May 2010. This value took estimates of the response system's general sensitivity, timeliness, specificity and positive predictive value in outbreak detection into account.

Signal generation and dissemination
Once a day, the response system searches for aberrations in the county-level incidence of HFM disease. Any warning signals generated are then automatically disseminated via short message service texts sent to the mobile phones of designated staff in the Centres for Disease Control and Prevention in the relevant counties. 13,25

Investigation and feedback
A health-care professional who receives a warning signal as a text message is expected to review the HFM disease cases that triggered the signal, further assess the possibility of an outbreak -by integrating information from other sources, such as information collected by direct contact with the reporting clinical and health-care agencies and -if there then seems to be a real threat of an outbreak (which meant that the warning signal became an alert signal)conduct a field investigation. 13,25 If an HFM disease outbreak is confirmed after field investigation, it should be reported to the public health emergency reporting system.
The health-care professionals who receive warning signals are expected to complete two simple, web-based forms, as soon as possible, so that details of how the professionals proceeded with signal verification and -if appropriate -field investigation can be viewed promptly by epidemiologists at higher levels. 13 In this way, high-level epidemiologists can carefully monitor and assess the risk of outbreak spread.

Reporting confirmed outbreaks
The Chinese public health emergency reporting system was initiated in 2004, to record outbreaks of infectious diseases identified by local epidemiologists. Aside from the procedures that form part of the response system, staff from local health departments are instructed to conduct a field investigation if, within 1 week, at least five HFM disease cases occur in the same setting -e.g. kindergarten or school -or at least three cases of the disease occur in the same village or community. Any outbreak confirmed by a field investigation should be reported to the public health emergency reporting system. 20,26

Evaluating the response system's effectiveness
The main objectives of our study were to evaluate the response system's capacity for identifying HFM disease outbreaks and the response system's impacts on the mean size and duration of an HFM disease outbreak and on the mean delays in the recording of an HFM disease outbreak to the public health emergency reporting system. The HFM disease outbreaks recorded in the public health emergency reporting system were used as the gold standard in our estimations of the response system's sensitivity, specificity and timeliness. The number of cases detected was used as the measure of the size of an outbreak. The number of days between the onset of symptoms in the first and last known cases that were related to the outbreak was used as the estimate of outbreak duration. Sensitivity was estimated by dividing the number of HFM disease outbreaks detected by the response system, by the corresponding number of such outbreaks recorded in the public health emergency reporting system. 9,27 Specificity was estimated by dividing the number of non-outbreak days on which no warning signal was generated for HFM disease -by the response system -by the total number of non-outbreak days. Time to detection was defined as the interval between the first case related to the outbreak being reported to the reporting system and the generation of the first warning signal about the outbreak by the response system. 27 Time from detection to report was defined as the interval between the generation of the first warning signal about the outbreak by the response system and the report of the outbreak to the public health emergency reporting system. Time to report -which was investigated both before and after the response system was implemented -was defined as the interval between symptom onset in the first case related to the outbreak and the report of the outbreak to the public health emergency reporting system. The mean size, duration and time to report of an HFM disease outbreak were estimated for the period 1 May 2008-30 April 2010 -i.e. before HFM disease was covered by the response system -and for the period 1 May 2010-30 April 2012i.e. after HFM disease was included in the response system's remit.

Statistical analyses
We used Pearson's χ 2 test to evaluate the significance of the response system's sensitivity in the detection of HFM disease outbreaks in three size categories: 3-10, 11-20 and more than 20 cases. Time to detection was investigated by one-way analysis of variance. Student's t-test was used to examine whether the mean size, duration and time to report of outbreaks were significantly different before and after HFM disease was included in the response system. All analyses were implemented in version 2.14.1 of the R statistical software package (R Foundation for Statistical Computing, Vienna, Austria).

Research
Detecting outbreaks of hand, foot and mouth disease in China Zhongjie Li et al.
every 100 days in each of the counties that had at least one signal. Initial verification indicated that 2361 (2.2%) of the signals merited being raised to alert status and field investigation. Field investigation of the response system's signals led to 573 HFM disease outbreaks being confirmed. The response system received the initial verification results for 94 920 (89.5%) of the signals within 24 h.
As 618 HFM disease outbreaks were recorded in the public health emergency reporting system in the period when 573 such outbreaks were identified in the response system, the overall sensitivity of the response system in the detection of HFM disease outbreaks was 92.7% ( Table 2). The response system's sensitivity was significantly higher for large outbreaks involving more than 20 cases than for small outbreaks that involved no more than 10 cases (99.3% versus 84.6%; P < 0.001). In the detection of HFM disease outbreaks, the overall specificity of the response system was 95.0% (19 74 324/2 078 361) and the overall mean time to detection was 2.1 days (95% CI: 1.8-2.3). The mean time to detection was 1.7 days for outbreaks that involved no more than 10 cases but 2.7 days for outbreaks that involved more than 20 cases. The mean time from detection to report in the public health emergency reporting system was 4.5 days (95% CI: 4.1-5.0).
In our investigation of the data recorded before HFM disease was included in the response system, the mean size (P = 0.982), duration (P = 0.572) and time to report (P = 0.358) of the HFM disease outbreaks detected between 1 May 2008 and 30 April 2009 were similar to those of the outbreaks detected in the following 12 months. Similarly, in our investigation of the data recorded after HFM disease was included in the response system, the mean size (P = 0.443), duration (P = 0.370) and time to report (P = 0.840) of the HFM disease outbreaks detected between 1 May 2010 and 30 April 2011 were similar to those of the outbreaks detected in the following 12 months. The outbreaks recorded in the two years immediately after HFM disease was included in the response system were generally smaller than those recorded over the previous two years, with mean sizes of 15.8 and 19.4 cases, respectively ( Table 3). The mean size of outbreaks that involved more than 20 cases was significantly less in the two years immediately after HFM disease was included in the response system than the corresponding value for the previous two years (29.2 versus 55 cases; P = 0.015).
The overall mean duration of an HFM disease outbreak was estimated to be 15.2 days for the study periods before and after HFM disease was included in  Detecting outbreaks of hand, foot and mouth disease in China Zhongjie Li et al.
the response system. However, the mean duration of outbreaks that involved more than 20 cases fell from 28.7 days in the two years before HFM disease was included in the response system to 26.0 days in the following two-year period. The corresponding falls in the mean number of days taken to report an HFM disease outbreak of any size -from 10.0 to 9.1 (P = 0.004) -and an HFM disease outbreak that involved more than 20 cases -from 12.7 to 10.5 (P = 0.020) -were significant.

Discussion
Our observations indicate that the response system had good sensitivity and specificity in the detection of HFM disease outbreaks and could lead to a reduction in the eventual size of an outbreak -by shortening the reporting time and so permitting an earlier response.
Our results are consistent with previous research that has found the C3 algorithm to be useful for the detection of aberrancy in the incidence of influenza, bacillary dysentery, HFM disease and other diseases. 22,23,27 We found that the response system's sensitivity in detecting outbreaks of HFM disease that became relatively large -i.e. 99.3% for outbreaks with more than 20 caseswas significantly higher than that for outbreaks that remained small -i.e. 84.6% for outbreaks with no more than 10 cases. Perhaps the outbreaks that grow large expand relatively rapidly and quickly present a large enough deviation from the baseline value for incidence to be easily detected. However, we made no attempt to investigate how responses to the detected outbreaks affected their final size. Overall, 45 HFM disease outbreaks -that were confirmed by health professionals at a time when HFM disease was included in the response system -were not detected by the response system. All 45 remained relatively small and occurred in kindergartens, elementary schools or rural villages. Efforts should be made to increase the sensitivity of the response system -e.g. by using high-resolution spatial detection methods [28][29][30] -to improve the prompt detection of outbreaks while they are small.
Although use of a C3 threshold of 1.3 resulted in good sensitivity, specificity and timeliness in the response system's detection of HFM disease outbreaks, it also resulted in a low positive predictive value. The health professionals who checked the data decided that only 2.2% of the warning signals that the response system generated for HFM disease merited field investigation. One cause of the low positive predictive value is that almost all of the HFM disease cases seen in China -over 99.6% according to the data that we analysed -are sporadic and never form part of an outbreak. A temporal cluster of sporadic cases may easily trigger a falsepositive warning signal in the response system. Such false signals need to be reduced by optimizing the algorithms and thresholds used for outbreak detection -perhaps according to the relevant baseline incidence of HFM disease. 31 The procedures for the verification of warning signals at county level also need to be simplified, to reduce the detrimental effects of so many false-positive signals on the morale and workloads of health professionals.
The early detection of potential outbreaks is important in minimizing the impact of HFM disease. 19 Inclusion of HFM disease in the national response system cut the time taken to report an outbreak of the disease by almost a day. Since the corresponding warning signals were generated a mean of 4.5 days before the outbreaks were reported, there is clearly scope to further reduce the mean time taken to report a confirmed outbreak. Early detection allows the early implementation of outbreak control measures -such as health surveys for the detection of other cases, case isolation, disinfection of affected settings, health education, promotion of hand hygiene, and closure of affected classes or schools -as well as the early treatment of cases and the prevention of the more severe complications of HFM disease.
One limitation of our study was that, for calculating the sensitivity, specificity and timeliness of the response system, we used the outbreaks reported to the public health emergency reporting system as the gold standard. It seems likely that some outbreaks of HFM disease are either never recorded by the public health emergency reporting system or are reported a long time after they have occurred. These issues need investigation. However, at the time of our study, we believed that the public health emergency reporting system was the best-functioning system for the collection of data on HFM disease outbreaks in China.