Automatic Outbreak Detection Algorithm versus Electronic Reporting System

To determine efficacy of automatic outbreak detection algorithms (AODAs), we analyzed 3,582 AODA signals and 4,427 reports of outbreaks caused by Campylobacter spp. or norovirus during 2005–2006 in Germany. Local health departments reported local outbreaks with higher sensitivity and positive predictive value than did AODAs.

I n 2001, the Robert Koch Institute, Germany's federal institute for infectious disease control, implemented an electronic system (SurvNet) for notifi able infectious disease surveillance (1,2). Local health departments electronically sent reports of confi rmed cases to state health departments, which forwarded them to Robert Koch Institute. SurvNet can link single case reports to outbreak reports in which local health departments report descriptive outbreak information in a standardized manner (reported outbreaks). Additionally, the same software organizes the electronic transmission of single case reports from peripheral databases from each local health department to databases of the respective state health department and fi nally to Robert Koch Institute. Automatic outbreak detection algorithms (AODAs), run weekly on this case-based data, generate signals when the observed number of cases per a specifi c week is higher than a defi ned threshold value (signal outbreaks).
To identify the need to follow up generated signals, one must know the positive predictive value of AODA. This knowledge could avoid overwork in local health departments because not every signal will require contacting the local offi ce for investigation.
Our goal was to assess the probability that a signal generated by AODA refl ects a real outbreak (Campylobacter spp. or norovirus) being reported by local health department. Previous studies have tested AODAs by comparing generated signals with simulated outbreaks superimposed on authentic syndromic surveillance data (3,4) or with a limited number of known natural outbreaks (5). In contrast to these approaches, we evaluated performance of AODA by comparing it with a large database of outbreaks elec-tronically reported by local health departments, which we considered to be the reference standard (2).

The Study
We considered a signal outbreak to be identical to a reported outbreak when 1) >1 signal was triggered within the same period as the fi rst and last case belonging to the particular reported outbreak, 2) the signal outbreak was associated with the identical geographic location on the municipal level (1 of the 430 municipalities) as the reported outbreak, and 3) the signal outbreak was associated with the identical pathogen (either Campylobacter spp. or norovirus). Using the data available as of June 1, 2007, we considered the number of reported outbreaks (a minimum of 4 cases because the algorithm cannot detect outbreaks with <4 cases), from week 5 of 2005 through week 4 of 2007.
During the study period, 118 and 4,309 outbreaks with >4 cases, associated with the pathogens Campylobacter spp. and norovirus, respectively, had been reported. The AODA had signaled 52 (44.1%) of the 118 reported Campylobacter spp. outbreaks and 2,538 (58.9%) of the 4,309 reported norovirus outbreaks (Table). The probability that a signal outbreak refl ected a reported outbreak (positive predictive value of AODA) was lower for Campylobacter spp. than for norovirus: 50 (6.4%) of 781 Campylobacter spp. signal outbreaks and 2,115 (75.5%) of 2,801 norovirus signal outbreaks were associated with reported outbreaks. The AODA may have triggered multiple signals during the outbreak if the threshold level was reached during several consecutive weeks ( Figure 1). Of the Campylobacter spp. outbreaks, 3 (6.0%) were each identifi ed by 2 different signals; of the norovirus outbreaks, 727 (28.6%) were identifi ed by multiple signals (2-20 signals per reported outbreak) (Table). Furthermore, 1 signal outbreak could correspond with different reported outbreaks when these occurred in the same local area and during the same period ( Figure 2). For Campylobacter spp., 4 (8.0%) of the outbreak signals could correspond to >1 reported outbreak; for norovirus, 760 (35.9%) of the signal outbreaks could correspond to 2-26 reported outbreaks (Table).

Conclusions
Germany´s electronic reporting system for infectious disease outbreaks provided a unique opportunity to compare the triggering of signals through AODA with the reporting of outbreaks identifi ed by local health departments. The probability of an outbreak signal being associated with a reported outbreak was much lower for Campylobacter spp. (6.4%) than for norovirus (75.5%). Furthermore, the fraction of cases as part of a reported outbreak was much lower for Campylobacter spp. (71.4%). Differences in route of transmission likely explain why Campylobacter spp. cases are generally more likely to occur sporadically and why norovirus cases are more likely to be part of an outbreak (6-9). These differences might result in a lower frequency of Campylobacter spp. outbreaks. The AODA might generate a signal when a higher than expected number of single cases is observed in a specifi c period and location, but this signal is likely to refl ect an in-creased number of sporadic cases; an increased number of norovirus cases is more likely to refl ect an occurring norovirus outbreak. An alternative possibility is that local health departments are more inclined to identify, investigate, and report norovirus outbreaks than Campylobacter spp. outbreaks (10). These differences demonstrate the importance of designing AODA specifi cally for the pathogens under surveillance.
For our analyses we used reported outbreaks as the reference standard by which to evaluate the AODA. Although this outbreak reporting is probably incomplete, we believe that it more closely identifi es the true number of outbreaks than does retrospectively identifying outbreaks (11) or simulating outbreaks (3,4). Thus, we believe it generates a better reference standard than that used in previous studies.
Our fi ndings question the usefulness of the AODA because a large number of generated signals were not confi rmed by the electronic outbreak reporting from local health departments. Our results suggest that AODAs are not useful for detecting outbreaks on a local level because the outbreaks are detected earlier and investigated by the local health department. AODAs might be more useful for detecting multicounty or even multistate outbreaks, which are more diffi cult to detect by a single local health department. The latter has been well demonstrated by AODA detection of various foodborne outbreaks in Germany (12,13). National surveillance should focus on the follow-up of signals that indicate potential multicounty or multistate outbreaks. We used the county level for the algorithm because we obtain the reported outbreaks on this level fi rst and we wanted to compare both systems. Our standard algorithms run also ‡During the duration of a reported outbreak, the detection algorithm may have triggered multiple signals during several consecutive weeks ( Figure 1). §One signal outbreak may correspond to multiple reported outbreaks if different outbreaks occur in the same municipality during the same period ( Figure 2). Figure 1. Example of 1 reported outbreak being detected by 3 signals. In this example, 3 signal outbreaks (S1, S2, S3) can be associated with 1 reported outbreak in same municipality and during the same period. Week No. cases reported Threshold value detection algorithm S1 S2 S3 Automatic Algorithm vs. Electronic Reporting on federal and state levels, but that was not the subject of this investigation. To enable local health departments to earlier discover multicounty outbreaks, a new version of SurvNet is being developed. This version will give local health departments the opportunity to include more information on the evidence and also the possibility of linking outbreaks from different counties (2). The Robert Koch Institute, along with the state health departments, will develop a standard operating procedure for how to communicate and follow up on signals generated by the AODA.
Our study suggests that the usefulness of AODA to detect local outbreaks is limited because local health departments generally detect local outbreaks earlier and in more detail than these algorithms. Investment in the development of user-friendly outbreak reporting tools for local health departments might therefore provide better information on outbreaks than extensive refi nements of AODAs. Week No. cases reported Reported outbreak 1 Reported outbreak 2 Reported outbreak 3 S1 Threshold value detection algorithm Figure 2. Example of 1 signal outbreak corresponding to multiple reported outbreaks. In this example, 1 signal outbreak (S1) can be associated with 3 reported outbreaks occuring in same municipality; threshold is reached in same week number.