Comparison of Enzootic Risk Measures for Predicting West Nile Disease, Los Angeles, California, USA, 2004–2010

The best model comprised enzootic surveillance data from avian, mosquito, and climate sources.


W est Nile virus (WNV; family Flaviviridae, genus
Flavivirus) is amplifi ed within a mosquito-bird cycle, with tangential transmission to equids and humans (1). Since the introduction of WNV into Los Angeles, California, USA, in 2003, our research (2)(3)(4)(5) has focused on surveillance indicators for enzootic WNV transmission and prediction of human cases. The Greater Los Angeles County Vector Control District (GLACVCD) serves >6 million of the ≈10 million residents of Los Angeles County and conducts year-round surveillance for WNV activity (6). In addition to having a robust surveillance dataset, Los Angeles County is a suitable location for evaluating environmental risk because the large human population enables the sensitive detection of dead birds (7), increases opportunities for human-vector contact, and experienced 2 outbreaks during the study period (6).
We compared the predictive ability of 3 measures of human risk by using time-series graphs, sensitivity, specifi city, positive predictive value (PPV), and concordance between human case onset and states of high risk based on enzootic transmission during 2004-2010. We believed that for operational decision support a successful risk measure should correctly 1) identify periods of low risk when few or no cases occur, 2) predict high or increased risk before human cases occur, and 3) identify periods of high risk concurrent with the occurrence of human cases .
The 3 measures of risk we compared were the California Mosquito-Borne Virus Risk Assessment (CMVRA), the vector index, and the Dynamic Continuous-Area Space-Time (DYCAST) system. The CMVRA (8) calculates risk on the basis of ranks of environmental variables for enzootic transmission and is used by health agencies throughout California to measure risk. At its inception, the CMVRA was evaluated retrospectively for its ability to detect cases of Western equine encephalomyelitis virus (family Togaviridae, genus Alphavirus) and St. Louis encephalitis virus (family Flaviviridae, genus Flavivirus) in California during low-, medium-and high-risk seasons (9). Additional assessment of the ability of CMVRA to track WNV cases in Bakersfi eld, California, produced impressive results during 2004 and 2007 (10,11).
The second method was the vector index, an estimate of the number of infected mosquitoes collected per trapnight. This index successfully determined human risk in Colorado (12,13) and is used by the Colorado Department of Public Health and Environment (www.cdphe.state.co.us/ dc/zoonosis/wnv/wnvsentinel.html).
The third method was the DYCAST (14) system, which provides an assessment of risk in time and space by using reports of dead birds from the California Department of Public Health Dead Bird Hotline. This risk estimate differs from the previous 2 in that the spatial scale is fi ne (0.44 km 2 grid cells), it is computationally more complex, and it does not rely on laboratory test results (15).
Understanding the characteristics of risk estimates to determine the best predictive measure for human cases is needed for several reasons. First, reducing the rate of falsepositive results will reduce message fatigue associated with repeated false warnings of high-risk conditions. Second, increasing the proportion of high-risk areas correctly identifi ed (sensitivity) can reduce the costs associated with emergency mosquito control by correctly focusing timely intervention. Third, a qualitative assessment of risk estimates that incorporates different variables for enzootic transmission enables understanding of the ability of different assemblages of surveillance data for predicting human risk. Overall, a better understanding of the tools used in decision support for emergency intervention can only improve the protection of human health.

Materials and Methods
The epidemiology of WNV in Los Angeles has been described in detail (6). Methods used for data collection for each risk assessment tool are summarized briefl y below and in detail (online Technical Appendix, wwwnc.cdc.gov/ EID/pdfs/11-1558-Techapp.pdf).

CMVRA
The CMVRA (8) calculated risk on the basis of average daily temperature, mosquito abundance and infection, counts of WNV RNA-positive dead birds, and sentinel chicken seroconversions over successive 2-week periods. Each variable was assigned to quintile ranks, and these categorical values were averaged to calculate a fi nal risk estimate. Thresholds <2.5 were considered low-risk (normal season) conditions; those 2.6-4.0 were considered medium-risk (emergency planning) conditions; and those >4.1 were considered high-risk (epidemic) conditions. Details of sampling, laboratory testing, and risk calculation are summarized in the online Technical Appendix. In the current study, temperature data were aggregated from the National Aeronautics and Space Administration Terrestrial Observation and Prediction System (16) at a 1-km 2 scale for the GLACVCD jurisdiction. Abundance anomalies for Culex pipiens quinquefasciatus mosquitoes collected by gravid traps (6) were calculated by comparing current 2-week estimates to 5-year averages for the same period. WNV infection incidence in Cx. p. quinquefasciatus mosquitoes was calculated from mosquito pool data by using the Excel (Microsoft, Redmond, WA, USA) add-in developed by Biggerstaff (17). Dead birds reported by the public and testing positive for WNV RNA and sentinel chicken seroconversions were ranked according to frequency and scale of occurrence for the broad region (Los Angeles County) and the specifi c region (within GLACVCD jurisdiction). Reports of sentinel chicken seroconversions from Los Angeles County outside the GLACVCD boundary were found on the California West Nile virus Web site (www.westnile.ca.gov). Human cases, recorded by the Los Angeles County Department of Public Health, Acute Communicable Disease Control, were excluded from the current risk calculations because they were used as an outcome measure.

Vector Index
The vector index also was calculated for 2-week time steps by using abundance (numbers per gravid trap per night) and infection incidence for Cx. p. quinquefasciatus mosquitoes collected by gravid traps by using the biascorrected maximum-likelihood estimate (6) (online Technical Appendix). Usually the species-specifi c maximum-likelihood estimate is multiplied by female mosquito abundance measured by CO 2 trap counts to yield an arbovirus equivalent of the entomologic inoculation rate in malaria epidemiology (18). Vector index estimates were stratifi ed into frequency percentiles by using SAS version 9.1 software (SAS Institute Inc., Cary NC, USA), and the percentiles were assessed individually for their effi cacy for predicting human cases.

DYCAST
For DYCAST, 0.44-km 2 grid cells were overlaid onto the Los Angeles County study area. There were 22,687 grid cells in Los Angeles County, but only 6,666 grid cells were within the GLACVCD boundary. We assessed the DYCAST risk estimates by using a predetermined Knox test signifi cance threshold of ≈0.10 = high risk. The Knox test statistically delineated signifi cantly positive groups of grid cells into clusters or hot spots. Unlike the other 2 methods, the DYCAST model assessed risk on a daily basis, providing a time and location of high risk on the basis of the spatial grouping of the number of reports of dead birds; data were independent of a predetermined spatial allocation of sampling assets and laboratory diagnostics. To make this method comparable with the previous 2 methods, we selected the minimum DYCAST value by grid for each 2-week period. The DYCAST model then was assessed by using daily and 2-week aggregations.
Another unique feature of the DYCAST model is the spatial resolution. The other 2 methods provide an assessment of high-risk conditions that can be anywhere within the GLACVCD boundary, whereas DYCAST delineates high-risk conditions within a defi ned space.
Again, to make our assessments comparable, we aggregated DYCAST high-and low-risk cells spatially by week, up to the spatial limit imposed by the GLACVCD boundary (6,666 cells). The new spatial aggregates were compared with human case occurrence to determine an optimal number of grid cells needed to establish a high-risk area. This comparison was performed by constructing a receiver operator characteristic (ROC) curve of the plotted sensitivity versus 1 -specifi city for all aggregated cell counts.

Reports of Human Cases
Reports of laboratory-confi rmed human cases, compiled by the Acute Communicable Disease Control program of the Los Angeles County Department of Public Health and occurring within GLACVCD, included West Nile fever (WNF) and West Nile neuroinvasive disease (WNND) diagnoses and asymptomatic viremic blood donors. Onset dates for symptomatic persons were adjusted backward 10 days to account for the intrinsic incubation period (19,20). Seven blood donors with viremia later became symptomatic for WNV disease and were added as WNV cases; the mean time from donation to symptom onset was 6.2 days (SD ±6.14, median 3.5). To account for earlier detection, the infection dates for all viremic blood donors were adjusted backward 4 days (10 latent days minus 6 induction days). As reported for Los Angeles County (6), the percentage of WNND among all reported WNV infections increased signifi cantly over time because of reduced physician requests for laboratory testing for febrile illness, thereby reducing the total number of human cases reported. In addition, unpublished data from elsewhere in California also indicate that relatively few persons hospitalized with neuroinvasive disease are tested for WNV, which possibly further reduced recent estimates of WNV-associated human disease. Because mosquito and public health agencies respond to reports of human cases regardless of diagnosis, we chose to use these data in the current analyses. The Institutional Review Board at the University of California, Davis, approved protocols for using human surveillance data (approval no. 201018171-1).

Analysis
Time-series graphs of the CMVRA and the vector index were plotted with human cases to depict which attained high-risk thresholds before human cases occurred. A true or false-positive fi nding was a time period identifi ed as high risk during which >1 or 0 human infections occurred, respectively. A true-negative period was a period identifi ed as low risk and during which no human infections occurred; conversely, false-negative periods were identifi ed as lowrisk periods when human infections occurred. Sensitivity was calculated as the proportion of high-risk periods correctly identifi ed; specifi city was the proportion of lowrisk periods correctly identifi ed (21). The PPV, likelihood ratio positive, and likelihood ratio negative were calculated as measures of relative precision (22).
ROC curves were plotted to defi ne optimum response thresholds. The area under the curve (AUC) was calculated to compare the 2 methods. ROC and AUC calculations were performed by using SAS version 9.1 and the Macro %ROC (http://support.sas.com/kb/25/addl/fusion25017_5_roc.sas. txt).
With the above analyses providing information about the accuracy of each risk assessment, a separate casecrossover study was performed by using the known onset information to create an estimate of the relative risk of acquiring WNV during high-risk periods. Illness onset dates for case-patients and asymptomatic viremic blood donors were lagged backward as described above. Mantel-Haenszel relative risks were calculated to determine whether high-risk values were signifi cantly associated with human infection (23)(24)(25). Mantel-Haenszel relative risks were calculated by using the proportion of highrisk periods before estimated infection as the expected frequency of exposure and the concordance odds of disease transmission occurring during a high-risk period by each model and threshold. Data aggregation and zonal statistics were performed by using PostgreSQL 8.3.7 and PostGIS 1.3.1.

CMVRA
Risk estimates (Figure 1, panel A) consistently reached emergency planning thresholds (threshold >2.6) before human case detection. In 2004, epidemic thresholds (>4.1) were reached by mid-August (Table 1) after 39 human cases had been reported. During the second epidemic in 2008, risk assessments reached epidemic thresholds after 8 human cases were identifi ed. Using the epidemic threshold, we identifi ed 13 true-positive intervals, 0 false-positive intervals, 151 true-negative intervals, and 28 false-negative intervals. Estimates using this method were driven by ranks for environmental conditions and infections in dead birds, followed by mosquito infection rates and abundance. Antecedent sentinel chicken seroconversions consistently ranked lowest on the 5-point scale until human cases occurred because they were temporally concordant (26).
Using the emergency planning threshold, we identifi ed 40 true-positive intervals, 28 false-positive intervals, 123 true-negative intervals, and 1 false-negative interval. Although there were more false-positive intervals, they represented high-risk periods before the onset of human cases because the threshold reached >2.6 at least 2 weeks before human cases occurred in all study years except 2008 (Table 1). On the basis of the advance warning that this risk estimate provided and the increase in sensitivity (Table  2), the 2.6 threshold was a better threshold for epidemic prediction.
We calculated sensitivity and specifi city separately for each study year by using the 2.6 emergency planning threshold ( Table 3). Use of this test validity revealed that sensitivity, i.e., correctly identifi ed high-risk periods, dipped in 2005, whereas specifi city, i.e., proportion of correctly identifi ed low-risk periods, was lowest in 2006 and 2008.

Vector Index
Vector index estimates (Figure 1, panel B) were calculated biweekly for the entire study period for Cx. p. quinquefasciatus mosquito collections and were driven exclusively by mosquito infection incidence. Using the 65th percentile (0.018) as the threshold, we identifi ed 38 true-positive, 37 false-positive, 116 true-negative, and 1 false-negative intervals. The frequency distribution of the vector index was highly right skewed and could not be

DYCAST
Positive DYCAST cells were observed before human case occurrence in 5 of the 7 study years ( Table 1). Counts of positive DYCAST grid cells compared with human case onset is presented in Figure 1, panel C. The DYCAST risk estimate, calculated by grouping the biweekly estimates, was used in the yearly comparisons of sensitivity and specifi city (Table 3). Temporal changes in sensitivity and specifi city showed the impact of reduced reporting of dead birds over time because the values for both measures of validity were highest in 2004 and declined to 0 or near 0 in all subsequent years.

Human Case Reports
A total of 389 cases of WNV disease were reported during the study period. Of these, 14 reports were missing onset date information and were not used to evaluate the risk estimates.

Analysis
The proportion of high-risk intervals correctly identifi ed (sensitivity) was greatest in the CMVRA when the 2.6 emergency planning threshold was used ( Table 2). The vector index provided the second highest sensitivity by using values just >0 (65th percentile). The greatest specifi city, i.e., proportion of low-risk intervals correctly identifi ed, was observed in the CMVRA at the epidemic threshold of 4.1, followed by the vector index at the 95th percentile; the PPV followed this fi nding. The likelihood ratio positive, i.e., the likelihood that a high-risk condition was identifi ed correctly when a human case occurred, was greatest for the vector index at the 95th percentile. The likelihood ratio negative, i.e., how much the odds of a human case decrease during low-risk conditions, was lowest in the emergency planning threshold of the CMVRA.
Discriminatory ability, as measured by the AUC, was greatest for the CMVRA (0.982), followed by the vector index (0.845) (Figure 2). Ideal response level cutoffs for the CMVRA as indicated in the ROC plots would be 1.8 and 2.6. The ideal response level for the vector index was more diffi cult to identify because of the obvious tradeoff between the sensitivity and specifi city as evidenced in the ROC plot. The DYCAST cell aggregates performed no better than chance with an AUC of 0.468, with worst performance occurring when a single positive cell was used to assess risk.
A case-crossover study was conducted for all cases and asymptomatic blood donors with a known illness onset or donation date. The relative risk, i.e., risk for WNV infection given exposure to high-risk conditions, was greatest when detected by the CMVRA by using the emergency planning threshold ( Table 2).

Discussion
Since the introduction of WNV into the United States in 1999, WNF and WNND have caused at least 31,365 illnesses and 1,250 deaths (27). Once considered to be a mild infl uenza-like illness, WNF is now understood to be an acute viral infection, often followed by months of illness associated with depression, altered moods, headaches, and fatigue (28)(29)(30). The illness associated with WNND, including meningitis, encephalitis, and acute fl accid paralysis, has been associated with persistent motor and cognitive defi cits and incomplete recovery (28,31). Reported cases of WNF and WNND underrepresent the actual number of WNV cases in the U.S. population (32,33), and symptomatic persons represent only a fraction of those infected. In addition to individual suffering, the medical and public health costs associated with WNV average >$40,000 per case (34) at a time when many health agencies are facing serious budgetary shortfalls. The individual health toll and associated medical costs present a strong case for active intervention. Current means to prevent WNV infection include integrated vector management by larval mosquito control to arrest viral amplifi cation and, in an outbreak, ground or aerial adulticide applications to eliminate infectious female mosquitoes and personal protection to avoid mosquito bites. Emergency application of adulticides became particularly controversial in California (35), even though it is the only method that targets mosquitoes capable of transmitting virus and is cost-effective for preventing human cases (36). In light of this controversy, mosquito control agencies in California are often hesitant to apply adulticides until epidemics appear imminent on the basis of available risk estimates or the occurrence of human cases. Our study comparatively evaluated 3 risk measures currently used as decision support tools for intervention and for predicting human cases.
By using only indicators of enzootic transmission, the CMVRA consistently produced estimates in the emergency planning range before human case occurrence; however, epidemic thresholds were not reached until after human cases had been detected. Risk assessment by this method required a robust arboviral surveillance program, with regular sampling for multiple surveillance indicators. The specifi city and PPV when the epidemic threshold of 4.1 was used were excellent; however, this was at the expense of adequate lead time for initiating intervention efforts before some human cases. Additionally, the sensitivity of the risk estimate was less than desirable at 0.317, meaning that fewer than one third of the high-risk periods were correctly identifi ed. The CMVRA using the 4.1 threshold was poor at predicting high-risk intervals but good at predicting lowrisk intervals.
The 2.6 emergency planning threshold for the CMVRA increased sensitivity and provided a predictive indication of human cases before their onset. The likelihood ratio positive was better than the DYCAST risk estimates, and the likelihood ratio negative was the best of all methods. In addition, the associated risk for human cases, measured by the Mantel-Haenszel relative risk, was the greatest. At the 2.6 emergency planning threshold, the CMVRA was excellent at predicting high-risk periods and good at predicting low-risk periods.
The vector index was simple to calculate and required only a mosquito surveillance and testing program, thereby saving costs associated with sentinel chicken maintenance and sampling and dead bird reporting and testing programs. Unfortunately, this measure did not have preestablished risk thresholds. In our study, it appeared that setting the threshold to >0 (i.e., whenever mosquito infection was detected) would be adequate for predicting human cases in urban settings, such as Los  Angeles, where Cx. p. quinquefaciatus mosquitoes are the primary vectors and temperatures generally permit viral amplifi cation. The estimates of the vector index increased before case occurrence in 5 of the 7 years. The sensitivity and specifi city were comparable with those of the CMVRA, but the likelihood ratio positive was the greatest of all risk estimates. The likelihood ratio negative was better than that of the DYCAST but not as good as that of the CMVRA. Therefore, the vector index was moderate at predicting high-risk periods and very good at predicting low-risk periods. The measure of risk associated with a high-risk value, assessed by the Mantel-Haenszel relative risk, was also better than the DYCAST risk estimate but not as good as either CMVRA threshold. The DYCAST risk estimate was useful in years with amplifi ed enzootic transmission, when dead birds were considered the primary WNV surveillance indicator (4,(37)(38)(39). However, after the initial epidemic, WNV activity has been progressively more diffi cult to predict by using DYCAST because of reduced reporting to the California Dead Bird Hotline. Whether this decrease resulted from truly decreased numbers of dead birds as bird populations became progressively more resistant to infection or to public apathy/decreased awareness was not possible to ascertain. Losing time precision by aggregating estimates clearly increased measures of validity, which considering the uncertainty regarding time between WNV exposure and disease onset seemed appropriate to improve predictive power. The sensitivity of the weekly DYCAST risk estimate was similar to that of the CMVRA, but the specifi city, PPV, likelihood ratio positive, and likelihood ratio negative were all uniformly worse than the other 2 methods, even when aggregated spatially. Additionally, the measure of relative risk associated with risk estimates was less than that of the CMVRA and the vector index.
In conclusion, critical decisions on intervention by using risk estimates require knowledge of the strengths and weaknesses of the selected method to respond in an adequate and timely manner to prevent human cases while reducing unnecessary response and costs associated with falsely identifi ed high-risk periods. The goals we set for a good WNV risk estimate were a balance of these attributes and were achieved best in urban and suburban Los Angeles by the CMVRA by using the 2.6 epidemic planning threshold. In light of this fi nding, an evaluation of the CMVRA should be done in other ecologic settings with transmission driven by other vector species to determine whether the threshold should be adjusted to provide better antecedent estimates of human risk.