Developing a sampling methodology for timely reporting of population‐based COVID‐19‐associated hospitalization surveillance in the United States, COVID‐NET 2020–2021

Abstract Background The COVID‐19‐Associated Hospitalization Surveillance Network (COVID‐NET) required a sampling methodology that allowed for production of timely population‐based clinical estimates to inform the ongoing US COVID‐19 pandemic response. Methods We developed a flexible sampling approach that considered reporting delays, differential hospitalized case burden across surveillance sites, and changing geographic and demographic trends over time. We incorporated weighting methods to adjust for the probability of selection and non‐response, and to calibrate the sampled case distribution to the population distribution on demographics. We additionally developed procedures for variance estimation. Results Between March 2020 and June 2021, 19,293 (10.4%) of all adult hospitalized cases were sampled for chart abstraction. Variance estimates for select variables of interest were within desired ranges. Conclusions COVID‐NET's sampling methodology allowed for reporting of robust and timely, population‐based data on the clinical epidemiology of COVID‐19‐associated hospitalizations and evolving trends over time, while attempting to reduce data collection burden on surveillance sites. Such methods may provide a general framework for other surveillance systems needing to quickly and efficiently collect and disseminate data for public health action.

hospitalizations and to provide timely data on clinical characteristics and outcomes of hospitalized patients with COVID-19 to inform the US pandemic response. Due to the overwhelming burden of COVID-19-associated hospitalizations and the substantial time and resources required to conduct detailed medical chart abstractions, it was not feasible to quickly abstract clinical data on every case. In the initial weeks of the pandemic, clinical estimates were generated based on data available at that time, which was not systematically collected and was therefore not generalizable to COVID-NET's surveillance area.
We developed a probability-based sampling strategy to allow for timely collection and dissemination of detailed clinical data on a representative sample of hospitalized cases on an ongoing basis throughout the pandemic. This sampling strategy would allow for continued monitoring of COVID-19 severity among hospitalized cases throughout the pandemic.
While established survey 3-6 and sampling [7][8][9][10][11] tenets would inform the sampling approach, COVID-NET faced several unique challenges. Due to reporting delays, the sampling frame from which the sample was selected (i.e., COVID-19-associated hospitalizations within the catchment area) was not completely ascertained when sampled cases needed to be drawn. Therefore, the sampling methodology needed to minimize the impact of case reporting delays. Also, the 14 surveillance catchment areas varied widely in population size (ranging from a small population center of 100,000 persons to a state of six million persons), yielding varying data collection burden across sites. Shifting geographic outbreaks and changing epidemiology over the course of the pandemic also yielded varying numbers of cases over time by site and age group and meant that sampling rates would need to change over time within sampling strata.
COVID-NET was designed using the infrastructure of the Influenza Hospitalization Surveillance Network (FluSurv-NET), an analogous surveillance platform that first employed sampling during the 2017-2018 season. [12][13][14] We aimed to devise a sampling methodology that (1) enabled timely collection and dissemination of clinical data, (2) produced robust population-based estimates while minimizing sampling and surveillance biases, (3) allowed for adaptability given the unknown pandemic trajectory and changing data needs, and (4) included explicit analytic provisions to account for the sample design. Here, we describe the sampling methods that were successfully employed to rapidly provide epidemiologic and clinical data on COVID-19-associated hospitalizations throughout the first 16 months of the US pandemic.

| Estimating population-based COVID-19-associated hospitalization rates
Surveillance sites transmit a minimum dataset (county, age, sex, race/ ethnicity, hospital admission date, and SARS-CoV-2 testing data) on all identified cases to CDC on a weekly basis to produce COVID-19-associated hospitalization rates stratified by key demographics.
Incidence rates are calculated using COVID-NET cases as the numerator and the National Center for Health Statistics' Vintage bridged-race postcensal population estimates 17 as the denominator. Rates are posted weekly to CDC's COVID-NET interactive 18 and COVID Data Tracker 2 webpages. On average, minimum data for each case are provided to CDC within 7 days of hospital admission ( Figure 1); however, reporting delays vary by surveillance site and epidemiologic week and can range from 1 day to several weeks. By 6 weeks from admission, >90% of cases have been reported.

| Characteristics of hospitalized COVID-NET cases
Using a standardized case report form (CRF) with >400 variables, trained surveillance staff conduct in-depth medical chart abstractions on sampled cases, including data on underlying medical conditions, intensive care unit (ICU) admission, mechanical ventilation, receipt of COVID-19 vaccination or treatment, and death during hospitalization.
Data on the prevalence of these clinical characteristics and outcomes are posted to COVID-NET's interactive web pages 19 but updated less frequently than hospitalization rates due to the substantial time needed to collect these data ( Figure 1); prior to the sampling strategy implementation, the delay ranged from weeks to months.

| General sampling approach
Because site, age, and hospital admission date were known for all cases upon first report to CDC, these variables were used to stratify cases for sampling. Sampling periods were selected because sampling needed to occur on an ongoing basis. For each sampling period, sample sizes were calculated for the entire surveillance network to achieve desired precision around clinical estimates of interest; samples were then distributed across the 14 surveillance sites. Sites drew random samples of cases based on CDC specifications, conducted medical chart abstractions on sampled cases, and transmitted data to CDC. CDC weighted the data to reflect the sample design and reported weighted clinical estimates for each period on its interactive website ( Figure 1). The following sections describe these steps in detail.

| Sampling by age group
Because different age groups experienced different rates of COVID-19-associated hospitalization, 2,18 COVID-NET stratified samples by age group (0-17, 18-49, 50-64, and ≥65 years). Doing so allowed for age groups to be sampled at rates inversely related to the age group's hospitalization burden. Sampling rates were adjusted over time as age groups experienced changing hospitalization rates. Because COVID-19-associated hospitalizations were less frequent in children, all cases <18 years were sampled; thus, the remaining sampling discussion focuses on adults.

| Sampling by period
In the initial months of the pandemic, sampling periods were established retrospectively based on hospitalization rate patterns. The first hospitalization wave occurred during March through May 2020 ( Figure 2); thus, the first sampling period included cases hospitalized during this period. A second hospitalization wave occurred during July through August 2020; thus, the second sampling period encompassed June through September 2020. Sampling rates were not determined until after the sampling period had ended, when the number of cases had been determined. Choosing longer periods over which to sample resulted in sites waiting longer to draw samples and led to substantial delays in availability of clinical data. Starting in October 2020, the sampling period shifted to a monthly pattern, allowing sites to more quickly sample and conduct chart abstractions. Monthly sampling also allowed CDC to rapidly adjust sampling rates in response to changes in hospitalization rates.

| Sampling by surveillance site
While each COVID-NET site followed standardized protocols for case ascertainment and data collection, case-finding and data acquisition methods varied across sites and were governed by the site's  20,21 Sample size calculations demonstrated that across the network, a minimum of 100 cases was necessary within each period and stratum. In practice, 100 cases were found to be insufficient because sites were not able to access patient charts for all the sampled cases. Additionally, we anticipated an increase in variance, above the level given by SRS, as a result of disproportionate allocation of the total sample size to strata and any non-response. As such, we adjusted the required sample size to at least 200 cases per stratum across the network. We also summarized the increase in variance by the weighting effect (WEFF). The formula for WEFF is shown below, where CV is the coefficient of variation, W i is the final weight for the ith case, and n is the sample size, for each period. 22 In November 2020, weighting effects were calculated using historical data from March through October 2020, and the resulting weighting effects were used to inflate November 2020 sample sizes calculated under SRS (with the weighting effect, $350 sampled cases were required across all sites rather than 200). After November 2020, weighting effects were considered, though not strictly applied in order to minimize burden on sites, when determining sample sizes.
Because at least 200-350 cases per strata were needed across all sites combined, we next established methods to produce site-specific sampling rates. While determining these sampling rates, the following objectives were considered: sampling rates for a specific surveillance period would be determined very soon after a period ended. However, due to reporting delays, it took up to 6 weeks after the end of a surveillance period for CDC to obtain final case counts; case counts for the most recent weeks of a period were particularly underestimated. To address this, CDC used a multiplier method developed in-house 23 to predict the total number of cases by site and age group within a given period, after accounting for historical reporting delays. Doing so allowed for sampling rates to be determined 1-2 weeks after the period ended ( Figure 1).
After sampling rates were determined for each period, based on CDC specifications, sites drew samples using auto-generated random numbers (1-100) that were prospectively assigned to each new case upon entry into the surveillance database. For example, if a site needed a 10% sample, cases assigned to random numbers yielding a 10% sample were drawn. While sites were required to abstract data on a minimum number of sampled cases, they could elect to conduct chart abstractions on all cases within specified strata to allow them more power for site-specific analyses. Because data on all cases are known for these strata, these strata are referred to as "certainty strata."

| Data collection and reporting
Sites completed chart abstractions with a goal of submitting data for all sampled cases within 3-4 weeks after each sample draw. Sites entered sampled case data into a standardized database and transmitted data weekly to CDC. Once a majority of sites completed at least 90% of chart abstractions for sampled cases for a given period, combined data for those sites were weighted and posted to CDC's interactive webpage. 19 Monthly clinical estimates were posted about 4-6 weeks after the end of each surveillance period, and cumulative estimates were also updated at this time ( Figure 1).

| Weighting process and variance estimation
Race/ethnicity and sex, used in the weighting process, were missing on 2% and 0.1% of sampled cases, respectively. Both fields were imputed using hot deck (single) imputation. 24 Clinical data were weighted to reflect the probability of selection, adjusted for nonresponse (CRF data completed for <100% of sampled cases), and raked to align the weighted case distribution in COVID-NET by site, age, sex, and race/ethnicity with the COVID-NET catchment population totals using methods previously described. 25 Because weight adjustments yielded more dispersed weights, weights were trimmed 25 to minimize overdispersion. The full weighting process is outlined in Table S1. After final sampling weights were created, Jackknife replicate weights 11 were created for variance estimation. A method that limited the number of Jackknife replicates was utilized to facilitate efficient computations and is described in Text S1. Across sampling periods and age groups, average sampling rates ranged from 5% to 44% among cases aged 18-49 years, 4% to 44% among those aged 50-64 years, and 2% to 46% among those aged ≥65 years (Table 1). Sampling rates stratified by site, age group, and time period are displayed in Table S2 and ranged from 1% to 100%.
After applying final sample weights, COVID-NET sample estimates were similar by age, sex, race/ethnicity, and site to all identified COVID-NET cases (Table S3). Figure 3 shows the weighted proportions and 95% confidence intervals of sampled cases who required ICU admission, mechanical ventilation, and who died in-hospital by month for the 14 COVID-NET states; Table S4 and Figure S1 show these results by age group.
All point estimates with prevalence ≥10% had RSEs < 0.3, while only a few point estimates with prevalence <10% had an RSE > 0.3. The 95% confidence interval half-widths for all point estimates with the exception of one were <10% (Table S4). Figure 3 also demonstrates the utility of these sampled data and shows the trends in clinical estimates and outcomes over time. For example, among all ages ICU admission declined from 37.8% in March 2020 to as low as 16.8% in February 2021. In-hospital deaths decreased during the initial months of the pandemic from a high of 16.5% in April 2020 to as low as 7.1% in June 2020 but increased to 13% in December of 2020 ( Figure 3).
Final sample weights differed by site, age group, and month. Table S5 shows the sample sizes, range of weights across the network, and weighting effect by month and age group. Weighting timeliness. This process allowed for estimation of monthly trends in disease severity 19,26 and timely descriptions of racial and ethnic disparities in rates of severe COVID-19. 28,29 Sample size determinations accounted for surveillance priorities and site resources and capacity. Selecting larger sample sizes and using similar sampling rates across all sites yielded higher precision.
Additionally, applying weighting effects to sample size calculations helped mitigate potential increases in variance due to differential sampling rates across sites, non-response adjustments, or raking. 22,30,31 However, during periods when case counts were unusually high and case burden was unequally distributed across sites, sampling at uniformly high rates was not feasible. For COVID-NET, it was preferred to have complete data on fewer representative cases than to have less complete, less representative data on a larger sample. When sample size requirements were lower, and differential sampling rates were applied across sites based on case burden, non-response was greatly reduced.
There are several limitations to COVID-NET surveillance. COVID-NET is a resource intensive surveillance platform which requires surveillance staff to ascertain cases and manually conduct medical chart abstractions using standardized protocols. While this approach yields accurate and consistent data across the platform, these high-quality data come at a potential cost; clinical data are available on a monthly F I G U R E 3 Weighted percentages and confidence intervals for select clinical interventions and outcomes by month among sampled cases, COVID-NET, a March 2020 to June 2021 rather than daily or weekly basis. COVID-NET data are also not nationally representative given that these resource intensive methods could not easily be implemented across all states; however, defined catchment areas allow for production of population-based estimates of the COVID-NET catchment areas rather than estimates generated from convenience samples. There were also several limitations related to the sampling approach. Reporting delays were modeled to allow for more timely sampling, but modeled sample sizes could be over or underestimated. Non-response adjustment and raking processes also required subjective decision-making considering surveillance goals and feasibility. While adjustments could be made on data with response rates much lower than 90%, doing so yielded a larger nonresponse adjustment factor that adversely impacted variance. Finally, the sampling guidance evolved rapidly as CDC learned and adapted to challenges, which yielded a process that was inconsistent at times. All authors read and approved the final manuscript.

DISCLAIMER
The findings and conclusions in this report are those of the author(s) and do not necessarily represent the official position of the Centers for Disease Control and Prevention (CDC).

PEER REVIEW
The peer review history for this article is available at https://publons. com/publon/10.1111/irv.13089.

DATA AVAILABILITY STATEMENT
Data are not publicly available. Please contact corresponding author Alissa O'Halloran (idg3@cdc.gov) with data-related questions.