FluWatchers: Evaluation of a crowdsourced influenza-like illness surveillance application for Canadian influenza seasons 2015–2016 to 2018–2019

Background: Sentinel influenza-like illness (ILI) surveillance is an essential component of a comprehensive influenza surveillance program. Community-based ILI surveillance systems that rely solely on sentinel healthcare practices omit important segments of the population, including those who do not seek medical care. Participatory surveillance, which relies on community participation in surveillance, may address some limitations of traditional ILI systems. Objective: We aimed to evaluate FluWatchers, a crowdsourced ILI application developed to complement and complete ILI surveillance in Canada. Methods: Using established frameworks for surveillance evaluations, we assessed the acceptability, reliability, accuracy and usefulness of the FluWatchers system 2015–2016, through 2018–2019. Evaluation indicators were compared against national surveillance indicators of ILI and of laboratory confirmed respiratory virus infections. Results: The acceptability of FluWatchers was demonstrated by growth of 50%–100% in season-over-season participation, and a consistent season-over-season retention of 80%. Reliability was greater for FluWatchers than for our traditional ILI system, although both systems had week-over-week fluctuations in the number of participants responding. FluWatchers’ ILI rates had moderate correlation with weekly influenza laboratory detection rates and other winter seasonal respiratory virus detections including respiratory syncytial virus and seasonal coronaviruses. Finally, FluWatchers has demonstrated its usefulness as a source of core FluWatch surveillance information and has the potential to fill data gaps in current programs for influenza surveillance and control. Conclusion: FluWatchers is an example of an innovative digital participatory surveillance program that was created to address limitations of traditional ILI surveillance in Canada. It fulfills the surveillance system evaluation criteria of acceptability, reliability, accuracy and usefulness. Suggested citation: Lee L, Desroches M, Mukhi S, Bancej C. FluWatchers: Evaluation of a crowdsourced influenza-like illness surveillance application for Canadian influenza seasons 2015–2016 to 2018–2019.


Introduction
FluWatch is Canada's national seasonal influenza surveillance program and consists of a network of laboratories, hospitals, physician offices, provincial and territorial ministries of health and Canadians (1). FluWatch consists of seven surveillance components (geographical spread, laboratory confirmed detections, syndromic influenza-like illness (ILI) surveillance, outbreak surveillance, severe outcome surveillance, strain characterization and antiviral resistance testing and vaccine monitoring) that work together to allow FluWatch to meet three main program objectives (detect, inform and enable).
The World Health Organization (WHO) defines the global standards for the collection, reporting and analysis of seasonal influenza surveillance data and provides a framework for influenza surveillance for member states (2). While the WHO does not mandate the exact surveillance components that every surveillance system must contain, it does recommend the inclusion of community-based surveillance of ILI as part of a comprehensive influenza surveillance system (2).
Developed in 1996, the Sentinel Practitioner ILI Reporting System (SPIR) is the primary source for ILI surveillance data for the Public Health Agency of Canada's (PHAC) FluWatch program (1). The SPIR consists of outpatient influenza data submitted by primary care practitioners or registered nurses. There are, however, three major limitations to SPIR: it is reliant on voluntary reporting from a convenience sample of volunteer sentinel physicians or registered nurses; only data from individuals who seek medical attention are captured; and data submission is highly manual and interrupts practitioner workflow.
A growing trend is the use of hybrid surveillance systems that use digital surveillance to complement traditional surveillance (3). One popular digital surveillance trend is participatory surveillance or crowdsourced surveillance. Participatory surveillance systems rely on volunteer members of the community to regularly share and report health information via the internet for disease surveillance (3,4). Relying on volunteers address various limitations of traditional ILI surveillance systems, such as reporting delays, low participation and exclusion of individuals who do not seek medical care.
The need to address the limitations of SPIR and the advantages presented by participatory surveillance prompted the FluWatch program to create FluWatchers, an online participatory syndromic surveillance platform to help improve and complement ILI surveillance in Canada.
The FluWatchers system was developed on the Canadian Network for Public Health Intelligence (CNPHI) platform, an established PHAC initiative developed and managed by the National Microbiology Laboratory (5). The CNPHI is a purpose-built scientific public health informatics and biosurveillance platform (6). Its infrastructure provides a secure, reliable and robust technical environment to facilitate and promote multi-jurisdictional collaboration, supporting the cross-domain and cross-discipline exchange of information, ideas and intelligence. The CNPHI was a natural choice to help develop the FluWatchers program, administer the weekly questionnaire and manage the data.
FluWatchers' participants complete a brief, weekly symptom-based report via an anonymous online questionnaire that asks whether the participant, and/or registered household members, have had a cough and/or fever in the past week and their influenza immunization status. Data on other symptoms, absenteeism and healthcare utilization are also collected from individuals reporting cough and fever. The weekly questionnaire is typically administered from October through May.
The objective of the present study is to present a formal evaluation of the FluWatchers program against four surveillance metrics that were adapted from the Centers for Disease Control and Prevention's Framework for Evaluating Public Health Surveillance Systems for Early Detection of Outbreaks (7) The SPIR program consists of sentinel practitioners who report the total number of patient visits and the number of patient visits presenting with ILI on a weekly basis. Influenza-like illness is defined as a sudden onset of fever and cough and with one or more of the following: sore throat, joint pain, muscle aches, fatigue, which could be due to the influenza virus (1).
The weekly percentage of visits for ILI is defined as the number of patient visits to healthcare providers presenting with ILI symptoms in a given week divided by the total number of weekly patient visits to healthcare providers as a whole for that same week.  (8). The RVDSS collects weekly data from provincial, regional and some hospital labs across Canada. Laboratories report on the number of tests performed and the number of tests positive for influenza and other respiratory viruses.

Respiratory Virus Detection Surveillance System
The weekly percentage of tests positive for influenza and all other respiratory viruses were used for this analysis. The weekly percentage of tests positive is defined as the number of positive tests for a given virus in a given week divided by the number of tests performed for a given virus for that same week.

FluWatchers
FluWatchers data consist of self-reported weekly episodes of cough and/or fever. For any participant reporting cough and fever, data on other symptoms experienced, absenteeism and healthcare utilization are collected. For the FluWatchers program, ILI is defined as a report of fever and cough.
The weekly percentage of FluWatchers reporting ILI is defined as the number of reports of cough and fever in a given week divided by the total number reports received by participants for that same week.

Measures
The four evaluation components were assessed as outlined in Table 1.

Reliability
Across four seasons, FluWatchers was consistently more reliable than SPIR (i.e. the denominator was more consistent week to week). The percentage of weeks where the denominator (number of weekly FluWatcher participants) was found to be within ±5% of a season median ranged from 55%-64% ( Table 2). This range is higher than the denominator (weekly number of patients seen) reported by SPIR sentinels, where only 26%-41% of reporting weeks were within ±5% of a respective season median. The percentage of weeks where the denominator was within ±10% or ±15% of a respective season median was always higher in the FluWatchers data. In the season 2018-2019 (Figure 2), the percentage of weeks where the denominator was found to be within ±5% of a season median was 65% in the FluWatchers data compared with 26% in the SPIR data. The percentage of weeks where the denominator was found to be within ±15% of the season median improved to 100% in the FluWatchers data and 65% in the SPIR data.

Accuracy
Across four seasons, when the weekly FluWatchers ILI rates were compared with the positivity rate of influenza from national surveillance system, there was a significant and strong correlation between the two datasets ( Figure 3).  Furthermore, when comparing the FluWatchers data to the positivity rate of other respiratory viruses across four seasons, there was either a weak or a negative correlation with adenovirus, enterovirus/rhinovirus, hMPV and parainfluenza ( Table 3). There was a moderate to strong correlation between the FluWatchers data and seasonal coronavirus and RSV in all seasons except the 2015-2016 season.
Weekly FluWatchers ILI rates were also compared to SPIR's weekly percentage of visits for ILI. The Pearson correlation coefficient varied across the four seasons ranging from moderate to strong ( Figure 4). As a validation measure, the weekly percentage of visits for ILI was compared with the percentage of tests positive for influenza. During the four seasons, the correlation was variable between SPIR ILI and laboratory data,

Usefulness
The FluWatchers program provided other data that had not previously been collected by PHAC or was not available weekly. These data include healthcare utilization, laboratory testing, vaccination status, absenteeism and demographic information such as age, gender, regular contact with patients and location (first three characters of a postal code-forward sortation area). Additionally, the FluWatchers program collected surveillance data from individuals who did not seek medical care or get tested.

Discussion
Our analyses show that the FluWatchers program fulfills the four surveillance evaluation areas assessed.
Acceptability-Canadians are willing to participate in FluWatchers, as reflected by an increase in uptake over the four seasons and a high retention rate. In its fourth year of surveillance, the number of participants was comparable to that seen in mature participatory ILI surveillance systems, some of which have been established as early as 2005 and in countries with populations larger than Canada (10,11). Some programs from countries with smaller populations than Canada, such as the Flutracking in Australia and the De Grote Griepmeting system in Belgium and the Netherlands, have between 15,000 and 50,000 registered users. The United States' Flu Near You has over 50,000 users from a national population of over 327 million (10,11). FluWatchers is still a small and relatively new program and has the potential to attract and retain more participants.
Reliability-FluWatcher participants have been consistently providing data. The percentage of weeks where the number of FluWatcher participants (denominator) was within either ±5%, ±10% or ±15% from the season median was always higher than that of the SPIR system. Influenza season in Canada often peaks around Christmas and New Years (late December, early January), when data providers such as practitioners and laboratories may be at reduced capacities. This affects the timing and the quality of data around peak influenza season. FluWatchers participation consistently dropped in late December and early January; however, the drop was not as drastic as that seen for SPIR participants. Receiving consistent and reliable data is important in surveillance to interpret trends.
Accuracy-The FluWatchers data appeared to track influenza in Canada with a moderate to strong positive correlation to our main influenza activity indicator, the percentage of laboratory tests positive for influenza. The timing of the peaks suggest that FluWatchers ILI data peaks before the influenza laboratory data. This is not unexpected because one of the aims of syndromic surveillance is to identify an increase illness activity before formal diagnoses are confirmed and reported to public health agencies (12). Usefulness-FluWatchers added value to the FluWatch surveillance program by filling gaps in data that is either not collected by PHAC or not available in a timely manner. Traditional surveillance programs within FluWatch typically capture the "tip of the iceberg" of influenza cases in Canada since only reports of positive laboratory confirmed cases are collected. FluWatchers may give us a better idea about the burden of influenza in Canada by capturing cases who did not seek medical attention or get tested for influenza, and by providing data on absenteeism and healthcare utilization. Additionally, the FluWatchers data can be used to inform work on initiatives such as the WHO's Pandemic Influenza Severity Assessment (PISA) (13). For example, "impact" is one of PISA's three main indicators, where school and work absenteeism due to influenza is recommended as a measurement of how an influenza epidemic affects society. Currently, data on the impact indicator is not currently available for Canada, and FluWatchers could potentially fill this gap with the weekly absenteeism data it collects.

Strengths and limitations
FluWatchers does have its limitations. The FluWatchers population differs from the Canadian population as seen in the 2016 Canadian Census: FluWatchers participants, while coming from all provinces and territories, from urban and rural settings, and all age groups, genders and influenza vaccine acceptance, under-represent the tails of Canada's age distribution and overrepresent females, urban-dwelling Canadians and those who engage in health promoting behaviors as indicated by high influenza vaccine coverage (9). This is not unique to Canada as other participatory surveillance systems around the world experience the same limitations but are still able demonstrate similar trends as traditional ILI sources (3,4,14,15). Despite this limitation, the FluWatchers data demonstrated positive attributes of other participatory surveillance systems, such as accuracy and sensitivity and being able to measure burden of illness (4,15). While the data provided by FluWatchers was comprehensive, further exploration of the data must be done and biases need to be quantified before using the data for other purposes than that of surveillance for ILI. The FluWatchers program is still in its infancy and public health practitioners can work towards using these data for other purposes, such as estimating vaccination coverage and effectiveness, informing disease transmission models, and supplying information for cost-benefit analyses of public health measures such as vaccination, as has been done by similar programs in other countries (4).

Conclusion
FluWatchers is an example of an effective and innovative surveillance program that was created to address limitations of traditional ILI surveillance in Canada. Currently, FluWatchers ILI rates are a formal indicator under syndromic surveillance and have been incorporated into Canada's weekly FluWatch report.