Study design

The Rockefeller Neuroscience Institute (RNI) team initiated a study approved by the institutional review board (IRB) at the West Virginia University Medical Center (#2003937069), Vanderbilt University Medical Center (#200685), and Thomas Jefferson University (#2004957109A001) to combine physiological and cognitive biometrics and self-reported symptoms information from individuals at risk for exposure to COVID-19 and potential contracture of a viral illness. We recruited study participants from each tertiary medical center by approaching front-line health care workers receiving regional referrals for COVID-19 patients. We asked each participant to 1) wear a smart ring device [16] with sensors that collect physiological measures such as body temperature, sleep, activity, heart rate, respiratory rate, heart rate variability; 2) use a custom mobile health app [17] to complete a brief symptoms diary [3], social exposure to potentially infected contacts, and measures of physical, emotional, and cognitive workload; (see S1 Table and S1 File) as well as the psychomotor vigilance cognitive task (PVT) [18] to measure attention and fatigue twice a day. All data are collected, structured, and organized into the RNI Cloud data lake for analysis. The RNI Cloud is a HIPAA compliant data platform hosted in Amazon Web Services (AWS) that supports all the security and legal requirements to protect the data’s privacy and integrity from the participants in the context of multi-center clinical studies [19].

We utilized a machine learning approach that combines features through probabilistic rules and provides a prediction. The training process consists of two steps. It combines subject reported symptoms (labeling model) to inform a predictive framework (forecast model) that uses physiological and cognitive signals to forecast suspicion of a viral illness (Fig 1). The dataset consisting of PVT and wearable data is split, 75% for training, and 25% reserved for testing the model [20] (see S4 File). The labeling and forecasting models are created from a set of rules combining one or more features. All rules are given a weight and combined to provide a final decision [21–23] (see S2 Table).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

Data Flow a) the labeling model, b) the forecast model. Each model takes as input three days of data (d, d-1, d-2).

https://doi.org/10.1371/journal.pone.0257997.g001

Labeling model

We use a rule-based approach to create an AI model that labels an individual’s self-reported symptoms as suspicious (or not) for presenting symptoms consistent with a viral illness. The labeling model is created based on the expert knowledge manually translated into decision rules (see below). The purpose of this model is to define if a person is being suspicious of an infectious disease (below we just say suspicious) based on its self-reported symptoms. Rules are based on those symptoms commonly present in a diagnosed viral-like condition and those more specific for SARS-Cov-2 (e.g., loss of taste and smell) [2, 24, 25]. Resulting rules (Table 1) assign, for instance, higher confidence on suspicion of a viral-illness for self-reported fever with the persistence of symptoms for more than two consecutive days. In comparison, lower confidence is assigned for stuffy nose and swollen eyes without fever (see S5 File). The rules and weights in this model were establish from clinical subject matter experts. In particular, the weights associated with the rules were chosen to minimize the labeling error assessed by medical experts. We also fine-tuned some rule weights by fitting the model to a small synthetic data set, which contained typical symptom combinations. The actual calculations of the labeling model’s output score is based on the machinery of the probabilistic logic, described in more detail in the next paragraph.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Labeling model rules.

https://doi.org/10.1371/journal.pone.0257997.t001

Forecasting model

The forecasting model was used to associate a label of suspicion for viral illness from the Labeling model to the features extracted from the user’s cognitive function assessment and physiological signals. The physiological features include (1) single day and (2) rolling averages over 28 days of the heart rate, heart rate variability, respiration rate, activity, sleep latency, sleep duration, composition (light, REM, deep), skin temperature, and sleep efficiency. Physiological features to the exclusion of skin temperature are measured during the night to remove noise due to varying daily activities. The daily cognitive task (PVT) is a sustained-attention, a reaction-timed task that measures the speed with which subjects respond to a visual stimulus [18]. From this data set, the algorithm extracts rules using an information gain-based approach and combines them in a predictive model using a probabilistic graphical network as follows.

The set of probabilistic rules comprises a Markov network. The joint distribution defined by the Markov network can be written as where x = (x₁,x₂,…,x_n,y) denotes a set of n+1 binary variables, out of which the first n are input variables, and y is the output variable. Here, f_j(x)∈{1,0} is a Boolean function corresponding to the rule, ω is a factor associated with the corresponding rule, Z is the normalization constant. In the current implementation, the relation between the rule’s factor ω and the weight ψ used in the supplementary materials is given by . More details on the fundamentals of the probabilistic logic can be found in [21–23]. With the joined distribution defined above, the model prediction s for every observation vector r = (r₁,r₂,…,r_n) is computed as the conditional probability of the output variable y as s = P(y = 1|r).

If a training set is available, the model’s parameters can be determined by the calibration process, which minimizes the prediction error. Suppose, for the i-th training example the model’s prediction is s_i, and the observed (ground truth) output is y_i. We define the cross-entropy loss function as

The calibration process uses the steepest gradient descent to find a combination of rules weights which minimizes the loss function. In our particular implementation we used Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm (L-BFGS).

Our model was developed on Stratyfy’s Probabilistic Rule Engine, a commercial machine learning platform [26] (see S3 File). In the application to our study, this general framework for creating a rule-based predictive model was applied as follows: In a preprocessing step, the data from the wearable device (e.g., heart rate, temperature, etc.) and the information for the mobile app (e.g., symptoms, results of the PVT, etc.) were collected, checked for completeness, and engineered variables were extracted. We found that, for our study, large gaps in the data had a significant negative impact on the predictive power of the model and, therefore, our efforts were concentrated on cases where most of the required information was actually available. We identified a number of engineered variables (for instance, a ratio of heart rate to heart rate variability) which helped significantly improve the model’s predictive power. In order to be used with probabilistic rules, continuous variables are discretized, and then discretized and categorical variables are converted into binary variables by one-hot encoding. The labeling model described above was used to construct the binary output variable, marking for each case days of potential onset of a viral infection. At this point, the setup fit into the context of a standard supervised learning problem: We needed to train a classifier to predict the onset of a disease based on the information available before the actual onset. We opted for the rule-based system described above for several reasons. A main reason was the transparency and interpretability of our model. In this case, our rule-based system produced models that were fairly small in size (20–50 rules) and still highly accurate. We compared our approach to standard approaches, for example gradient boosting, and found the rule-based approach most promising. Note that, in this study, the rule-based models were used in two ways. In the labelling model, the rules, together with the confidences, were developed and specified by clinical experts. To create the forecasting model, the rules were extracted from the available data via rule mining. For this purpose, we used the Association Rule Mining algorithm [27], which is based on the co-occurrences frequency analysis. After extracting the rules, the weights of the rules were determined by the calibration process outlined above.

Validation of the model

Model performance was tested with K-fold cross-validation with in our case we perform four rounds of validation (K = 4). One round of cross-validation involves portioning the dataset into complementary subsets, performing the training on one subset and the validation on the other. To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (averaged) over the rounds to give an estimate of the model’s predictive performance. The entire dataset is divided 4 times as 75% for training and 25% for validating the model. The results are then average across the 4 runs of training-validation. The model weights in the final model are obtained by using training dataset of the model. We measure the model’s performances at various threshold settings. We also used the area under the curve (AUC) of the receiver operating characteristic (ROC) curve as a threshold-invariant performance measure. Additionally, we report the model’s learning performances, i.e., how much data is required to reach the stability of the model. Learning is achieved when adding more data does not significantly impact the performance of the model.