Continuous prediction of a time intervals-related pattern’s completion

Abstract

In many daily applications, such as meteorology or patient data, the starting and ending times of the events are stored in a database, resulting in time interval data. Discovering patterns from time interval data can reveal informative patterns, in which the time intervals are related by temporal relations, such as before or overlaps. When multiple temporal variables are sampled in a variety of forms, and frequencies, as well as irregular events that may or may not have a duration, time intervals patterns can be a powerful way to discover temporal knowledge, since these temporal variables can be transformed into a uniform format of time intervals. Predicting the completion of such patterns can be used when the pattern ends with an event of interest, such as the recovery of a patient, or an undesirable event, such as a medical complication. In recent years, an increasing number of studies have been published on time intervals-related patterns (TIRPs), their discovery, and their use as features for classification. However, as far as we know, no study has investigated the prediction of the completion of a TIRP. The main challenge in performing such a completion prediction occurs when the time intervals are coinciding and not finished yet which introduces uncertainty in the evolving temporal relations, and thus on the TIRP’s evolution process. To overcome this challenge, we propose a new structure to represent the TIRP’s evolution process and calculate the TIRP’s completion probabilities over time. We introduce two continuous prediction models (CPMs), segmented continuous prediction model (SCPM), and fully continuous prediction model (FCPM) to estimate the TIRP’s completion probability. With the SCPM, the TIRP’s completion probability changes only at the TIRP’s time intervals’ starting or ending point. The FCPM incorporates, in addition, the duration between the TIRP’s time intervals’ starting and ending time points. A rigorous evaluation of four real-life medical and non-medical datasets was performed. The FCPM outperformed the SCPM and the baseline models (random forest, artificial neural network, and recurrent neural network) for all datasets. However, there is a trade-off between the prediction performance and their earliness since the new TIRP’s time intervals’ starting and ending time points are revealed over time, which increases the CPM’s prediction performance.

Appendices

Appendix A Efficient TIRP’s candidates generator

We introduce an efficient TIRP’s candidates generator (Algorithm 3), which exploits the transitivity property to generate all possible TIRPs that can evolve from a given TIRP-prefix with multiple unfinished STIs. The algorithm works by iterating over all possible temporal relations between each pair of adjacent unfinished STIs, according to the lexicographical order, and inferring temporal relations between the remaining unfinished STI pairs using Allen’s transition Table [26]. The relevant temporal relations that can eventually evolve given unfinished STIs $A^*$, $B^*$ and $C^*$ are presented in Table 1. Given the temporal relation that eventually evolved between STIs $A$ and $B$, r( $A$,$B$), and the temporal relation between STIs $B$ and $C$, r( $B$,$C$), can be used to infer the optional temporal relations between STIs $A$ and $C$, r( $A$,$C$).

Table 1 The Allen’s transition table that is relevant for the unfinished coinciding STIs challenge

Algorithm 3 takes a TIRP-prefix as inputs and stores the set of possible TIRPs that can evolve from it in the variable fnlTIRPCnddts. The given TIRP-prefix might include both finished STIs ($\hat{IS}_f$) or unfinished STIs ($\hat{IS}_*$). Only the temporal relations among pairs of unfinished STIs are unknown and need to be determined. First, fnlTIRPCnddts is initialized to the empty set (line 1), and a variable unfSTIsLen is set to the number of unfinished STIs in the TIRP-prefix (line 2). Then, the function initGenAdjacentUnfSTIs enumerates all possible temporal relations between all pairs of adjacent (according to the lexicographical order) unfinished STIs of the TIRP-prefix. Thus, for $\check{k}$ unfinished STIs, the function outputs $3^{\check{k}-1}$ possible candidates, stored in the variable initTIRPCanddts (line 3). In lines 4–6, the algorithm iterates over initTIRPCanddts. Each candidate c is expanded using the function expandTIRPCand (Algorithm 4), which returns all TIRP candidates derived from c using Allen’s transition table (line 5). They are added to fnlTIRPCnddts, which is finally returned by the algorithm.

Efficient TIRP’s Candidates Generator

Fig. 17

A TIRP-prefix including unfinished STIs $A^*$, $B^*$, $C^*$, and $D^*$, where $A^*{\texttt {+}}< B^*{\texttt {+}}< C^*{\texttt {+}} < D^*{\texttt {+}}$ and all possible TIRPs that it can evolve into, presented in two ways: (i) TIRP-prefix schematic and (ii) TIRP-prefix half matrix representation

The function expandTIRPCand takes a parameter adjJump which specifies the “adjacency jump,” i.e., the distance between the analyzed adjacent unfinished STIs. For example, in Fig. 17, given the temporal relations between the adjacent unfinished STIs: $A$ overlaps $B$, $B$ overlaps $C$, and $C$ overlaps D, and while considering the second-adjacent unfinished STIs (adjJump=2), r(A, C) is must be overlaps based on r(A, B) and r(B, C). Similarly, r(B, D) must be overlaps based on the r(B, C) and r(C, D). Then, the recursive procedure proceeds by inferring all the temporal relations among c’s adjJump+1 adjacent unfinished STIs. For example, while considering the third-adjacent unfinished STIs in Fig. 17, r(A, D) can be inferred from r(A, B) and r(B, D).

Algorithm 4 presents the recursive expandTIRPCand function. The function takes as arguments the current TIRP candidate to expand (c), the number of TIRP-prefix’s unfinished STIs (unfSTIsLen), the adjacency jump between unfinished STIs (adjJump), and an index i of the earliest unfinished STI to consider. The function returns a set of c’s expanded TIRPs candidates. In lines 2–3, a function getRel is called, which returns the temporal relation between two given unfinished STIs. Note that the temporal relation between the i-th and $i+1$-th unfinished STIs (stored in fstRel) is known since it was determined during the enumeration step of Algorithm 3. Also, the temporal relation between the $i+1$-th and $i+$adjJump-th unfinished STIs (stored in scdRel) is known since it has been determined earlier in the recursive procedure. Based on fstRel and scdRel, the temporal relations among the i-th and $i+$adjJump-th unfinished STIs are inferred using Allen’s transition table and stored in inferredRels (line 4).

Expand TIRP’s Candidates (expandTIRPCand function)

Then, the algorithm iterates over the inferredRels (lines 5–12) with the current relation stored in the variable rel, and in each iteration executes the following steps: (a) updates c’s temporal relation between the i-th and i+adjJump-th unfinished STIs, which is stored in extC (line 6); (b) if there are more relations between adjJump-adjacent STIs to infer, calls itself recursively with extC and i+1 adding the result to fnlCnds (lines 7–8); (c) otherwise, if there are more relations between adjJump+1-adjacent STIs to infer, calls itself recursively with extC and adjJump+1 adding the result to fnlCnds (lines 9–10); (d) otherwise, adds extC to fnlCnds (lines 11–12). Lastly, fnlCands is returned.

Given a TIRP-prefix with STI series IS of size k, in which there are $\check{k}$ unfinished STIs, the overall time complexity of creating the TIRP’s candidates, using Algorithms 3 and 4, is bounded in the worst case by the following expression: $O(3^{(\check{k}^2-\check{k})/2})$. The base of the exponent in the upper bound (i.e., three) represents the number of possible temporal relations between each pair of unfinished STIs. The formula $(\check{k}^2-\check{k})/2$ follows from the binomial coefficient $\left( {\begin{array}{c}\check{k}\\ 2\end{array}}\right) =\check{k}(\check{k}-1)/2=(\check{k}^2-\check{k})/2$, where two stands for pairs of temporal relations and $\check{k}$ is the number of unfinished STIs. However, this worst-case, upper-bound complexity expression is far from the practical situation for the algorithms. The reduction is achieved by using the transitivity of temporal relations. The overall number of generated candidates is smaller than the naive generation, which does not exploit the transitivity property.

Appendix B Cutoffs used for the knowledge-based abstraction

The knowledge-based abstraction was performed using cutoffs defined by a domain expert, in which the states for each dataset were as follows:

• CSP dataset:

  • MAP [mmHg]: $\le$60, (60,90], >90

  • CVP [mmHg]: $\le$5, (5,17], >17

  • FiO2 [%]: $\le$41, (41,60], >60

  • HR [bpm]: $\le$60, (60,110], >110

  • PEEP [cmH2O]: $\le$10, >10

  • TMP [$^\circ$C]: $\le$35, (35,38.5], >38.5

  • BE [mEq/L]: $\le$-6, (-6,-3], >-3)

  • CI: [$L/min/m^2$] $\le$2.5, >2.5

  • Glucose [mmol/L]: $\le$2.5, (2.5,10] >10

  • CKMB [%]: $\le$25, (25,50], >50

• AHE dataset:

  • HR [bpm]: $\le$50, (50, 100], >100

  • RESP [breath/min]: $\le$7, (7,20], >20

  • SpO2 [%]: $\le$88, (88,100], >100

  • SABP [mmHg]: $\le$90, (90,140], >140

• DBT dataset:

  • Blood Glucose [mg/dL]: $\le$100, (100,125], (126,200], >200

  • HbA1C [%]: $\le$7, (7,9], (9,10.5], >10.5

  • LDL Cholesterol [mg/dL]: $\le$100, (100,130], (130,160], >160

  • Creatinine [mg/dL]: $\le$1, (1,1.5], (1.5,2.5], (2.5,4], >4

  • Albumin [g/dL]: $\le$3.5, >3.5

The cutoffs for the EFIF dataset were not applicable.

Appendix C imbalance ratio

The TIRPs’ instances imbalance ratio was defined as the number of instances of a complete TIRP divided by the total number of instances. Figure 18 presents the number of TIRPs’ instances imbalance ratio and the performance prediction while using the early warning strategies for each TIRPs’ instances imbalance ratio and values of $\tau$. As can be shown in Fig. 18, the imbalance ratio was lower than 0.3 for most of the TIRPs. Overall, there was a positive correlation between the AUPRC performances and the imbalance ratio for all datasets. In contrast, there was no significant correlation between the imbalance ratio and the AUROC.

Fig. 18

The imbalance ratio was lower than 0.3 for most of the TIRPs, and there was a positive correlation between the AUPRC performances and the imbalance ratio for all datasets

Appendix D Parameters of baseline models

The parameters of each model are selected after testing the performance of each combination (not in a greedy comparison approach), and here we describe the parameters that performed best.

Random forest (RF) [51] classifier utilizes an ensemble learning approach, a technique that combines the decisions from multiple models. We used 100 trees in the forest for the RF model and a maximum depth of 3 for a tree, in which the combinations of 30, 50, 100, and 200 trees in the forest and depths of 2, 3, 5, or 10 were tested. Bootstrap was used when building trees and out-of-bag samples to estimate the generalization accuracy. RF was implemented with Python 3.6 Scikit-Learn (https://scikit-learn.org) version 0.22.1.

The artificial neural network (ANN) [52] is comprised of neuron layers, which contain an input layer, hidden layers, and an output layer. In this paper, the fully connected ANN architecture was used, in which all neurons in one layer are connected to all neurons in the next layer. We used ANN with two hidden layers with 50 neurons for each hidden layer and with the activation function ReLU [56]. We trained all models using a maximum epoch of 20, a batch size of 16, and a learning rate of 0.001 with gradually decreasing. A validation set was used to measure the generalization error by randomly taking 20 percent of all training data. We used early stopping on the validation set, in which lower than a change of 0.001 was not considered an improvement for the loss. For ANN, combinations of one, two, three, or five hidden layers with 20, 50, or 100 neurons and batch sizes of 16, 32, or 64 were tested. ANN was implemented with Python 3.6 Scikit-Learn (https://scikit-learn.org) version 0.22.1.

The sequential deep learning recurrent neural network (RNN) [53] is designed to learn the data dependencies in a sequence. We used RNN with five hidden units per layer and a recurrent dropout of 0.2 and an activation function, ReLU [56]. We trained all models using a maximum epoch of 20, a batch size of 16, and a learning rate of 0.001 with a gradually decreasing. A validation set was used to measure the generalization error by randomly taking 20 percent of all training data. We used early stopping on the validation set, in which lower than a change of 0.001 for more than two epochs was not considered an improvement for the loss. For RNN, combinations of one, two, three, or five hidden units per layer, dropout of 0.2, 0.3, or 0.5, with batch sizes of 16, 32, or 64 were tested. RNN was implemented with Keras (https://keras.io) version 2.2.5.

For parameters we did not specify, we used the default.

Appendix E Preliminary analysis

1.1 E.1 TIRP’s completion prediction relative to the end

This analysis’s goal was to evaluate the prediction of a TIRP’s completion at the instances’ different portions over time using the CPMs (Sect. 5.2.1). The TIRP-prefixes’ instances were evaluated at each time stamp until the end of the entity’s data (i.e., patient), in which the decisions for the TIRP’s completion were determined at different instances’ revealed time portions. Then, the metrics (Sect. 5.3) were computed for all instances for each revealed time portion. The setup for this analysis was the same as described in Sect. 5.2.1. This preliminary analysis was excluded from the Evaluation and Results (Sects. 5–6) since it examines the TIRP’s completion prediction in retrospect, in which the instances’ different portions of decision points over time were selected based on the end of the entity’s data.

The CSP, AHE, and DBT datasets were abstracted with eleven combinations of temporal abstraction methods (KB, GRAD, EWD, EFD, and SAX) and the number of symbols (2, 3, 4, and KB), in which the TIRPs were discovered for each combination. The EFIF dataset was not abstracted using KB and resulted in ten combinations of temporal abstraction methods (GRAD, EWD, EFD, and SAX) and the number of symbols (2, 3, and 4). All five continuous prediction models (SCPM, FCPM, RF, ANN, and RNN) were evaluated on the TIRP-prefixes’ detected instances.

1.2 E.2 Analysis results

The results are based on 2,392 10-fold cross-validation runs on 1,094 TIRPs for the CSP dataset, 158 TIRPs for the AHE dataset, 766 TIRPs for the DBT dataset, and 374 TIRPs for the EFIF dataset.

We first evaluated the overall performance of the five continuous prediction models: SCPM, FCPM, RF, ANN, and RNN on predicting a TIRP’s completion at the instances’ revealed portions of time. Figure 19 presents the mean AUROC, and AUPRC results over the instances’ revealed portions of time, in which each point on the graph represents the mean performance results for the different TIRPs.

As expected, it can be seen from Fig. 19 that as long as time goes by for each instance, the continuous prediction models provided more accurate predictions, resulting in better AUROC and AUPRC performance over time. In all datasets, the FCPM performed best and the SCPM worst in terms of AUROC. However, in terms of AUPRC, the SCPM performed worst at making relatively early decisions regarding the instances of TIRP’s completion and drastically improved as long as time goes. As a result, the SCPM performed with better AUPRC than the other baselines in all datasets at making a relatively late decision regarding the instances TIRP’s completion. Also, for the DBT and EFIF datasets, even though the FCPM performed with better AUPRC than other baseline models at making relatively early decisions, SCPM performed better at making relatively late decisions. Overall, the FCPM performed better than the baseline models in terms of AUROC and AUPRC. The AUPRC performance of the SCPM was poor at making relatively early decisions but was improved drastically as long as time went by. As a result, the AUPRC performances of FCPM and SCPM were close at making relatively late decisions regarding a TIRP’s completion.

Fig. 19

The FCPM performed better than the baseline models in terms of AUROC and AUPRC. The AUPRC performance of the SCPM was poor at making relatively early decisions but was improved drastically as long as time went by. As a result, the AUPRC performances of FCPM and SCPM were close at making relatively late decisions regarding a TIRP’s completion

The following results in this analysis are only involved in the FCPM results as we demonstrated its superiority.

Next, we evaluated the TIRP’s completion at the instances’ revealed portions of time by the FCPM for each temporal abstraction method (KB, GRAD, EWD, EFD, and SAX). Figure 20 presents the number of TIRPs per temporal abstraction method and the mean AUROC and AUPRC results over the instances’ revealed portions of time. Each point on the chart represents the mean performance results of the FCPM in providing the completions predictions for the different TIRPs.

Fig. 20

FCPM provided less accurate predictions for TIRPs discovered using GRAD and more accurate predictions for TIRPs discovered using KB and EWD

Figure 20 shows that the CSP, DBT, and EFIF datasets abstracted with EWD resulted in more TIRPs that end with the target event than other temporal abstraction methods. However, the AHE dataset that was abstracted with EFD resulted in more TIRPs that end with the target event. As can be seen from all datasets, the CPMs provided less accurate predictions for TIRPs discovered using GRAD. The prediction performances of EFD and SAX were quite similar, but SAX performed slightly better. Looking at the overall mean results, FCPM provided more accurate predictions for TIRPs discovered using KB and EWD. However, EWD performed much better than KB for the DBT dataset.

Fig. 21

CPMs provided more accurate predictions over time for TIRPs discovered using two symbols per variable or KB than three and four symbols per variable

Fig. 22

FCPM performed slightly better over time for TIRPs discovered using EWD with two symbols per variable. In contrast, FCPM performed poorly over time for TIRPs discovered using EWD with four symbols per variable

We also evaluated the TIRP’s completion at the instances’ revealed portions of time provided by the FCPM for each number of symbols (two, three, four, and a varied number of symbols for KB). Figure 21 presents the number of TIRPs per number of symbols and the mean AUROC and AUPRC results over the instances’ revealed portions of time. Each point on the graph represents the mean performance of the FCPM at a instances’ revealed portion of time, averaging the results of the different TIRPs.

Figure 21 shows that CPMs provided more accurate predictions over time, in terms of AUROC, for TIRPs discovered using two symbols per variable or KB. In terms of AUPRC, two symbols and KB performed better for the CSP dataset. For the AHE, DBT, and EFIF datasets, four symbols per variable performed better than other numbers of symbols at making relatively early decisions, but as long as time went by, two symbols and KB closed the gap and performed better at making relatively late decisions. The poor performance of the three symbols per variable is related to the poor performance of the GRAD abstraction, which was tested only with three symbols. Looking at the overall mean results, CPMs provided more accurate predictions over time for TIRPs discovered using two symbols per variable or KB than three and four symbols per variable.

Lastly, we evaluated the performance of a different number of symbols (two, three, four, and a varied number of symbols for KB) only on TIRPs discovered using EWD and KB. Figure 22 presents the number of TIRPs per temporal abstraction and number of symbols and the mean AUROC and AUPRC result over the instances’ revealed portions of time. Each point on the graph represents the mean performance of the FCPM, at a instances revealed portion of time, in providing the completions probabilities for the different TIRPs.

Figure 22 shows the CSP and DBT datasets that were abstracted with EWD with two symbols per variable resulting in more TIRPs that end with the target event than other temporal abstraction methods. In contrast, for the AHE dataset, there was much difference between the EWD or KB and the number of TIRPs. For the EFIF dataset, the different number of symbols per variable resulted in a similar number of TIRPs. EWD with two and four symbols per variable resulted in more accurate predictions over time. In addition, it can be shown that FCPM provided more accurate predictions over time, in terms of AUROC, for TIRPs discovered using KB for the CSP and AHE datasets. In contrast, for the DBT dataset, FCPM provided less accurate predictions with KB. In addition, in terms of AUPRC, FCPM performed poorly with KB for the AHE and DBT datasets. For the DBT dataset, EWD performed better than KB with all numbers of symbols. Overall, FCPM performed slightly better over time for TIRPs discovered using EWD with two symbols per variable. In contrast, FCPM performed poorly over time for TIRPs discovered using EWD with four symbols per variable.

In summary, the FCPM performed better than the baseline models in terms of AUROC and AUPRC. The AUPRC performance of the SCPM was poor at making relatively early decisions but improved as time went by. Moreover, FCPM provided less accurate predictions for TIRPs discovered using GRAD and more accurate predictions for TIRPs discovered using KB and EWD. While comparing the predictions over time provided by FCPM, TIRPs discovered that using two symbols per variable or KB led to better results. More specifically, FCPM performed slightly better over time for TIRPs discovered using EWD with two symbols per variable.