Optimal classification of Gaussian processes in homo- and heteroscedastic settings

Abstract

A procedure to derive optimal discrimination rules is formulated for binary functional classification problems in which the instances available for induction are characterized by random trajectories sampled from different Gaussian processes, depending on the class label. Specifically, these optimal rules are derived as the asymptotic form of the quadratic discriminant for the discretely monitored trajectories in the limit that the set of monitoring points becomes dense in the interval on which the processes are defined. The main goal of this work is to provide a detailed analysis of such optimal rules in the dense monitoring limit, with a particular focus on elucidating the mechanisms by which near-perfect classification arises. In the general case, the quadratic discriminant includes terms that are singular in this limit. If such singularities do not cancel out, one obtains near-perfect classification, which means that the error approaches zero asymptotically, for infinite sample sizes. This singular limit is a consequence of the orthogonality of the probability measures associated with the stochastic processes from which the trajectories are sampled. As a further novel result of this analysis, we formulate rules to determine whether two Gaussian processes are equivalent or mutually singular (orthogonal).

Appendices

A Discrete monitoring

In the derivations carried out, the processes X are monitored at a set of appropriately chosen discrete times $\left\{ t_i \right\} _{i=i}^N \in {\mathcal {I}}^N$. The integrals that appear (e.g., in the definitions of the inner products) are then approximated by Riemann sums

$$\begin{aligned} \int _{t \in {\mathcal {I}}} h(t) \mathrm{d}t \approx \frac{1}{N} \sum _{n=1}^N h(t_n). \end{aligned}$$

(93)

For functions that are continuous in ${\mathcal {I}}$, these Riemman sums converge to the corresponding definite integrals in the limit of dense monitoring

$$\begin{aligned} \lim _{N \rightarrow \infty } \frac{1}{N} \sum _{n=1}^N h(t_n) = \int _{t \in {\mathcal {I}}} h(t) \mathrm{d}t \quad \forall h \in {\mathcal {C}}\left[ I\right] . \end{aligned}$$

(94)

Let $K_0$ and $K_1$ be symmetric, strictly positive kernels that are continuous in ${\mathcal {I}}$. Let the corresponding RKHS’s be infinite-dimensional. In the discretized representation, the kernel functions $\left\{ K_i(s,t); s,t \in {\mathcal {I}} \right\} _{i=0}^1$ is approximated by ${\mathbf {K}}_i$, the corresponding $N \times N$ Gram matrices, whose elements are

$$\begin{aligned} \left( {\mathbf {K}}_i\right) _{mn} = K_i(t_n,t_m), \quad n,m = 1, 2,\ldots N, \end{aligned}$$

(95)

for $i = 0,1.$ Let $\left\{ \nu _{ij} = \right\} _{j=1}^N$ be the (positive) eigenvalues of matrix ${\mathbf {K}}_i$. Theorem 3.4 of Baker (1977) can be used to show that, in the limit of dense monitoring,

$$\begin{aligned} \lim _{N \rightarrow \infty } \frac{\nu _{j}}{\varDelta T} = \lambda _{j}, \quad j = 1,2, \ldots ,N \end{aligned}$$

(96)

where $\left\{ \lambda _{i1} \ge \lambda _{i2} \ge \ldots \ge \lambda _{iN} > 0 \right\}$ are the largest N eigenvalues of ${\mathcal {K}}_i$, the covariance operator associated with the kernel $K_i$.

Therefore, the spectrum of the Gram matrix ${\mathbf {K}}_i$ converges to the spectrum of the covariance operator ${\mathcal {K}}_i$. In particular, the ratio of the determinants of the Gram matrix

$$\begin{aligned} \lim _{N \rightarrow \infty } \frac{\left| {\mathbf {K}}_1\right| }{\left| {\mathbf {K}}_0\right| }= & {} \lim _{N \rightarrow \infty } \prod _{j=1}^N \frac{\nu _{1j}}{\nu _{0j}} \nonumber \\= & {} \lim _{N \rightarrow \infty } \prod _{j = 1}^N \frac{\lambda _{1j}}{\lambda _{0j}} \equiv \frac{\left| {\mathcal {K}}_1\right| }{\left| {\mathcal {K}}_0\right| }, \end{aligned}$$

(97)

can be used to define the ratio $\frac{\left| {\mathcal {K}}_1\right| }{\left| {\mathcal {K}}_0\right| }$ when the corresponding Gaussian processes are equivalent (${\mathbb {P}}_0 \sim {\mathbb {P}}_1$), in which case the limit exists (is finite) and is different from zero.

B Setup for the experiment with financial data

The setup of the experiment is as follows: Let $\left\{ S_i(t_0),\right.$$\left. S_i(t_1),\ldots , S_i(t_L) \right\}$ be the time series of asset market prices for stock i monitored at the equally spaced instants

$$\begin{aligned} t_n = t_0 + n \varDelta T; \ n = 0,1,\ldots , L, \end{aligned}$$

where $L = M (N_B + 1) - 1$. In the data analyzed $\varDelta T$ is 1 day. Therefore, the quantity $S_i(t_n)$ is the closing price of the corresponding stock on the nth day of the period considered.

The time series is broken up into M segments of length $N_B +1$, with $N_B = 2^B$ for some integer B

$$\begin{aligned} \left\{ S_i\left( t_0^{[m]}\right) , S_i\left( t_1^{[m]}\right) , \ldots , S_i\left( t_{N_{B}}^{[m]}\right) \right\} _{m=1}^M, \end{aligned}$$

where $t_n^{[m]} = t_{n+ (m-1)N_B}$, with $n = 0, 1, \ldots , N_B$, and $m = 1,2,\ldots , M$. These M time series of $N_B+1$ prices are then transformed into the corresponding time series of log-returns

$$\begin{aligned} \left\{ X_i\left( t_0^{[m]}\right) , X_i\left( t_1^{[m]}\right) , \ldots , X_i\left( t_{N_B}^{[m]}\right) \right\} _{m=1}^M, \end{aligned}$$

(98)

where

$$\begin{aligned} X_i\left( t_n^{[m]}\right) = \log \frac{S_i\left( t_{n}^{[m]}\right) }{S_i\left( t_{0}^{[m]}\right) },\quad n = 0,1,\ldots N_B. \end{aligned}$$

The goal is to discriminate between different stocks on the basis of the corresponding time series of log-returns. In particular, we will analyze how the accuracy of the predictions depends on the monitoring frequency. For this reason, discrimination is made on the basis of $N_b+1$ subsampled values within each segment

$$\begin{aligned} \left\{ X_i\left( t_{0}^{[m]}\right) , X_i\left( t_{ n_b}^{[m]}\right) , X_i\left( t_{2 n_b}^{[m]}\right) , \ldots , X_i\left( t_{N_b n_b}^{[m]}\right) \right\} , \end{aligned}$$

where $N_b = 2^b$, and $n_b = 2^{B-b}$ with $b = 0,1,\ldots ,B$. As an illustration, for $b = 0$, only two inputs in each time series are used for discrimination

$$\begin{aligned} \left\{ X_i\left( t_{0}^{[m]}\right) , X_i\left( t_{N_B}^{[m]}\right) \right\} . \end{aligned}$$

For $b = B$ ($n_B = 1$) the complete time series given by Eq. (98) is used as input to the different classifiers. The higher monitoring the frequency is, the closer the problem is to a functional paradigm.