2023 A Karhunen–Loève Theorem for Random Flows in Hilbert spaces

We develop a generalisation of Mercer’s theorem to operator-valued kernels in inﬁnite dimensional Hilbert spaces. We then apply our result to deduce a Karhunen-Loève theorem, valid for mean-square continuous Hilbertian functional data, i


Introduction
The Karhunen-Loève theorem is a fundamental result on stochastic processes, playing a central role in their probabilistic construction, numerical analysis and statistical inference (Adler, 2010;Le Maître and Knio, 2010;Hsing and Eubank, 2015).In its simplest form, it provides a countable decomposition of a time-indexed real-valued random function into a "bi-orthogonal" Fourier series that separates its stochastic from its functional components: the basis functions are deterministic orthogonal functions, and their coefficients are uncorrelated random variables.When the basis functions are ordered by decreasing coefficient variance, the k-truncated expansion provides the best k-dimensional approximation of the process in mean square.Importantly, for random functions that are mean-square continuous, the series can be interpreted pointwise.These properties explain the catalytic role the expansion has played (and continues to play) in the field of Functional Data Analysis (Wang et al., 2016) -in fact, the field's very origin traces to Grenander's use of the expansion as a coordinate system for inference on random functions (Grenander, 1950).
Traditionally, functional data analysis focused on real-valued functions defined on a compact interval.Increasingly, though, the complexity of functional data escapes this context.Modern functional data can come in the form of functional flows (random maps from an interval into a function space) or more generally functional random fields (random maps from a Euclidean set into a function space).With the aim of making such data amenable to the tools of functional data analysis, we establish a generalisation of the Karhunen-Loève expansion to random flows (and more generally random fields) valued in an abstract separable Hilbert space, possibly infinite dimensional (Theorem 2).To do so, we first establish a version of Mercer's theorem for non-negative definite kernels valued in separable Hilbert spaces (Theorem 1).
Let H be a separable Hilbert space with inner product ⟨⋅, ⋅⟩ H ∶ H × H → R and induced norm ⋅ H ∶ H → R + , with dim(H) ∈ N ∪ {∞}.We denote by B(H), B 1 (H) and B 2 (H) the space of bounded, trace-class (TC) and Hilbert-Schmidt (HS) linear operators on H, respectively, with corresponding norms: where the adjoint B * of a linear operator We say that a bounded operator B is compact if for any bounded sequence {h n } n≥1 in H, {Bh n } n≥1 contains a convergent subsequence.Let B ∈ B(H) be compact and self adjoint.Denote by {e j } j≥1 its eigenvectors, with corresponding eigenvalues {λ j } j≥1 , ordered so that λ 1 ≥ λ 2 ≥ . . . .Then {e j } j≥1 comprises a Complete Orthonormal System (CONS) for Im(B) and we may write B = ∑ j≥1 λ j e j ⊗ H e j .
Let T be any compact subset of a Euclidean space.We denote by L 2 (T , H) the Hilbert space of square integrable H-valued functions f ∶ T → H, i.e., T , H) be a (mean zero) random flow in H with finite second moment, E χ 2 L 2 (T ,H) < ∞.We will refer to χ as Hilbertian flow, as in Kim et al. (2020), though it could also be a termed a Hilbertian field when dim(T ) ≥ 2. Denote by C ∈ B(L 2 (T , H)) its covariance operator: or, equivalently: Note that C is nonnegative-definite and trace-class.In particular, C is compact and selfadjoint; therefore, it admits the following spectral decomposition in terms of its eigenvalueeigenfuction pairs (e.g.Hsing and Eubank (2015, Theorem 7.2.6)): where λ 1 > λ 2 > ⋯ > 0 are the eigenvalues and Φ k ∶ L 2 (T , H) → L 2 (T , H) the corresponding eigenfunctions for C , forming a complete orthonormal system.
Consequently, χ admits the following decomposition where the convergence is understood in the mean L 2 (T , H) norm sense lim and where it may furthermore be shown that {⟨log Ξ , Φ k ⟩ L 2 (T ,H) } k≥1 are uncorrelated random variables with zero mean.In particular, by Hsing and Eubank (2015, Theorem 7.2.8), the above decomposition is optimal: for any N < ∞ and any CONS .
We ask the following questions: Is expansion (2) interpretable pointwise in t ∈ T ?Is the convergence (3) valid uniformly over t ∈ T ?
When H = R, and assuming mean-square continuity of χ, these two questions have been long known to admit a positive answer in the form of the celebrated Karhunen-Loève theorem (Karhunen (1946), Loeve (1948); see also Kac and Siegert (1947)), whose proof fundamentally relies on Mercer's theorem on the decomposition of real-valued kernels (Mercer, 1909).Extensions to random fields valued in d-dimensional Euclidean space H = R d have also been tackled (Withers, 1974), but the general, infinite-dimensional case has not been addressed, and does not straightforwardly follow from the case H = R d .

Mercer's Theorem for Operator-Valued Kernels
Consider a function K ∶ T × T → B 1 (H).We refer to such a function K as an operatorvalued kernel.We say that K is continuous if: for every s, t ∈ T .We say that K is symmetric if K(s, t) = K(t, s) * .Finally, we say that K is non-negative definite if for every n ≥ 1 and sequences {v j } j=1,...,n ⊂ T , {h j } j=1,...,n ⊂ H: Given an operator-valued kernel, we may define an integral operator where the integral in ( 5) is to be understood as a Bochner integral.
Lemma 1.Let K be a continuous kernel.
(iii) K is non-negative definite if and only if A K is non-negative definite.
Proof.(i) Let {e i } i≥1 be a CONS for H. Then P n ∶= ∑ n i=1 e i ⊗ e i converges strongly to the identity.For n ≥ 1, let K n (⋅, ⋅) ∶= P n K(⋅, ⋅)P n .Note that A Kn is compact, being of finite rank.To prove that A K is compact it thus suffices to show that A Kn → A K strongly, as n → ∞.Now: By continuity, to prove compactness of A K it thus suffices to show that: Notice that P n K(s, t)P n → K(s, t) strongly.Because K(s, t) is Hilbert-Schmidt, the convergence also holds in operator norm.Indeed for g, h ∈ H we can write ⟨g, (K(s, t) and the conclusion follows.
(ii) If K is symmetric, for any f, g ∈ L 2 (T , H): (iii) We follow Hsing and Eubank (Theorem 4.6.4 2015).Given n > 0 let δ n be chosen so that K (s 2 , t 2 ) − K (s 1 , t 1 ) < n −1 whenever d ((s 1 , t 1 ) , (s 2 , t 2 )) < δ n .As T is a compact metric space, there exists a finite partition {E ni } of T such that each E ni has diameter less than δ n .Let v i be an arbitrary point of E ni and, for all (s, t) ∈ E ni × E nj , define K n (s, t) to be K (v i , v j ).The (uniform) continuity of K now has the consequence that max (s,t)∈T ×T K(s, t) − K n (s, t) < n −1 .Now let A Kn be the integral operator with kernel K n .With this choice, we find that, for any f ∈ L 2 (T , H), which, by (4), proves the positiveness of A K .Conversely, suppose that for some {v j } j=1,...,n ⊂ T , {f j } j=1,...,n ⊂ H.As K is uniformly continuous, there exist measurable disjoint sets E 1 , . . ., E m ⊂ T with E i > 0, v i ∈ E i for all i such that: This implies that: due to the mean-value theorem.Upon observing that the last expression is simply In particular, if K is a continuous, symmetric and non-negative definite kernel, A K admits a spectral decomposition in terms of its eigenvalue-eigenfunction pairs.That is, the eigenfunctions for A K , {Φ j } j≥1 , form a CONS for L 2 (T , H), and if {λ j } j≥1 denote the corresponding eigenvalues, one may write: Furthermore, the following lemma establishes that the eigenfunctions of A K are uniformly continuous.
Lemma 2. Let K be a continous kernel.For each f ∈ L 2 (T , H), (A K f )(⋅) is uniformly continuous.
Proof.By compactness of T and uniform continuity of K, for any given ǫ > 0 there exists δ > 0 such that K (u, s) − K (u, t) B 1 (H) < ǫ for all u, s, t ∈ T with s − t < δ.Then: The following lemma will be instrumental in the proof of our generalisation to Mercer's theorem.
Lemma 3. (a) For any t ∈ T , is well-defined and uniformly continuous wrt to ⋅ B 1 (H) .Furthermore, the sum converges uniformly.
Proof.(a) Let and take A Kn to be the integral operator with kernel K n .Note that K n is continuous, by continuity of K and by Lemma 2. For any f ∈ L 2 (T , H): and A Kn must be non-negative definite.This implies, by Lemma 1 (iii), that K n is non-negative definite and hence K n (t, t) ≥ 0, thereby proving ( 7).
(b) Let {e i } i≥1 be a CONS for H. First, note that: Then, by Cauchy-Schwarz, we obtain that: H) .
(c) Fix ε > 0. By (b) we may conclude that there exists N ε such that: Furthermore, for any N , uniform continuity of the Φ j (⋅) entails the existence of δ > 0 such that: By the last two displays, it is then clear that there exists δ such that: Theorem 1 (Mercer's Theorem for Operator-Valued Kernels).Let K be a continuous, symmetric and non-negative definite kernel, and denote by A K the corresponding integral operator.Let {λ j , Φ j } j≥1 be the eigenvalue and eigenfunction pairs of A K , where λ j ∈ R and Φ j ∈ L 2 (T , H). Then: for all s, t, with the series converging absolutely and uniformly.
Proof.For different continuous kernels K 1 (s, t) and K 2 (s, t), it is straightforward to construct a function f such that ∫ T K 2 (s, t)f (s)ds and ∫ T K 1 (s, t)f (s)ds differ.Thus, K is the unique operator kernel that defines A K .the integral operator with the continuous kernel ∑ ∞ j=1 λ j Φ j (s) ⊗ Φ j (t) has the same eigen-decomposition as A K and is therefore the same operator.Thus, K(s, t) = ∑ ∞ j=1 λ j Φ j (s)⊗Φ j (t) for all s, t ∈ T with the right-hand side converging absolutely and uniformly as a consequence of Lemma 3.
Finally, we can now see that the integral operator A K in ( 5) is trace class.
Proposition 1.Let the continuous kernel K be symmetric and non-negative definite and A K the corresponding integral operator.Then: Proof.We see that, for any CONS {e i } i≥1 of H: where we have employed Parseval's equality, and have exchanged of the order of summation and integration by Fubini's Theorem.

Karhunen-Loève Theorem for Hilbertian Flows
Let {χ(t) ∶ t ∈ T } be a stochastic process on a probability space (Ω, F , P), taking values in H.We may define its mean m ∶ T → H by: and its covariance kernel K ∶ T × T → B(H) by: provided the above expectations are well defined as Bochner integrals (see Hsing and Eubank (2015, Definition 7.2.1).
We say that a process {χ(t) ∶ t ∈ T } is a second-order process if (8) and ( 9) are well defined for every t ∈ T .Note that the covariance kernel is a symmetric operator-valued kernel, i.e.: K(s, t) = K(t, s) * We say that χ(⋅) is mean-square continuous if: = K(t, t) + K(s, s) − K(t, s) − K(t, s) B 1 (H) → 0, as s → t.
Note that the kernel K is non-negative definite, in the sense of (4).Indeed, for n ≥ 1 and any sequences {v j } j=1,...,n ⊂ T , {f j } j=1,...,n ⊂ H: We may finally state: Theorem 2 (Karhunen-Loève Expansion for Hilbertian Flows/Fields).Let {χ(t) ∶ t ∈ T } be a mean-squared continuous second-order process.Define: 2H → 0 for any t ∈ T and any sequence {t n } n≥1 converging to t.Lemma 4. Let {χ(t) ∶ t ∈ T } be a second-order process.Then χ is mean-square continuous if and only if its mean and covariance functions are continuous wrt ⋅ H and ⋅ B 1 (H) , respectively.Proof.Assume without loss of generality that the process is centered, i.e. that m = 0.