Gaussian and sparse processes are limits of generalized Poisson processes
Introduction
In their landmark paper on linear prediction [1], H.W. Bode and C.E. Shannon proposed that “a noise can be thought of as made up of a large number of closely spaced and very short impulses.” In this work, we formulate this intuitive interpretation in a mathematically rigorous way. This allows us to extend this intuition beyond noise and to draw additional properties for the class of stochastic processes that can be linearly transformed into a white noise. More precisely, we show that the law of these processes can be approximated as closely as desired by generalized Poisson processes, which can also be viewed as random L-splines.
Let us define the first ingredient of our work. Splines are continuous-domain functions characterized by a sequence of knots and sample values. They provide a powerful framework to build discrete descriptions of continuous objects in sampling theory [2]. Initially defined as piecewise-polynomial functions [3], they were further generalized, starting from their connection with differential operators [4], [5], [6]. Let L be a suitable linear differential operator such as the derivative. Then, the function is a non-uniform L-spline if is a sum of weighted and shifted Dirac impulses. The are the weights and the the knots of the spline. Deterministic splines associated to various differential operators are depicted in Fig. 1a and 1b. Note that the knots and weights can also be random, yielding stochastic splines.
The second main ingredient is a generative model of stochastic processes. Specifically, we consider linear differential equations of the form where L is a differential operator called the whitening operator and w is a d-dimensional Lévy noise or innovation. Examples of such stochastic processes are illustrated in Fig. 1c and 1d.
Our goal in this paper is to build a bridge between linear stochastic differential equations (SDE) and splines. By comparing (1) and (2), one easily realizes that the differential operator L connects the random and deterministic frameworks. The link is even stronger when one notices that compound-Poisson white noises can be written as [7]. This means that the random processes that are solution of are (random) L-splines.
Our main result thus uncovers the link between splines and random processes through the use of Poisson processes. A Poisson noise is made of a sparse sequence of weighted impulses whose jumps follow a common law. The average density of impulses λ is the primary parameter of such a Poisson white noise. Upon increasing λ, one increases the average number of impulses by unit of time. Meanwhile, the intensity of the impulses is governed by the common law of the jumps of the noise. Upon decreasing this intensity, one makes the weights of the impulses smaller. By combining these two effects, the intuitive conceptualization of a white noise proposed by Bode and Shannon in [1] can be recovered.
Theorem 1 Every random process s solution of (2) is the limit in law of a sequence of generalized Poisson processes driven by compound-Poisson white noises and whitened by L.
We shall see that the convergence procedure is based on a coupled increase of the average density and a decrease of the intensity of the impulses of the Poisson noises. This is in the spirit of Bode and Shannon's quote and is, in fact, true for any Lévy noise.
Our motivation to study stochastic differential equations such as (2) comes from signal processing. Random processes and random fields are notorious tools to model uncertainty and statistics of signals. Gaussian processes are by far the most studied stochastic models because of their fundamental properties (e.g., stability, finite variance, central-limit theorem) and their relative ease of use. They are the principal actors within the classical paradigm of statistical signal processing [8]. Many fractal-type signals are modeled as self-similar Gaussian processes [9], [10], [11], [12].
However, lots of real-world signals are empirically observed to be inherently sparse, a property that is incompatible with Gaussianity [13], [14], [15]. In order to overcome the limitations of Gaussian model, several other stochastic models have been proposed for the study of sparse signals. They include infinite-variance [12], [16], [17] or piecewise-constant models based on compound Poisson laws [7]. Interestingly, the compound Poisson processes, which are playing a crucial role in Theorem 1, are shown to be sparsest among the family of Lévy processes in the sense of approximation theory [18] and of information theory [19].
In this paper, we model signals as continuous-domain random processes defined over that are solutions of a differential equation driven by Lévy noise. They are called generalized Lévy processes. We thus follow the general approach of [20] which includes the models mentioned above. The common feature of these processes is that their probability distributions are always infinitely divisible, meaning that they can be decomposed as sums of any length of independent and identically distributed random variables. Infinite divisibility is a key concept of continuous-domain random processes [21] and will be at the heart of our work. In order to embrace the largest variety of random models, we rely on the framework of generalized random processes, which is the probabilistic version of the theory of generalized functions of L. Schwartz [22]. Initially introduced independently by K. Itô [23] and I. Gelfand [24], the framework has been developed extensively by these two authors in [25] and [26]. A thrilling aspect of generalized Lévy processes is their ability to model sparse signals, as soon as the underlying white noise is non-Gaussian. This has been demonstrated both experimentally [27] and theoretically [18], [28]. For this reason, the non-Gaussian members of the Lévy family are referred to as sparse stochastic processes [20], [29] to reflect their very compressibility nature.
Several behaviors can be observed within the family of generalized Lévy processes. For instance, self-similar Gaussian processes exhibit fractal behaviors. In one dimension, they include the fractional Brownian motion [11] and its higher-order extensions [30]. In higher dimensions, our framework covers the family of fractional Gaussian fields [31], [32], [33], [34] and finite-variance self-similar fields that appear to converge to fractional Gaussian fields at large scales [35], [36]. The self-similarity property is also compatible with the family of α-stable processes [37] or fields [38], which have unbounded variances (when being non-Gaussian). More generally, every process considered in our framework is part of the Lévy family, including Laplace processes [39] and Student's processes [40]. Upon varying the operator L, one recovers Lévy processes [41], CARMA processes [42], [43], and their multivariate generalizations [44], [45], [20]. Unlike those examples, the compound-Poisson processes, although members of the Lévy family, are piecewise-constant and have a finite rate of innovation (FRI) in the sense of [46]. For a signal, being FRI means that a finite quantity of information is sufficient to reconstruct it over a bounded domain.
The present paper is an extension of our previous contribution [47].1 We believe that Theorem 1 is relevant for the conceptualization of random processes that are solution of linear SDE. Starting from the L-spline interpretation of generalized Poisson processes, the statistics of a more general process can be understood as a limit of the statistics of random L-splines. In general, the studied processes that are solution of (2) do not have a finite rate of innovation, except if the underlying white noise is Poisson. The convergence result helps us understand why non-Poisson processes do not have a finite rate of innovation. They in fact correspond to infinitely many impulses per unit of time as they can be approximated by FRI processes with an increasing and asymptotically infinite rate of innovation.
Interesting connections can also be drawn with some classical finite-dimension convergence results in probability theory. As mentioned earlier, there is a direct correspondence between Lévy white noises and infinitely divisible random variables. It is well known that any infinitely divisible random variable is the limit in law of a sequence of compound-Poisson random variables [21, Corollary 8.8]. Theorem 1 is the generalization of this result from real random variables to random processes that are solution of a linear SDE.
The paper is organized as follows: In Sections 2 and 3, we introduce the concepts of L-splines and generalized Lévy processes, respectively. A special emphasis is put on generalized Poisson processes in Section 4 as they embrace both generalized Lévy processes and (random) L-splines. Our main contribution is Theorem 1, which is proven in Section 5. Section 6 contains illustrative examples in the one- and two-dimensional settings, followed by concluding remarks in Section 7.
Section snippets
Nonuniform L-splines
We denote by the space of rapidly decaying functions from to , that is, the space of functions that decay faster than any polynomial together with their derivatives. Its topological dual is , the Schwartz space of tempered generalized function [22]. We denote by the duality product between and . A linear and continuous operator L from to is spline-admissible if
- •
it is shift-invariant, meaning that for every and ;
Generalized Lévy processes
In this section, we briefly introduce the main tools and concepts for the characterization of Gaussian and sparse stochastic processes. For a more comprehensive description, we refer the reader to [20]. First, let us recall that a real random variable X is a measurable function from a probability space to , endowed with the Borelian σ-field. The law of X is the probability measure on such that . The characteristic function of X is the (conjugate) Fourier
Generalized Poisson processes: a bridge between L-splines and generalized Lévy processes
Generalized Poisson processes are generalized Lévy processes driven by impulsive noise. They can be interpreted as random L-splines, which makes them conceptually more accessible than other generalized Lévy processes.
Definition 8 Let and let P be a probability law on such that there exists for which . The impulsive noise w with rate and amplitude probability law P is the process with characteristic functional
According to [7, Theorem 1]
Generalized Lévy processes as limits of generalized Poisson processes
This section is dedicated to the proof of Theorem 1. We start with some notations. The characteristic function of a compound-Poisson law with rate λ and jump law P is given by with the characteristic function of P. If f is a Lévy exponent, then one denotes by the compound-Poisson probability law with rate and by law of jumps the infinitely divisible law with characteristic function . The characteristic function of is therefore and the Lévy exponent of
Simulations
Here, we illustrate the convergence result of Theorem 1 on generalized Lévy processes of three types, namely
- •
Gaussian processes based on Gaussian white noise, which are non-sparse;
- •
Laplace processes based on Laplace noise, which are sparse and have finite variance;
- •
Cauchy processes based on Cauchy white noise, our prototypical example of infinite-variance sparse model.
Conclusion
Our main result in this work is the proof that any generalized Lévy process is the limit in law of generalized Poisson processes obeying the same equation, but where w corresponds to an appropriate impulsive Poisson noises. In addition, we showed that generalized Poisson processes are random L-splines. In the asymptotic regime, generalized Lévy processes can thus conveniently be described using splines.
Theorem 1 is interesting in practice as it provides a new way of generating
Acknowledgements
This work was supported in part by the European Research Council under Grant H2020-ERC (ERC grant agreement No. 692726 – GlobalBioIm), and in part by the Swiss National Science Foundation under Grant 200020_162343/1.
References (61)
- et al.
Polyharmonic cardinal splines
J. Approx. Theory
(1990) - et al.
Operator scaling stable random fields
Stochastic Process. Appl.
(2007) - et al.
CARMA generalized random processes
J. Statist. Plann. Inference
(2010) - et al.
A simplified derivation of linear least square smoothing and prediction theory
Proc. IRE
(1950) Splines: a perfect fit for signal and image processing
IEEE Signal Process. Mag.
(1999)Cardinal Spline Interpolation
(1973)- et al.
L-splines
Numer. Math.
(1967) - et al.
Fractional splines and wavelets
SIAM Rev.
(2000) - et al.
Stochastic models for sparse and piecewise-smooth signals
IEEE Trans. Signal Process.
(2011) - et al.
Foundations of Signal Processing
(2014)
Self-similarity: part II—optimal estimation of fractal processes
IEEE Trans. Signal Process.
The Fractal Geometry of Nature
Fractional Brownian motions, fractional noises and applications
SIAM Rev.
Stochastic fractal models for image processing
IEEE Signal Process. Mag.
Pattern Theory: The Stochastic Analysis of Real-World Signals
Stochastic models for generic images
Quart. Appl. Math.
On advances in statistical modeling of natural images
J. Math. Imaging Vision
Signal Processing with Alpha-Stable Distributions and Applications
The n-term approximation of periodic generalized Lévy processes
How compressible are sparse innovation processes?
An Introduction to Sparse Stochastic Processes
Lévy Processes and Infinitely Divisible Distributions, vol. 68
Théorie des distributions
Stationary random distributions
Kyoto J. Math.
Generalized random processes
Dokl. Akad. Nauk SSSR
Generalized Functions, vol. 4: Applications of Harmonic Analysis
Foundations of Stochastic Differential Equations in Infinite Dimensional Spaces, vol. 47
Sparse stochastic processes and discretization of linear inverse problems
IEEE Trans. Image Process.
Wavelet analysis of the Besov regularity of Lévy white noises
Cited by (6)
The domain of definition of the Lévy white noise
2021, Stochastic Processes and their ApplicationsStochastic Hyperkernel Convolution Trains and h-Counting Processes
2023, IEEE AccessThe Wavelet Compressibility of Compound Poisson Processes
2022, IEEE Transactions on Information TheoryGenerating Sparse Stochastic Processes Using Matched Splines
2020, IEEE Transactions on Signal Processing