Machine Learning Application of Generalized Gaussian Radial Basis Function and Its Reproducing Kernel Theory

Abstract

Gaussian Radial Basis Function Kernels are the most-often-employed kernel function in artificial intelligence for providing the optimal results in contrast to their respective counterparts. However, our understanding surrounding the utilization of the Generalized Gaussian Radial Basis Function across different machine learning algorithms, such as kernel regression, support vector machines, and pattern recognition via neural networks is incomplete. The results delivered by the Generalized Gaussian Radial Basis Function Kernel in the previously mentioned applications remarkably outperforms those of the Gaussian Radial Basis Function Kernel, the Sigmoid function, and the ReLU function in terms of accuracy and misclassification. This article provides a concrete illustration of the utilization of the Generalized Gaussian Radial Basis Function Kernel as mentioned earlier. We also provide an explicit description of the reproducing kernel Hilbert space by embedding the Generalized Gaussian Radial Basis Function as an $L^2-$measure, which is utilized in implementing the analysis support vector machine. Finally, we provide the conclusion that we draw from the empirical experiments considered in the manuscript along with the possible future directions in terms of spectral decomposition of the Generalized Gaussian Radial Basis Function.

1. Introduction

Contemporary data science, which revolves around diverse architectural frameworks of machine learning, signifies crucial potential across a wide range of applications. These applications span from various fields such as physics and engineering [1] to areas like genomic-assisted prediction [2] and predictive modeling for cancer tumors [3]. The central task of (automated) machine learning is to devise an optimal activation function or optimal kernel function, depending upon the learning architecture, that is best-suited for the specific scientific model. Therefore, it is not hard to see that machine learning applications and many others across various scientific realms duly take advantage of various mathematical functions that arise in basic function theory and incorporate them along with their Hilbert space theories. This symbiotic relationship between advanced machine learning architecture avenues of artificial intelligence and the raw availability of various classes of mathematical functions is aptly captured by pictorial representations produced by the National Academy of Sciences in [4]. This does not only simply depict the importance of such functions but also questions and inquires about the possibility of the existence and applicability of other new, unexplored, better math functions, which can advance the horizons of artificial intelligence, for instance in [5].

Across the rich and diverse history of machine learning, numerous activation functions, along with their incorporation in a kernel sense, have been proposed. Certain important examples of activation functions are the Sigmoid function or S-Shaped curves in Figure 1, like the hyperbolic tangent function (given as ${\displaystyle \frac{1}{1+e^{-x}}}$ cf. [6]). The class of Sigmoid functions is employed to classify data objects where the output is supposed to be constrained on a limited range (usually on $[0,1]$).

The other class of important and popular activation functions comprises ReLU activation functions in Figure 2, defined by $f\left(x\right)=x$ if $x>0$, which are employed in machine learning problems of reinforcement learning, quantification, and classification. If one adds a piece of function defined as $f\left(x\right)=\alpha x$ for $x\le 0$, where $\alpha \ne 0$ to the actual definition of ReLU, we obtain the $\alpha$ReLU activation function (see (29) in this article), which is often also referred to as LeakyReLU. It is essential to acknowledge that both ReLU and the $\alpha$ReLU activation function have discontinuities at the origin; these lead to the failure of the gradient descent optimizer by producing undefined gradients [7].

Furthermore, one of the most important and interesting classes of mathematical functions that leverage crucial platforms of machine learning or deep learning, to name a few, are radial basis functions, defined as follows (cf. [8]).

Definition 1.

A function $\mathrm{\Phi }:{\mathbf{R}}^d\to \mathbf{R}$ is referred to as radial if there exists a univariate function $\varphi :\left\{0\right\}\cup {\mathbf{R}}_{+}\to \mathbf{R}$, such that

$$\begin{array}{ccc}\hfill \mathrm{\Phi }\left(\mathit{x}\right)=\varphi \left(r\right),& \hfill & \hfill wherer=\parallel \mathit{x}\parallel .\end{array}$$

Here, $\parallel •\parallel$ is some norm defined on ${\mathbf{R}}^d$, usually ${\parallel •\parallel }_2$, which is the Euclidean norm.

One such special type of mathematical radial basis function is the Gaussian Radial Basis Function (GRBF), which is given as:

$$\begin{array}{c}\hfill g_{{\sigma }^2}{\left(r\right):=}_{\mathrm{def}}exp\left(-{\sigma }^2{r}^2\right),\end{array}$$

where $\sigma >0$. The Gaussian Radial Basis Function is an activation function that possesses a bell-shaped curve (top in Figure 3); it has impact-full applications in modeling random variables, which are distributed according to the Gaussian probability distribution. For instance, a neural network architecture predicting the speed of a car might employ Gaussian kernel regression, since the speed of the car follows the Gaussian probability distribution.

Moreover, in recent years, learning algorithms of support vector machines (SVMs) have played a key role in data-mining methods; these are workhorse machine learning tools in industry and science. The crucial explanation for this aspect is the unique ability of SVMs to embed in higher-dimensional nonlinear spaces for nonlinear classifiers. The theoretical underpinnings of SVMs revolve around the kernel function, which essentially enables one to operate in a high-dimensional space without computing coordinates of data but by simply computing inner products between all pairs of data in the feature space (cf. page 197 in [9]). To that end, we define the most commonly used kernel function, that is, the Gaussian Radial Basis Function Kernel.

Let ${\parallel •\parallel }_2$ be the usual Euclidean norm on ${\mathbf{R}}^d$; then, the $g(\parallel •-•{\parallel }_2)$ function is comfortably synonymous to its kernel notion, which is famously referred to as the Gaussian Radial Basis Function Kernel, given as $K_{\sigma }{(\mathit{x},\mathit{z}):=}_{\mathrm{def}} g_{{\sigma }^2}{(\parallel \mathit{x}-\mathit{z}\parallel }_2)$ for $x,z\in {\mathbf{R}}^d$. Explicitly, that is

$$\begin{array}{c}\hfill K_{\sigma }(\mathit{x},\mathit{z})=exp\left(-{\sigma }^2{\parallel \mathit{x}-\mathit{z}\parallel }_2^2\right).\end{array}$$

The Gaussian Radial Basis Function Kernel (Figure 4) is a building block for various learning architectures, such as spatial statistics [10], dynamical system identification [11], Gaussian processes for machine learning [12], etc. Additionally, it is also employed for the classification of the existence of objects (cf. [13]). This particular study extends the idea of the Gaussian Radial Basis Function Kernel to what is referred to as the Generalized Gaussian Radial Basis Function Kernel (GGRBF Kernel in Figure 5), as introduced in [14], on the avenues of machine learning and deep learning.

Definition 2.

Let $\sigma >0$ and ${\sigma }_0\ge 0$. Then, the Generalized Gaussian Radial Basis Function Kernel for $\mathit{x}$ and $\mathit{z}\in {\mathbf{R}}^d$ is defined as:

$$\begin{array}{cc}\hfill K_{\sigma ,{\sigma }_0}\left(\mathit{x}-\mathit{z}\right):{=}_{def}& \left[g_{{\sigma }^2}\left(r\right)exp\left(g_{{\sigma }_0^2}\left(r\right)-1\right){\circ (\parallel \mathit{x}-\mathit{z}\parallel }_2)\right]\hfill \\ \hfill =& g_{{\sigma }^2}({\parallel \mathit{x}-\mathit{z}\parallel }_2)e^{\left(g_{{\sigma }_0^2}{(\parallel \mathit{x}-\mathit{z}\parallel }_2)-1\right)}\hfill \\ \hfill =& e^{-{\sigma }^2{\parallel \mathit{x}-\mathit{z}\parallel }_2^2}{e}^{e^{-{\sigma }_0^2\left({\parallel \mathit{x}-\mathit{z}\parallel }_2^2\right)}-1}.\hfill \end{array}$$

(1)

Now that we have introduced the Generalized Gaussian Radial Basis Function Kernel, let us briefly discuss the motivation of the present article with respect to it.

1.1. Motivation

The applicability of the Generalized Gaussian Radial Basis Function was introduced in [15] to provide better results in contrast to the Gaussian Radial Basis Function results; here, we specifically refer to the results of convergence and stability for interpolation problems on Franke’s test function and Runge’s function or solving the system with Tikhonov regularization and Riley’s algorithm. Drawing significant inspiration from [15], the application of the unexplored Generalized Gaussian Radial Basis Function Kernel was documented in [14], specifically in the contexts of SVMs, kernel regression, and pattern recognition, through the activation function in a neural network. These state-of-the-art methods in learning architectures are leveraged by a peculiar topic from the Hilbert function space called the reproducing kernel Hilbert space (RKHS) [16]. The analysis from the reproducing kernel theory perspective for the Gaussian Radial Basis Function Kernel has already been established by [17]; in the paper, an investigation related to the norms, orthonormal basis, and feature space was presented. However, with the present empirical evidence supporting better results obtained by employing the Generalized Gaussian Radial Basis Function Kernel, it becomes important to perform the same investigation for the Generalized Gaussian Radial Basis Function Kernel.

Note that, if ${\sigma }_0=0$, then we obtain the traditional Gaussian Radial Basis Function Kernel. This is demonstrated in Figure 6.

1.2. Contributions

This article is motivated by documenting the application of the Generalized Gaussian Radial Basis Function Kernel on various machine learning and deep learning architectures. In the process of executing the Generalized Gaussian Radial Basis Function Kernel over learning architectures, we discover that the results produced by the Generalized Gaussian Radial Basis Function Kernel excel in terms of comparative accuracy or misclassification; these are achieved in contrast to the results obtained from the existing counterparts. These intriguing and promising findings naturally demand an exploration of the Generalized Gaussian Radial Basis Function from the perspective of mathematical function theory, which starts with the identification of the Hilbert space, generated by it as an $L^2-$measure. Furthermore, with the established Hilbert function space theory, we are also able to address the corresponding reproducing kernel theory by virtue of Aronszajn’s theorem (see [16]).

Having aptly addressed the Hilbert function theory generated by the Generalized Gaussian Radial Basis Function, this study immediately provides its application in learning architectures. The detailed investigation on the reproducing kernel for the Hilbert space of the Generalized Gaussian Radial Basis Function is proved to be universal (see in Section 4) on the restrictions over ${\mathbf{R}}^d\times {\mathbf{R}}^d$; this makes it ready-to-use for learning architecture problems, such as standard binary classification through SVMs, function approximation through kernel regression, and pattern recognition through deep convolutional neural networks. In particular, we determine that the results related to the Hilbert function theory of the Generalized Gaussian Radial Basis Function Kernel can be leveraged for SVM applications by addressing the hinge loss function L and minimal $L-$risk losses as well.

Finally, this article also provides promising future directions in light of the spectral decomposition of the Generalized Gaussian Radial Basis Function.

1.3. Plan of the Article

The present article is organized as follows: we provide the essential preliminaries of reproducing kernel theory in Section 2. Then, we give the results from the Hilbert space function theory, such as the orthonormal basis and the reproducing kernel of the Generalized Gaussian Radial Basis Function Kernel in Section 3. Then, we provide the functionality of the reproducing kernel in a real sense in Section 4; this is followed by compelling empirical comparison results in Section 5. Lastly, we provide our conclusions and results in Section 6 and describe our future research directions in Section 7.

2. Notation and Preliminaries

2.1. Hypergeometric Function Notation

We begin first by recalling the basic calculus results related to the Generalized Hypergeometric Function ${ }_p{F}_q\left[\begin{array}{cccc}{a}_1\hfill & a_2\hfill & \cdots \hfill & a_p\hfill \\ b_1\hfill & b_2\hfill & \cdots \hfill & b_q\hfill \end{array};z\right]$ [18].

Definition 3.

We define the Pochhammer symbol [19] (rising factorial notation) as

$${\left(a\right)}_k=\frac{\mathrm{\Gamma }(a+k)}{\mathrm{\Gamma }\left(a\right)}=a(a+1)\cdots (a+k-1).$$

Then, we define Generalized Hypergeometric Function ${ }_p{F}_q\left[\begin{array}{cccc}{a}_1\hfill & a_2\hfill & \cdots \hfill & a_p\hfill \\ b_1\hfill & b_2\hfill & \cdots \hfill & b_q\hfill \end{array};z\right]$, which is given as

$${ }_p{F}_q\left[\begin{array}{cccc}{a}_1\hfill & a_2\hfill & \cdots \hfill & a_p\hfill \\ b_1\hfill & b_2\hfill & \cdots \hfill & b_q\hfill \end{array};z\right]:{=}_{def}\sum _{l=0}^{\infty }\frac{{\prod }_{i=1}^p{\left(a_i\right)}_l}{{\prod }_{i=1}^p{\left(b_i\right)}_l}\frac{z^l}{l!}.$$

(2)

Example 1.

The summation ${\sum }_{l=0}^{\infty }\frac{1}{{\left(l+x\right)}^{n+1}}\frac{1}{l!}$ can be represented as:

$$\begin{array}{cc}\hfill \sum _{l=0}^{\infty }\frac{1}{{\left(l+x\right)}^{n+1}}\frac{1}{l!}=& \sum _{l=0}^{\infty }{\left(\frac{\mathrm{\Gamma }(l+x)}{\mathrm{\Gamma }\left(l+x+1\right)}\right)}^{n+1}\frac{1}{l!}\hfill \\ \hfill =& \sum _{l=0}^{\infty }{\left(\frac{{\left(x\right)}_l\mathrm{\Gamma }\left(x\right)}{{\left(x+1\right)}_l\mathrm{\Gamma }(x+1)}\right)}^{n+1}\frac{1}{l!}\hfill \\ \hfill =& \sum _{l=0}^{\infty }{\left(\frac{{\left(x\right)}_l\mathrm{\Gamma }\left(x\right)}{{\left(x+1\right)}_{l}x\mathrm{\Gamma }\left(x\right)}\right)}^{n+1}\frac{1}{l!}\hfill \\ \hfill =& \frac{1}{x^{n+1}}\sum _{l=0}^{\infty }\frac{{\left(x\right)}_l^{n+1}}{{\left(x+1\right)}_l^{n+1}}\frac{1}{l!}\hfill \\ \hfill =& \frac{1}{x^{n+1}}n+1F_{n+1}\left[\begin{array}{c}x\\ x+1\end{array};1\right]\hfill & \hfill & \left(use\right(2\left)\right).\hfill \end{array}$$

The example presented above will be useful in addressing further details such as the orthonormal basis in the present manuscript. To avoid heavy notation clutter, we write

$${ }_{n+1}{F}_{n+1}\left[\begin{array}{c}x\\ x+1\end{array};1\right]:{=}_{notation}{\mathcal{F}}_{n,x,1}$$

(3)

from now onward where necessary. Note that ${\mathcal{F}}_{n,\infty ,1}=e$.

2.2. Field and Space Notations

The set of natural numbers in union with 0 is denoted by $\mathbf{W}$, that is $\mathbf{W}:=0,1,2,\dots$. We use Kronecker delta ${\delta }_{nm}$ on non-negative integers n and m to show that ${\delta }_{nm}=1$; this is the case whenever $n=m$ and ${\delta }_{nm}=0$, if $n\ne m$. We denote a complex number, $z=x+iy$, where x and $y\in \mathbf{R}$. With that z, its conjugate part is given as $\overline{z}=x-iy$, along with its absolute value as ${\left|z\right|}^2=z·\overline{z}=x^2+y^2$. We reserve symbol $\mathbf{K}$ to treat with a choice of fields on which we will operate; in particular, $\mathbf{K}$ can either be $\mathbf{R}$ or $\mathbf{C}$.

If X denotes the space of input values (sometimes a closed subset of ${\mathbf{R}}^d$), then we write $C\left(X\right)$ as the space of continuous functions $f:X\to \mathbf{R}$; on the other hand, $C^m\left(X\right)$ represents the space of the $m-$times differentiable function, including $m=\infty .$

2.3. Tensor Product Notation

We recall the tensor product between two functions; say that $f_1,f_2:X\to \mathbf{K}$ is given as $f_1\otimes f_2:X\times X\to \mathbf{K}$. Then, for all $x,x^{\prime }\in X$, the tensor product $f_1\otimes f_2$ is defined as $f_1\otimes f_2(x,x^{\prime }):=f_1\left(x\right)f_2\left(x^{\prime }\right)$.

2.4. Preliminaries

Definition 4.

Let $X=\varnothing$; then, a function $k:X\times X\to \mathbf{K}$ is referred to as the kernel on X if there exists a $\mathbf{K}-$Hilbert space $\left(H,{\langle ·,·\rangle }_H\right)$ accompanied by a map $\mathrm{\Phi }:X\to H$ such that $\forall x,x^{\prime }\in X$, we have

$$\begin{array}{c}\hfill k(x,x^{\prime })={\langle \mathrm{\Phi }\left(x^{\prime }\right),\mathrm{\Phi }\left(x\right)\rangle }_H.\end{array}$$

(4)

We regard Φ as the feature map and H as the feature space of k.

Now that we have introduced the basic notion from the kernel theory in the definition provided above, we can now comfortably define the building block of this article: reproducing kernel Hilbert space—RKHS.

Definition 5.

Let $X=\varnothing$ and $\left(H,{\langle ·,·\rangle }_H\right)$ be the Hilbert function space over X.

1. 

The space H is referred to as the reproducing kernel Hilbert space (RKHS) if $\forall x\in X$, the evaluation functional ${\epsilon }_x:H\to \mathbf{K}$ defined as ${\epsilon }_x\left(f\right):=f\left(x\right),f\in H$ is continuous.

Definition 6.

A function $k:X\times X\to \mathbf{K}$ is called reproducing kernel of H if we have:

1. 

$k(·,x)\in H\forall x\in X$, that is ${\parallel k(·,x)\parallel }_H<\infty$, and

2. 

$k(·,·)$ has the reproducing property; that is

$$\begin{array}{c}\hfill f\left(x\right)={\langle f,k(·,x)\rangle }_H\forall f\in Handx\in X.\end{array}$$

It is worthwhile mentioning that the norm convergence yields the point-wise convergence inside RKHS. This fact can be readily learned due to the continuity of the evaluation function. This is demonstrated as follows for an arbitrary $f\in H$ and ${\left\{f_n\right\}}_n\in H$ with $\parallel f-f_n{\parallel }_H\to 0$ as $n\to \infty$; then,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{f}_n\left(x\right)=\underset{n\to \infty }{lim}{\epsilon }_x\left(f_n\right){=}_{(\mathrm{continuity}\mathrm{of}{\epsilon }_x)}{\epsilon }_x\left(f\right)=f\left(x\right).\end{array}$$

Now, we will state an important theorem from [16], which dictates the relationship between the reproducing kernel of the RKHS H and the orthonormal basis of it.

Theorem 1

(Aronszajn’s Theorem [16]). Let H be an RKHS over an nonempty set X; then, $k:X\times X\to \mathbf{K}$, defined as $k(x,x^{\prime }):={\langle {\epsilon }_x,{\epsilon }_{x^{\prime }}\rangle }_H$ for $x,x^{\prime }\in X$ is the only reproducing kernel of H. Additionally, for some index set, $\mathcal{I}$, if we have ${\left\{{\mathbf{e}}_i\right\}}_{i\in \mathcal{I}}$ as an orthonormal basis, then for all $x,x^{\prime }\in X$, we have

$$\begin{array}{c}\hfill k(x,x^{\prime })=\sum _{i\in \mathcal{I}}{\mathbf{e}}_i\left(x\right)\overline{{\mathbf{e}}_i\left(x^{\prime }\right)},\end{array}$$

(5)

with an absolute convergence.

3. Function Space of Generalized Gaussian Radial Basis Measure

Let $d\in \mathbf{N}$, $\sigma >0$ and ${\sigma }_0\ge 0$ and f be holomorphic functions $f:{\mathbf{C}}^d\to \mathbf{C}$; we first write the measure of our interest:

$$\begin{array}{c}\hfill d{\mu }_{\sigma ,{\sigma }_0,d}\left(\mathit{z}\right):=e^{-{\sigma }^2{\left|z\right|}^2}{e}^{e^{-{\sigma }_0^2{\left|z\right|}^2}-1}d{V}_{{\mathbf{C}}^d}\left(\mathit{z}\right).\end{array}$$

(6)

Here, ‘$d{V}_{{\mathbf{C}}^d}\left(\mathit{z}\right)$’ is the usual Lebesgue measure on the entire ${\mathbf{C}}^d$. For $d=1$, we write simply $d{\mu }_{\sigma ,{\sigma }_0}\left(z\right)$ to denote the typical Lebesgue area measure on $\mathbf{C}$. We now provide the inner product associated with this measure as:

$$\begin{array}{c}\hfill {\langle f,g\rangle }_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}:={\mathcal{N}}_{\sigma ,{\sigma }_0,d}{\int }_{{\mathbf{C}}^d}f\left(\mathit{z}\right)\overline{g\left(\mathit{z}\right)}d{\mu }_{\sigma ,{\sigma }_0,d}\left(\mathit{z}\right).\end{array}$$

(7)

Here, ‘${\mathcal{N}}_{\sigma ,{\sigma }_0,d}$’ is the normalization constant whose value is explicitly given as ${\left({\displaystyle \frac{e{\sigma }^2}{2\pi }}\right)}^d$. Once we have define the inner product for the space, the norm for holomorphic function $f:{\mathbf{C}}^d\to \mathbf{C}$ is:

$$\begin{array}{cc}\hfill {\parallel f\parallel }_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}^2:=& {\left(\frac{e{\sigma }^2}{2\pi }\right)}^d{\int }_{{\mathbf{C}}^d}{\left|f\left(\mathit{z}\right)\right|}^{2}d{\mu }_{\sigma ,{\sigma }_0,d}\left(\mathit{z}\right).\hfill \end{array}$$

(8)

We write the following for the collection of holomorphic functions $f:{\mathbf{C}}^d\to \mathbf{C}$, whose ${\parallel •\parallel }_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ is finite, that is:

$$\begin{array}{c}\hfill H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}:=\left\{\mathrm{holomorphic}\mathrm{function}f:{\mathbf{C}}^d\to \mathbf{C}:{\parallel f\parallel }_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}<\infty \right\}.\end{array}$$

(9)

Once we have defined norm in (8) and the associated Hilbert space in (9), we can provide the following formulation, which makes the Hilbert space $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ as an RKHS.

Theorem 2.

For all $\sigma >0$, ${\sigma }_0\ge 0$ and all compact sets $K\subset {\mathbf{C}}^d$, there exists a positive constant $c_{\sigma ,{\sigma }_0,d}$ such that for all $\mathit{z}\in K$ and $f\in H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$, we have

$$\begin{array}{c}\hfill \left|f\left(\mathit{z}\right)\right|\le c_{\sigma ,{\sigma }_0,d}{\parallel f\parallel }_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}.\end{array}$$

(10)

Proof. 

Denote ${\mathbb{B}}_{(0,1)}$ as the complex unit ball in $\mathbf{C}$. Define

$$\begin{array}{c}\hfill {\mathrm{c}}_{\sigma ,{\sigma }_0,d}:=\underset{\mathit{z}\in K+{\mathbb{B}}_{(0,1)}^d}{sup}\left\{e^{-{\sigma }^2{\left|\mathit{z}\right|}^2}{e}^{e^{-{\sigma }_0^2{\left|\mathit{z}\right|}^2}-1}\right\}.\end{array}$$

In the spirit of ([17], Lemma-3, Page 4639), we have

$$\begin{array}{cc}\hfill \prod _{j=1}^d{r}_j{\left|f\left(\mathit{z}\right)\right|}^2\le & \frac{1}{{\left(2\pi \right)}^d}\prod _{j=1}^d{r}_j{\int }_{{[0,2\pi ]}^d}{\left|f\left(z_1+r_1{e}^{i{\theta }_1},\dots ,z_1+r_d{e}^{i{\theta }_d}\right)\right|}^{2}d\theta .\hfill \end{array}$$

Integration of the above with respect to $\left(r_1,\dots ,r_d\right)\in {[0,1]}^d$ yields:

$$\begin{array}{cc}\hfill {\left|f\left(\mathit{z}\right)\right|}^2\le & \frac{1}{{\left(2\pi \right)}^d}{\int }_{\mathit{z}+{\mathbb{B}}_{(0,1)}^d}{\left|f\left({\mathit{z}}^{\prime }\right)\right|}^{2}dV\left({\mathit{z}}^{\prime }\right)\hfill \\ \hfill \le & \frac{{\mathrm{c}}_{\sigma ,{\sigma }_0,d}}{{\left(2\pi \right)}^d}{\int }_{\mathit{z}+{\mathbb{B}}_{(0,1)}^d}{\left|f\left({\mathit{z}}^{\prime }\right)\right|}^2{e}^{-{\sigma }^2{\left|{\mathit{z}}^{\prime }\right|}^2}{e}^{e^{-{\sigma }_0^2{\left|{\mathit{z}}^{\prime }\right|}^2}-1}dV\left({\mathit{z}}^{\prime }\right)\hfill \\ \hfill \le & \frac{{\mathrm{c}}_{\sigma ,{\sigma }_0,d}}{{\left(e{\sigma }^2\right)}^d}{\parallel f\parallel }_{\sigma ,{\sigma }_0,d}^2.\hfill \end{array}$$

In particular, we have $c_{\sigma ,{\sigma }_0,d}=\sqrt{{\displaystyle \frac{{\mathrm{c}}_{\sigma ,{\sigma }_0,d}}{{\left(e{\sigma }^2\right)}^d}}}$. Hence, the result is established.    □

Establishing Theorem 2 immediately yields that $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ is indeed an RKHS and we state here as an important corollary.

Corollary 1.

The Hilbert space $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ identified by the inner product ${\langle ·,·\rangle }_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ is an RKHS for all $\sigma >0$ and $\sigma \ge 0$.

Orthonormal Basis

We will need following technical result to establish the orthonormal basis for the RKHS $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$.

Theorem 3.

For every $\sigma >0,{\sigma }_0\ge 0$ and $n,m\in \mathbf{W}$, we have

$$\begin{array}{c}\hfill {\int }_{\mathbf{C}}{z}^n\overline{z^m}d{\mu }_{\sigma ,{\sigma }_0}\left(z\right)=\left(\sqrt{\frac{2\pi n!}{e{\sigma }^{2n+2}}{\mathcal{F}}_{n,\widehat{\sigma },1}}\sqrt{\frac{2\pi m!}{e{\sigma }^{2m+2}}{\mathcal{F}}_{m,\widehat{\sigma },1}}\right){\delta }_{nm}\end{array}$$

(11)

where $\widehat{\sigma }={\displaystyle \frac{{\sigma }_0^2}{{\sigma }^2}}$ and ${\mathcal{F}}_{n,\widehat{\sigma },1}$ are defined in (3).

Proof. 

Employ the polar coordinate of $z=r{e}^{i\theta }$ to have:

$$\begin{array}{cc}\hfill {\int }_{\mathbf{C}}{z}^n\overline{z^m}d{\mu }_{\sigma ,{\sigma }_0}\left(z\right)=& {\int }_0^{\infty }{r}^{n+m}{e}^{-{\sigma }^2{r}^2}{e}^{e^{-{\sigma }_0^2{r}^2}-1}rdr{\int }_0^{2\pi }{e}^{i\left(n-m\right)\theta }d\theta .\hfill \end{array}$$

(12)

The quantity ${\int }_0^{\infty }{r}^{n+m}{e}^{-{\sigma }^2{r}^2}{e}^{e^{-{\sigma }_0^2{r}^2}-1}rdr{\int }_0^{2\pi }{e}^{i\left(n-m\right)\theta }d\theta$ is 0 when $n\ne m$. Now, assume that $n=m$ in (12), then:

$$\begin{array}{cc}\hfill {\int }_{\mathbf{C}}{z}^n\overline{z^m}d{\mu }_{\sigma ,{\sigma }_0}\left(z\right)=& 2\pi {\int }_0^{\infty }{r}^{2n}{e}^{-{\sigma }^2{r}^2}{e}^{e^{-{\sigma }_0^2{r}^2}-1}rdr\hfill \\ \hfill =& \frac{2\pi }{e}{\int }_0^{\infty }{t}^n{e}^{-{\sigma }^{2}t}{e}^{e^{-{\sigma }_0^{2}t}}dt\hfill & \hfill & (\mathrm{put}t=r^2)\hfill \\ \hfill =& \frac{2\pi }{e}{\int }_0^{\infty }{\left(\frac{s}{{\sigma }^2}\right)}^n{e}^{-s}{e}^{e^{-{\sigma }_0^2/{\sigma }^{2}s}}\frac{ds}{{\sigma }^2}\hfill & \hfill & (\mathrm{put}s={\sigma }^{2}t)\hfill \\ \hfill =& \frac{2\pi }{e{\left({\sigma }^2\right)}^{n+1}}{\int }_0^{\infty }{s}^n{e}^{-s}{e}^{e^{-{\sigma }_0^2/{\sigma }^{2}s}}ds\hfill \\ \hfill =& \frac{2\pi }{e{\left({\sigma }^2\right)}^{n+1}}{\int }_1^0{\left(\log \frac{1}{y}\right)}^n{e}^{y^{{\sigma }_0^2/{\sigma }^2}}(-dy)\hfill & \hfill & (\mathrm{put}{e}^{-s}=y)\hfill \\ \mathrm{(13)}& \hfill =& \frac{2\pi }{e{\left({\sigma }^2\right)}^{n+1}}{\int }_0^1{\left(\log \frac{1}{y}\right)}^n\left(\sum _{l=0}^{\infty }\frac{{\left(y^{{\sigma }_0^2/{\sigma }^2}\right)}^l}{l!}\right)dy\hfill \mathrm{(14)}& \hfill =& \frac{2\pi }{e{\left({\sigma }^2\right)}^{n+1}}\sum _{l=0}^{\infty }\frac{1}{l!}{\int }_0^1{\left(\log \frac{1}{y}\right)}^n{y}^{l\widehat{\sigma }}dy\hfill & \hfill & (\mathrm{use}\widehat{\sigma }={\displaystyle \frac{{\sigma }_0^2}{{\sigma }^2}})\hfill \mathrm{(15)}& \hfill =& \frac{2\pi }{e{\left({\sigma }^2\right)}^{n+1}}\sum _{l=0}^{\infty }\frac{1}{l!}\frac{\mathrm{\Gamma }(n+1)}{{(l\widehat{\sigma }+1)}^{n+1}}\hfill \hfill =& \frac{2\pi n!}{e{\sigma }^{2n+2}}{\mathcal{F}}_{n,\widehat{\sigma },1}\hfill & \hfill & (\mathrm{use}\left(3\right)).\hfill \end{array}$$

We used the entry 4.272(6)), Page 551 in [20] for the result in (15). The summation over l in (13) is interchanged with integral with respect to $dy$ for $0\le y\le 1$ by virtue of Fubini–Tonelli’s theorem as ${\int }_0^1{\left(\log \frac{1}{y}\right)}^n{e}^{y^{{\sigma }_0^2/{\sigma }^2}}dy$ is finite which is demonstrated as:

$$\begin{array}{c}\hfill \left|{\int }_0^1{\left(\log \frac{1}{y}\right)}^n{e}^{y^{{\sigma }_0^2/{\sigma }^2}}dy\right|\le {\int }_0^1\left|{\left(\log \frac{1}{y}\right)}^n{e}^{y^{{\sigma }_0^2/{\sigma }^2}}\right|dy\le e{\int }_0^1\left|{\left(\log \frac{1}{y}\right)}^n\right|dy=e\mathrm{\Gamma }(n+1)<\infty .\end{array}$$

Thus, the result prevails.    □

We will employ the notation $\widehat{\sigma }={\displaystyle \frac{{\sigma }_0^2}{{\sigma }^2}}$ for $\sigma >0$ and ${\sigma }_0\ge 0$ wherever it is required. In the light of Theorem 1, we have to determine the orthonormal basis of $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$.

Theorem 4.

Let $\sigma >0$, ${\sigma }_0\ge 0$ and

$$\begin{array}{c}\hfill {\mathbf{e}}_n\left(z\right):=\sqrt{\frac{{\sigma }^{2n}}{n!{\mathcal{F}}_{n,\widehat{\sigma },1}}}{z}^n\end{array}$$

(16)

for some $z\in \mathbf{C}$. Then, the tensor–product system ${\left({\mathbf{e}}_{n_1}\otimes \cdots \otimes {\mathbf{e}}_{n_d}\right)}_{n_1,\dots ,n_d\ge 0}$ forms the orthonormal basis of $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$.

Proof. 

We establish our result for $d=1$ for an initial basic understanding. For this, let us show that ${\left\{{\mathbf{e}}_n\right\}}_{n\in \mathbf{W}}$ forms an orthonormal system. So, consider $z\in \mathbf{C}$ and let $m,n\in \mathbf{W}$. Then,

$$\begin{array}{cc}\hfill {\langle {\mathbf{e}}_n,{\mathbf{e}}_m\rangle }_{\sigma ,{\sigma }_0}=& \frac{e{\sigma }^2}{2\pi }{\int }_{\mathbf{C}}{\mathbf{e}}_n\left(z\right)\overline{{\mathbf{e}}_m\left(z\right)}d{\mu }_{\sigma ,{\sigma }_0}\left(z\right)\hfill \\ \hfill =& \frac{e{\sigma }^2}{2\pi }\sqrt{\frac{{\sigma }^{2n}}{n!{\mathcal{F}}_{n,\widehat{\sigma },1}}}\sqrt{\frac{{\sigma }^{2m}}{m!{\mathcal{F}}_{m,\widehat{\sigma },1}}}{\int }_{\mathbf{C}}{z}^n\overline{z^m}d{\mu }_{\sigma ,{\sigma }_0}\left(z\right)\hfill \\ \hfill =& \left\{\begin{array}{cc}1\hfill & \mathrm{if}n=m\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\hfill & \hfill & (\mathrm{use}\mathrm{Theorem}3).\hfill \end{array}$$

We use the ‘polarplot’ function available in MATLAB R2023b to construct the polar plot of ${\mathbf{e}}_n\left(z\right)$. These results are given in Figure 7, where the numbers $1,2,$ and 3 are the radius values in the polar plot.

The above result concludes that ${\left\{{\mathbf{e}}_n\right\}}_{n\in \mathbf{W}}$ is actually an orthonormal system. To this end, we have to establish that it is also complete. So, for this, pick a holomorphic function $f\in H_{\sigma ,{\sigma }_0,\mathbf{C}}$ with $f\left(z\right)={\sum }_{l=0}^{\infty }{a}_l{z}^l$ and observe that

$$\begin{array}{cc}\hfill {〈f,{\mathbf{e}}_n〉}_{\sigma ,{\sigma }_0}=& \frac{e{\sigma }^2}{2\pi }{\int }_{\mathbf{C}}f\left(z\right)\overline{{\mathbf{e}}_n\left(z\right)}d{\mu }_{\sigma ,{\sigma }_0}\left(z\right)\hfill \\ \hfill =& \frac{e{\sigma }^2}{2\pi }\sum _{l=0}^{\infty }{a}_l{\int }_{\mathbf{C}}{z}^l\overline{{\mathbf{e}}_n\left(z\right)}d{\mu }_{\sigma ,{\sigma }_0}\left(z\right)\hfill \\ \hfill =& \frac{e{\sigma }^2}{2\pi }\sqrt{\frac{{\sigma }^{2n}}{n!{\mathcal{F}}_{n,\widehat{\sigma },1}}}\sum _{l=0}^{\infty }{a}_l{\int }_{\mathbf{C}}{z}^l\overline{z^n}d{\mu }_{\sigma ,{\sigma }_0}\left(z\right)\hfill \\ \hfill =& \frac{e{\sigma }^2}{2\pi }\sqrt{\frac{{\sigma }^{2n}}{n!{\mathcal{F}}_{n,\widehat{\sigma },1}}}\sum _{l=0}^{\infty }{a}_l\left(\sqrt{\frac{2\pi l!}{e{\sigma }^{2l+2}}{\mathcal{F}}_{n,\widehat{\sigma },1}}\sqrt{\frac{2\pi n!}{e{\sigma }^{2n+2}}{\mathcal{F}}_{n,\widehat{\sigma },1}}\right){\delta }_{ln}\hfill \\ \hfill =& \sqrt{\frac{{\sigma }^{2n}}{n!{\mathcal{F}}_{n,\widehat{\sigma },1}}}{a}_n\frac{n!{\mathcal{F}}_{n,\widehat{\sigma },1}}{{\sigma }^{2n}}\hfill \\ \hfill =& {\left[\sqrt{\frac{{\sigma }^{2n}}{n!{\mathcal{F}}_{n,\widehat{\sigma },1}}}\right]}^{-1}{a}_n.\hfill \end{array}$$

Since the constant is ${\displaystyle \frac{{\sigma }^{2n}}{n!{\mathcal{F}}_{n,\widehat{\sigma },1}}}\ne 0$ for any choice of n, the condition that $\langle f,{\mathbf{e}}_n\rangle =0$ for all $n\in \mathbf{W}$ yields that $a_n=0$ for all $n\in \mathbf{W}$, which results in the conclusion that $f\equiv 0$. Therefore, ${\left\{{\mathbf{e}}_n\right\}}_{n\in \mathbf{W}}$ is complete. Now, we establish these results in a $d-$dimensional situation by employing the tensor product notation of Section 2.3. To this end, we see that

$$\begin{array}{c}\hfill {\langle {\mathbf{e}}_{n_1}\otimes \cdots \otimes {\mathbf{e}}_{n_d},{\mathbf{e}}_{m_1}\otimes \cdots \otimes {\mathbf{e}}_{m_d}\rangle }_{\sigma ,{\sigma }_0,d}=\prod _{j=1}^d{\langle {\mathbf{e}}_{n_j},{\mathbf{e}}_{m_j}\rangle }_{\sigma ,{\sigma }_0}.\end{array}$$

Hence, the orthonormality of ${\left\{{\mathbf{e}}_{n_1}\otimes \cdots \otimes {\mathbf{e}}_{n_d}\right\}}_{n_1,\dots n_d\in ,{\mathbf{W}}^d}$ is established due to the orthonormality of each ${\langle {\mathbf{e}}_{n_j},{\mathbf{e}}_{m_j}\rangle }_{\sigma ,{\sigma }_0}$. We still need to ensure that this $d-$dimensional orthonormal system is complete. Now, observe

$$\begin{array}{cc}\hfill {\langle f,{\mathbf{e}}_{n_1}\otimes \cdots \otimes {\mathbf{e}}_{n_d}\rangle }_{\sigma ,{\sigma }_0,d}=& {\left(\frac{e{\sigma }^2}{2\pi }\right)}^d{\int }_{{\mathbf{C}}^d}f\left(\mathit{z}\right)\overline{{\mathbf{e}}_{n_1}\otimes \cdots \otimes {\mathbf{e}}_{n_d}\left(\mathit{z}\right)}d{\mu }_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}\left(\mathit{z}\right)\hfill \\ \hfill =& {\left(\frac{e{\sigma }^2}{2\pi }\right)}^d\sum _{l_1,\dots ,l_d}^{\infty }{a}_{l_1,\dots ,l_d}{\mathbb{I}}_{l,d},\hfill \end{array}$$

where ${\mathbb{I}}_{l,d}={\int }_{{\mathbf{C}}^d}{\mathit{z}}^l\left({\mathbf{e}}_{n_1}\otimes \cdots \otimes {\mathbf{e}}_{n_d}\left(\overline{\mathit{z}}\right)\right)d{\mu }_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}\left(\mathit{z}\right)$. We further can simplify ${\mathbb{I}}_d$, as follows:

$$\begin{array}{cc}\hfill {\mathbb{I}}_{l,d}=& {\int }_{{\mathbf{C}}^d}{\mathit{z}}^l{\mathbf{e}}_{n_1}\left(\overline{z_1}\right)\wedge \cdots \wedge {\mathbf{e}}_{n_d}\left(\overline{z_d}\right)d{\mu }_{\sigma ,{\sigma }_0}\left(z_1\right)\wedge \cdots \wedge d{\mu }_{\sigma ,{\sigma }_0}\left(z_d\right)\hfill \\ \hfill =& \prod _{j=1}^d\left({\int }_{\mathbf{C}}{z}_j^{l_j}{\mathbf{e}}_{n_j}\left(\overline{z_j}\right)d{\mu }_{\sigma ,{\sigma }_0}\left(z_j\right)\right)\hfill \\ \hfill =& \prod _{j=1}^d\left({\int }_{\mathbf{C}}{z}_j^{l_j}{\overline{z_j}}^{n_j}d{\mu }_{\sigma ,{\sigma }_0}\left(z_j\right)\right)\hfill \\ \hfill =& \prod _{j=1}^d\left(\sqrt{\frac{2\pi l_j!}{e{\sigma }^{2l_j+2}}{\mathcal{F}}_{l_j,\widehat{\sigma },1}}\sqrt{\frac{2\pi n_j!}{e{\sigma }^{2n_j+2}}{\mathcal{F}}_{n_j,\widehat{\sigma },1}}\right){\delta }_{l_j{n}_j}{a}_{l_1,\dots ,l_d}.\hfill \end{array}$$

Finally,

$$\begin{array}{cc}\hfill {\left(\frac{e{\sigma }^2}{2\pi }\right)}^d\sum _{l_1,\dots ,l_d}^{\infty }{a}_{l_1,\dots ,l_d}{\mathbb{I}}_{l,d}=& \left(\prod _{j=1}^d{\left[\sqrt{\frac{{\sigma }^{2n_j}}{n_j!{\mathcal{F}}_{n_j,\widehat{\sigma },1}}}\right]}^{-1}\right)a_{n_1,\dots ,n_d}.\hfill \end{array}$$

The further result for completeness in $d-$dimension follows a routine procedure from single-dimension case, as already discussed before.    □

The following theorem provides the reproducing kernel for the RKHS $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ defined in (9).

Theorem 5.

For $\sigma >0,{\sigma }_0\ge 0$ and $\widehat{\sigma }={\displaystyle \frac{{\sigma }^2}{{\sigma }_0^2}}$, the reproducing kernel for the RKHS $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ is indeed (because Theorem 1) given as

$$\begin{array}{c}\hfill K\left(\mathit{z},\mathit{w}\right):=\sum _{n_1,\dots ,n_d=0}^{\infty }{\lambda }_n{\left(\mathit{z}\overline{\mathit{w}}\right)}^n,\end{array}$$

(17)

where multi-index notation is employed: $n=\left(n_1,\dots ,n_d\right)$ and ${\lambda }_n={\prod }_{i=1}^d{\displaystyle \frac{{\sigma }^{2n_i}}{n_i!{\mathcal{F}}_{n_i,\widehat{\sigma },1}}}$.

Proof. 

We will demonstrate the desired proof as follows:

• For $\mathit{w}\in {\mathbf{C}}^d$, we will show that $\parallel K\left(·,\mathit{w}\right){\parallel }_{\sigma ,{\sigma }_0,d}<\infty$. For this, consider that

  $$\begin{array}{cc}\hfill \parallel K\left(·,\mathit{w}\right){\parallel }_{\sigma ,{\sigma }_0,d}^2=& {\left(\frac{e{\sigma }^2}{2\pi }\right)}^d{\int }_{{\mathbf{C}}^d}{\left|K\left(\mathit{z},\mathit{w}\right)\right|}^{2}d{\mu }_{\sigma ,{\sigma }_0,d}\left(\mathit{z}\right)\hfill \\ \mathrm{(18)}& \hfill =& {\left(\frac{e{\sigma }^2}{2\pi }\right)}^d\sum _{n_1,\dots ,n_d}^{\infty }\sum _{m_1,\dots ,m_d}^{\infty }{\lambda }_n{\lambda }_m{\mathit{w}}^n{\overline{\mathit{w}}}^m{\int }_{{\mathbf{C}}^d}{\mathit{z}}^n{\overline{\mathit{z}}}^{m}d{\mu }_{\sigma ,{\sigma }_0,d}\left(\mathit{z}\right)\hfill \mathrm{(19)}& \hfill =& {\left(\frac{e{\sigma }^2}{2\pi }\right)}^d\sum _{n_1,\dots ,n_d}^{\infty }{\lambda }_n^2{\left|\mathit{w}\right|}^{2n}\left(\prod _{n_i=1}^d{\int }_{\mathbf{C}}{z}_i^{n_i}{\overline{z_i}}^{m_i}d{\mu }_{\sigma ,{\sigma }_0}\left(z_i\right)\right)\hfill \hfill =& {\left(\frac{e{\sigma }^2}{2\pi }\right)}^d\sum _{n_1,\dots ,n_d}^{\infty }{\lambda }_n^2{\left|\mathit{w}\right|}^{2n}\left(\prod _{n_i=1}^d\frac{2\pi n_i!}{{\sigma }^{2n_i+2}}{\mathcal{F}}_{n_i,\widehat{\sigma },1}\right)\hfill \\ \hfill =& \sum _{n_1,\dots ,n_d}^{\infty }\frac{{\left|\mathit{w}\right|}^{2n}}{{\prod }_{i=1}^d{\displaystyle \frac{n_i!}{{\sigma }^{2n_i}}}{\mathcal{F}}_{n_i,\widehat{\sigma },1}}.\hfill \end{array}$$

  We used the result of Theorem 4 from (18) to (19). For all $\mathit{w}\in {\mathbf{C}}^d$, the quantity ${\sum }_{n_1,\dots ,n_d}^{\infty }{\displaystyle \frac{{\left|\mathit{w}\right|}^{2n}}{{\prod }_{i=1}^d\left({\displaystyle \frac{n_i!}{{\sigma }^{2n_i}}}\right){\mathcal{F}}_{n_i,\widehat{\sigma },1}}}$ achieves convergence. This implies that $\parallel K\left(·,\mathit{w}\right){\parallel }_{\sigma ,{\sigma }_0,d}<\infty$ for all $\mathit{w}\in {\mathbf{C}}^d$. Therefore, the kernel function is $K(·,\mathit{w})\in H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$.
• In order to establish the reproducing property of $K(·,\mathit{w})$, pick an arbitrary $f={\sum }_{n_1,\dots ,n_d}^{\infty }{a}_{n_1,\dots ,n_d}{\mathit{w}}^n\in H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$. Then, consider the inner product of f with $K(·,\mathit{w})$, as follows:

  $$\begin{array}{cc}\hfill {\langle f,K(·,\mathit{w})\rangle }_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}=& {\left(\frac{e{\sigma }^2}{2\pi }\right)}^d{\int }_{{\mathbf{C}}^d}f\left(\mathit{z}\right)\overline{K\left(\mathit{z},\mathit{w}\right)}d{\mu }_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}\left(\mathit{z}\right)\hfill \\ \hfill =& {\left(\frac{e{\sigma }^2}{2\pi }\right)}^d\sum _{n_1,\dots ,n_d}^{\infty }\sum _{l_1,\dots ,l_d}^{\infty }{a}_{n_1,\dots ,n_d}{\lambda }_{l_1,\dots ,l_d}{\mathit{w}}^n\left(\prod _{i=1}^d\frac{2\pi }{e{\sigma }^2}\frac{{\sigma }^{2n_i}}{n_i!{\mathcal{F}}_{n_i,\widehat{\sigma },1}}\right)\hfill \\ \hfill =& \sum _{n_1,\dots ,n_d=0}^{\infty }{a}_{n_1,\dots ,n_d}{\mathit{w}}^n\hfill \\ \hfill =& f\left(\mathit{w}\right).\hfill \end{array}$$

Hence, the desired result is achieved.    □

The proof in the preceding theorem to demonstrate reproducing kernel nature of $K\left(\mathit{z},\mathit{w}\right)$ of RKHS $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ utilizes the basic machinery borrowed from the two-part definition for the reproducing kernel given in Definition 6. Now that we have determined the reproducing kernel theory for the Generalized Gaussian Radial Basis Function Kernel, we will investigate their functionality in terms of universality, which makes them applicable in addressing the learning architecture problems, such as standard binary classification through SVMs, function approximation through kernel regression, and pattern recognition through deep convolutional neural networks.

4. Restriction and Universality of Reproducing Kernel of RKHS ${\mathbf{H}}_{\mathbf{\sigma },{\mathbf{\sigma }}_{\mathbf{0}},{\mathbf{C}}^{\mathbf{d}}}$

It is crucial to address the fact that the restriction of the reproducing kernel $K\left(\mathit{z},\mathit{w}\right)$ on ${\mathbf{R}}^d\times {\mathbf{R}}^d$ is also a kernel function, which eventually will assist us in establishing the universal nature of the $K\left(\mathit{z},\mathit{w}\right)$.

Lemma 1

(Restriction of kernels [21]). Let k be a kernel on X, let $\tilde{X}$ be a set, and let $A:\tilde{X}\to X$ be a map. Then, $\tilde{k}$ defined by $\tilde{k}(x,x^{\prime }):=k\left(A\left(x\right),A\left(\tilde{x}\right)\right),x,\tilde{x}\in X$, which is a kernel on $\tilde{X}$. In particular, if $\tilde{X}\subset X$, then $k_{|\tilde{X}\times \tilde{X}}$ is a kernel.

In other words, the conclusion of the above lemma is given for all $x,x^{\prime }\in {\mathbf{R}}^d$; if $k(x,x^{\prime })\in \mathbf{R}$, then k is also a kernel in the real sense (cf. [21]). This notion of restricting the kernel from $\mathbf{C}$ to $\mathbf{R}$ can allow the restriction of $K\left(\mathit{z},\mathit{w}\right)$ for the RKHS $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ on ${\mathbf{R}}^d\times {\mathbf{R}}^d$ to have kernel in a real sense. Note that the ${\lambda }_n^{\prime }$s present in the reproducing kernel $K(\mathit{z},\mathit{w})$ in (17) are strictly positive real numbers for all $n\in \mathbf{W}$; also, observe that for all $x,x^{\prime }\in {\mathbf{R}}^d$, we have

$$\begin{array}{c}\hfill K_{|{\mathbf{R}}^d\times {\mathbf{R}}^d}(x,x^{\prime })=\sum _{n_1,\dots ,n_d=0}^{\infty }{\lambda }_n{\left(x\overline{x^{\prime }}\right)}^n=\left[\sum _{n_1,\dots ,n_d=0}^{\infty }{\lambda }_n{\left(x{x}^{\prime }\right)}^n\right]\in \mathbf{R},\end{array}$$

(20)

which implies that the restriction of $K(·,·)$ on ${\mathbf{R}}^d\times {\mathbf{R}}^d$ is a kernel. Now, we shall recall the notion of universal kernel.

Definition 7

(Universal Kernels [21]). A continuous kernel k on a compact metric space X is called universal if the RKHS H of k is dense in $C\left(X\right)$. This means that, for every function $g\in C\left(X\right)$ and all $ϵ>0$, there exists $f\in H$, such that ${\parallel f-g\parallel }_{\infty }\le ϵ.$

Now that we have defined the notion of universal kernel function in the above definition, we provide an example of the same.

Example 2

(Universal Taylor Kernels). For a fixed $R\in (0,\infty ]$ and a $C^{\infty }-$function $f:(-R,R)\to \mathbf{R}$ that can be expanded into its Taylor series at 0, i.e.,

$$\begin{array}{ccc}\hfill f\left(t\right)=\sum _{n=0}^{\infty }{a}_n{t}^n& \hfill & \hfill t\in (-R,R).\end{array}$$

(21)

Let $X\left\{x\in {\mathbf{R}}^d:{\parallel x\parallel }_2<\sqrt{R}\right\}$. If we have $a_n>0$ for all $n\ge 0$ in (21), then k given by

$$\begin{array}{ccc}\hfill k(x,x^{\prime }):=f\left(\langle x,x^{\prime }\rangle \right),& \hfill & \hfill x,x^{\prime }\in X,\end{array}$$

(22)

is a universal kernel on every compact subset of X.

With the help of universal Taylor kernels, as given above, we can see that the reproducing kernel of the RKHS $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ is a universal Taylor kernel, followed by its restriction on ${\mathbf{R}}^d\times {\mathbf{R}}^d$. This motivating result is captured in the following important theorem.

Theorem 6.

Let X be a compact subset of ${\mathbf{R}}^d$, $\sigma >0$ and $\sigma \ge 0$. Then, the kernel $K_{|{\mathbf{R}}^d\times {\mathbf{R}}^d}$ defined in (20) is universal.

Proof. 

Let $\sigma >0$ and ${\sigma }_0\ge 0$. Recall $a_n$ from (21) and ${\lambda }_n$ from (17) and set both of them equal to each other, that is, $a_n={\lambda }_n$ for all $n\in \mathbf{W}$. With these choices of $a_n$ for all $n\in \mathbf{W}$, we now define a $C^{\infty }-$function $f:\left(-R,R\right)\to \mathbf{R}$ as

$$\begin{array}{ccc}\hfill f\left(t\right)=\sum _{n=0}^{\infty }{\lambda }_n{t}^n& \hfill & \hfill t\in (-R,R).\end{array}$$

(23)

We learn that, for $x,x^{\prime }\in {\mathbf{R}}^d$, whose Euclidean norm is bounded by $\sqrt{R}$, we can express $K_{|{\mathbf{R}}^d\times {\mathbf{R}}^d}\left(x,x^{\prime }\right)$ in terms of f as defined in (23) as:

$$\begin{array}{c}\hfill K_{|{\mathbf{R}}^d\times {\mathbf{R}}^d}\left(x,x^{\prime }\right)=\sum _{n=0}^{\infty }{\lambda }_n{\left(x{x}^{\prime }\right)}^n=\sum _{n=0}^{\infty }{\lambda }_n{\left(\langle x,x^{\prime }\rangle \right)}^n=f\left(\langle x,x^{\prime }\rangle \right)\end{array}$$

(24)

The establishment of $K_{|{\mathbf{R}}^d\times {\mathbf{R}}^d}\left(x,x^{\prime }\right)$ for $x,x^{\prime }\in {\mathbf{R}}^d$ in terms of $C^{\infty }-$function f and in combination of ${\lambda }_n>0$ for all $n\in \mathbf{W}$ makes $K_{|{\mathbf{R}}^d\times {\mathbf{R}}^d}$ a universal kernel by the virtue of Example 2.    □

The establishment of the universal nature of the reproducing kernel $K(·,·)$ of the RKHS $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$ with restriction on ${\mathbf{R}}^d\times {\mathbf{R}}^d$ in Theorem 6 allows one to approximate arbitrary continuous functions on compact subsets of ${\mathbf{R}}^d$ with those functions that can be represented against the reproducing kernel $K(·,·)$ to the inner product, as defined in (7) or consequently whose norm given in (8) is finite.

5. Empirical Evidence and Results Comparison

We present the empirical evidence for the application of the Generalized Gaussian Radial Basis Function Kernel on various basic yet important learning architecture routines. The technical details of every experiment presented here are provided in Section 6.

5.1. Kernel Regression

We know that kernel regression aims to approximate or interpolate arbitrary continuous functions through the available choices of universal kernel functions on compact subsets of ${\mathbf{R}}^d$. Usually, these functions are radial basis functions that are employed in the kernel sense to achieve the approximation. In the current example given as follows, we leverage the universal nature of both Gaussian Radial Basis Function Kernel (cf. Corollary 4.58 in [21]) and Generalized Gaussian Radial Basis Function Kernel from Theorem 6.

5.1.1. Example 1

Kernel regression of $f\left(x\right)=e^{\frac{(1-9x^2)}{4}}+tanx+x^{\frac{1}{6}}+sin{x}^{\vartheta \left(n\right)}$ is performed via both GRBF and (1) and the respective results are given below in Figure 8. Here, $\vartheta \left(n\right)$ is uniform random distributed number in $(0,1)$, $x=n\in {\mathbf{Z}}_{100}$.

5.1.2. Example 2

Kernel regression of $f\left(x\right)=e^{sinx-sin{x}^2}+\sqrt{2\pi }|x+cos\vartheta \left(n\right)|$ is performed via both GRBF and (1) and the respective results are given below in Figure 9. Here, $\vartheta \left(n\right)$ is uniform random distributed number in $(0,1)$, $x=n\in {\mathbf{Z}}_{100}$.

5.2. Support Vector Machine

Let us briefly address how the results derived in Section 3 can be used to understand the analysis of SVM. To that end, we define an important notion in SVMs referred to as hinge loss function, denoted by $L(·,·)$ as:

$$\begin{array}{ccc}\hfill L(y,t):=\underset{y\in Y:=\left\{-1,1\right\}}{max}\left\{0,1-yt\right\}& \hfill & \hfill t\in \mathbf{R}.\end{array}$$

(25)

Let $X\subset {\mathbf{R}}^d$ be a nonempty open subset and P be a probability measure on $X\times Y$; then, the risk induced by the hinge loss function is $L-$risk, which is defined with help of $f:X\to \mathbf{R}-$measurable function as:

$$\begin{array}{c}\hfill {\mathcal{R}}_{L,P}\left(f\right):={\int }_{X\times Y}L\left(y,f\left(x\right)\right)dP(x,y).\end{array}$$

(26)

Additionally, the minimal $L-$risk is given as

$$\begin{array}{c}\hfill {\mathcal{R}}_{L,P}^{*}:=\underset{f:X\to \mathbf{R}}{inf}{\mathcal{R}}_{L,P}\left(f\right).\end{array}$$

(27)

It is important to recall here that the RKHSs of universal kernels approximate the minimal $L-$risk losses. It is performed in the following corollary (cf. Corollary 5.29 in [21]).

Corollary 2.

Let X be a compact metric space, H be the RKHS of a universal kernel on X, P be a distribution on $X\times Y$, and $L:X\times Y\times \mathbf{R}\to [0,\infty )$ be a continuous $P-$integrable loss. Then, we have ${inf}_{f\in H}\left[{\mathcal{R}}_{L,P}\left(f\right)\right]={\mathcal{R}}_{L,P}^{*}.$

Now, consider $f\in H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$; then, let the restriction of f on ${\mathbf{R}}^d$ be denoted by $f_{|{\mathbf{R}}^d}$; then, the aforementioned result forces us to establish the relation

$$\begin{array}{c}\hfill \underset{f_{|{\mathbf{R}}^d}\in H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}}{inf}\left[{\mathcal{R}}_{L,P}\left(f\right)\right]={\mathcal{R}}_{L,P}^{*}.\end{array}$$

(28)

The setup presented in (28) enables to have consistent results concerning the classifiers of SVMs, which simply means that the classifiers that are based on minimization of a regularized risk can asymptotically learn in every classification task [22]. With this theory of SVM around the RKHS $H_{\sigma ,{\sigma }_0,{\mathbf{C}}^d}$, we are now ready to ensure the application of the Generalized Gaussian Radial Basis Function Kernel in SVMs.

SVMs are implemented for the standard binary data classification (cf. [23]) data generated in top left of Figure 10; this is performed via a different choices of kernels; these kernels include the traditional Sigmoid function, the Gaussian Radial Basis Function Kernel, and the Generalized Gaussian Radial Basis Function Kernel in (1).

Of the available three kernels—the Generalized Gaussian Radial Basis Function in (1), the Sigmoid, and the Gaussian Radial Basis Function Kernel—the Generalized Gaussian Radial Basis Function Kernel yields the lowest misclassification for the data of 101 sampled points.

5.3. Neural Network

The work in this section revolves around the application of universal approximation theorem (adapted from [24]) of Cybenko-1989 [25], Hornik-1991 [26], and Pinkus-1999 [27], as follows:

Theorem 7

(Universal Approximation Theorem in Neural Network). Let $\rho :\mathbf{R}\to \mathbf{R}$ be any continuous function. Let ${\mathcal{N}}_d^{\rho }$ represent the class of feed-forward neural networks with activation function ρ with $d-$neurons in the input layer, one neuron in the output layer, and one hidden layer with an arbitrary number of neurons. Let $X\subseteq {\mathbf{R}}^d$ be compact. Then, ${\mathcal{N}}_d^{\rho }$ is dense in $C\left(X\right)$ if and only if ρ is non-polynomial.

Now, under the application of the universal approximation theorem, one can have $\rho \left(r\right)=g_{{\sigma }^2}\left(r\right)exp\left(g_{{\sigma }_0^2}\left(r\right)-1\right)$ in Theorem 7; then, we realize that the Generalized Gaussian Radial Basis Function is an activation function, as it is a non-polynomial function. In the pursuit of an application of the Generalized Gaussian Radial Basis Function, we have following examples that provide the evidence of the optimal results that are yielded by the Generalized Gaussian Radial Basis Function in (1) employment both as in the usage of the activation function and the neural net layer in deep convolution neural nets (DCNNs).

5.3.1. Activation Function

Consider an $\alpha$ReLU activation function, a simple modification of the powerful traditional ReLU activation function from NN (AF-NN), defined as

$$\begin{array}{c}\hfill f\left(x\right)\left\{\begin{array}{cc}x\hfill & \mathrm{if}x>0\hfill \\ \alpha x\hfill & \mathrm{if}x\le 0.\hfill \end{array}\right.\end{array}$$

(29)

In the present experiment, two seven-layered NNs are constructed: one with the activation function defined by (29) and one with Generalized Gaussian Radial Basis Function in (1). Thereafter, the following performance

Table 1. Performance summary for GGRBF with an accuracy of $\mathbf{97.88}\%$.

Training with GGRBF (on Single CPU)
Epoch	Iteration	Runtime (hh:mm:ss)	Accuracy (%)	Batch Loss	Learning Rate
1	1	00:00:04	8.59	2.5329	0.0010
2	50	00:00:10	75.78	1.0087	0.0010
3	100	00:00:15	93.75	0.4377	0.0010
4	150	00:00:21	99.22	0.2452	0.0010
6	200	00:00:26	99.22	0.1790	0.0010
7	250	00:00:31	100.00	0.1028	0.0010
8	300	00:00:36	100.00	0.0681	0.0010
9	350	00:00:41	100.00	0.0458	0.0010
10	390	00:00:45	100.00	0.0382	0.0010

and

Table 2. Performance summary for (29) with accuracy of $93.48\%$.

Training with (29) (on Single CPU)
Epoch	Iteration	Runtime (hh:mm:ss)	Accuracy (%)	Batch Loss	Learning Rate
1	1	00:00:08	8.59	2.9976	0.0010
2	50	00:00:15	78.91	0.6053	0.0010
3	100	00:00:19	86.72	0.4108	0.0010
4	150	00:00:25	90.62	0.3122	0.0010
6	200	00:00:30	96.88	0.1632	0.0010
7	250	00:00:36	99.22	0.0695	0.0010
8	300	00:00:41	99.22	0.0778	0.0010
9	350	00:00:46	100.00	0.0390	0.0010
10	390	00:00:49	100.00	0.0355	0.0010

yield the respective results and comparison for the NNs.

Furthermore, the training progress data for the individual neural network architectures are presented next. The legends for accuracy and loss are same for both of the neural network architectures and their respective information is present in between the Figure 11 and Figure 12.

5.3.2. DCNN

The above experiments demonstrate the concrete functionality of the Generalized Gaussian Radial Basis Function in (1) as an activation function; this finding further motivates the performance of the deep convolution neural net. Therefore, in pursuit of this, a typical 7-layered deep convolution neural net is constructed; it has an activation function defined by (29) and one with (1). In the following experiment given in Figure 13 and Figure 14, the training data contain 1500–$28\times 28$ gray-scale letter images of A, B, and C in a 4-D array.

6. Results and Technical Details

Now that we have successfully constructed the Hilbert function space theory around the Generalized Gaussian Radial Basis Function, when it is viewed as the $L^2-$measure, we immediately employ it in various basic artificial intelligence learning architectures. The learning architectures, namely kernel regression, SVM, and DCNN, that are considered here are quite basic yet well-versed in the field of machine learning and artificial intelligence. These can be regarded as the building blocks of several other artificial intelligence routines. Following

Table 3. Compilation of technical results.

AI Learning Architecture	Activation Function	Reference	Min. Error	Misclass. %	Accuracy %	Runtime (s)
Kernel Regression	GRBF	Figure 8	0.00230	-	-	-
Kernel Regression	GGRBF	Figure 8	$\mathbf{0.00097}$	-	-	-
Kernel Regression	GRBF	Figure 9	0.00100	-	-	-
Kernel Regression	GGRBF	Figure 9	$\mathbf{0.00043}$	-	-	-
SVM	GRBF	Figure 10	-	5.75	94.25	-
SVM	Sigmoid	Figure 10	-	4.50	95.50	-
SVM	GGRBF	Figure 10	-	$\mathbf{3.75}$	$\mathbf{96.25}$	-
AF in NN	GGRBF	Table 1	-	$\mathbf{2.12}$	$\mathbf{97.88}$	0.43077
AF in NN	$\alpha$ReLU	Table 2	-	6.52	93.48	1.8935
DCNN	GGRBF	Figure 13	-	$\mathbf{3.67}$	$\mathbf{96.33}$	-
DCNN	$\alpha$ReLU	Figure 14	-	8.13	91.87	-

is essentially a compilation of the results documented for the various experiments we have discussed so far in comparison with the Generalized Gaussian Radial Basis Function Kernel as opposed to its available counterpart(s), where none of hyper-parameters were optimized for the available kernel functions.

The empirical evidence recorded here simply directs us to conclude that the results prompted by the Generalized Gaussian Radial Basis Function outperform every considered learning architecture routine. A detailed discussion of the empirical evidence collected is presented here:

• We considered two experiments for the kernel regression of the functions mentioned in Figure 8 and Figure 9. These functions, as easily can be seen, are not well-behaved functions, since they involve transcendental behavior along with function discontinuities (due to $tanx$ or $x^{1/6}$, etc.) followed by uniform randomness due to the $\vartheta \left(n\right)$ of $101-$random points. In both of the experiments of this nature, we record the minimum error obtained by employing either the Gaussian Radial Basis Function Kernel or the Generalized Gaussian Radial Basis Function Kernel. From this experiment, we learn that the minimum error recorded in both of them is remarkably small, in the order of ∼${10}^{-4}$, if we use the Generalized Gaussian Radial Basis Function Kernel in contrast to the Gaussian Radial Basis Function Kernel, which provides a minimum error in the order of ∼${10}^{-3}$.
• Next, we consider the experiment of support vector machines for binary classification, in which we generated a random set of points (one can think of them as data) within the unit circle. We identified the random points spread across the circle with the positive class if it lies either in the first or the third quadrant; otherwise, we consider it to be within the negative class. After we generate and identify these datasets, we then identify the support vectors and the decision boundary by employing three (custom) kernel functions over MATLAB R2023b: Gaussian Radial Basis Function Kernel, Sigmoid Kernel, and Generalized Gaussian Radial Basis Function Kernel. We then register the out-of-sample misclassification rate by using 10-fold cross-validation from the aforementioned kernel functions. Here, we immediately learn that the misclassification yielded by Gaussian Radial Basis Function Kernel is comparatively high among all three employed kernel functions; the Generalized Gaussian Radial Basis Function Kernel yields the lowest misclassification rate: upward of $96\%$.
• For the activation function application in the domain of neural networks, we consider a practical example of pattern recognition for the synthetic hand-written English letter dataset in the form of a $4-D$ array (lettersTrainSet in MATLAB). As already mentioned in the respective subsection, two $7-$layered neural nets were constructed whose layer names are given as follows:

  (a)

  Image input;

  (b)

  2D convolution;

  (c)

  Batch normalization;

  (d)

  (29) vs. Generalized Gaussian Radial Basis Function;

  (e)

  Ten fully connected layers;

  (f)

  Softmax;

  (g)

  Classification layer.

  It should be noted that, from the above $7-$layered neural network constructed, both of the neural network training experiments came to a complete stop when the maximum epoch of 30 amounting to 330 iterations were finished. Additionally, the sequence of inserting the neural net layer was the same as that which was given above from (a)–(g). Once the desired neural net is set with both of the activation functions, we immediately learn that the neural net containing the activation function of the Generalized Gaussian Radial Basis Function Kernel as one of its layers simply outperforms its opponent, i.e., the neural net layer containing the activation function of (29).
• With the establishment of the basics of the neural network architecture in the previous listing point, we train and execute a deep convolution neural net for the representative images of the alphabet characters A, B, and C. A list of the seven layers involved in the construction of a deep convolution neural network is presented as follows:

  (a)

  Image input;

  (b)

  2D convolution;

  (c)

  (29) vs. Generalized Gaussian Radial Basis Function;

  (d)

  A 2$\times 2$ max pooling with stride [2 2] and padding [0 0 0 0];

  (e)

  Three fully connected layers;

  (f)

  Softmax;

  (g)

  Classification layer.

  Again, the sequence of inserting the neural net layers was the same as what is given above from (a)–(g) to construct the present DCNN. Similarly, the result for the successful classification performances by the DCNN which contains the activation layer of Generalized Gaussian Radial Basis Function Kernel achieves an accuracy upward of $96\%$. However, we fail to see the match-able competitiveness by the DCNN, which contains the (29) as the activation layer.

7. Future Directions

Eigen Function Expansion of GGRBF

We recall the Mercer’s theorem from ([12], Theorem 4.2, Page 96).

Theorem 8

(Mercer’s Theorem). Let $\left(\mathcal{X},\mu \right)$ be a finite measure space and $k\in L_{\infty }\left({\mathcal{X}}^2,{\mu }^2\right)$ be a kernel, such that $T_k:L_2\left(\mathcal{X},\mu \right)\to L_2\left(\mathcal{X},\mu \right)$ is a definite positive. Let ${\left\{{\varphi }_i\right\}}_i\in L_2\left(\mathcal{X},\mu \right)$ be the normalized eigenfunctions of $T_k$ associated with the eigenvalues ${\left\{{\mathrm{\Lambda }}_i\right\}}_i$. Then,

1. 

the eigenvalues ${\left\{\mathrm{\Lambda }\right\}}_i$ are absolutely summable

2. 

$k\left(\mathit{x},{\mathit{x}}^{\prime }\right)={\sum }_{i=0}^{\infty }{\mathrm{\Lambda }}_i{\varphi }_i\left(\mathit{x}\right){\varphi }_i{\left({\mathit{x}}^{\prime }\right)}^{*}$ holds ${\mu }^2$ almost everywhere, where the series converges absolutely and uniformly ${\mu }^2$ almost everywhere.

Example 3.

With the application of Theorem 8, we can provide the eigen function decomposition of $K_{\sigma }\left(x,z\right)=e^{-{\sigma }^2{\left(x-z\right)}^2}$ for $x,z\in \mathbf{R}$, that is:

$$\begin{array}{cc}\hfill e^{-{\sigma }^2{\left(x-z\right)}^2}=& \sum _{i=0}^{\infty }{\mathrm{\Lambda }}_i{\varphi }_i\left(x\right){\varphi }_i\left(z\right),\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill {\mathrm{\Lambda }}_i=& \frac{\alpha {\sigma }^{2i}}{{\left(\frac{{\alpha }^2}{2}\left(1+\sqrt{1+{\left(\frac{2\sigma }{\alpha }\right)}^2}\right)+\frac{{\sigma }^2}{2}\right)}^{i+1/2}},\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill {\varphi }_i\left(x\right)=& \frac{\sqrt[8]{1+{\left(1+\frac{2\sigma }{\alpha }\right)}^2}}{\sqrt{2^{i}i!}}{e}^{-\left(\sqrt{1+{\left(\frac{2\sigma }{\alpha }\right)}^2}-1\right)\frac{{\alpha }^2{x}^2}{2}}{H}_i\left(\sqrt[4]{1+{\left(\frac{2\sigma }{\alpha }\right)}^2}\alpha x\right).\hfill \end{array}$$

(32)

The expression ${\left\{H_i\left(•\right)\right\}}_i$ in (32) are the Hermite polynomials which are $L_2$ orthonormal against the weight ${\displaystyle \frac{\alpha }{\pi }}{e}^{-{\alpha }^2{x}^2}$, that is:

$$\begin{array}{c}\hfill {\int }_{\mathbf{R}}{\varphi }_n\left(x\right){\varphi }_m\left(x\right)\frac{\alpha }{\pi }{e}^{-{\alpha }^2{x}^2}dx={\delta }_{nm}.\end{array}$$

(33)

We have the graphical representation of first seven Hermite polynomials in Figure 15a,b.

Based on the Hermite polynomials [28] and as its application for the eigen decomposition analysis for the Gaussian Radial Basis Function Kernel, we have the following promising future research direction. In the spirit of the application of Mercer’s theorem, we know the eigen function decomposition of the Gaussian Radial Basis Function Kernel. However, presently, we are not fortunate enough to have such a decomposition for the Generalized Gaussian Radial Basis Function Kernel. A preliminary investigation towards the desired eigen function decomposition of the Generalized Gaussian Radial Basis Function Kernel (from [29,30,31]) directs us toward the incorporation of a new variety of functions, as defined in (34).

$$\begin{array}{cc}\hfill {\mathcal{H}}_n\left(x\right):=& {(-1)}^n{e}^{a{x}^2}{e}^{-e^{-b{x}^2}+1}\frac{d^n}{d{x}^n}\left(e^{-a{x}^2}{e}^{e^{-b{x}^2}-1}\right)\hfill \end{array}$$

(34)

into our desired eigen function decomposition analysis. Here, $a>0$ and $b\ge 0$; therefore, if $a=1\&b=0$, then ${\mathcal{H}}_i=H_i$. The first two expressions for ${\mathcal{H}}_n\left(x\right)$ are explicitly given as:

$$\begin{array}{cc}\hfill {\mathcal{H}}_1\left(x\right)=& 2ax+2bx{e}^{-b{x}^2}\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill {\mathcal{H}}_2\left(x\right)=& -2a+4b^2{x}^2{e}^{-b{x}^2}-2b{e}^{-b{x}^2}+{\left(2ax+2bx{e}^{-b{x}^2}\right)}^2.\hfill \end{array}$$

(36)

The real constants a and b present in (34) corresponds to the respective constants present in (1) as $a={\sigma }^2$ and $b={\sigma }_0^2$. Next, see the graphical presentation of the first seven functions from the family defined in (34) in Figure 16 and Figure 17.

Future Direction 1.

We remain in a position lacking in knowledge of eigenvalues for the Generalized Gaussian Radial Basis Function Kernel. Additionally, we still need to investigate whether the function introduced in (34) is an orthonormal in the sense that the traditional Hermite polynomials are. Therefore, it will be interesting to understand the analysis of the function given in (34).

8. Discussion

In the present manuscript, we explore and introduce the Hilbert function space theory; in particular, the RKHS was generated by the Generalized Gaussian Radial Basis Function $g_{{\sigma }^2}\left(r\right)exp\left(g_{{\sigma }_0^2}\left(r\right)-1\right)$ as an $L^2-$measure, followed by an interesting application of it in the learning architectures of machine learning and deep learning. Throughout this article, we have learned from practical results that it is indeed beneficial to employ the Generalized Gaussian Radial Basis Function Kernel; here, one might achieve phenomenal results compared to those derived using its corresponding counterparts. Even though it is quite advantageous to use the Generalized Gaussian Radial Basis Function Kernel in learning architectures, we also encounter a few shortcomings in this endeavor. Oftentimes, the result yielded by the Generalized Gaussian Radial Basis Function is similar to the results that one might obtain using the preexisting and well-studied activation functions, such as Gaussian error linear unit (GELU) (cf. [32]), whose operation on x is given as

$$\begin{array}{c}\hfill GELU\left(x\right):=\frac{x}{2}\left(1+erf\left(\frac{\phantom{x}x}{\sqrt{2}}\right)\right).\end{array}$$

(37)

For instance, the comparison of the results based on choice of either Generalized Gaussian Radial Function Kernel or Gaussian error linear unit as an activation function is given in

Table 4. Performance summary for GGRBF in comparison with (37) with accuracy of $\mathbf{98.18}\%$.

Training with GGRBF (on a Single CPU)
Epoch	Iteration	Runtime (hh:mm:ss)	Accuracy (%)	Batch Loss	Learning Rate
1	1	00:00:00	11.72	2.5329	0.0010
2	50	00:00:04	72.66	1.0087	0.0010
3	100	00:00:09	84.38	0.4377	0.0010
4	150	00:00:13	89.06	0.2452	0.0010
6	200	00:00:18	98.44	0.1790	0.0010
7	250	00:00:23	100.00	0.1028	0.0010
8	300	00:00:27	100.00	0.0681	0.0010
9	350	00:00:32	100.00	0.0458	0.0010
11	400	00:00:36	100.00	0.0382	0.0010
12	450	00:00:41	100.00	0.0382	0.0010
13	500	00:00:45	100.00	0.0382	0.0010
14	546	00:00:50	100.00	0.0382	0.0010

and

Table 5. Performance summary for (37) in comparison with GGRBF with accuracy of $98.06\%$.

Training with (37) (on a Single CPU)
Epoch	Iteration	Runtime (hh:mm:ss)	Accuracy (%)	Batch Loss	Learning Rate
1	1	00:00:00	10.94	2.7556	0.0010
2	50	00:00:04	82.81	0.5065	0.0010
3	100	00:00:07	97.66	0.1923	0.0010
4	150	00:00:11	100.00	0.0738	0.0010
6	200	00:00:14	100.00	0.0446	0.0010
7	250	00:00:28	100.00	0.0262	0.0010
8	300	00:00:21	100.00	0.0211	0.0010
9	350	00:00:25	100.00	0.0173	0.0010
11	400	00:00:29	100.00	0.0099	0.0010
12	450	00:00:32	100.00	0.0085	0.0010
13	500	00:00:36	100.00	0.0085	0.0010
14	546	00:00:39	100.00	0.0068	0.0010

, respectively. We see that the results from both of the cases are comparable in terms of acquiring an accuracy rate. The key point to note here is that the Generalized Gaussian Radial Basis Function Kernel is a brand-new novel mathematical function to operate on; hence, it demands a detailed, in-depth investigation, which is initiated by its RKHS in this article, before it can be exploited to its full potential on the venues of artificial intelligence. Out of the other few major concerns in connection with the Generalized Gaussian Radial Basis Function Kernel is its mathematical structure, i.e., the presence of an extra exponential of an exponential of an argument. Hence, the simple presence of this can often slow the training process of learning architectures in which the Generalized Gaussian Radial Basis Function Kernel is employed; therefore, this might pose a challenge in attaining the desired results.