Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations

Highlights

• Rates and complexity for smooth function approximation in Hölder norms by ReQU neural networks.

• Explicit and uniform bounds for weights of the approximating neural network.

• Exponential convergence rates for analytic functions.

Abstract

This paper investigates the approximation properties of deep neural networks with piecewise-polynomial activation functions. We derive the required depth, width, and sparsity of a deep neural network to approximate any Hölder smooth function up to a given approximation error in Hölder norms in such a way that all weights of this neural network are bounded by 1. The latter feature is essential to control generalization errors in many statistical and machine learning applications.

Introduction

Neural networks have recently gained much attention due to their impressive performance in many complicated practical tasks, including image processing (LeCun, Bengio, & Hinton, 2015), generative modeling (Goodfellow et al., 2014), reinforcement learning (Mnih et al., 2015), numerical solution of PDEs, e.g., Geist et al., 2021, Han et al., 2018, and optimal control (Chen et al., 2019, Onken et al., 2022). This makes them extremely useful in design of self-driving vehicles (Li, Ota, & Dong, 2018) and robot control systems, e.g., Bozek et al., 2020, Cembrano et al., 1994, González-Álvarez et al., 2022. One of the reasons for such a success of neural networks is their expressiveness, that is, the ability to approximate functions with any desired accuracy. The question of expressiveness of neural networks has a long history and goes back to the papers (Cybenko, 1989, Funahashi, 1989, Hornik, 1991). In particular, in Cybenko (1989), the author showed that one hidden layer is enough to approximate any continuous function $f$ with any prescribed accuracy $\varepsilon >0$. However, further analysis revealed the fact that deep neural networks may require much fewer parameters than the shallow ones to approximate $f$ with the same accuracy. Many efforts were put in recent years to understand the fidelity of deep neural networks. In a pioneering work (Yarotsky, 2017), the author showed that for any target function $f$ from the Sobolev space ${\mathcal{W}}^{n,\infty }\left({[0,1]}^d\right)$ there is a neural network with $O\left({\varepsilon }^{-d/n}\right)$ parameters and ReLU activation function, that approximates $f$ within the accuracy $\varepsilon$ with respect to the $L_{\infty }$-norm on the unit cube ${[0,1]}^d$. Further works in this direction considered various smoothness classes of the target functions (Gühring and Raslan, 2021, Li et al., 2020a, Lu et al., 2021, Shen et al., 2021), neural networks with diverse activations (De Ryck et al., 2021, Gühring and Raslan, 2021, Jiao et al., 2021, Langer, 2021), domains of more complicated shape (Shen, Yang, & Zhang, 2020), and measured the approximation errors with respect to different norms (De Ryck et al., 2021, Gühring and Raslan, 2021, Schmidt-Hieber, 2020, Yarotsky, 2017). Several authors also considered the expressiveness of neural networks with different architectures. This includes wide neural networks of logarithmic (Gühring and Raslan, 2021, Schmidt-Hieber, 2020, Yarotsky, 2017) or even constant depth (De Ryck et al., 2021, Li et al., 2020a, Li et al., 2020b, Shen et al., 2021), or deep and narrow networks (Hanin, 2019, Kidger and Lyons, 2020, Park et al., 2021). Most of the existing results on the expressiveness of neural networks measure the quality of approximation with respect to either the $L_{\infty }$- or $L_p$-norm, $p⩾1$. Much fewer papers study the approximation of derivatives of smooth functions. These rare exceptions include Gühring et al., 2020, Gühring and Raslan, 2021 and De Ryck et al. (2021).

In the present paper, we focus on feed-forward neural networks with piecewise-polynomial activation functions of the form ${\sigma }^{\mathsf{ReQU}}\left(x\right)={(x\vee 0)}^2$. Neural networks with such activations are known to successfully approximate smooth functions from the Sobolev and Besov spaces with respect to the $L_{\infty }$- and $L_p$-norms (see, for instance, Abdeljawad and Grohs, 2022, Ali and Nouy, 2021, Chen et al., 2022, Gribonval et al., 2022, Klusowski and Barron, 2018, Li et al., 2020a, Li et al., 2020b, Siegel and Xu, 2022). We continue this line of research and study the ability of such neural networks to approximate not only smooth functions themselves but also their derivatives. We derive the non-asymptotic upper bounds on the Hölder norm between the target function and its approximation from a class of sparsely connected neural networks with ReQU activations. In particular, we show that, for any $f$ from a Hölder ball ${\mathcal{H}}^{\beta }({[0,1]}^d,H)$, $H>0$, $\beta >2$, (see Section 2 for the definition) and any $\varepsilon >0$, there exists a neural network with ReQU activation functions, that uniformly approximates the target function in the norms of the Hölder spaces ${\mathcal{H}}^{\ell }\left({[0,1]}^d\right)$ for all $\ell \in \{0,\dots ,\lfloor \beta \rfloor \}$. Here and further in the paper, $\lfloor \beta \rfloor$ stands for the largest integer which is strictly smaller than $\beta$. A simplified statement of our main result is given below.

Theorem 1 simplified version of Theorem 2

Fix $\beta >2$ and $p,d\in \mathbb{N}$. Then, for any $H>0$, for any $f:{[0,1]}^d\to {\mathbb{R}}^p$, $f\in {\mathcal{H}}^{\beta }({[0,1]}^d,H)$ and any integer $K⩾2$, there exists a neural network $h_f:{[0,1]}^d\to {\mathbb{R}}^p$ with ReQU activation functions such that it has $O(logd+\lfloor \beta \rfloor +loglogH)$ layers, at most $O(p\vee d{(K+\lfloor \beta \rfloor )}^d)$ neurons in each layer and $O\left(p(d\beta +d^2+loglogH){(K+\lfloor \beta \rfloor )}^d\right)$ non-zero weights taking their values in $[-1,1]$. Moreover, it holds that

$${\Vert f-h_f\Vert }_{{\mathcal{H}}^{\ell }\left({[0,1]}^d\right)}⩽\frac{C^{\beta d}H{\beta }^{\ell }}{K^{\beta -\ell }}\text{for all }\ell \in \{0,\dots ,\lfloor \beta \rfloor \}\text{,}$$

where $C$ is a universal constant.

We provide explicit expressions for the hidden constants in Theorem 2. The main contributions of our work can be summarized as follows.

• Given a smooth target function $f\in {\mathcal{H}}^{\beta }({[0,1]}^d,H)$, we construct a neural network, that simultaneously approximates all the derivatives of $f$ up to order $\lfloor \beta \rfloor$ with optimal dependence of the precision on the number of non-zero weights. That is, if we denote the number of non-zero weights in the network by $N$, then it holds that ${\Vert f-h_f\Vert }_{{\mathcal{H}}^{\ell }\left({[0,1]}^d\right)}=O\left(N^{-(\beta -\ell )/d}\right)$ simultaneously for all $\ell \in \{0,\dots ,\lfloor \beta \rfloor \}$.

• The constructed neural network has almost the same smoothness as the target function itself while approximating it with the optimal accuracy. This property turns out to be very useful in many applications including the approximation of PDEs and density transformations where we need to use derivatives of the approximation.

• The weights of the approximating neural network are bounded in absolute values by $1$. The latter property plays a crucial role in deriving bounds on the generalization error of empirical risk minimizers in terms of the covering number of the corresponding parametric class of neural networks. Note that the upper bounds on the weights provided in De Ryck et al., 2021, Gühring et al., 2020, Gühring and Raslan, 2021 blow up as the approximation error decreases.

The rest of the paper is organized as follows. In Section 2, we introduce necessary definitions and notations. Section 3 contains the statement of our main result, Theorem 2, followed by a detailed comparison with the existing literature. We then present numerical experiments in Section 4. The proofs are collected in Section 5. Some auxiliary facts are deferred to Appendix.