Approximation rates for neural networks with general activation functions

Abstract

We prove some new results concerning the approximation rate of neural networks with general activation functions. Our first result concerns the rate of approximation of a two layer neural network with a polynomially-decaying non-sigmoidal activation function. We extend the dimension independent approximation rates previously obtained to this new class of activation functions. Our second result gives a weaker, but still dimension independent, approximation rate for a larger class of activation functions, removing the polynomial decay assumption. This result applies to any bounded, integrable activation function. Finally, we show that a stratified sampling approach can be used to improve the approximation rate for polynomially decaying activation functions under mild additional assumptions.

Introduction

Deep neural networks have recently revolutionized a variety of areas of machine learning, including computer vision and speech recognition (LeCun, Bengio, & Hinton, 2015). A deep neural network with $l$ layers is a statistical model which takes the following form

$$f(x;\theta )=h_{W_l,b_l}\circ \sigma \circ h_{W_{l-1},b_{l-1}}\circ \sigma \circ \cdots \circ \sigma \circ h_{W_1,b_1},$$

where $h_{W_i,b_i}\left(x\right)=W_{i}x+b_i$ is an affine linear function, $\sigma$ is a fixed activation function which is applied pointwise and $\theta =\{W_1,\dots ,W_l,b_1,\dots ,b_l\}$ are the parameters of the model.

The approximation properties of neural networks have received a lot of attention, with many positive results. For example, in Ellacott (1994) and Leshno, Lin, Pinkus, and Schocken (1993) it is shown that neural networks can approximate any function on a compact set as long as the activation function is not a polynomial, i.e. that the set

$${\Sigma }_d\left(\sigma \right)=\mathrm{span}\left\{\sigma (\omega \cdot x+b):\omega \in {\mathbb{R}}^d,b\in \mathbb{R}\right\}$$

is dense in $C\left(\Omega \right)$ for any compact $\Omega \subset {\mathbb{R}}^d$. An earlier result of this form can be found in Hornik, 1991, Hornik, 1993 and shows that derivatives can be approximated arbitrarily accurately as well. An elementary and constructive proof for $C^{\infty }$ functions can be found in Attali and Pagès (1997).

In addition, quantitative estimates on the order of approximation are obtained for sigmoidal activation functions in Barron (1993) and for periodic activation functions in Mhaskar and Micchelli, 1994, Mhaskar and Micchelli, 1995. Results for general activation functions can be found in Hornik, Stinchcombe, White, and Auer (1994). A remarkable feature of these results in that the approximation rate is $O\left(n^{-1∕2}\right)$, where $n$ is the number of hidden neurons, which shows that neural networks can overcome the curse of dimensionality. Results concerning the approximation properties of generalized translation networks (a generalization of two-layer neural networks) for smooth and analytic functions are obtained in Mhaskar (1996). Approximation estimates for multilayer convolutional neural networks are considered in Zhou (2018) and multilayer networks with rectified linear activation functions in Yarotsky (2017). A comparison of the effect of depth vs. width on the expressive power of neural networks is presented in Lu, Pu, Wang, Hu, and Wang (2017).

An optimal approximation rate in terms of highly smooth Sobolev norms is given in Petrushev (1998). This work differs from previous work and the current work in that it considers approximation of highly smooth functions, for which proof techniques based on the Hilbert space structure of Sobolev spaces can be used. In contrast, the line of reasoning initially pursued in Barron (1993) and continued in this work makes significantly weaker assumptions on the function to be approximated.

A review of a variety of known results, especially for networks with one hidden layer, can be found in Pinkus (1999). More recently, these results have been improved by a factor of $n^{1∕d}$ in Klusowski and Barron (2016) using the idea of stratified sampling, based in part on the techniques in Makovoz (1996).

Our work, like much of the previous work, focuses on the case of two-layer neural networks. A two layer neural network can be written in the following particularly simple way

$$f(x;\theta )=\sum _{i=1}^n{\beta }_i\sigma ({\omega }_i\cdot x+b_i),$$

where $\theta =\{{\omega }_1,\dots ,{\omega }_n,b_1,\dots ,b_n\}$ are parameters and $n$ is the number of hidden neurons in the model.

In this work, we study the how the approximation properties of two-layer neural networks depend on the number of hidden neurons. In particular, we consider the class of functions where the number of hidden neurons is bounded,

$${\Sigma }_d^n\left(\sigma \right)=\left\{\sum _{i=1}^n{\beta }_i\sigma ({\omega }_i\cdot x+b_i):{\omega }_i\in {\mathbb{R}}^d,b_i,{\beta }_i\in \mathbb{R}\right\},$$

and prove Theorem 2 concerning the order of approximation as $n\to \infty$ for activation functions with polynomial decay, and Theorem 3, which applies to neural networks with periodic activation functions. Our results make the assumption that the function to be approximated, $f:{\mathbb{R}}^d\to \mathbb{R}$, has bounded Barron norm

$${\Vert f\Vert }_{{ℬ}^s}={\int }_{{\mathbb{R}}^d}{(1+\left|\omega \right|)}^s\left|\hat{f}\left(\omega \right)\right|d\omega ,$$

and we consider the problem of approximating $f$ on a bounded domain $\Omega$. This is a significantly weaker assumption than the strong smoothness assumption made in Kainen, Kurkova, and Vogt (2007) and Petrushev (1998). Similar results appear in Barron (1993) and Hornik et al. (1994), but we have improved their bound by a logarithmic factor for exponentially decaying activation functions and generalized these results to polynomially decaying activation functions. We also leverage this result to obtain a somewhat worse, though still dimension independent, approximation rate without the polynomial decay condition. This result ultimately applies to every activation function of bounded variation. Finally, we extend the stratified sampling argument in Klusowski and Barron (2016) to polynomially decaying activation functions in Theorem 5 and to periodic activation functions in Theorem 6. This gives an improvement on the asymptotic rate of convergence under mild additional assumptions.

The paper is organized as follows. In the second section, we discuss some basic results concerning the Fourier transform. We use these results to provide a simplified proof using Fourier analysis of the density result in Leshno et al. (1993) under the mild additional assumption of polynomial growth on $\sigma$. Then, in the third section, we study the order of approximation and prove Theorem 2, Theorem 3, extending the result in Barron (1993) and Hornik et al. (1994) to polynomially decaying activation functions, respectively periodic activation functions, and removing a logarithmic factor in the rate of approximation. In the fourth section, we provide a new argument using an approximate integral representation to obtain dimension independent results without the polynomial decay condition in Theorem 4. In the fifth section, we use a stratified sampling argument to prove Theorem 5, Theorem 6, which improves upon the convergence rates in Theorem 2, Theorem 3 under mild additional assumptions. This generalizes the results in Klusowski and Barron (2016) to more general activation functions. Finally, we give concluding remarks and further research directions in the conclusion.