Asymptotic properties of parallel Bayesian kernel density estimators

Abstract

In this article, we perform an asymptotic analysis of Bayesian parallel kernel density estimators introduced by Neiswanger et al. (in: Proceedings of the thirtieth conference on uncertainty in artificial intelligence, AUAI Press, pp 623–632, 2014). We derive the asymptotic expansion of the mean integrated squared error for the full data posterior estimator and investigate the properties of asymptotically optimal bandwidth parameters. Our analysis demonstrates that partitioning data into subsets requires a non-trivial choice of bandwidth parameters that optimizes the estimation error.

Appendix

1.1 Kernel density estimators and asymptotic error analysis

In this section, we will use the following notation. The function f denotes a probability density, and its kernel density estimator is given by

$$\begin{aligned} \hat{f}(x; X_1, X_2,\ldots , X_N, h )=\frac{1}{N h}\sum _{i=1}^{N}K \left( \frac{x-X_{i}}{h}\right) . \end{aligned}$$

(33)

where $X_1,X_2,\ldots X_n \sim f$ are i.i.d. samples.

Lemma 6

(Bias expansion) Let K satisfy (H3) and (H4). Let f be a probability density function satisfying (H5) and (H6). Let $\widehat{f}_{n,h}(x)$ be an estimation of f given by (33). Then,

• (i) $\text {bias}(\widehat{f}_{n,h})$ is given by

  $$\begin{aligned} \begin{aligned} \big [\text {bias}(\widehat{f}_{n,h})\big ] (x)&= \mathbb {E}\big [\widehat{f}_{n,h}(x)\big ]-f(x)=\frac{h^2k_2f''(x)}{2}+ {[E_b(f,K)](x ; h)}, \end{aligned} \end{aligned}$$

  (34)

  where

  $$\begin{aligned} E_b(x;h) :=\int _{\mathbb {R}}K(t)\left( \int _{x}^{x-ht}\frac{f'''(z)(x-ht -z)^2}{2}\,\mathrm{d}z\right) \,\mathrm{d}t. \end{aligned}$$

• (ii) For all $n\ge 1$ and $h>0$, the term $E_b(\cdot ; n,h)$ satisfies the bounds

  $$\begin{aligned} \begin{aligned} |E_b(x ; h)| \,&\le \, \frac{C k_3}{6}h^3, \quad x\in \mathbb {R},\\ \int _{\mathbb {R}} |E_b(x ; h)|\, \mathrm{d}x \,&\le C\frac{k_3}{6} h^3 ,\\ \int _{\mathbb {R}} |E_b(x ; n,h)|^2\, \mathrm{d}x \,&\le \,\frac{C^2 k_3^2}{36} h^6, \end{aligned} \end{aligned}$$

  (35)

  for some constant C.

• (iii) The square-integrated $bias(\widehat{f}_{n,k})$ satisfies

  $$\begin{aligned} \int _{\mathbb {R}} \text {bias}^2(\widehat{f}_{n,k}) \, \mathrm{d}x \, = \, \frac{h^4 k_2^2}{4}\int _{\mathbb {R}} (f''(x))^2 \, \mathrm{d}x + \mathcal {E}_b(n,h) \, < \, \infty \end{aligned}$$

  with

  $$\begin{aligned} |\mathcal {E}_b(n,h)| \le C_b \left( k_2 + \frac{k_3}{6}h\right) \frac{k_3 h^5}{6}, \end{aligned}$$

  (36)

  for some constant $C_b$, and all $n \ge 1$, $h>0$.

Proof

Using (33) and the fact that $X_i$, $i=1,\ldots ,n$ are i.i.d., we obtain

$$\begin{aligned} \begin{aligned} \text {bias}_{n,h}(x)&=\mathbb {E}\big [\widehat{f}_{n,h}(x)\big ]-f(x) \\&=\frac{1}{h}\mathbb {E}\left[ K\left( \frac{x-X_1}{h}\right) \right] -f(x)\\&=\frac{1}{h}\int _{\mathbb {R}}K\left( \frac{x-y}{h}\right) f(y)\,\hbox {d}y-f(x)\\&=\int _{\mathbb {R}}K(t)(f(x-ht)-f(x))\,\hbox {d}t, \end{aligned} \end{aligned}$$

where we used the substitution $t=(x-y)/h$. Employing Taylor’s theorem with an error term in integral form and using (H3), we get

$$\begin{aligned} \begin{aligned}&\text {bias}_{n,h}(x) \\&\quad =\int _{\mathbb {R}}K(t)\left( -htf'(x)+\frac{h^2t^2}{2}f''(x) + \int _{x}^{x-ht}\frac{f'''(z)(x-ht-z)^2}{2}\,\hbox {d}z\right) \,\hbox {d}t\\&\quad =\frac{h^2f''(x)}{2}\int _{\mathbb {R}}t^2K(t)\,\hbox {d}t+\int _{\mathbb {R}}K(t)\left( \int _{x}^{x-ht}\frac{f'''(z)(x-ht-z)^2}{2}\,\hbox {d}z \right) \,\hbox {d}t, \end{aligned} \end{aligned}$$

which proves (i).

By (H4), we have

$$\begin{aligned} |E_b(x ; n,h)|\leqslant C\left( \int _{\mathbb {R}}K(t)\left| \int _{x}^{x-ht}\frac{(x-ht -z)^2}{2}\,\hbox {d}z\right| \, \hbox {d}t\right) = \frac{C k_3 }{6} h^3, \end{aligned}$$

(37)

and by (H6), using the substitution $\alpha =x-ht-z$ and employing Tonelli’s theorem, we obtain

$$\begin{aligned} \begin{aligned}&\int _{\mathbb {R}} |E_b(x ; n,h)|\,\hbox {d}x \\&\quad \le \, \int _{\mathbb {R}}\int _{\mathbb {R}}K(t)\int _{x-\frac{h}{2}(|t|+t)}^{x+\frac{h}{2}(|t|-t)}\frac{|f'''(z)|(x-ht -z)^2}{2}\,\hbox {d}z\,\hbox {d}t\,\hbox {d}x\\&\quad = \, \int _{\mathbb {R}}K(t)\int _{-\frac{h}{2}(|t|+t)}^{\frac{h}{2}(|t|-t)}\left( \left( \int _{\mathbb {R}}|f'''(x-ht-\alpha )|\,\hbox {d}x\right) \frac{\alpha ^2}{2} \right) \,\hbox {d}\alpha \,\hbox {d}t\\&\quad \le \, C\int _{\mathbb {R}}K(t)\left( \int _{-\frac{h}{2}(|t|-t)}^{\frac{h}{2}(|t|+t)}\frac{\alpha ^2}{2}\,\hbox {d}\alpha \,\right) \hbox {d}t =\frac{h^3}{6}C k_3. \end{aligned} \end{aligned}$$

(38)

Thus, combining the two bounds above, we conclude

$$\begin{aligned} \int _{\mathbb {R}} |E_b(x ; n,h)|^2 \, \hbox {d}x \le \frac{C k_3 }{6}h^3 \int _{\mathbb {R}} |E_b(x ; n,h)| \, \hbox {d}x \le \frac{C^2 k_3^2 }{36}h^6. \end{aligned}$$

Observe that

$$\begin{aligned} \begin{aligned} \text {bias}^2(\widehat{f}_{n,h})(x) = \frac{h^4k_2^2 }{4}(f''(x))^2 + h^2k_2 f''(x) E_b(x ; n,h) + E_b^2(x; n,h). \end{aligned} \end{aligned}$$

(39)

By (H5), (37), and (38)

$$\begin{aligned} \begin{aligned} \big |\mathcal {E}_b(n,h)\big |&:=\left| \int _{\mathbb {R}} \left( h^2k_2 f''(x)E_b(x ; n,h) + E_b^2(x; n,h)\right) \, \hbox {d}x \right| \\&\le \left( h^2k_2 C + \frac{C k_3}{6}h^3\right) \int _{\mathbb {R}} |E_b(x ; n,h)|\\&\le \left( h^2k_2 C + \frac{C k_3}{6}h^3\right) \frac{h^3}{6}C k_3. \end{aligned} \end{aligned}$$

(40)

By (H5) and (H6), we have $\int _{\mathbb {R}} (f''(x))^2 \, \hbox {d}x < \infty$. Hence, by setting $C_b=C^2$, using (39) and (40), we obtain (36). $\square$

Lemma 7

(Variation expansion) Let K satisfy (H3) and (H4), with $r=2$. Let f satisfy (H5) and (H6), and $\widehat{f}_{n,h}(x)$ be the estimator of f given by (33). Then,

• (i) $\mathbb {V}(\widehat{f}_{n,h})$ is given by

  $$\begin{aligned} \big [\mathbb {V}(\widehat{f}_{n,h})\big ](x) =f(x)\frac{1}{nh}\int _{\mathbb {R}}K^2(t)\,\mathrm{d}t+ E_{V}(x ; n,h), \quad x \in \mathbb {R}\end{aligned}$$

  (41)

  with

  $$\begin{aligned} \begin{aligned} E_V(x;n,h)&= -\frac{1}{n}\left( \int _{\mathbb {R}} t K^2(t) \int _{0}^{1} f'(x-htu)\, \mathrm{d}u \,\mathrm{d}t \right. \\&\left. \quad +\Big (f(x)+\text {bias}(\widehat{f}_{n,h})(x)\right) ^2 \bigg ). \end{aligned} \end{aligned}$$

  (42)

• (ii) The term $E_V(x ; n,h)$ satisfies

  $$\begin{aligned} \begin{aligned} \mathcal {E}_V(n,h)&=\left| \int _{\mathbb {R}} E_V(x)\,\mathrm{d}x\right| \\&\le \frac{C_V}{n} \left( 2 + h^2 k_2 + \big (k_2 + \frac{k_3}{3}h\big ) \frac{h^5}{6}k_3\right) . \end{aligned} \end{aligned}$$

  (43)

Proof

Using (34) and the fact that $X_i$, $i=1,\dots ,n$ are i.i.d., we obtain

$$\begin{aligned} \mathbb {V}(\widehat{f}_{n,h}(x))= & {} \mathbb {V}\left( \frac{1}{h}K\left( \frac{x-X_1}{h}\right) \right) \\= & {} \frac{1}{n}\int _{\mathbb {R}}\frac{1}{h^2}K^2 \left( \frac{x-y}{h}\right) f(y)\,\hbox {d}y-\frac{1}{n}\left( \int _{\mathbb {R}}\frac{1}{h}K\left( \frac{x-y}{h}\right) f(y)\,\hbox {d}y\right) ^2\\= & {} \frac{1}{nh}\int _{\mathbb {R}}K^2(t)f(x-ht)\,\hbox {d}t -\frac{1}{n}\Big (f(x)+\text {bias}(\widehat{f}_{n,h})(x)\Big )^2\\= & {} \frac{1}{nh}\int _{\mathbb {R}}K^2(t)f(x)\,\hbox {d}t +\frac{1}{nh}\int _{\mathbb {R}}K^2(t)\left( \int _{x}^{x-ht}f'(z)\,\hbox {d}z\,\right) \hbox {d}t \\&-\frac{1}{n}\Big (f(x)+\text {bias}(\widehat{f}_{n,h})(x)\Big )^2\\= & {} \frac{1}{nh}\int _{\mathbb {R}}K^2(t)f(x)\,\hbox {d}t -\frac{1}{n}\int _{\mathbb {R}}tK^2(t)\int _{0}^{1}f'(x-htu)\,\hbox {d}u\,\hbox {d}t \\&-\frac{1}{n}\Big (f(x)+\text {bias}(\widehat{f}_{n,h})(x)\Big )^2,\\ \end{aligned}$$

which proves (41) and (42).

We next estimate the terms

$$\begin{aligned} \begin{aligned} E_1(x)&:=\int _{\mathbb {R}}t K^2(t)\bigg (\int _{0}^{1}f'(x-htu)\,\hbox {d}u\bigg )\,\hbox {d}t, \\ E_2(x)&:= \Big (f(x) + \text {bias}(\widehat{f}_{n,h})(x)\Big )^2. \end{aligned} \end{aligned}$$

Observe that (H5)–(H6) imply

$$\begin{aligned} \int _{\mathbb {R}} |f'(x)|\, \hbox {d}x = \int _{\mathbb {R}} |f'(x+\alpha )|\, \hbox {d}x :=I_1\, < \, \infty \end{aligned}$$

for any $\alpha \in \mathbb {R}$. Then, using Tonelli’s Theorem and (H4), we obtain

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}} |E_1(x)|\, \hbox {d}x&\le \int _{\mathbb {R}} |t|K^2(t)\left( \int _{\mathbb {R}} \int _{0}^{1} |f'(x-htu)|\,\hbox {d}u\,\hbox {d}x \right) \, \hbox {d}t\\&\le \int _{\mathbb {R}} |t|K^2(t) \left( \int _{0}^{1} \left( \int _{\mathbb {R}} |f'(x-htu)|\,\hbox {d}x \right) \,\hbox {d}u \right) \, \hbox {d}t \le I_1 k_1.\\ \end{aligned} \end{aligned}$$

Since $E_1$ is integrable, we can use Fubini’s theorem, and this yields

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}} E_1(x)\, \hbox {d}x&= \int _{\mathbb {R}} t K^2(t)\left( \int _{\mathbb {R}} \int _{0}^{1} f'(x-htu)\,\hbox {d}u\,\hbox {d}x \right) \, \hbox {d}t\\&= \int _{\mathbb {R}} tK^2(t) \left( \int _{0}^{1} \left( \int _{\mathbb {R}} f'(x-htu)\,\hbox {d}x \right) \,\hbox {d}u \right) \, \hbox {d}t = 0, \end{aligned} \end{aligned}$$

where we used the fact that $\lim _{x\rightarrow \pm \infty }f(x) = 0$. Next, by (H5) and (35), we get

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}} |E_2(x)| \, \hbox {d}x&\le 2 \int _{\mathbb {R}} \Big (f^2(x) + \text {bias}^2(\widehat{f}_{n,h})(x) \Big )\, \hbox {d}x\\&\le 2 C + C h^2 k_2 + \left( k_2 C + \frac{C k_3}{6}h\right) \frac{h^5}{3}C k_3. \end{aligned} \end{aligned}$$

Combining the above estimates, we obtain (43). $\square$

Lemma 8

(Kernel autocorrelation) Let K satisfy (H3) and (H4), then the function

$$\begin{aligned} K_2(z)=\int _{\mathbb {R}}K(s)K(s-z)\,\mathrm{d}s \ge 0, \quad z \in \mathbb {R}\end{aligned}$$

satisfies

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}}K_2(z)\,\mathrm{d}z&=1, \quad \int _{\mathbb {R}}z\,K_2(z)\,\mathrm{d}z&=0. \end{aligned} \end{aligned}$$

Moreover, for any sufficiently smooth f(x)

$$\begin{aligned} \int \frac{1}{h}K_2\left( \frac{z-x}{h}\right) f(z)\,\mathrm{d}z=f(x) +E_{C,f} \quad \text {with} \quad |E_{C,f}|\le \Vert f''\Vert _{\infty }k_2h^2. \end{aligned}$$

Proof

Since $K \ge 0$, we have $K_2 \ge 0$. Moreover, we have

$$\begin{aligned} \int _{\mathbb {R}}K_2(z)\,\hbox {d}z=\iint _{\mathbb {R}\times \mathbb {R}}K(s)K(s-z)\,\hbox {d}z\hbox {d}s=1 \end{aligned}$$

and this proves the first property. Similarly, recalling that $\int z K(z) \,\hbox {d}z =0$, we obtain

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}}z\,K_2(z)\,\hbox {d}z&=\int _\mathbb {R}K(s)\int _\mathbb {R}(z-s+s)K(s-z)\,\hbox {d}z\,\hbox {d}s=0. \end{aligned} \end{aligned}$$

Next, we take any smooth function f and compute

$$\begin{aligned} \begin{aligned}&\int \frac{1}{h}K_2\left( \frac{z-x}{h}\right) f(z)\,\hbox {d}z =\int K_2\left( u\right) f(x-hu)\,\hbox {d}u\\&\quad =f(x)+\int K_2(u)\int _{x}^{x-hu}f''(t)(t-x+hu)\,\hbox {d}t\,\hbox {d}u. \end{aligned} \end{aligned}$$

Finally, we estimate the last term in the above formula as follows:

$$\begin{aligned} \begin{aligned}&\left| \int K_2(u)\int _{x}^{x-hu}f''(t)(t-x+hu)\,\hbox {d}t\,\hbox {d}u\right| \\&\quad \le \Vert f''\Vert _{\infty }\int K_2(u)\frac{h^2u^2}{2}\,\hbox {d}u\\&\quad = \frac{ \Vert f''\Vert _{\infty }h^2}{2}\left( \int K(s)\int (s-u)^2K(s-u)\,\hbox {d}u\hbox {d}s \right. \\&\quad \qquad \qquad \qquad \qquad \left. +\int s^2K(s)\int K(s-u)\,\hbox {d}u\hbox {d}s\right) \\&\quad \le \Vert f''\Vert _{\infty }k_2h^2. \end{aligned} \end{aligned}$$

$\square$

Lemma 9

(Product expectation) Let K satisfy (H3) and (H4), with $r=2$. Let f be a probability density function that satisfies (H5) and (H6), and let $\widehat{f}_{n,h}(x)$ be an estimate of f given by (33). Then,

$$\begin{aligned} \mathbb {E}[\widehat{f}_{n,h}(x)\widehat{f}_{n,h}(y)]-\mathbb {E}[\widehat{f}_{n,h}(x)]\mathbb {E}[\widehat{f}_{n,h}(y)]=\frac{1}{Nh}f(x) K_2\left( \frac{x-y}{h} \right) -E_{\varPi }, \end{aligned}$$

(44)

where the error term

$$\begin{aligned} E_{\varPi }= & {} \frac{1}{N} \int \left( \, s K (s) K\left( s - \frac{x-y}{h} \right) \left( \int _{0}^{1}f'(x-shu)\,\mathrm{d}u \right) \right) \mathrm{d}s \\&+\,\frac{1}{N} \mathbb {E}[\widehat{f}(x)]\mathbb {E}[\widehat{f}(y)] \end{aligned}$$

satisfies

$$\begin{aligned} \begin{aligned}&|E_{\varPi }(x,y)|\le \frac{C_\varPi }{N}, \qquad \left| \int \int E_{\varPi }(x,y) \, \mathrm{d}x \mathrm{d}y\right| \le \frac{1}{N}\left( 1+\frac{Ck_3h^3}{6}\right) ^2\\&\int \int | E_{\varPi }(x,y)|\mathrm{d}x \mathrm{d}y \le \frac{1}{N}\left( 1+k_1\,C\frac{C k_2h^2}{2}+\frac{Ck_3h^3}{6}\right) ^2, \end{aligned} \end{aligned}$$

for some constant $C_\varPi$ and constants C given in (H6) and $K_2$ defined in Lemma 8.

Proof

By the definition of the estimator $\widehat{f}$, we have

$$\begin{aligned} \begin{aligned} \mathbb {E}\Big (\widehat{f}(x) \widehat{f}(y) \Big )&=\mathbb {E}\left( \frac{1}{N^2h^2}\sum _{i,j=1}^N K\left( \frac{x-X_i}{h}\right) K\left( \frac{y-X_j}{h}\right) \right) . \end{aligned} \end{aligned}$$

Since all $\{X_i\}_{i=1}^{N}$ are i.i.d., we can split the calculation into two parts, one for the part, where the indexes coincide and the part, where indexes are different. We then can use the independence of the samples to simplify the calculation

$$\begin{aligned} \begin{aligned} \mathbb {E}\Big (\widehat{f}(x) \widehat{f}(y) \Big )&=\frac{1}{N^2h^2} \mathbb {E}\left( \sum _{i=j} K\left( \frac{x-X_i}{h}\right) K\left( \frac{y-X_i}{h}\right) \right) \\&\quad +\frac{1}{N^2h^2} \mathbb {E}\left( \sum _{i \ne j} K\left( \frac{x-X_i}{h}\right) K\left( \frac{y-X_j}{h}\right) \right) \\&= \frac{1}{Nh^2}\left[ \mathbb {E}\left( K\left( \frac{x-X}{h}\right) K\left( \frac{y-X}{h} \right) \right) \right] \\&\quad + \Big (1-\frac{1}{N}\Big )\mathbb {E}[\widehat{f}(x)]\mathbb {E}[\widehat{f}(y)], \end{aligned} \end{aligned}$$

(45)

where $X=X_1$. The first expectation term in (45) can be expanded as

$$\begin{aligned} \begin{aligned}&\frac{1}{Nh^2}\mathbb {E}\left[ K\left( \frac{x-X}{h}\right) K\left( \frac{y-X}{h} \right) \right] \\&\quad = \frac{1}{Nh^2} \int \, K \left( \frac{x-t}{h}\right) K\left( \frac{y-t}{h} \right) \, f(t) \, \hbox {d}t\\&\quad =\frac{1}{Nh} \int \, K (s) K\left( s - \frac{x-y}{h} \right) \, \left( f(x)+\int _{x}^{x-sh}f'(z)\,\hbox {d}z \right) \, \hbox {d}s\\&\quad = f(x)\frac{1}{Nh} K_2\left( \frac{x-y}{h} \right) \\&\qquad -\frac{1}{N} \int \, s K (s) K\left( s - \frac{x-y}{h} \right) \, \left( \int _{0}^{1}f'\left( x-shu\right) \,\hbox {d}u \right) \, \hbox {d}s.\\ \end{aligned} \end{aligned}$$

Let us denote

$$\begin{aligned} E_{\varPi ,1}= & {} \frac{1}{N} \int \left( \, s K (s) K\left( s - \frac{x-y}{h} \right) \left( \int _{0}^{1}f'(x-shu)\,\hbox {d}u \right) \right) \hbox {d}s, \\ E_{\varPi ,2}= & {} \frac{1}{N}\mathbb {E}[\widehat{f}(x)]\mathbb {E}[\widehat{f}(y)]. \end{aligned}$$

Then, we obtain

$$\begin{aligned} \begin{aligned}&\mathbb {E}\Big (\widehat{f}_{n,h}(x) \widehat{f}_{n,h}(y) \Big ) - \mathbb {E}[\widehat{f}_{n,h}(x)]\mathbb {E}[\widehat{f}_{n,h}(y)]\\&\quad = f(x)\frac{1}{Nh} K_2 \left( \frac{x-y}{h} \right) \hbox {d}s-(E_{\varPi ,1}+E_{\varPi ,2}),\\ \end{aligned} \end{aligned}$$

and this establishes (44).

Observe that (H3), (H4), and (H5) imply

$$\begin{aligned} |E_{\varPi ,1}|\le \frac{C\,k_1}{N}. \end{aligned}$$

Next, according to (34) and (35)

$$\begin{aligned} |\mathbb {E}[\widehat{f}(x)]|\le C+\frac{C k_2 h^2}{2}+\frac{C k_3 h^3}{6} \quad \text {for all} \quad x\in \mathbb {R}, \end{aligned}$$

where C is a maximum of constants from (H5) and hence

$$\begin{aligned} |E_{\varPi ,2}| \le \frac{1}{N}\Big ( C+\frac{C k_2 h^2}{2}+\frac{C k_3 h^3}{6} \Big )^2. \end{aligned}$$

Combining the above estimate, we conclude that

$$\begin{aligned} |E_{\varPi }|=|E_{\varPi ,1}+E_{\varPi ,2}| \le \frac{1}{N}\left( Ck_1+\left( C+\frac{C k_2 h^2}{2}+\frac{C k_3 h^3}{6}\right) ^2\right) . \end{aligned}$$

To obtain bounds on the integral of the error term, let us consider each component of the error separately. The term $E_{\varPi ,1}$ is integrable

$$\begin{aligned} \begin{aligned}&\iint |E_{\varPi ,1}(x,y)|\,\hbox {d}x\hbox {d}y \\&\quad \le \frac{1}{N} \iiint _{\mathbb {R}^3} \, |s|\, K (s) K\left( s - \frac{x-y}{h} \right) \, \left( \int _{0}^{1}|f'\left( x-shu\right) |\,\hbox {d}u \right) \hbox {d}s\,\hbox {d}x\,\hbox {d}y\\&\quad \le \frac{1}{N} \int _{\mathbb {R}} \, |s|\, K (s) \, \left( \int _{0}^{1} \int _\mathbb {R}|f'\left( x-shu\right) |\,\hbox {d}x \,\hbox {d}u \right) \hbox {d}s\le \frac{k_1\,C}{N}. \end{aligned} \end{aligned}$$

(46)

Next using Fubini’s theorem, we obtain

$$\begin{aligned} \begin{aligned}&\left| \iint E_{\varPi ,1}(x,y)\,\hbox {d}x\hbox {d}y\right| \\&\quad \le \frac{1}{N} \left| \iiint _{\mathbb {R}^3} \, s\, K (s) K\left( s - \frac{x-y}{h} \right) \, \left( \int _{0}^{1}f'\left( x-shu\right) \,\hbox {d}u \right) \hbox {d}s\, \hbox {d}x\,\hbox {d}y\right| \\&\quad = \frac{1}{N} \left| \int _{\mathbb {R}} \, s\, K (s) \, \left( \int _{0}^{1}\int _\mathbb {R}f'\left( x-shu\right) \,\hbox {d}x\,\hbox {d}u \right) \, \hbox {d}s\right| =0. \end{aligned} \end{aligned}$$

Therefore, using Lemma 6, (34), (35), and the Hypothesis (H6), we obtain

$$\begin{aligned} \begin{aligned} \left| \iint _{\mathbb {R}^2} E_{\varPi }(x,y)\,\hbox {d}x\hbox {d}y\right|&=\frac{1}{N} \left| \int _{\mathbb {R}} \mathbb {E}[\widehat{f}(x)]\,\hbox {d}x\right| ^2 \le \frac{1}{N}\left( 1+\frac{Ck_3h^3}{6}\right) ^2. \end{aligned} \end{aligned}$$

Finally, directly from (46), (34), and (35), we obtain

$$\begin{aligned} \begin{aligned} \iint _{\mathbb {R}^2} \left| E_{\varPi }(x,y)\right| \,\hbox {d}x\hbox {d}y&\le \iint _{\mathbb {R}^2} \left| E_{\varPi ,1}(x,y)\right| \hbox {d}x\hbox {d}y+ \iint _{\mathbb {R}^2} \left| E_{\varPi ,2}(x,y)\right| \hbox {d}x\hbox {d}y\\&\le \frac{k_1\,C}{N} + \frac{1}{N}\left( 1+\frac{Ck_2h^2}{2}+\frac{Ck_3h^3}{6}\right) ^2 .\\ \end{aligned} \end{aligned}$$

$\square$

Theorem 3

(MISE expansion) Let K satisfy (H3) and (H4), with $r=2$. Let f be a probability density function that satisfies (H5) and (H6), and let $\widehat{f}_{n,h}(x)$ be an estimate of f given by (33). Then,

$$\begin{aligned} \mathrm{MISE}(\widehat{f}_{n,h}) =\frac{h^4k^2_2}{4}\int _{\mathbb {R}}(f''(x))^2\mathrm{d}x + \frac{1}{nh} f(x)\int _{\mathbb {R}} K^2(t) \,\mathrm{d}t + \mathcal {E}_b(n,h)+\mathcal {E}_V(n,h) \end{aligned}$$

with $\mathcal {E}_b$ and $\mathcal {E}_V$ defined in (40) and (43), respectively. Moreover, for every $H>0$, there exists $C_{f,K,H}$ such that

$$\begin{aligned} |\mathcal {E}_b(h,n)+\mathcal {E}_V(h,n)| \, \leqslant \, C_{f,K,H} \Big ( h^5 + \frac{1}{n}\Big ) \end{aligned}$$

for all $n \ge 1$ and $H \ge h > 0$.

Proof

It is easy to show (see Silverman 1986) that

$$\begin{aligned} \begin{aligned} \mathrm{MISE}(\widehat{f}_{n,h})&=\int _{\mathbb {R}}\mathbb {E}[\widehat{f}_{n,h}(x)-f(x)]^2\,\hbox {d}x\\&= \int _{\mathbb {R}}\big (\text {bias}(\widehat{f}_{n,h})(x)\big )^2\, \hbox {d}x +\int _{\mathbb {R}}\mathbb {V}(\widehat{f}_{n,h}(x))\,\hbox {d}x, \end{aligned} \end{aligned}$$

and hence the result follows from Lemma 6 and Lemma 7. $\square$