Nonparametric Regression

Abstract

In this chapter we will study nonparametric regression, also known as “learning a function” in the jargon of machine learning. We are given n pairs of observations (x ₁, Y ₁), . . ., (x _n, Y _n) as in Figures 5.1, 5.2 and 5.3. The response variable Y is related to the covariate x by the equations

$$ Y_i = r(x_i + \in _i ),{\text{ }}\mathbb{E}{\text{(}} \in _i {\text{)}} = {\text{0, }}i = 1, \ldots ,n $$

((5.1))

where r is the regression function. The variable x is also called a feature. We want to estimate (or “learn”) the function r under weak assumptions. The estimator of r(x) is denoted by $\widehat{r_n }(x)$. We also refer to $\widehat{r_n }(x)$ as a smoother. At first, we will make the simplifying assumption that the variance $\mathbb{V}{\text{(}} \in _i {\text{) = }}\sigma ^{\text{2}}$ does not depend on x. We will relax this assumption later.