Optimal upper and lower bounds for the true and empirical excess risks in heteroscedastic least-squares regression

We consider the estimation of a bounded regression function with nonparametric heteroscedastic noise and random design. We study the true and empirical excess risks of the least-squares estimator on finite-dimensional vector spaces. We give upper and lower bounds on these quantities that are nonasymptotic and optimal to first order, allowing the dimension to depend on sample size. These bounds show the equivalence between the true and empirical excess risks when, among other things, the least-squares estimator is consistent in sup-norm with the projection of the regression function onto the considered model. Consistency in the sup-norm is then proved for suitable histogram models and more general models of piecewise polynomials that are endowed with a localized basis structure.


Introduction
A few years ago, Birgé and Massart [6] introduced a data-driven calibration method for penalized criteria in model selection, called the Slope Heuristics. Their algorithm is based on the concept of the minimal penalty, under which a model selection procedure fails. Given the shape of the ideal penalty, which in their Gaussian setting is a known function of the dimension of the considered models, the algorithm first provides a data-driven estimate of the minimal penalty. This is done by taking advantage of a sudden change in the behavior of the model selection procedure around this level of penalty. Then, the algorithm selects a model by using a penalty that is twice the estimated minimal penalty. Birgé and Massart prove in [6] that an asymptotically optimal penalty is twice the minimal one, in the sense that the associated selected model achieves a nonasymptotic oracle inequality with leading constant converging to one when the sample size tends to infinity.
The slope heuristics algorithm has been recently extended by Arlot and Massart [2] to the selection of M-estimators, whenever the number of models is not more than polynomial in the sample size. Arlot and Massart highlight that, in this context, the mean of the empirical excess risk on each model should be a good, rather general candidate for the -unknown -minimal penalty. In addition, they note that an optimal penalty is roughly given by the sum of the true and the empirical excess risks on each model. A key fact underlying the asymptotic optimality of the slope heuristics algorithm is the equivalence -in the sense that the ratio tends to one when the sample size tends to infinity -between the true and empirical excess risk, for each model which is likely to be selected. Generally, these models are of moderate dimension, typically between (log (n)) c and n/ (log (n)) c , where c is a positive constant and n is the sample size. This equivalence leads, quite straightforwardly, to the factor two between the minimal penalty and the optimal one.
Arlot and Massart prove in [2], by considering the selection of finite-dimensional models of histograms in heteroscedastic regression with a random design, that the slope heuristics algorithm is asymptotically optimal. The authors conjecture in [2], Section 1, that the restriction to histograms is "mainly technical", and that the slope heuristics "remains valid at least in the general least squares regression framework".
The first motivation of the present paper is thus to tackle the challenging mathematical problem raised by Arlot and Massart in [2], concerning the validity slope heuristics. More precisely, we isolate the question of the equivalence, for a fixed model, between the true and empirical excess risks. As emphasized in [2], this constitutes the principal part of the conjecture, since other arguments leading to model selection results are now well understood. We thus postpone model selection issues to a forthcoming paper, and focus on the fixed model case.
We consider least squares regression with heteroscedastic noise and random design, using a finite-dimensional linear model. Our analysis is nonasymptotic in the sense that our results are available for a fixed value of the sample size. It is also worth noticing that the dimension of the considered model is allowed to depend on the sample size and consequently is not treated as a constant of the problem. In order to determine the possible equivalence between the true and empirical excess risks, we investigate upper and lower deviation bounds for each quantity. We obtain first order optimal bounds, thus exhibiting the first part of the asymptotic expansion of the excess risks. This requires to determine not only the right rates of convergence, but also the optimal constant on the leading order term. We give two examples of models that satisfy our conditions: models of histograms and models of piecewise polynomials, whenever the partition defining these models satisfy some regularity condition with respect to the unknown distribution of data. Our results concerning histograms roughly recover those derived for a fixed model by Arlot and Massart [2], but with different techniques. Moreover, the case of piecewise polynomials strictly extend these results, and thus tends to confirm Arlot and Massart conjecture on the validity of the slope heuristics.
We believe that our deviation bounds, especially those concerning the true excess risk, are interesting by themselves. Indeed, the optimization of the excess risk is, from a general perspective, at the core of many nonparametric approaches, especially those related to statistical learning theory. Hence, any sharp control of this quantity is likely to be useful in many contexts.
In the general bounded M-estimation framework, rates of convergence and upper bounds for the excess risk are now well understood, see [18], [17], [13], [4], [10]. However, the values of the constants in these deviation bounds are suboptimal -or even unknown -, due in particular to the use of chaining techniques. Concerning lower deviation bounds, there is no convincing contribution to our knowledge, except the work of Bartlett and Mendelson [4], where an additional assumption on the behavior of underlying empirical process is used to derive such a result. However, this assumption is in general hard to check.
More specific frameworks, such as least squares regression with a fixed design on linear models (see for instance [6], [3] and [1]), least squares estimation of density on linear models (see [?] and references therein), or least squares regression on histograms as in [2], allow for sharp, explicit computations that lead to optimal upper and lower bounds for the excess risks. Hence, a natural question is: is there a framework, between the general one and the special cases, that would allow to derive deviation bounds that are optimal at the first order ? In other words, how far could optimal results concerning deviation bounds been extended ? The results presented in this article can be seen as a first attempt to answer these questions.
The article is organized as follows. We present the statistical framework in Section 2, where we show in particular the existence of an expansion of the least squares regression contrast into the sum of a linear and a quadratic part. In Section 3, we detail the main steps of our approach at a heuristic level and give a summary of the results presented in the paper. We then derive some general results in Section 4. These theorems are then applied to the case of histograms and piecewise polynomials in Sections 5 and 6 respectively, where in particular, explicit rates of convergence in sup-norm are derived. Finally, the proofs are postponed to the end of the article.

Regression framework and notations 2.1 least squares estimator
Let (X , T X ) be a measurable space and set Z = X ×R. We assume that ξ i = (X i, Y i ) ∈ X ×R, i ∈ {1, ..., n}, are n i.i.d. observations with distribution P . The marginal law of X i is denoted by P X . We assume that the data satisfy the following relation where s * ∈ L 2 P X , ε i are i.i.d. random variables with mean 0 and variance 1 conditionally to X i and σ : X −→R is a heteroscedastic noise level. A generic random variable of law P , independent of (ξ 1 , ..., ξ n ), is denoted by ξ = (X , Y ) . Hence, s * is the regression function of Y with respect to X, to be estimated. Given a finite dimensional linear vector space M , that we will call a "model", we denote by s M the linear projection of s * onto M in L 2 P X and by D the linear dimension of M .
We consider on the model M a least squares estimator s n (possibly non unique), defined as follows s n ∈ arg min So, if we denote by the empirical distribution of the data and by K : L 2 P X −→ L 1 (P ) the least squares contrast, defined by K (s) = (x, y) ∈ Z → (y − s (x)) 2 , s ∈ L 2 P X , we then remark that s n belongs to the general class of M-estimators, as it satisfies s n ∈ arg min s∈M {P n (K (s))} .

Excess risk and contrast
As defined in (3), s n is the empirical risk minimizer of the least squares contrast. The regression function s * can be defined as the minimizer in L 2 P X of the mean of the contrast over the unknown law P , s * = arg min is called the risk of the function s. In particular we have P Ks * = E σ 2 (X) . We first notice that for any s ∈ L 2 P X , if we denote by its quadratic norm, then we have, by (1) above, The quantity P Ks − P Ks * is called the excess risk of s. Now, if we denote by s M the linear projection of s * onto M in L 2 P X , we have P Ks M − P Ks * = inf and for all s ∈ M P X (s · (s M − s * )) = 0 .
From (4), we deduce that s M = arg min s∈M P K (s) .
Our goal is to study the performance of the least squares estimator, that we measure by its excess risk. So we are mainly interested in the random quantity P (Ks n − Ks * ) . Moreover, as we can write P (Ks n − Ks * ) = P (Ks n − Ks M ) + P (Ks M − Ks * ) we naturally focus on the quantity P (Ks n − Ks M ) ≥ 0 that we want to bound from upper and from below, with high probability. We will often call this last quantity the excess risk of the estimator on M or the true excess risk of s n , in opposition to the empirical excess risk for which the expectation is taken over the empirical measure, The following lemma establishes the expansion of the regression contrast around s M on M . This expansion exhibits a linear part and a quadratic parts.

Outline of the approach
Having introduced the framework and notations in Section 2 above, we are now able to explain more precisely the major steps of our approach to the problem of deriving optimal upper and lower bounds for the excess risks. As mentioned in the introduction, one of our main motivations is to determine whether the true excess risk is equivalent to the empirical one or not: Indeed, such an equivalence is a keystone to justify the slope heuristics, a data-driven calibration method first proposed by Birgé and Massart [6] in a Gaussian setting and then extended by Arlot and Massart [2] to the selection of M-estimators. The goal of this section is twofold. Firstly, it helps the reader to understand the role of the assumptions made in the forthcoming sections. Secondly, it provides an outline of the proof of our main result, Theorem 2 below. We suggest the reader interested in our proofs to read this section before entering the proofs. We start by rewriting the lower and upper bound problems, for the true and empirical excess risks. Let C and α be two positive numbers. The question of bounding the true excess risk from upper and with high probability can be stated as follows: find, at a fixed α > 0, the smallest C > 0 such that We then write, by definition of the M-estimator s n as a minimizer of the empirical excess risk over the model M , where are subsets of the model M , localized in terms of excess risk. As a matter of fact, M C is the closed ball of radius C in (M, · 2 ). In the same manner, the question of bounding the true excess risk from below and with high probability is formalized as follows: find the larger C > 0 such that We then have, by definition of the M-estimator s n , Expressions obtained in (12) and (13) and also The study of the excess risk thus reduces to the control of the following suprema, on the spheres D L of radius L in (M, · 2 ), of the empirical process indexed by contrasted increments of functions in M around the projection s M of the target, sup Similarly, the empirical excess risk can be written, by definition of the M-estimator s n , Hence, the study of the empirical excess risk reduces again to the control of the quantities given in (16). As these quantities are (local) suprema of an empirical process, we can handle, under the right hypotheses, the deviations from their mean via the use of concentration inequalities -deviations from the right being described with optimal constants by Bousquet inequality (Bousquet, [8], recalled in Section 7.5 at the end of the present paper) and deviations from left being controlled with sharp constants by Klein and Rio inequality (Klein and Rio [12], also recalled in Section 7.5). We can thus expect that, under standard assumptions, the deviations are negligible compared to the means with large enough probability, at least for radii L not too small, Remark 1 It is worth noting that the above computations, which allow to investigate both upper and lower bound problems, only rely on the definition of s n as a minimizer of the empirical risk over the model M , and not on the particular structure of the least squares contrast. Thus, formula (12), (13), (14), (15) and (17) are general facts of M-estimation -whenever the projection s M of the target onto the model M exists. Moreover, although presented in a quite different manner, our computations related to the control of the true excess risk are in essence very similar to those developed by Bartlett and Mendelson in [4], concerning what they call "a direct analysis of the empirical minimization algorithm". Indeed, the authors highlight in Section 3 of [4] that, under rather mild hypotheses, the true excess risk is essentially the maximizer of the function V n (L) − L, where we set Now, combining (12), (13), (14) and (15), it is easily seen that in the case where s n is unique and where we have in fact the following exact formula, So, if (18) is satisfied with high probability, we recover Bartlett and Mendelson's observation, which is In Theorem 3.1 of [4], a precise sense is given to (20), in a rather general framework. In particular, a lower bound for the excess risk is given but only through an additional condition controlling the supremum of the empirical process of interest itself over a subset of functions of "small" excess risks. This additional condition remains the major restriction concerning the related result of Bartlett and Mendelson. In the following, we show in our more restricted framework how to take advantage of the linearity of the model, as well as the existence of an expansion of the least squares contrast around the projection s M of the target, to derive lower bounds without additional assumptions on the behavior of the empirical process of interest. Moreover, our methodology allow to explicitly calculate the first order of the quantity given at the right side of (20), thus exhibiting a rather simple complexity term controlling the rate of convergence of the excess risk in the regression setting and relating some geometrical characteristics of the model M to the unknown law P of data. (17) and (19) above show that the true and empirical excess risks are of different nature, in the sense that the first one is referred to the arguments of the function

Remark 2 Formula
whereas the second one is measured from the values of the function Γ n . Hence, the equivalence between the true and the empirical excess risks, when satisfied, is in general not straightforward. It is a consequence of the following "fixed point type" equation, Considering that the approximation stated in (18) is suitably satisfied, it remains to get an asymptotic first order expansion of its right-hand term. Such a control is obtained through the use of the least squares contrast expansion given in (6). Indeed, using (6), we get In order to show that the residual term is negligible compared with the principal part, it is natural to use a contraction principle (see Theorem 4.12 of [15], also recalled in Section 7.5). Indeed, arguments of the empirical process appearing in the residual term are related to the square of the arguments defining the empirical process in the principal part. Moreover, it appears by using the contraction principle, that the ratio of the residual term over the principal part is roughly given by the supremum norm of the indexes: sup s∈DL |(s − s M ) (x)| (see Lemma 14 in Section 7.4 for more details). Now, using assumption (H3) of Section 4.1, concerning the unit envelope of the linear model M , we get that the last quantity is of order √ DL.
Since the values L of interest are typically of order D/n, the quantity controlling the ratio is not sharp enough as it does not converge to zero as soon as the dimension D is of order at least √ n. We thus have to refine our analysis in order to be able to neglect the residual term. The assumption of sup-norm consistency, of the least squares estimator s n toward the projection s M of the target onto the model M , appears here to be essential. Indeed, if assumption (H5) of Section 4.1 is satisfied, then all the above computations can be restricted with high probability to the subset where belongs the estimator s n , this subset being more precisely R n,D,α ≪ 1 being the rate of convergence in sup-norm of s n toward s M , defined in (H5). In particular, the spheres of interest D L are now replaced in the calculations by their intersectionD L with the ball of radius The ratio between the consequently modified residual term and principal part of (21) is then roughly controlled by R n,D,α (see again Lemma 14 in Section 7.4), a quantity indeed converging to zero as desired. Hence, under the assumption (H5), we get A legitimate and important question is: how restrictive is assumption (H5) of consistency in sup-norm of the least squares estimator ? We prove in Lemma 5 of Section 5 that this assumption is satisfied for models of histograms defined on a partition satisfying some regularity condition, at a rate of convergence of order D ln (n) /n. Moreover, in Lemma 8, Section 6, we extend this result for models of piecewise polynomials uniformly bounded in their degrees, again under some lower-regularity assumption on the partition defining the model; the rate of convergence being also preserved. A systematical study of consistency in sup-norm of least squares estimators, on more general finite-dimensional linear models, is also postponed to a forthcoming paper.
The control of the right-hand side of (23), which is needed to be sharp, is particularly technical, and is essentially contained in Lemmas 12 and 13 of Section 7.4. Let us shortly describe the mathematical figures underlying this control. First, by bounding the variance of the considered supremum of the empirical process -by using a result due to Ledoux [14], see Theorem 24 and also Corollary 25 in Section 7.5 -, we roughly get, for values of L of interest, Then, by assuming that the model M is fulfilled with a localized orthonormal basis, as stated in assumption (H4) of Section 4.1, it can be shown that the localization on the ball B L∞ (s M , R M,n,α ) can be removed from the right-hand side of (24), in the sense that The property of localized basis is standard in model selection theory (see for instance Chapter 7 of [16]) and was first introduced by Birgé and Massart in [5], also for deriving sharp exponential bounds in a M-estimation context. We show in Lemmas 4 and 7 that this assumption is satisfied for models of histograms and piecewise polynomials respectively, when they satisfy a certain regularity assumption concerning the underlying partition. Finally, as D L is a sphere in (M, · 2 ), we simply get, by the use of Cauchy-Schwarz inequality, that the right-hand side of (25) is equal to (L/n) · D k=1 Var ψ 1,M · ϕ k , where (ϕ k ) D k=1 is an orthonormal basis of M . Gathering our arguments, we then obtain where K 2 1,M := 1 DM DM k=1 Var ψ 1,M · ϕ k . As shown in Section 4.3 below, the (normalized) complexity term K 1,M is independent of the choice of the basis (ϕ k ) D k=1 and is, under our assumptions, of the order of a constant. Concerning the empirical excess risk, we have In particular, the equivalence In Theorem 2 below, a precise, non-asymptotic sense, is given to equivalences described in (26) and (27). This is done under the structural constraints stated in conditions (H4) and (H5), for models of reasonable dimension. Moreover, we give in Theorem 3 upper bounds for the true and empirical excess risks, that are less precise than the bounds of Theorem 2, but that are also valid for models of small dimension. Corollaries of these theorems are given in the case of histograms and piecewise polynomials, in Corollaries 6 and 9 respectively. Indeed, we show that in these particular cases, our general conditions (H4) and (H5) essentially reduce to a simple lower-regularity assumption on the underlying partition.

True and empirical excess risk bounds
In this section, we derive under general constraints on the linear model M , upper and lower bounds for the true and empirical excess risk, that are optimal -and equal -at the first order. In particular, we show that the true excess risk is equivalent to the empirical one when the model is of reasonable dimension. For smaller dimensions, we only achieve some upper bounds.

Main assumptions
We turn now to the statement of some assumptions that will be needed to derive our results in Section 4.2. These assumptions will be further discussed in Section 4.3.

Boundedness assumptions:
• (H1) The data and the linear projection of the target onto M are bounded: a positive finite constant A exists such that |Y i | ≤ A a.s. and Hence, from (H1) we deduce that and that there exists a constant σ max > 0 such that Moreover, as ψ 1,M (z) = −2 (y − s M (x)) for all z = (x, y) ∈ Z, we also deduce that • (H2) The heteroscedastic noise level σ is uniformly bounded from below: a positive finite constant σ min exists such that 0 < σ min ≤ σ (X i ) a.s.
Models with localized basis in L 2 P X : Let us define a function Ψ M on X , that we call the unit envelope of M , such that |s (x)| .
As M is a finite dimensional real vector space, the supremum in (33) can also be taken over a countable subset of M , so Ψ M is a measurable function.
• (H3) The unit envelope of M is uniformly bounded on X : a positive constant A 3,M exists such that The following assumption is stronger than (H3).

Remark 3 (H4) implies (H3) and in that case
The assumption of consistency in sup-norm: In order to handle second order terms in the expansion of the contrast (6), we assume that the least squares estimator is consistent for the sup-norm on the space X . More precisely, this requirement can be stated as follows.
• (H5) Assumption of consistency in sup-norm: for any then for every α > 0, we can find a positive integer n 1 and a positive constant A cons satisfying the following property: there exists R n,D,α > 0 depending on D, n and α, such that and by setting it holds for all n ≥ n 1 ,

Theorems
We state here the general results of this article, that will be applied in Section 5 and 6 in the case of piecewise constant functions and piecewise polynomials respectively.
They are optimal at the first order since the leading constants are equal for upper and lower bounds. Moreover, Theorem 2 states the equivalence with high probability of the true and empirical excess risks for models of reasonable dimensions. We notice that second orders are smaller for the empirical excess risk than for the true one. Indeed, when normalized by the first order, the deviations of the empirical excess risk are square of the deviations of the true one. Our bounds also give another evidence of the concentration phenomenon of the empirical excess risk exhibited by Boucheron and Massart [7] in the slightly different context of M-estimation with bounded contrast where some margin condition hold. Notice that considering the lower bound of the empirical excess risk given in (43), we do not need to assume the consistency of the least squares estimator s n towards the linear projection s M .
We turn now to upper bounds in probability for the true and empirical excess risks on models with possibly small dimensions. In this context, we do not achieve sharp or explicit constants in the rates of convergence.
Theorem 3 Let α, A + > 0 be fixed and let M be a linear model of finite dimension Assume that assumptions (H1), (H3) and (H5) hold. Then a positive constant A u exists, only depending on A, A cons , A 3,M and α, such that for all n ≥ n 0 (A cons , n 1 ), and Notice that on contrary to the situation of Theorem 2, we do not assume that (H2) hold. This assumption states that the noise level is uniformly bounded away from zero over the space X , and allows in Theorem 2 to derive lower bounds for the true and empirical excess risks, as well as to achieve sharp constants in the deviation bounds for models of reasonable dimensions. In Theorem 3, we just derive upper bounds and assumption (H2) is not needed. The price to pay is that constants in the rates of convergence derived in (44) and (45) are possibly larger than the corresponding ones of Theorem 2, but our results still hold true for small models. Moreover, in the case of models with reasonable dimensions, that is dimensions satisfying assumption (37) of Theorem 2, the rate of decay is preserved compared to Theorem 2 and is proportional to D/n. The proofs of the above theorems can be found in Section 7.3.

Some additional comments
Let us first comment on the assumptions given in Section 4.1. Assumptions (28) and (H2) are rather mild and can also be found in the work of Arlot and Massart [2] related to the case of histograms, where they are respectively denoted by (Ab) and (An). These assumptions state respectively that the response variable Y is uniformly bounded and that the noise level is uniformly bounded away from zero. In [2], Arlot and Massart also notice that their results can be extended to the unbounded case, where assumption (Ab) is replaced by some condition on the moments of the noise, and where (An) is weakened into mild regularity conditions for the noise level. We believe that moments conditions on the noise, in the spirit of assumptions stated by Arlot and Massart, could also been taken into account in our study in order to weaken (28), but at the prize of many technical efforts that are beyond the scope of the present paper. However, we explain at the end of this section how condition (H2) can be relaxed -see hypothesis (H2bis) below.
In assumption (H4) we require that the model M is provided with an orthonormal localized basis in L 2 P X . This property is convenient when dealing with the L ∞ -structure on the model, and this allows us to control the sup-norm of the functions in the model by the sup-norm of the vector of their coordinates in the localized basis. For examples of models with localized basis, and their use in a model selection framework, we refer for instance to Section 7.4.2 of Massart [16], where it is shown that models of histograms, piecewise polynomials and compactly supported wavelets are typical examples of models with localized basis for the L 2 (Leb) structure, considering that X ⊂R k . In Sections 5 and 6, we show that models of piecewise constant and piecewise polynomials respectively can also have a localized basis for the L 2 P X structure, under rather mild assumptions on P X . Assumption (H4) is needed in Theorem 2, whereas in Theorem 3 we only use the weaker assumption (H3) on the unit envelope of the model M , relating the L 2 -structure of the model to the L ∞ -structure. In fact, assumption (H4) allows us in the proof of Theorem 2 to achieve sharp lower bounds for the quantities of interest, whereas in Theorem 3 we only give upper bounds in the case of small models. We ask in assumption (H5) that the M-estimator is consistent towards the linear projection s M of s * onto the model M , at a rate at least better than (ln n) −1/2 . This can be considered as a rather strong assumption, but it is essential for our methodology. Moreover, we show in Sections 5 and 6 that this assumption is satisfied under mild conditions for histogram models and models of piecewise polynomials respectively, both at the rate R n,D,α ∝ D ln n n .
Secondly, let us comment on the rates of convergence given in Theorem 2 for models of reasonable dimensions. As we can see in Theorem 2, the rate of estimation in a fixed model M of reasonable dimension is determined at the first order by a key quantity that relates the structure of the model to the unknown law P of data. We call this quantity the complexity of the model M and we denote it by C M . More precisely, let us define Var ψ 1,M · ϕ k for a localized orthonormal basis (ϕ k ) D k=1 of (M, · 2 ) . Notice that K 1,M is well defined as it does not depend on the choice of the basis (ϕ k ) D k=1 . Indeed, since we have P ψ 1,M · ϕ k = 0, we deduce that Now observe that, by using Cauchy-Schwarz inequality in Definition (33), as pointed out by Birgé and Massart [5], we get and so On the one hand, if we assume (H1) then we obtain by elementary computations On the other hand, (H2) implies To fix ideas, let us explicitly compute K 2 1,M in a simple case. Consider homoscedastic regression on a histogram model M , in which the homoscedastic noise level σ is such that thus we deduce, by (46) and the previous equality, that where the conditional variance V [U |A ] of a variable U with respect to the event A is defined to be By (47), we explicitly get A careful look at the proof of Theorem 2 given in Section 7.3 show that condition (H2) is only used through the lower bound (49), and thus (H2) can be replaced by the following slightly more general assumption : (H2bis) Lower bound on the normalized complexity K 1,M : a positive constant A min exists such that When (H2) holds, we see from Inequality 49 that (H2bis) is satisfied with A min = 2σ min . For suitable models we can have for a positive constant A − Ψ and for all x ∈ X , and this allows to consider vanishing noise level, as we then have by (47), As we will see in Sections 5 and 6, Inequality (51) can be satisfied for histogram and piecewise polynomial models on a partition achieving some upper regularity assumption with respect to the law P X .

The histogram case
In this section, we particularize the results stated in Section 4 to the case of piecewise constant functions. We show that under a lower regularity assumption on the considered partition, the assumption (H4) of existence of a localized basis in L 2 P X and (H5) of consistency in sup-norm of the M-estimator towards the linear projection s M are satisfied.

Existence of a localized basis
The following lemma states the existence of an orthonormal localized basis for piecewise constant functions in L 2 P X , on a partition which is lower-regular for the law P X .

Lemma 4
Let consider a linear model M of histograms defined on a finite partition P on X , and write |P| = D the dimension of M . Moreover, assume that for a positive finite constant c M,P , Set, for I ∈ P, ϕ I = P X (I) −1/2 1 I .
Then the family (ϕ I ) I∈ΛM is an orthonormal basis in L 2 P X and we have, Condition (52) can also be found in Arlot and Massart [2] and is named lower regularity of the partition P for the law P X . It is easy to see that the lower regularity of the partition is equivalent to the property of localized basis in the case of histograms, i.e. (52) is equivalent to (53). The proof of Lemma 4 is straightforward and can be found in Section 7.1.

Rates of convergence in sup-norm
The following lemma allows to derive property (H5) for histogram models.
Lemma 5 Consider a linear model M of histograms defined on a finite partition P of X , and denote by |P| = D the dimension of M . Assume that Inequality (28) holds, that is, a positive constant A exists such that |Y | ≤ A a.s. Moreover, assume that for some positive finite constant c M,P , and that D ≤ A + n (ln n) −2 ≤ n for some positive finite constant A + . Then, for any α > 0 and for all n ≥ n 0 (α, c M,P , A + ), there exists an event of probability at least 1 − n −α on which s n exists, is unique and it holds, In Lemma 5 we thus achieve the convergence in sup-norm of the regressogram s n towards the linear projection s M at the rate D ln (n) /n . It is worth noticing that for a model of histograms satisfying the assumptions of Lemma 5, if we set A cons = L A,cM,P ,α A + , n 1 = n 0 (α, c M,P , A + ) and R n,D,α = L A+,A,cM,P ,α D ln n n , then Assumption (H5) is satisfied. To derive Inequality (55), we need to assume that the response variable Y is almost surely bounded and that the considered partition is lower-regular for the law P X . Hence, we fit again with the framework of [2] and we can thus view the general set of assumptions exposed in Section 4.1 as a natural generalization for linear models of the framework developed in [2] in the case of histograms. The proof of Lemma 5 can be found in Section 7.1.

Bounds for the excess risks
The next results is a straightforward application of Lemmas 4, 5 and Theorems 2, 3.
then there exists a positive finite constant A 0 , only depending on α, A, σ min , A − , A + , c M,P such that, by setting we have, for all n ≥ n 0 (A − , A + , A, , σ min , c M,P , α), and If (56) holds together with (H1) and if we assume that then a positive constant A u exists, only depending on A, c M,P , A + and α, such that for all n ≥ n 0 (A, c M,P , A + , α), We recover in Corollary 6 the general results of Section 4.2 for the case of histograms on a lower-regular partition. Moreover, in the case of histograms, assumption (29) which is part of (H1) is a straightforward consequence of (28). Indeed, we easily see that the projection s M of the regression function s * onto the model of piecewise constant functions with respect to P can be written Under (28), we have |E [Y |X ∈ I ]| ≤ Y ∞ ≤ A for every I ∈ P and we deduce by (59) that s M ∞ ≤ A.

Comments
Our bounds in Corollary 6 are obtained by following a general methodology that consists, among other things, in expanding the contrast and to take advantage of explicit computations that can be derived on the linear part of the contrast -for more details, see the proofs in Section 7.3 below. It is then instructive to compare them to the best available results in this special case. Let us compare them to the bounds obtained by Arlot and Massart in [2], in the case of a fixed model. Such results can be found in Propositions 10, 11 and 12 of [2]. The strategy adopted by the authors in this case is as follows. They first notice that the mean of the empirical excess risk on histograms is given by Then they derive concentration inequalities for the true excess risk and its empirical counterpart around their mean. Finally, the authors compare the mean of the true excess risk to the mean of the empirical excess risk. More precisely, using our notations, inequality (34) of Proposition 10 in [2] states that for every x ≥ 0 there exists an event of probability at least 1 − e 1−x on which, for some absolute constant L. One can notice that inequality (60), which is a special case of general concentration inequalities given by Boucheron and Massart [7], involves the bias of the model P (Ks M − Ks * ). By pointing out that the bias term arises from the use of some margin conditions that are satisfied for bounded regression, we believe that it can be removed from Proposition 10 of [2], since in the case of histograms models for bounded regression, some margin-like conditions hold, that are directly pointed at the linear projection s M . Apart for the bias term, the deviations of the empirical excess risk are then of the order considering the same probability of event as ours, inequality (60) becomes significantly better than inequality (58) for large models. Concentration inequalities for the true excess risk given in Proposition 11 of [2] give a magnitude of deviations that is again smaller than ours for sufficiently large models and that is in fact closer to ε 2 n than ε n , where ε n is defined in Corollary 6. But the mean of the true excess risk has to be compared to the mean of the empirical excess risk and it is remarkable that in Proposition 12 of [2] where such a result is given in a way that seems very sharp, there is a term lower bounded by , due to the lower regularity assumption on the partition. This tends to indicate that, up to a logarithmic factor, the term proportional to D ln n n 1/4 appearing in ε n is not improvable in general, and that the empirical excess risk concentrates better around its mean than the true excess risk. We conclude that the bounds given in Proposition 10, 11 and 12 of [2] are essentially more accurate than ours, apart for the bias term involved in concentration inequalities of Proposition 10, but this term could be removed as explained above. Furthermore, concentration inequalities for the empirical excess risk are significantly sharper than ours for large models. Arlot and Massart [2] also propose generalizations in the case of unbounded noise and when the noise level vanishes. The unbounded case seems to be beyond the reach of our strategy, due to our repeated use of Bousquet and Klein-Rio's inequalities along the proofs. However, we recover the case of vanishing noise level for histogram models, when the partition is upper regular with respect to the law P X , a condition also needed in [2] in this case. Indeed, we have noticed in Section 4.3 that assumption (H2) can be weakened into (H2bis), where we assume that for some positive constant A min . So, it suffices to bound from below the normalized complexity. We have from identity (47), . Moreover, from identity (46), we have in the case of histograms, Now, if we assume the upper regularity of the partition P with respect to P X , that is for some positive constant c + M,P , we then have

The case of piecewise polynomials
In this Section, we generalize the results given in Section 5 for models of piecewise constant functions to models of piecewise polynomials uniformly bounded in their degrees.

Existence of a localized basis
The following lemma states the existence of a localized orthonormal basis in (M, · 2 ), where M is a model of piecewise polynomials and X = [0, 1] is the unit interval.

Lemma 7 Let
Leb denote the Lebesgue measure on [0, 1]. Let assume that X = [0, 1] and that P X has a density f with respect to Leb satisfying, for a positive constant c min , Consider a linear model M of piecewise polynomials on [0, 1] with degree r or smaller, defined on a finite partition P made of intervals. Then there exists an orthonormal basis ϕ I,j , I ∈ P, j ∈ {0, ..., r} of (M, · 2 ) such that, for all j ∈ {0, ..., r} , ϕ I,j is supported by the element I of P, and a constant L r,cmin depending only on r, c min exists, satisfying for all I ∈ P, max j∈{0,...,r} a constant L r,cmin,c M,Leb depending only on r, c min and c M,Leb exists, such that for all β = β I,j I∈P,j∈{0,...,r} ∈ R D , where D = (r + 1) |P| is the dimension of M .
Lemma 7 states that if X = [0, 1] is the unit interval and if P X has a density with respect to the Lebesgue measure Leb on X , which is uniformly bounded away form zero, then there exists an orthonormal basis in (M, · 2 ) satisfying good enough properties in terms of the sup-norm of its elements. Moreover, if we assume the lower regularity of the partition with respect to Leb, then the orthonormal basis is localized and the constant of localization given in (64) depend on the maximal degree r. We notice that in the case of piecewise constant functions we do not need to assume the existence of a density for P X or to restrict ourselves to the unit interval. The proof of Lemma 7 can be found in Section 7.2.

Rates of convergence in sup-norm
The following lemma allows to derive property (H5) for piecewise polynomials.
Lemma 8 Assume that Inequality (28) holds, that is a positive constant A exists such that |Y | ≤ A a.s. Denote by Leb the Lebesgue measure on [0, 1]. Assume that X = [0, 1] and that P X has a density f with respect to Leb, satisfying for positive constants c min and c max , Consider a linear model M of piecewise polynomials on [0, 1] with degree less than r, defined on a finite partition P made of intervals, that satisfies for some finite positive constants c M,Leb Assume moreover that D ≤ A + n (ln n) −2 for a positive finite constant A + . Then, for any α > 0, there exists an event of probability at least 1 − n −α such that s n exists, is unique on this event and it holds, for all n ≥ n 0 (r, A + , c min , c M,Leb , α), In Lemma 5, we thus obtain the convergence in sup-norm of the M-estimator s n toward the linear projection s M at the rate D ln (n) /n . It is worth noting that, for a model of piecewise polynomials satisfying the assumptions of Lemma 5, if we set A cons = L A,r,A+,cmin,cmax,c M,Leb ,α A + , R n,D,α = L A,r,A+,cmin,cmax,c M,Leb ,α D ln n n , then Assumption (H5) is satisfied. The proof of Lemma 8 can be found in Section 7.2.

Bounds for the excess risks
The forthcoming result is a straightforward application of Lemmas 7, 8 and Theorems 2, 3.

Corollary 9
Denote by Leb the Lebesgue measure on [0, 1] and fix some positive finite constant α. Assume that X = [0, 1] and that P X has a density f with respect to Leb satisfying, for some positive finite constants c min and c max , Consider a linear model M of piecewise polynomials on [0, 1] with degree less than r, defined on a finite partition P made of intervals, that satisfy for a finite constant c M,Leb , Assume that (H1) and (H2) hold. Then, if there exist some positive finite constants A − and A + such that then there exists a positive finite constant A 0 , depending on α, A, σ min , A − , A + , r, c M,Leb , c min and c max such that, by setting we have, for all n ≥ n 0 (A − , A + , A, r, σ min , c M,Leb , c min , c max , α), Moreover, if (68) and (69) hold together with (H1) and if we assume that then a positive constant A u exists, only depending on A + , A, r, c M,Leb , c min and α, such that for all n ≥ n 0 (A + , A, r, c min , c max , c M,Leb , α), We derive in Corollary 9 optimal upper and lower bounds for the excess risk and its empirical counterpart in the case of models of piecewise polynomials uniformly bounded in their degree, with reasonable dimension. We give also upper bounds for models of possibly small dimension, without assumption (H2). Notice that we need stronger assumptions than in the case of histograms. Namely, we require the existence of a density uniformly bounded from above and from below for the unknown law P X , with respect to the Lebesgue measure on the unit interval. However, we recover essentially the bounds of Corollary 6, since by Lemma 8, we still have R n,D,α ∝ D ln (n) /n. Moreover, as in the case of histograms, assumption (29) which is part of (H1), is a straightforward consequence of (28). Indeed, we easily see that the projection s M of the regression function s * onto the model of piecewise polynomials with respect to P can be written where ϕ I,j is the orthonormal basis given in Lemma 7. It is then easy to show, using (62) and (28), that s M ∞ ≤ L A,r,cmin,cmax . Again, we can consider vanishing noise at the prize to ask that the partition is upper regular with respect to Leb. By (H2bis) of Section 4.3, if we show that for a positive constant A min instead of (H2), then the conclusions of Corollary 9 still hold. Now, from identity (47) we have Moreover, from identity (46), it holds in the case of piecewise polynomials, for all x ∈ X , Furthermore, if we ask that |P| sup for some positive constant c + M,P , then by using (68), (70) and (71), we obtain for all x ∈ X , and so A min = 2 c max × c + M,P × (r + 1)

Proofs
We begin with the simpler proofs of Sections 5 and 6, in Sections 7.1 and 7.2 respectively. The proofs of Theorems 2 and 3 of Section 4.2 can be found in Section 7.3.

Proofs of Section 5
Proof of Lemma 4. It suffices to observe that We now intend to prove (55) under the assumptions of Lemma 5.
Proof of Lemma 5. Along the proof, we denote by abusing the notation, for any I ∈ P, P (I) := P (I × R) = P X (I) and P n (I) := P n (I × R) .
Let α > 0 be fixed and let β > 0 to be chosen later. We first show that, since we have D ≤ A + n (ln n) −2 , it holds with large probability and for all n sufficiently large, inf I∈P P n (I) > 0 .

Since
1 I ∞ ≤ 1 and E 1 2 I = P (I) , we get by Bernstein's inequality (230), for any x > 0 and I ∈ P, Further note that by (54), D ≥ c 2 M,P P (I) −1 > 0 for any I ∈ P, and thus by taking x = β ln n, we easily deduce from inequality (72) that there exists a positive constant L (1) β,cM,P,A + only depending on c M,P and β such that, for any I ∈ P, Now, as D ≤ A + n (ln n) −2 for some positive constant A + , a positive integer n 0 (β, c M,P , A + ) exists such that Therefore we get, for all n ≥ n 0 (β, c M,P , A + ), Introduce the event Ω + = {∀I ∈ P, P n (I) > 0} .
We have shown that Moreover, on the event Ω + , the least squares estimator s n exists, is unique and it holds Moreover, by Bernstein's inequality (230), as we get for all I ∈ P, By putting x = β ln n in the latter inequality and using the fact that D ≥ c 2 M,P P (I) −1 it follows that there exists a positive constant L A,cM,P ,β,A+ only depending on A, c M,P and β such that Now define A,cM,P ,β,A+ D ln n n .

Proofs of Section 6
Under the assumptions of Lemma 7, we intend to establish (64).
Proof of Lemma 7. Let I be any interval of [0, 1] and w a positive measurable function on I. Denote by L 2 (I, Leb) the space of square integrable functions on I with respect to the Lebesgue measure Leb and set This space is equipped with the natural inner product Write . I,w its associated norm. Now, consider an interval I of P with bounds a and b, a < b. Also denote by f |I : x ∈ I −→ f (x) the restriction of the density f to the interval I. We readily have for g, h ∈ L 2 I, f |I , Define the function f I from [0, 1] to R + by If (p I,0 , p I,1 , ...p I,r ) is an orthonormal family of polynomials in L 2 [0, 1] , f I then by setting, for all x ∈ I, j ∈ {0, ..., r},φ we deduce from equality (81) that φ I,j r j=0 is an orthonormal family of polynomials in L 2 I, f |I such that deg φ I,j = deg (p I,j ). Now, it is a classical fact of orthogonal polynomials theory (see for example Theorems 1.11 and 1.12 of [9]) that there exists a unique family (q I,0 , q I,1 , ...q I,r ) of orthogonal polynomials on [0, 1] such that deg (q I,j ) = j and the coefficient of the highest monomial x j of q I,j is equal to 1. Moreover, each q I,j has j distinct real roots belonging to ]0, 1[. Thus, we can write x − α k I,j , α k I,j ∈ ]0, 1[ and α k I,j = α l I,j for k = l .
Clearly, q I,j ∞ ≤ 1. Moreover, Finally, introduce p I,j = q I,j −1 [0,1],f I q I,j and denote by ϕ I,j its associated orthonormal family of L 2 I, f |I . Then, by considering the extension ϕ I,j ofφ I,j to [0, 1] by adding null values, it is readily checked that the family ϕ I,j , I ∈ P, j ∈ {0, ..., r} is an orthonormal basis of (M, · 2 ) . In addition, where in the last inequality we used the fact that For all j ∈ {0, ..., r}, ϕ I,j is supported by the element I of P, hence we deduce from (83) that the orthonormal basis ϕ I,j , I ∈ P, j ∈ {0, ..., r} of (M, · 2 ) satisfies (62) with To conclude, observe that In order to avoid cumbersome notation, we define a total ordering on the set as follows. Let ≺ be a binary relation on I × I such that (I k , j) ≺ (I l , i) if (k < l or (k = l and j < i)) , and consider the total ordering defined to be So, from the definition of , the vector β = β I k ,j (I k ,j)∈I ∈ R D has coordinate β I k ,j at position (r + 1) k+j+1 and the matrix A = A (I k ,j),(I l ,i) (I k ,j),(I l ,i)∈I×I ∈ R D×D , has coefficient A (I k ,j),(I l ,i) at line (r + 1) k + j + 1 and column (r + 1) l + i + 1. Now, for some s = (I k ,j)∈I β I k ,j ϕ I k ,j ∈ M , we have β I k ,j P n yϕ I k ,j (x) + (I k ,j),(I l ,i)∈I×I β I k ,j β I l ,i P n ϕ I k ,j ϕ I l ,i .
Hence, by taking the derivative with respect to β I k ,j in the last quantity, We see that if β (n) = β (n) I k ,j (I k ,j)∈I ∈ R D is a critical point of and by combining (86) with the fact that P ϕ I k ,j 2 = 1 , for all (I k , j) ∈ I and P ϕ I k ,j ϕ I l ,i = 0 if (I k , j) = (I l , i) , we deduce that β (n) satisfies the following random linear system, where X y,n = P n yϕ I k ,j (x) (I k ,j)∈I ∈ R D , I D is the identity matrix of dimension D and L n,D = (L n,D ) (I k ,j),(I l ,i) (I k ,j),(I l ,i)∈I×I is a D × D matrix satisfying (L n,D ) (I k ,j),(I l ,i) = (P n − P ) ϕ I k ,j ϕ I l ,i . (99) in Lemma 10 below, one can find a positive integer n 0 (r, A + , c min , c M,Leb , γ) such that for all n ≥ n 0 , we have on an event Ω n of probability at least 1 − 3Dn −γ ,

Now, by inequality
where for a D × D matrix L, the operator norm · associated to the sup-norm on vectors is Then we deduce from (88) that (I D + L n,D ) is a non-singular D × D matrix and, as a consequence, that the linear system (87) admits a unique solution β (n) on Ω n for any n ≥ n 0 (r, A + , c min , c M,Leb , γ). Moreover, since P n y − (I k ,j)∈I β I k ,j ϕ I k ,j (x) 2 is a nonnegative quadratic functional with respect to β I k ,j (I k ,j)∈I ∈ R D we can easily deduce that on Ω n , β (n) achieves the unique minimum of P n y − (I k ,j)∈I β I k ,j ϕ I k ,j (x) 2 on R D . In other words, is the unique least squares estimator on M , and by (87) it holds, Now, as ϕ I k ,j and ϕ I l ,i have disjoint supports when k = l, it holds ϕ I k ,j ϕ I l ,i = 0 whenever k = l, and so equation (89) reduces to β (n) Moreover, recalling that s M = (I k ,j)∈I P yϕ I k ,j (x) ϕ I k ,j , it holds where the first inequality comes from the fact that ϕ I k ,j and ϕ I l ,i have disjoint supports when k = l. We next turn to the control of the right-hand side of (91). Let the index (I k , j) be fixed. By subtracting the quantity 1 + r i=0 (P n − P ) ϕ I k ,j ϕ I k ,i × P yϕ I k ,j (x) in each side of equation (90), we get Moreover, by Inequality (100) of Lemma 10, we have for all n ≥ n 0 (r, A + , c min , c M,Leb , γ), on the event Ω n . We thus deduce that β (n) and r i=0 (P n − P ) ϕ I k ,j ϕ I k ,i × P yϕ I k ,j (x) ≤ L r,A+,cmin,c M,Leb ,γ ln n n Leb (I k ) × P yϕ I k ,j (x) . (95) Moreover, by (28), (65) and (85) we have ≤ Ac max L r,cmin Leb (I k ) ≤ L A,r,cmin,cmax Leb (I k ) .
Hence, using inequalities (94), (97) and inequality (101) of Lemma 10 in equation (92), we obtain that To conclude, simply take γ = ln 3 ln 2 + α + 1, so that it holds for n ≥ 2, P [Ω c n ] ≤ n −α which implies (67). It remains to prove the following lemma that has been used all along the proof. Also recall that for a D × D matrix L, the operator norm · associated to the sup-norm on the vectors is Then, under the assumptions of Lemma 8, a positive integer n 0 (r, A + , c min , c M,Leb , γ) exists such that, for all n ≥ n 0 (r, A + , c min , c M,Leb , γ), the following inequalities hold on an event Ω n of probability at least 1 − 3Dn −γ , Proof of Lemma 10. Let us begin with the proof of inequality (101). Let the index (I k , j) ∈ I be fixed. By using Bernstein's inequality (230) and observing that, by (28), and, by (28), (85) and (66), By taking x = γ ln n in inequality (102), we obtain that Now, as D ≤ A + n (ln n) −2 , we deduce from (103) that for some well chosen positive constant L A,A+,r,cmin,c M,Leb ,γ , we have P (P n − P ) yϕ I k ,j (x) ≥ L A,A+,r,cmin,c M,Leb ,γ ln n n ≤ 2n −γ and by setting Hence the expected bound (101) holds on Ω n , for all n ≥ 1. We turn now to the proof of inequality (100). Let the index (I k , j) ∈ I be fixed. By Cauchy-Schwarz inequality, we have Let write By Cauchy-Schwarz inequality again, it holds Then, Bousquet's inequality (231), applied with ε = 1 and F =B I k , implies that where, by (85), and injecting the last bound in (108) we get In addition, we have .
Now, since by (66) and the fact that D ≤ A + n (ln n) −2 we have we obtain from (111) that a positive constant L r,A+,cmin,c M,Leb ,γ exists, depending only on γ, r, A + , c min and c M,Leb such that Finally, define and, by (105), the expected bound (100) holds on Ω (2) n , for all n ≥ n 0 (r, A + , c min , c M,Leb , γ). Next, notice that for a D × D matrix L = L (I k ,j),(I l ,i) (I k ,j),(I l ,i)∈I×I we have the following classical formula, Applied to the matrix of interest L n,D , this gives L n,D = max Thus, using formula (115), inequalities (100), (66) and (113) give that for all n ≥ n 0 (r, A + , c min , c M,Leb , γ), we have on Ω Finally, by setting Ω n = Ω n , we have P (Ω n ) ≥ 1 − 3Dn −γ , and inequalities (100), (99) and (101) are satisfied on Ω n for all n ≥ n 0 (r, A + , c min , c M,Leb , γ), which completes the proof of Lemma 10.

Proofs of Section 4
In order to express the quantities of interest in the proofs of Theorems 2 and 3, we need preliminary definitions. Let α > 0 be fixed and for R n,D,α defined in (H5), see Section 4.1, we set Recall that, by Lemma 1 of Section 2.2, the contrasted functions satisfy, for every s ∈ M and z = (x, y) ∈ X ×R, where ψ 1,M (z) = −2 (y − s M (x)) and ψ 2 (t) = t 2 , for all t ∈ R. For convenience, we will use the following notation, for any s ∈ M , Note that, for all s ∈ M , P ψ 1,M · s = 0 (118) and by (H1) inequality (32) holds true, that is Also, for K 1,M defined in Section 4.3, we have Var ψ 1,M · ϕ k for any orthonormal basis (ϕ k ) D k=1 of (M, · 2 ) . Moreover, inequality (48) holds under (H1) and we have and so, for any orthonormal basis (ϕ k ) D k=1 of (M, · 2 ), it holds for all k ∈ {1, ..., D}, as P ϕ 2 k = 1,

Proofs of the theorems
The proof of Theorem 2 relies on Lemmas 16, 17 and 18 stated in Section 7.4, and that give sharp estimates of suprema of the empirical process on the contrasted functions over slices of interest.
Proof of Theorem 2. Let α > 0 be fixed and let ϕ = (ϕ k ) D k=1 be an orthonormal basis of (M, · 2 ) satisfying (H4). We divide the proof of Theorem 2 into four parts, corresponding to the four Inequalities (39), (40), (41) and (42). The values of A 0 and A ∞ , respectively defined in (38) and (116), will then be chosen at the end of the proof.
Proof of Inequality (41). Let C = D 8n K 2 1,M > 0 and let r = 2. By (120) and (121)  so from (153) we deduce that Remark 4 Notice that in the proof of inequality (41), we do not need to assume the consistency of the least squares estimator s n towards the projection s M . Straightforward adaptations of Lemma 18 allow to takẽ ν n = max ln n D , D ln n n instead of the quantity ν n defined in (117). This readily gives the expected bound (43) of Theorem 2.
Proof of Theorem 3. We localize our analysis in the subset Unlike in the proof of Theorem 2, see (116), we need not to consider the quantityR n,D,α , a radius possibly larger than R n,D,α . Indeed, the use ofR n,D,α rather than R n,D,α in the proof of Theorem 2 is only needed in Lemma 12, where we derive a sharp lower bound for the mean of the supremum of the empirical process indexed by the contrasted functions centered by the contrasted projection over a slice of interest. To prove Theorem 3, we just need upper bounds, and Lemma 12 is avoided as well as the use ofR n,D,α . Let us define several slices of excess risk on the model M : for any C ≥ 0, I. Proof of Inequality (44). Let C 1 > 0 to be fixed later, satisfying We have by (H5), for all n ≥ n 1 , and also Moreover, it holds Now, from inequality (181) of Lemma 11 applied with β = α, we get In addition, we handle the empirical process indexed by the second order terms by straightforward modifications of Lemmas 14 and 15 as well as their proofs. It thus holds, by the same type of arguments as those given in Lemma 14, Moreover, using (164), the same type of arguments as those leading to inequality (208) of Lemma 15, allow to show that for any q ≥ 1 and j ∈ N * , for all x > 0, Hence, taking x = γ ln n in (165) and using the fact that C − = Dn −1 ≥ n −1 , we get Now, by straightforward modifications of the proof of Lemma 15, we get that for all n ≥ n 0 (A cons ), Combining (162), (163) and (167), we have on an event of probability at least 1 − 2n −α , for all n ≥ n 0 (A cons ), Now, as R n,D,α ≤ A cons (ln n) −1/2 , we deduce from (168) that for with L A,Acons,A3,M ,α large enough, it holds with probability at least 1 − 2n −α and for all n ≥ n 0 (A cons ), and so by using (160) and (161), this yields inequality (44).
II. Proof of Inequality (45). Let C 2 > 0 to be fixed later, satisfying We have by (H5), for all n ≥ n 1 , Moreover, we have on Ω ∞,α , where C 1 is defined in the first part of the proof dedicated to the establishment of inequality (44). Moreover, let us recall that in the first part of the proof, we have proved that an event of probability at least 1 − 2n −α exists, that we call Ω 1 , such that it holds on this event, for all n ≥ n 0 (A cons ), = sup Now, we have on Ω 1 , for all n ≥ n 0 (A cons ), and also, by (174)

Technical Lemmas
We state here some lemmas needed in the proofs of Section 7.3. First, in Lemmas 11, 12 and 13, we derive some controls, from above and from below, of the empirical process indexed by the "linear parts" of the contrasted functions over slices of interest. Secondly, we give upper bounds in Lemmas 14 and 15 for the empirical process indexed by the "quadratic parts" of the contrasted functions over slices of interest. And finally, we use all these results in Lemmas 16, 17 and 18 to derive upper and lower bounds for the empirical process indexed by the contrasted functions over slices of interest.
We are now ready to prove the lower bound (185) for the expected value of the largest increment of the empirical process over F (C,rC] .
Proof of Lemma 12. Let us begin with the lower bound of a result that will be need further in the proof. Introduce for all k ∈ {1, ..., D}, and observe that the excess risk on M of D k=1 β k,n ϕ k + s M ∈ M is equal to rC. We also set By Lemma 13 we have that for all Moreover, by (H4), we get on the eventΩ, Comparing inequality (194) with (193) and using (191), we obtain the following lower bound for all n ≥ n 0 (A − , A + , A, B 2 , r M (ϕ) , σ min , β), We take β = 4, and we must have Since D ≤ A + n (ln n) −2 and K 1,M ≥ 2σ min under (H2), we get, for all n ≥ n 0 (A, A + , r M (ϕ) , σ min ), and so, by combining (195) Now, as D ≥ A − (ln n) 2 we have for all n ≥ n 0 (A − ), D −1/2 ≤ 1/2. Moreover, we have K 1,M ≥ 2σ min by (H2) and rC ≥ A l Dn −1 , so we finally deduce from (197) that, for all n ≥ n 0 (A − , A + , A, B 2 , A l , r M (ϕ) , σ min ), We turn now to the lower bound of E sup s∈F (C,rC] (P n − P ) ψ 1,M · (s M − s) . First observe that s ∈ F (C,rC] implies that (2s M − s) ∈ F (C,rC] , so that In the next step, we apply Corollary 25. More precisely, using notations of Corollary 25, we set Hence, a positive constant L A,A l ,Au,σmin ( max 4A √ holds) exists such that, by setting κ n = L A,A l ,Au,σmin √ D we get, using (200), that, for all n ≥ n 0 (A − , A + , A l , A u , A, B 2 , r M (ϕ) , A cons , σ min ), Furthermore, since D ≥ A − (ln n) 2 , we have for all n ≥ n 0 (A − , A, A u , A l , σ min ), κ n ∈ (0, 1) .
So, using (199) and Corollary 25, it holds for all n ≥ n 0 ( Let us now turn to the control of second order terms appearing in the expansion of the least squares contrast, see (6). Let us define After straightforward computations using that ψ 2 (t) = t 2 for all t ∈ R and assuming (H3), we get that, for all x ∈ X , Lemma 14 Let C ≥ 0. Under (H3), it holds Proof. We define the Rademacher process R n on a class F of measurable functions from X to R, to be where ε i are independent Rademacher random variables also independent from the X i . By the usual symmetrization argument we have Taking the expectation with respect to the Rademacher variables, we get where the functions ϕ i : R −→ R are defined by Then by (202) we deduce that ϕ i is a contraction mapping with ϕ i (0) = 0. We thus apply Theorem 21 to get and so we derive successively the following upper bounds in mean, to complete the proof, it remains to observe that, by (203), In the following Lemma, we provide uniform upper bounds for the supremum of the empirical process of second order terms in the contrast expansion when the considered slices are not too small.
Proof. First notice that, as A + n (ln n) −2 ≥ D, we have by (34), By consequence, for all n ≥ n 0 (A ∞ , A cons , A + ),R n,D,α ≤ 1 . We thus have, for all n ≥ n 0 (A ∞ , A cons , A + ), and by monotonicity of the collection F C , for some q > 1 and J = |ln(C−∧1)| ln q + 1, it holds Simple computations show that, since D ≥ 1 and C − ≥ A l D n ≥ A l n , one can find a constant L A l ,q such that J ≤ L A l ,q ln n.
Moreover, by monotonicity of C −→ sup s∈FC |(P n − P ) (ψ 2 • (s − s M ))|, we have uniformly in C ∈ q j−1 C − , q j C − , Hence, taking the convention sup s∈∅ |(P n − P ) (ψ 2 • (s − s M ))| = 0, we get for all n ≥ n 0 (A ∞ , A cons , A + ) and any L > 0, Now, for any L > 0, Given j ∈ {1, ..., J} , Lemma 14 yields and next, we apply Bousquet's inequality (232) to handle the deviations around the mean. We have and, for all s ∈ F q j C− , It follows that, for ε = 1 and all x > 0, (208) By consequence, as D ≥ A − (ln n) 2 and asR n,D,α ≤ 1 for all n ≥ n 0 (A ∞ , A cons , A + ), taking x = γ ln n in (208) for some γ > 0, easy computations show that a positive constant L A−,A l ,γ independent of j exists such that for all n ≥ n 0 (A ∞ , A cons , A + ), Hence, using (207), we get for all n ≥ n 0 (A ∞ , A cons , A + ), And finally, as J ≤ L A l ,q ln n, taking γ = β + 1 and q = 2 gives the result for all n ≥ n 0 (A ∞ , A cons , A + , A l ).
Having controlled the residual empirical process driven by the remainder terms in the expansion of the contrast, and having proved sharp bounds for the expectation of the increments of the main empirical process on the slices, it remains to combine the above lemmas in order to establish the probability estimates controlling the empirical excess risk on the slices.
where the last bound follows from Cauchy-Schwarz inequality. Hence, we deduce from Lemma 11 that (212) Moreover, as C ≥ A l D n , we derive from Lemma 15 that it holds, for all n ≥ n 0 (A ∞ , A cons , A + , A l ), Finally, noticing thatR  (117), then for all n ≥ n 0 (A ∞ , A cons , A + ), Moreover, when we only assume C ≥ 0, we have for all n ≥ n 0 (A ∞ , A cons , A + ), Proof. First observe that In addition, observe that which concludes the proof.

Probabilistic Tools
We recall here the main probabilistic results that are instrumental in our proofs. Let us begin with the L p -version of Hoffmann-Jørgensen's inequality, that can be found for example in [15], Proposition 6.10, p.157.
Theorem 19 For any independent mean zero random variables Y j , j = 1, ..., n taking values in a Banach space (B, . ) and satisfying E [ Y j p ] < +∞ for some p ≥ 1, we have where B p is a universal constant depending only on p.
We will use this theorem for p = 2 in order to control suprema of empirical processes. In order to be more specific, let F be a class of measurable functions from a measurable space Z to R and (X 1 , ..., X n ) be independent variables of common law P taking values in Z. We then denote by B = l ∞ (F ) the space of uniformly bounded functions on F and, for any b ∈ B, we set b = sup f ∈F |b (f )|. Thus (B, . ) is a Banach space. Indeed we shall apply Theorem 19 to the independent random variables, with mean zero and taking values in B, defined by More precisely, we will use the following result, which is a straightforward application of Theorem 19. Denote by δ Xi the empirical measure associated to the sample (X 1 , ..., X n ) and by P n − P F = sup f ∈F |(P n − P ) (f )| the supremum of the empirical process over F .

Corollary 20
If F is a class of measurable functions from a measurable space Z to R satisfying and (X 1 , ..., X n ) are n i.i.d. random variables taking values in Z, then an absolute constant B 2 exists such that, Another tool we need is a comparison theorem for Rademacher processes, see Theorem 4.12 of [15]. A function ϕ : R → R is called a contraction if |ϕ (u) − ϕ (v)| ≤ |u − v| for all u, v ∈ R. Moreover, for a subset T ⊂ R n we set h (t) T = h T = sup t∈T |h (t)| .
Theorem 21 Let (ε 1 , ..., ε n ) be n i.i.d. Rademacher variables and F : R + −→ R + be a convex and increasing function. Furthermore, let ϕ i : R −→ R, i ≤ n, be contractions such that ϕ i (0) = 0. Then, for any bounded subset T ⊂ R n , The next tool is the well known Bernstein's inequality, that can be found for example in [16], Proposition 2.9.
Theorem 22 (Bernstein's inequality) Let (X 1 , ..., X n ) be independent real valued random variables and define Assuming that v = 1 n n i=1 E X 2 i < ∞ and |X i | ≤ b a.s.
we have, for every x > 0, We turn now to concentration inequalities for the empirical process around its mean. Bousquet's inequality [8] provides optimal constants for the deviations at the right. Klein-Rio's inequality [12] gives sharp constants for the deviations at the left, that slightly improves Klein's inequality [11].
Theorem 23 Let (ξ 1 , ..., ξ n ) be n i.i.d. random variables having common law P and taking values in a measurable space Z. If F is a class of measurable functions from Z to R satisfying |f (ξ i ) − P f | ≤ b a.s., for all f ∈ F , i ≤ n, then, by setting we have, for all x ≥ 0, Bousquet's inequality : and we can deduce that, for all ε, x > 0, it holds Klein-Rio's inequality : and again, we can deduce that, for all ε, x > 0, it holds The following result is due to Ledoux [14]. We will use it along the proofs through Corollary 25 which is stated below. From now on, we set for short Z = P n − P F .
Theorem 24 Let (ξ 1 , ..., ξ n ) be independent random with values in some measurable space (Z, T ) and F be some countable class of real-valued measurable functions from Z. Let ξ ′ 1 , ..., ξ ′ n be independent from (ξ 1 , ..., ξ n ) and with the same distribution. Setting Corollary 25 Under notations of Theorem 23, if some κ n ∈ (0, 1) exists such that The result then follows from straightforward calculations.