Relative deviation learning bounds and generalization with unbounded loss functions

Abstract

We present an extensive analysis of relative deviation bounds, including detailed proofs of two-sided inequalities and their implications. We also give detailed proofs of two-sided generalization bounds that hold in the general case of unbounded loss functions, under the assumption that a moment of the loss is bounded. We then illustrate how to apply these results in a sample application: the analysis of importance weighting.

Appendices

Appendix

Lemmas in Support of Section 3

Lemma 4

Let 1 < α ≤ 2 and for any η > 0, let $f\colon (0, +\infty ) \times (0, +\infty ) \to \mathbb {R}$ be the function defined by $f\colon (x, y) \mapsto \frac {x - y}{\sqrt [\alpha ]{x + y + \eta }}$. Then, f is a strictly increasing function of x and a strictly decreasing function of y.

Proof

f is differentiable over its domain of definition and for all $(x, y) \in (0, +\infty ) \times (0, +\infty )$,

$$\begin{array}{@{}rcl@{}} && \frac{\partial f}{\partial x}(x, y) = \frac{(x + y + \eta)^{\frac{1}{\alpha}} - \frac{x - y}{\alpha} (x + y + \eta)^{\frac{1}{\alpha} - 1}}{(x + y + \eta)^{\frac{2}{\alpha}}} = \frac{\frac{\alpha - 1}{\alpha} x + \frac{\alpha + 1}{\alpha} y + \eta}{(x + y + \eta)^{1 + \frac{1}{\alpha}}} > 0\\ && \frac{\partial f}{\partial y}(x, y) = \frac{- (x + y + \eta)^{\frac{1}{\alpha}} - \frac{x - y}{\alpha} (x + y + \eta)^{\frac{1}{\alpha} - 1}}{(x + y + \eta)^{\frac{2}{\alpha}}} = -\frac{\frac{\alpha + 1}{\alpha} x + \frac{\alpha - 1}{\alpha} y + \eta}{(x + y + \eta)^{1 + \frac{1}{\alpha}}} < 0. \end{array} $$

□

Proofs in Support of Section 4

3.1 Proof of Theorem 3

Proof

We prove the first statement. The second statement can be shown in a very similar way. Fix 1 < α ≤ 2 and 𝜖 > 0 and $\mathcal {S}$ assume that for any h ∈ H and t ≥ 0, the following holds:

$$ \frac{\Pr[L(h, z) > t] - \widehat{\Pr}[L(h, z) > t] }{\sqrt[\alpha]{\Pr[L(h, z) > t] + \tau}} \leq \epsilon. $$

(17)

We show that this implies that for any h ∈ H, $\frac {\mathcal {L}(h) - \widehat {\mathcal {L}}_{S}(h)}{\sqrt [\alpha ]{\mathcal {L}_{\alpha }(h) + \tau }} \leq {\Gamma }(\alpha , \epsilon ) \epsilon$. By the properties of the Lebesgue integral, we can write

$$\begin{array}{@{}rcl@{}} && \mathcal{L}(h) = \mathrm{E}_{z \sim D}[L(h, z)] = {\int}_{0}^{+\infty} \Pr[L(h, z) > t] dt \\ && \widehat{\mathcal{L}}(h) = \mathrm{E}_{z \sim \widehat{D}}[L(h, z)] = {\int}_{0}^{+\infty} \widehat{\Pr}[L(h, z) > t] dt, \end{array} $$

and, similarly,

$$\mathcal{L}_{\alpha}(h) = \mathcal{L}_{\alpha}(h) = {\int}_{0}^{+\infty} \Pr[L^{\alpha}(h, z) > t] dt = {\int}_{0}^{+\infty} \alpha t^{\alpha - 1} \Pr[L(h, z) > t] dt. $$

In what follows, we use the notation $I_{\alpha } = \mathcal {L}_{\alpha }(h) + \tau$. Let $t_{0} = s I_{\alpha }^{\frac {1}{\alpha }}$ and $t_{1} = t_{0} \left [ \frac {1}{\epsilon } \right ]^{\frac {1}{\alpha - 1}}$ for s > 0. To bound $\mathcal {L}(h) - \widehat {\mathcal {L}}(h)$, we simply bound $\Pr [L(h, z) > t] - \widehat {\Pr }[L(h, z) > t]$ by $\Pr [L(h, z) > t]$ for large values of t, that is t > t₁, and use inequality (17) for smaller values of t:

$$\begin{array}{@{}rcl@{}} \mathcal{L}(h) - \widehat{\mathcal{L}}(h) &=& {\int}_{0}^{+\infty} \Pr[L(h, z) > t] - \widehat{\Pr}[L(h, z) > t] dt\\ & \leq& {\int}_{0}^{t_{1}} \epsilon \sqrt[\alpha]{\Pr[L(h, z) > t] + \tau} dt + {\int}_{t_{1}}^{+\infty} \Pr[L(h, z) > t] dt. \end{array} $$

For relatively small values of t, $\Pr [L(h, z) > t]$ is close to one. Thus, we can write

$$\begin{array}{@{}rcl@{}} \mathcal{L}(h) - \widehat{\mathcal{L}}(h) &\!\leq\!& {\int}_{0}^{t_{0}} \epsilon \sqrt[\alpha]{1 + \tau} dt + {\int}_{t_{0}}^{t_{1}} \epsilon \sqrt[\alpha]{\Pr[L(h, z) \!>\! t] + \tau} dt + {\int}_{t_{1}}^{+\infty} \Pr[L(h, z) \!>\! t]dt \\ &=& {\int}_{0}^{+\infty} f(t) g(t) dt, \end{array} $$

with

$$f(t) = \left\{\begin{array}{ll} \gamma_{1} I_{\alpha}^{\frac{\alpha - 1}{\alpha^{2}}} \epsilon \sqrt[\alpha]{1 + \tau} & \text{if } 0 \leq t \leq t_{0}\\ \gamma_{2} \left[ \alpha t^{\alpha - 1} (\Pr[L(h, z) > t] + \tau) \right]^{\frac{1}{\alpha}} \epsilon & \text{if } t_{0}< t \leq t_{1}\\ \gamma_{2} \left[ \alpha t^{\alpha - 1} \Pr[L(h, z) > t] \right]^{\frac{1}{\alpha}} \epsilon & \text{if } t_{1} < t. \end{array}\right. \quad g(t) = \left\{\begin{array}{ll} \frac{1}{\gamma_{1} I_{\alpha}^{\frac{\alpha - 1}{\alpha^{2}}}} & \text{if } 0 \leq t \leq t_{0}\\ \frac{1}{\gamma_{2} (\alpha t^{\alpha - 1})^{\frac{1}{\alpha}}} & \text{if } t_{0} < t \leq t_{1}\\ \frac{\Pr[L(h, z) > t]^{\frac{\alpha - 1}{\alpha}}}{\gamma_{2} (\alpha t^{\alpha - 1})^{\frac{1}{\alpha}}} \frac{1}{\epsilon} & \text{if } t_{1} < t, \end{array}\right. $$

where γ₁,γ₂ are positive parameters that we shall select later. Now, since α > 1, by Hölder’s inequality,

$$\begin{array}{@{}rcl@{}} \mathcal{L}(h) - \widehat{\mathcal{L}}(h) & \leq& \left[ {\int}_{0}^{+\infty} f(t)^{\alpha} dt \right]^{\frac{1}{\alpha}} \left[ {\int}_{0}^{+\infty} g(t)^{\frac{\alpha}{\alpha - 1}} dt \right]^{\frac{\alpha - 1}{\alpha}}. \end{array} $$

The first integral on the right-hand side can be bounded as follows:

$$\begin{array}{@{}rcl@{}} {\int}_{0}^{+\infty} f(t)^{\alpha} dt & = & {\int}_{0}^{t_{0}} (1\! +\! \tau) (\gamma_{1} I_{\alpha}^{\frac{\alpha - 1}{\alpha^{2}}} \epsilon)^{\alpha} dt + \gamma_{2}^{\alpha} \epsilon^{\alpha} \tau {\int}_{t_{0}}^{t_{1}} \alpha t^{\alpha - 1} dt + \gamma_{2}^{\alpha} {\int}_{t_{0}}^{+\infty} \alpha t^{\alpha - 1} \Pr[L(h, z) > t] \epsilon^{\alpha} dt\\ & \!\leq\!& (1 + \tau) \gamma_{1}^{\alpha} I_{\alpha}^{\frac{\alpha - 1}{\alpha}} t_{0} \epsilon^{\alpha} + \gamma_{2}^{\alpha} \epsilon^{\alpha} \tau (t_{1}^{\alpha} - t_{0}^{\alpha})+ \gamma_{2}^{\alpha} \epsilon^{\alpha} I_{\alpha} \\ & \!\leq\!& (\gamma_{1}^{\alpha} (1 + \tau) s + \gamma_{2}^{\alpha} (1 + s^{\alpha} (1/\epsilon)^{\frac{\alpha}{\alpha - 1}} \tau) ) \epsilon^{\alpha} I_{\alpha}\\ & \!\leq\!& (\gamma_{1}^{\alpha} (1 + \tau) s + \gamma_{2}^{\alpha} (1 + s^{\alpha} \tau^{\frac{1}{\alpha}})) \epsilon^{\alpha} I_{\alpha}. \end{array} $$

Since $t_{1}/t_{0} = (1/\epsilon )^{\frac {1}{\alpha - 1}}$, the second one can be computed and bounded following

$$\begin{array}{@{}rcl@{}} {\int}_{0}^{+\infty} g(t)^{\frac{\alpha}{\alpha - 1}} dt & =& {\int}_{0}^{t_{0}} \frac{dt}{\gamma_{1}^{\frac{\alpha}{\alpha - 1}} I_{\alpha}^{\frac{1}{\alpha}}} + {\int}_{t_{0}}^{t_{1}} \frac{1}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}} \alpha^{\frac{1}{\alpha - 1}} } \frac{dt}{t} + {\int}_{t_{1}}^{+\infty} \frac{\Pr[L(h, z) > t]}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}} \alpha^{\frac{1}{\alpha - 1}} \epsilon^{\frac{\alpha}{\alpha - 1}} t} dt\\ & = &\frac{s}{\gamma_{1}^{\frac{\alpha}{\alpha - 1}}} + \frac{1}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}} (\alpha - 1) \alpha^{\frac{1}{\alpha - 1}} } \log \frac{1}{\epsilon} + {\int}_{t_{1}}^{+\infty} \frac{\alpha t^{\alpha - 1} \Pr[L(h, z) > t]}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}} (\alpha \epsilon)^{\frac{\alpha}{\alpha - 1}} t^{\alpha} } dt\\ & \leq &\frac{s}{\gamma_{1}^{\frac{\alpha}{\alpha - 1}}} + \frac{1}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}} (\alpha - 1) \alpha^{\frac{1}{\alpha - 1}} } \log \frac{1}{\epsilon} + {\int}_{t_{1}}^{+\infty} \frac{\alpha t^{\alpha - 1} \Pr[L(h, z) > t]}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}} (\alpha \epsilon)^{\frac{\alpha}{\alpha - 1}} t_{1}^{\alpha}} dt\\ & \leq& \frac{s}{\gamma_{1}^{\frac{\alpha}{\alpha - 1}}} + \frac{1}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}} (\alpha - 1) \alpha^{\frac{1}{\alpha - 1}} } \log \frac{1}{\epsilon} + \frac{I_{\alpha}}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}} (\alpha \epsilon)^{\frac{\alpha}{\alpha - 1}} s^{\alpha} I_{\alpha} (\frac{1}{\epsilon})^{\frac{\alpha}{\alpha - 1}}} \\ & =& \frac{s}{\gamma_{1}^{\frac{\alpha}{\alpha - 1}}} + \frac{1}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}}} \left( \frac{1}{(\alpha - 1) \alpha^{\frac{1}{\alpha - 1}} } \log \frac{1}{\epsilon} + \frac{1}{\alpha^{\frac{\alpha}{\alpha - 1}} s^{\alpha} } \right). \end{array} $$

Combining the bounds obtained for these integrals yields directly

$$\begin{array}{@{}rcl@{}} &&\mathcal{L}(h) - \widehat{\mathcal{L}}(h) \\ & \leq& \left[ (\gamma_{1}^{\alpha} (1 + \tau) s + \gamma_{2}^{\alpha} (1 + s^{\alpha} \tau^{\frac{1}{\alpha}})) \epsilon^{\alpha} I_{\alpha} \right]^{\frac{1}{\alpha}} \left[ \frac{s}{\gamma_{1}^{\frac{\alpha}{\alpha - 1}}} + \frac{1}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}}} \left( \frac{1}{(\alpha - 1) \alpha^{\frac{1}{\alpha - 1}} } \log \frac{1}{\epsilon} + \frac{1}{\alpha^{\frac{\alpha}{\alpha - 1}} s^{\alpha} } \right) \right]^{\frac{\alpha - 1}{\alpha}} \\ & =& (\gamma_{1}^{\alpha} (1 + \tau) s + \gamma_{2}^{\alpha} (1 + s^{\alpha} \tau^{\frac{1}{\alpha}}))^{\frac{1}{\alpha}} \left[ \frac{s}{\gamma_{1}^{\frac{\alpha}{\alpha - 1}}} + \frac{1}{\gamma_{2}^{\frac{\alpha}{\alpha - 1}}} \left( \frac{1}{(\alpha - 1) \alpha^{\frac{1}{\alpha - 1}} } \log \frac{1}{\epsilon} + \frac{1}{\alpha^{\frac{\alpha}{\alpha - 1}} s^{\alpha} } \right) \right]^{\frac{\alpha - 1}{\alpha}} \epsilon I_{\alpha}^{\frac{1}{\alpha}}. \end{array} $$

Observe that the expression on the right-hand side can be rewritten as $\| \mathbf {u} \|_{\alpha } \| \mathbf {v} \|_{\frac {\alpha }{\alpha - 1}} \ \epsilon I_{\alpha }^{\frac {1}{\alpha }}$ where the vectors u and v are defined by $\mathbf {u} = (\gamma _{1} (1 + \tau )^{\frac {1}{\alpha }} s^{\frac {1}{\alpha }}, \gamma _{2} (1 + s^{\alpha } \tau ^{\frac {1}{\alpha }})^{\frac {1}{\alpha }})$ and $\mathbf {v} = (v_{1}, v_{2}) = \left (\frac {s^{\frac {\alpha - 1}{\alpha }}}{\gamma _{1}}, \frac {1}{\gamma _{2}} \left [ \frac {1}{(\alpha - 1) \alpha ^{\frac {1}{\alpha - 1}} } \log \frac {1}{\epsilon } + \frac {1}{\alpha ^{\frac {\alpha }{\alpha - 1}} s^{\alpha } } \right ]^{\frac {\alpha - 1}{\alpha }} \right )$. The inner product u ⋅v does not depend on γ₁ and γ₂, and equality holds if and only if the vectors u and $\mathbf {v}^{\prime } = (v_{1}^{\frac {1}{\alpha - 1}}, v_{2}^{\frac {1}{\alpha - 1}})$ are collinear (as we can see by applying Hölder’s inequality).

γ₁ and γ₂ can be chosen so that ${\det }(\mathbf {u}, \mathbf {v}^{\prime }) = 0$, since this condition can be rewritten as

$$ s^{\frac{1}{\alpha}} (1 + \tau)^{\frac{1}{\alpha}} \frac{\gamma_{1}}{\gamma_{2}^{\frac{1}{\alpha - 1}}} \left[ \frac{1}{(\alpha - 1) \alpha^{\frac{1}{\alpha - 1}} } \log \frac{1}{\epsilon} + \frac{1}{\alpha^{\frac{\alpha}{\alpha - 1}} s^{\alpha} } \right]^{\frac{1}{\alpha}} - s^{\frac{1}{\alpha}} (1 + s^{\alpha} \tau^{\frac{1}{\alpha}})^{\frac{1}{\alpha}} \frac{\gamma_{2}}{\gamma_{1}^{\frac{1}{\alpha - 1}}} = 0, $$

(18)

or equivalently,

$$ \left( \frac{\gamma_{1}}{\gamma_{2}} \right)^{\frac{\alpha}{\alpha - 1}} \left[ \frac{1}{(\alpha - 1) \alpha^{\frac{1}{\alpha - 1}} } \log \frac{1}{\epsilon} + \frac{1}{\alpha^{\frac{\alpha}{\alpha - 1}} s^{\alpha} } \right]^{\frac{1}{\alpha}} - (1 + s^{\alpha} \tau^{\frac{1}{\alpha}})^{\frac{1}{\alpha}} = 0. $$

(19)

Thus, for such values of γ₁ and γ₂, the following inequality holds:

$$\begin{array}{@{}rcl@{}} \mathcal{L}(h) - \widehat{\mathcal{L}}(h) & \leq (\mathbf{u} \cdot \mathbf{v}) \epsilon I_{\alpha}^{\frac{1}{\alpha}} = f(s) \epsilon I_{\alpha}^{\frac{1}{\alpha}}, \end{array} $$

with

$$\begin{array}{@{}rcl@{}} f(s) & =& (1 + \tau)^{\frac{1}{\alpha}} s + (1 + s^{\alpha} \tau^{\frac{1}{\alpha}})^{\frac{1}{\alpha}} \left[ \frac{1}{(\alpha - 1) \alpha^{\frac{1}{\alpha - 1}} } \log \frac{1}{\epsilon} + \frac{1}{\alpha^{\frac{\alpha}{\alpha - 1}} s^{\alpha} } \right]^{\frac{\alpha - 1}{\alpha}} \\ & =& (1 + \tau)^{\frac{1}{\alpha}} s + \frac{(1 + s^{\alpha} \tau^{\frac{1}{\alpha}})^{\frac{1}{\alpha}}}{\alpha} \left[ \frac{\alpha}{(\alpha - 1) } \log \frac{1}{\epsilon} + \frac{1}{s^{\alpha} } \right]^{\frac{\alpha - 1}{\alpha}}. \end{array} $$

Setting $s = \frac {\alpha - 1}{\alpha }$ yields the statement of the theorem. □

3.2 Proof of Proposition 1

Proof

We prove the first inequality. The second can be proven in a very similar way. Fix α > 2 and h ∈ H. As in the proof of Theorem 3, we bound $\Pr [L(h, z) > t]$ by 1 for t close to 0, say t ≤ t₀ for some t₀ > 0 that we shall later determine. We can write

$${\int}_{0}^{+\infty} \sqrt{\Pr[L(h, z) > t]} dt \leq {\int}_{0}^{t_{0}} 1 dt + {\int}_{t_{0}}^{+\infty} \sqrt{\Pr[L(h, z) > t]} dt = {\int}_{0}^{+\infty} f(t) g(t) dt, $$

with functions f and g defined as follows:

$$f(t) = \left\{\begin{array}{ll} \gamma I_{\alpha}^{\frac{\alpha - 1}{2 \alpha}} & \text{if } 0 \leq t \leq t_{0}\\ \alpha^{\frac{1}{2}} t^{\frac{\alpha - 1}{2}} \Pr[L(h, z) > t]^{\frac{1}{2}} & \text{if } t_{0} < t. \end{array}\right. \quad g(t) = \left\{\begin{array}{ll} \frac{1}{\gamma I_{\alpha}^{\frac{\alpha - 1}{2 \alpha}}} & \text{if } 0 \leq t \leq t_{0}\\ \frac{1}{\alpha^{\frac{1}{2}} t^{\frac{\alpha - 1}{2}}} & \text{if } t_{0} < t, \end{array}\right. $$

where $I_{\alpha } = \mathcal {L}_{\alpha }(h)$ and where γ is a positive parameter that we shall select later. By the Cauchy-Schwarz inequality,

$${\int}_{0}^{+\infty} \sqrt{\Pr[L(h, z) > t]} dt \leq \left( {\int}_{0}^{+\infty} f(t)^{2} dt \right)^{\frac{1}{2}} \left( {\int}_{0}^{+\infty} g(t)^{2} dt \right)^{\frac{1}{2}}. $$

Thus, we can write

$$\begin{array}{@{}rcl@{}} &&{\int}_{0}^{+\infty} \sqrt{\Pr[L(h, z) > t]} dt \\ &\leq& \left( \gamma^{2} I_{\alpha}^{\frac{\alpha - 1}{\alpha}} t_{0} + {\int}_{t_{0}}^{+\infty} \alpha t^{\alpha - 1} \Pr[L(h, z) > t] dt \right)^{\frac{1}{2}} \left( \frac{t_{0}}{\gamma^{2} I_{\alpha}^{\frac{\alpha - 1}{\alpha}}} + {\int}_{t_{0}}^{+\infty} \frac{1}{\alpha t^{\alpha - 1}} dt \right)^{\frac{1}{2}}\\ &\leq& \left( \gamma^{2} I_{\alpha}^{\frac{\alpha - 1}{\alpha}} t_{0} + I_{\alpha} \right)^{\frac{1}{2}} \left( \frac{t_{0}}{\gamma^{2} I_{\alpha}^{\frac{\alpha - 1}{\alpha}}} + \frac{1}{\alpha (\alpha - 2) t_{0}^{\alpha - 2}} \right)^{\frac{1}{2}}. \end{array} $$

Introducing t₁ with $t_{0} = I_{\alpha }^{1/\alpha } t_{1}$ leads to

$$\begin{array}{@{}rcl@{}} {\int}_{0}^{+\infty} \sqrt{\Pr[L(h, z) > t]} dt & \leq& \left( \gamma^{2} I_{\alpha} t_{1} + I_{\alpha} \right)^{\frac{1}{2}} \left( \frac{t_{1}}{\gamma^{2} I_{\alpha}^{\frac{\alpha - 2}{\alpha}}} + \frac{1}{\alpha (\alpha - 2) t_{1}^{\alpha - 2} I_{\alpha}^{\frac{\alpha - 2}{\alpha}}} \right)^{\frac{1}{2}}\\ & \leq& \left( \gamma^{2} t_{1} + 1 \right)^{\frac{1}{2}} \left( \frac{t_{1}}{\gamma^{2}} + \frac{1}{\alpha (\alpha - 2) t_{1}^{\alpha - 2}} \right)^{\frac{1}{2}} I_{\alpha}^{\frac{1}{\alpha}}. \end{array} $$

We now seek to minimize the expression $\left (\gamma ^{2} t_{1} + 1 \right )^{\frac {1}{2}} \left (\frac {t_{1}}{\gamma ^{2}} + \frac {1}{\alpha (\alpha - 2) t_{1}^{\alpha - 2}} \right )^{\frac {1}{2}}$, first as a function of γ. This expression can be viewed as the product of the norms of the vectors $\mathbf {u} = (\gamma t_{1}^{\frac {1}{2}}, 1)$ and $\mathbf {v} = (\frac {t_{1}^{\frac {1}{2}}}{\gamma }, \frac {1}{\sqrt {\alpha (\alpha - 2)} t_{1}^{\frac {\alpha - 2}{2}}})$, with a constant inner product (not depending on γ). Thus, by the properties of the Cauchy-Schwarz inequality, it is minimized for collinear vectors and in that case equals their inner product:

$$\mathbf{u} \cdot \mathbf{v} = t_{1} + \frac{1}{\sqrt{\alpha (\alpha - 2)} t_{1}^{\frac{\alpha - 2}{2}}}. $$

Differentiating this last expression with respect to t₁ and setting the result to zero gives the minimizing value of t₁: $(\frac {2}{\alpha - 2} \sqrt {\alpha (\alpha - 2)})^{-\frac {2}{\alpha }} = \left (\frac {1}{2} \sqrt {\frac {\alpha - 2}{\alpha }} \right )^{\frac {2}{\alpha }}$. For that value of t₁,

$$\mathbf{u} \cdot \mathbf{v} = \left( 1 + \frac{2}{\alpha - 2} \right) t_{1} = \frac{\alpha}{\alpha - 2} \left( \frac{1}{2} \sqrt{\frac{\alpha - 2}{\alpha}} \right)^{\frac{2}{\alpha}} = \left( \frac{1}{2} \right)^{\frac{2}{\alpha}} \left( \frac{\alpha - 2}{\alpha} \right)^{\frac{1 - \alpha}{\alpha}}, $$

which concludes the proof. □