Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities

Abstract

In this paper, we present the Lipschitz regularization theory and algorithms for a novel Loss-Sensitive Generative Adversarial Network (LS-GAN). Specifically, it trains a loss function to distinguish between real and fake samples by designated margins, while learning a generator alternately to produce realistic samples by minimizing their losses. The LS-GAN further regularizes its loss function with a Lipschitz regularity condition on the density of real data, yielding a regularized model that can better generalize to produce new data from a reasonable number of training examples than the classic GAN. We will further present a Generalized LS-GAN (GLS-GAN) and show it contains a large family of regularized GAN models, including both LS-GAN and Wasserstein GAN, as its special cases. Compared with the other GAN models, we will conduct experiments to show both LS-GAN and GLS-GAN exhibit competitive ability in generating new images in terms of the Minimum Reconstruction Error (MRE) assessed on a separate test set. We further extend the LS-GAN to a conditional form for supervised and semi-supervised learning problems, and demonstrate its outstanding performance on image classification tasks.

Appendices

Proof of Lemma 1

To prove Lemma 1, we need the following lemma.

Lemma 5

For two probability densities $p({\mathbf {x}})$ and $q({\mathbf {x}})$, if $p({\mathbf {x}})\ge \eta q({\mathbf {x}})$ almost everywhere, we have

$$\begin{aligned} \int _{\mathbf {x}} |p({\mathbf {x}})-q({\mathbf {x}})|d{\mathbf {x}} \le \dfrac{2(1-\eta )}{\eta } \end{aligned}$$

for $\eta \in (0,1]$.

Proof

We have the following equalities and inequalities:

$$\begin{aligned} \begin{aligned} \int _{{\mathbf {x}}} |p({\mathbf {x}})-q({\mathbf {x}})|d{\mathbf {x}}&=\int _{\mathbf {x}} \mathbb {1}_{\left[ p({\mathbf {x}})\ge q({\mathbf {x}})\right] } (p({\mathbf {x}})-q({\mathbf {x}}))d{\mathbf {x}}\\&\quad +\,\int _{\mathbf {x}} \mathbb {1}_{\left[ p({\mathbf {x}})<q({\mathbf {x}})\right] } (q({\mathbf {x}})-p({\mathbf {x}}))d{\mathbf {x}}\\&=\int _{\mathbf {x}} (1-\mathbb {1}_{\left[ p({\mathbf {x}})< q({\mathbf {x}})\right] }) (p({\mathbf {x}})-q({\mathbf {x}}))d{\mathbf {x}}\\&\quad +\,\int _{\mathbf {x}} \mathbb {1}_{\left[ p({\mathbf {x}})<q({\mathbf {x}})\right] } (q({\mathbf {x}})-p({\mathbf {x}}))d{\mathbf {x}}\\&=2\int _{\mathbf {x}} \mathbb {1}_{\left[ p({\mathbf {x}})< q({\mathbf {x}})\right] } (q({\mathbf {x}})-p({\mathbf {x}}))d{\mathbf {x}}\\&\le 2\left( \dfrac{1}{\eta }-1\right) \int _{\mathbf {x}} \mathbb {1}_{\left[ p({\mathbf {x}})< q({\mathbf {x}})\right] } p({\mathbf {x}}) d{\mathbf {x}}\\&\le \dfrac{2(1-\eta )}{\eta } \end{aligned} \end{aligned}$$

(16)

This completes the proof. $\square$

Now we can prove Lemma 1.

Proof

Suppose $(\theta ^*,\phi ^*)$ is a Nash equilibrium for the problems (4) and (5).

Then, on one hand, we have

$$\begin{aligned} \begin{aligned} S(\theta ^*,\phi ^*)&\ge \mathop {\mathbb {E}}\limits _{{\mathbf {x}}\sim P_{data}({\mathbf {x}})} L_{\theta ^*}({\mathbf {x}})\\&\quad +\, \lambda \mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}({\mathbf {x}})\\ {\mathbf {z}}_G\sim P_{G^*}({\mathbf {z}}_G) \end{array}} \big (\varDelta ({\mathbf {x}}, {\mathbf {z}}_G) +L_{\theta ^*}({\mathbf {x}})-L_{\theta ^*}({\mathbf {z}}_G)\big )\\&=\int _{\mathbf {x}} P_{data}({\mathbf {x}}) L_{\theta ^*}({\mathbf {x}}) d{\mathbf {x}} + \lambda \mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}({\mathbf {x}}) \\ {\mathbf {z}}_G\sim P_{G^*}({\mathbf {z}}_G) \end{array}} \varDelta ({\mathbf {x}}, {\mathbf {z}}_G) \\&\quad +\,\lambda \int _{\mathbf {x}} P_{data}({\mathbf {x}}) L_{\theta ^*}({\mathbf {x}}) d{\mathbf {x}} \\&\quad -\, \lambda \int _{{\mathbf {z}}_G} P_{G^*}({\mathbf {z}}_G) L_{\theta ^*}({\mathbf {z}}_G) d {\mathbf {z}}_G\\&=\int _{\mathbf {x}} \big ((1+\lambda )P_{data}({\mathbf {x}})- \lambda P_{G^*}({\mathbf {x}})\big )L_{\theta ^*}({\mathbf {x}}) d{\mathbf {x}} \\&\quad +\,\lambda \mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}({\mathbf {x}}) \\ {\mathbf {z}}_G\sim P_{G^*}({\mathbf {z}}_G) \end{array}}\varDelta ({\mathbf {x}}, {\mathbf {z}}_G) \end{aligned} \end{aligned}$$

(17)

where the first inequality follows from $(a)_+\ge a$.

We also have $T(\theta ^*,\phi ^*)\le T(\theta ^*,\phi )$ for any $G_\phi$ as $\phi ^*$ minimizes $T(\theta ^*,\phi )$. In particular, we can replace $P_G({\mathbf {x}})$ in $T(\theta ^*,\phi )$ with $P_{data}({\mathbf {x}})$, which yields

$$\begin{aligned} \mathop \int \limits _{{\mathbf {x}}}L_{\theta ^*}({\mathbf {x}})P_{G^*}({\mathbf {x}})d{\mathbf {x}} \le \mathop \int \limits _{{\mathbf {x}}}L_{\theta ^*}({\mathbf {x}})P_{data}({\mathbf {x}})d{\mathbf {x}}. \end{aligned}$$

Applying this inequality into (17) leads to

$$\begin{aligned} S(\theta ^*,\phi ^*)\ge & {} \int _{\mathbf {x}} P_{data}({\mathbf {x}})L_{\theta ^*}({\mathbf {x}}) d{\mathbf {x}} +\lambda \mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}({\mathbf {x}}) \\ {\mathbf {z}}_G\sim P_{G^*}({\mathbf {z}}_G) \end{array}}\varDelta ({\mathbf {x}}, {\mathbf {z}}_G)\nonumber \\\ge & {} \lambda \mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}({\mathbf {x}}) \\ {\mathbf {z}}_G\sim P_{G^*}({\mathbf {z}}_G) \end{array}}\varDelta ({\mathbf {x}}, {\mathbf {z}}_G) \end{aligned}$$

(18)

where the last inequality follows as $L_\theta ({\mathbf {x}})$ is nonnegative.

On the other hand, consider a particular loss function

$$\begin{aligned} L_{\theta _0}({\mathbf {x}})=\alpha \big (-(1+\lambda ) P_{data}({\mathbf {x}})+\lambda P_{G^*}({\mathbf {x}})\big )_+ \end{aligned}$$

(19)

When $\alpha$ is a sufficiently small positive coefficient, $L_{\theta _0}({\mathbf {x}})$ is a nonexpansive function (i.e., a function with Lipschitz constant no larger than 1.). This follows from the assumption that $P_{data}$ and $P_G$ are Lipschitz. In this case, we have

$$\begin{aligned} \varDelta ({\mathbf {x}}, {\mathbf {z}}_G) +L_{\theta _0}({\mathbf {x}})-L_{\theta _0}({\mathbf {z}}_G) \ge 0 \end{aligned}$$

(20)

By placing this $L_{\theta _0}({\mathbf {x}})$ into $S(\theta ,\phi ^*)$, one can show that

$$\begin{aligned} \begin{aligned} S(\theta _0,\phi ^*)&=\int _{\mathbf {x}} \big ((1+\lambda )P_{data}({\mathbf {x}})- \lambda P_{G^*}({\mathbf {x}})\big )L_{\theta _0}({\mathbf {x}}) d{\mathbf {x}}\\&\quad +\,\lambda \mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}({\mathbf {x}}) \\ {\mathbf {z}}_G\sim P_{G^*}({\mathbf {z}}_G) \end{array}}\varDelta ({\mathbf {x}}, {\mathbf {z}}_G)\\&=-\alpha \mathop \int \limits _{{\mathbf {x}}} \big (-(1+\lambda )P_{data}({\mathbf {x}})+\lambda P_{G^*}({\mathbf {x}})\big )_+^2 d{\mathbf {x}}\\&\quad +\,\lambda \mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}({\mathbf {x}}) \\ {\mathbf {z}}_G\sim P_{G^*}({\mathbf {z}}_G) \end{array}}\varDelta ({\mathbf {x}}, {\mathbf {z}}_G)\\ \end{aligned} \end{aligned}$$

where the first equality uses Eq. (20), and the second equality is obtained by substituting $L_{\theta _0}({\mathbf {x}})$ in Eq. (19) into the equation.

Assuming that $(1+\lambda )P_{data}({\mathbf {x}})- \lambda P_{G^*}({\mathbf {x}})<0$ on a set of nonzero measure, the above equation would be strictly upper bounded by $\lambda \mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}({\mathbf {x}}) \\ {\mathbf {z}}_G\sim P_{G^*}({\mathbf {z}}_G) \end{array}}\varDelta ({\mathbf {x}}, {\mathbf {z}}_G)$ and we have

$$\begin{aligned} S(\theta ^*,\phi ^*)\le S(\theta _0,\phi ^*)<\lambda \mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}({\mathbf {x}}) \\ {\mathbf {z}}_G\sim P_{G^*}({\mathbf {z}}_G) \end{array}}\varDelta ({\mathbf {x}}, {\mathbf {z}}_G) \end{aligned}$$

(21)

This results in a contradiction with Eq. (18). Therefore, we must have

$$\begin{aligned} P_{data}({\mathbf {x}}) \ge \dfrac{\lambda }{1+\lambda }P_{G^*}({\mathbf {x}}) \end{aligned}$$

(22)

for almost everywhere. By Lemma 5, we have

$$\begin{aligned} \int _{{\mathbf {x}}}|P_{data}({\mathbf {x}})-P_{G^*}({\mathbf {x}})|d{\mathbf {x}} \le \dfrac{2}{\lambda } \end{aligned}$$

Let $\lambda \rightarrow +\infty$, this leads to

$$\begin{aligned} \int _{{\mathbf {x}}}|P_{data}({\mathbf {x}})-P_{G^*}({\mathbf {x}})|d{\mathbf {x}} \rightarrow 0 \end{aligned}$$

This proves that $P_{G^*}({\mathbf {x}})$ converges to $P_{data}({\mathbf {x}})$ as $\lambda \rightarrow +\infty$. $\square$

Proof of Lemma 3

Proof

Suppose a pair of $(f_w^*, g_\phi ^*)$ jointly solve the WGAN problem.

Then, on one hand, we have

$$\begin{aligned} U(f_w^*,g_\phi ^*)=\int _x f_w^*({\mathbf {x}}) P_{data}({\mathbf {x}})d{\mathbf {x}} - \int _x f_w^* ({\mathbf {x}})P_{g_\phi ^*}({\mathbf {x}})d{\mathbf {x}}\le 0 \end{aligned}$$

(23)

where the inequality follows from $V(f_w^*,g_\phi ^*)\ge V(f_w^*,g_\phi )$ by replacing $P_{g_\phi }({\mathbf {x}})$ with $P_{data}({\mathbf {x}})$.

Consider a particular $f_w({\mathbf {x}})\triangleq \alpha (P_{data}({\mathbf {x}})-P_{g_\phi ^*}({\mathbf {x}}))_+$. Since $P_{data}({\mathbf {x}})$ and $P_{g_\phi ^*}$ are Lipschitz by assumption, when $\alpha$ is sufficiently small, it can be shown that $f_w({\mathbf {x}})\in {\mathcal {L}}_1$.

Substituting this $f_w$ into $U(f_w, g_\phi ^*)$, we get

$$\begin{aligned} U(f_w,g_\phi ^*)=\alpha \int _{\mathbf {x}} (P_{data}({\mathbf {x}})-P_{g_\phi ^*}({\mathbf {x}}))_+^2 d{\mathbf {x}} \end{aligned}$$

Let us assume $P_{data}({\mathbf {x}}) > P_{g_\phi ^*}({\mathbf {x}})$ on a set of nonzero measure, we would have

$$\begin{aligned} U(f_w^*,g_\phi ^*)\ge U(f_w,g_\phi ^*) > 0 \end{aligned}$$

This leads to a contradiction with (23), so we must have

$$\begin{aligned} P_{data}({\mathbf {x}}) \le P_{g_\phi ^*}({\mathbf {x}}) \end{aligned}$$

almost everywhere.

Hence, by Lemma 5, we prove the conclusion that

$$\begin{aligned} \int _{{\mathbf {x}}}|P_{data}({\mathbf {x}})-P_{g_\phi ^*}({\mathbf {x}})|d{\mathbf {x}}=0. \end{aligned}$$

$\square$

Proof of Theorem 2

For simplicity, throughout this section, we disregard the first loss minimization term in $S(\theta ,\phi ^*)$ and $S_m(\theta ,\phi ^*)$, since the role of the first term would vanish as $\lambda$ goes to $+\infty$. However, even if it is involved, the following proof still holds with only some minor changes.

To prove Theorem 2, we need the following lemma.

Lemma 6

For all loss functions $L_\theta$, with at least the probability of $1-\eta$, we have

$$\begin{aligned} |S_m(\theta ,\phi ^*)-S(\theta ,\phi ^*)|\le \varepsilon \end{aligned}$$

when the number of samples

$$\begin{aligned} m\ge \dfrac{C B_\varDelta ^2(\kappa +1)^2 \big (N \log (\kappa _L N/\varepsilon )+\log (1/\eta )\big )}{\varepsilon ^2} \end{aligned}$$

with a sufficiently large constant C.

The proof of this lemma needs to apply the McDiarmid’s inequality and the fact that $(\cdot )_+$ is an 1-Lipschitz to bound the difference $|S_m(\theta ,\phi ^*)-S(\theta ,\phi ^*)|$ for a loss function. Then, to get the union bound over all loss functions, a standard $\epsilon$-net (Arora et al. 2017) will be constructed to yield finite points that are dense enough to cover the parameter space of the loss functions. The proof details are given below.

Proof

For a loss function $L_\theta$, we compute $S_m(\theta ,\phi ^*)$ over a set of m samples $\{({\mathbf {x}}_i, {{\mathbf {z}}_G}_i)|1\le i\le m\}$ drawn from $P_{data}$ and $P_{G^*}$ respectively.

To apply the McDiarmid’s inequality, we need to bound the change of this function when a sample is changed. Denote by $S^i_m(\theta ,\phi ^*)$ when the jth sample is replaced with ${\mathbf {x}}'_i$ and ${{\mathbf {z}}'_G}_i$. Then we have

$$\begin{aligned} \begin{aligned} |S_m(\theta ,\phi ^*)-S^i_m(\theta ,\phi ^*)|&=\dfrac{1}{m}|\big (\varDelta ({\mathbf {x}}_i, {{\mathbf {z}}_G}_i)+L_\theta ({\mathbf {x}}_i)-L_\theta ({{\mathbf {z}}_G}_i)\big )_+\\&\quad -\,\big (\varDelta ({\mathbf {x}}'_i, {{\mathbf {z}}'_G}_i)+L_\theta ({\mathbf {x}}'_i)-L_\theta ({{\mathbf {z}}'_G}_i)\big )_+|\\&\le \dfrac{1}{m}|\varDelta ({\mathbf {x}}_i, {{\mathbf {z}}_G}_i)-\varDelta ({\mathbf {x}}'_i, {{\mathbf {z}}'_G}_i)|\\&\quad +\,\dfrac{1}{m}|L_\theta ({\mathbf {x}}_i)-L_\theta ({\mathbf {x}}'_i)|\\&\quad +\,\dfrac{1}{m}|L_\theta ({{\mathbf {z}}_G}_i)-L_\theta ({{\mathbf {z}}'_G}_i)|\\&\le \dfrac{1}{m}\big (2B_\varDelta +\kappa \varDelta ({\mathbf {x}}_i,{\mathbf {x}}'_i) + \kappa \varDelta ({{\mathbf {z}}_G}_i,{{\mathbf {z}}'_G}_i)\big )\\&\le \dfrac{2}{m}(1+\kappa )B_\varDelta \end{aligned} \end{aligned}$$

where the first inequality uses the fact that $(\cdot )_+$ is 1-Lipschitz, the second inequality follows from that $\varDelta ({\mathbf {x}}, {\mathbf {z}}_G)$ is bounded by $B_\varDelta$ and $L_\theta ({\mathbf {x}})$ is $\kappa$-Lipschitz in ${\mathbf {x}}$.

Now we can apply the McDiarmid’s inequality. Noting that

$$\begin{aligned} S(\theta ,\phi ^*)=\mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}_i\sim P_{data}\\ {{\mathbf {z}}_G}_i\sim P_G\\ i=1,\ldots ,m \end{array}}S_m(\theta ,\phi ^*), \end{aligned}$$

we have

$$\begin{aligned} P(|S_m(\theta ,\phi ^*)-S(\theta ,\phi ^*)|\ge \varepsilon /2)\le 2\exp \left( -\dfrac{\varepsilon ^2 m}{8(1+\kappa )^2B_\varDelta ^2}\right) \end{aligned}$$

(24)

The above bound applies to a single loss function $L_\theta$. To get the union bound, we consider a $\varepsilon /8\kappa _L$-net ${\mathcal {N}}$, i.e., for any $L_\theta$, there is a $\theta '\in {\mathcal {N}}$ in this net so that $\Vert \theta -\theta '\Vert \le \varepsilon /8\kappa _L$. This standard net can be constructed to contain finite loss functions such that $|{\mathcal {N}}|\le O(N\log (\kappa _L N/\varepsilon ))$, where N is the number of parameters in a loss function. Note that we implicitly assume the parameter space of the loss function is bounded so we can construct such a net containing finite points here.

Therefore, we have the following union bound for all $\theta \in {\mathcal {N}}$ that, with probability $1-\eta$,

$$\begin{aligned} |S_m(\theta ,\phi ^*)-S(\theta ,\phi ^*)|\le \dfrac{\varepsilon }{2} \end{aligned}$$

when $m\ge \dfrac{C B_\varDelta ^2(\kappa +1)^2 \big (N \log (\kappa _L N/\varepsilon )+\log (1/\eta )\big )}{\varepsilon ^2}$.

The last step is to obtain the union bound for all loss functions beyond ${\mathcal {N}}$. To show that, we consider the following inequality

$$\begin{aligned} \begin{aligned}&|S(\theta ,\phi ^*)-S(\theta ',\phi ^*)|\\&\quad =|\mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}\\ {\mathbf {z}}_G\sim P_G \end{array}} \big (\varDelta ({\mathbf {x}}, {\mathbf {z}}_G)+ L_{\theta }({\mathbf {x}})- L_{\theta }({\mathbf {z}}_G)\big )_+\\&\qquad -\,\mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data}\\ {\mathbf {z}}_G\sim P_G \end{array}} \big (\varDelta ({\mathbf {x}}, {\mathbf {z}}_G)+ L_{\theta '}({\mathbf {x}})- L_{\theta '}({\mathbf {z}}_G)\big )_+|\\&\quad \le \mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {x}}\sim P_{data} \end{array}}|L_{\theta }({\mathbf {x}})-L_{\theta '}({\mathbf {x}})|\\&\qquad +\,\mathop {\mathbb {E}}\limits _{\begin{array}{c} {\mathbf {z}}_G\sim P_G \end{array}}|L_{\theta }({\mathbf {z}}_G)-L_{\theta '}({\mathbf {z}}_G)|\\&\quad \le 2\kappa _L\Vert \theta -\theta '\Vert \end{aligned} \end{aligned}$$

where the first inequality uses that fact that $(\cdot )_+$ is 1-Lipschitz again, and the second inequality follows from that $L_\theta$ is $\kappa _L$-Lipschitz in $\theta$. Similarly, we can also show that

$$\begin{aligned} |S_m(\theta ,\phi ^*)-S_m(\theta ',\phi ^*)| \le 2\kappa _L\Vert \theta -\theta '\Vert \end{aligned}$$

Now we can derive the union bound over all loss functions. For any $\theta$, by construction we can find a $\theta '\in {\mathcal {N}}$ such that $\Vert \theta -\theta '\Vert \le \varepsilon /8\kappa _L$. Then, with probability $1-\eta$, we have

$$\begin{aligned} \begin{aligned}&|S_m(\theta ,\phi ^*)-S(\theta ,\phi ^*)|\\&\quad \le |S_m(\theta ,\phi ^*)-S_m(\theta ',\phi ^*)| +|S_m(\theta ',\phi ^*)-S(\theta ',\phi ^*)|\\&\qquad +\,|S(\theta ',\phi ^*)-S(\theta ,\phi ^*)|\\&\quad \le 2\kappa _L\Vert \theta -\theta '\Vert +\dfrac{\varepsilon }{2}+2\kappa _L\Vert \theta -\theta '\Vert \\&\quad \le \dfrac{\varepsilon }{4}+\dfrac{\varepsilon }{2}+\dfrac{\varepsilon }{4}=\varepsilon \end{aligned} \end{aligned}$$

This proves the lemma. $\square$

Now we can prove Theorem 2.

Proof

First let us bound $S_m-S$. Consider $L_{\theta ^*}$ that minimizes $S(\theta ,\phi ^*)$. Then with probability $1-\eta$, when $m\ge \dfrac{C N B_\varDelta ^2(\kappa +1)^2\log (\kappa _L N/\eta \varepsilon )}{\varepsilon ^2}$, we have

$$\begin{aligned} S_m - S \le S_m(\theta ^*,\phi ^*) - S(\theta ^*,\phi ^*)\le \varepsilon \end{aligned}$$

where the first inequality follows from the inequality $S_m\le S_m(\theta ^*,\phi ^*)$ as $\theta ^*$ may not minimize $S_m$, and the second inequality is a direct application of the above lemma. Similarly, we can prove the other direction. With probability $1-\eta$, we have

$$\begin{aligned} S - S_m \ge S(\theta ^*,\phi ^*)- S_m(\theta ^*,\phi ^*) \ge \varepsilon \end{aligned}$$

Finally, a more rigourous discussion about the generalizability should consider that $G_{\phi ^*}$ is updated iteratively. Therefore we have a sequence of $G_{\phi ^*}^{(t)}$ generated over T iterations for $t=1,\ldots ,T$. Thus, a union bound over all generators should be considered in (24), and this makes the required number of training examples m become

$$\begin{aligned} m\ge \dfrac{C B_\varDelta ^2(\kappa +1)^2 \big (N \log (\kappa _L N/\varepsilon )+\log (T/\eta )\big )}{\varepsilon ^2}. \end{aligned}$$

However, the iteration number T is usually much smaller than the model size N (which is often hundreds of thousands), and thus this factor will not affect the above lower bound of m. $\square$

Proof of Theorem 4 and Corollary 2

We prove Theorem 4 as follows.

Proof

First, the existence of a minimizer follows from the fact that the functions in ${\mathcal {F}}_\kappa$ form a compact set, and the objective function is convex.

To prove the minimizer has the two forms in (10), for each $L_\theta \in {\mathcal {F}}_\kappa$, let us consider

$$\begin{aligned} \widehat{L}_{\theta }({\mathbf {x}})= & {} \max _{1\le i\le n+m}\big \{\big (L_\theta ({\mathbf {x}}^{(i)})-\kappa \varDelta ({\mathbf {x}},{\mathbf {x}}^{(i)})\big )_+\big \},\\ \widetilde{L}_{\theta }({\mathbf {x}})= & {} \min _{1\le i\le n+m}\big \{L_\theta ({\mathbf {x}}^{(i)})+\kappa \varDelta ({\mathbf {x}},{\mathbf {x}}^{(i)})\} \end{aligned}$$

It is not hard to verify that $\widehat{L}_\theta ({\mathbf {x}}^{(i)})=L_\theta ({\mathbf {x}}^{(i)})$ and $\widetilde{L}_\theta ({\mathbf {x}}^{(i)})=L_\theta ({\mathbf {x}}^{(i)})$ for $1\le i\le n+m$.

Indeed, by noting that $L_{\theta }$ has its Lipschitz constant bounded by $\kappa$, we have $L_{\theta }({\mathbf {x}}^{(j)})-L_{\theta }({\mathbf {x}}^{(i)})\le \kappa \varDelta ({\mathbf {x}}^{(i)},{\mathbf {x}}^{(j)})$, and thus

$$\begin{aligned} L_{\theta }({\mathbf {x}}^{(j)})-\kappa \varDelta ({\mathbf {x}}^{(i)},{\mathbf {x}}^{(j)})\le L_{\theta }({\mathbf {x}}^{(i)}) \end{aligned}$$

Because $L_{\theta }({\mathbf {x}}^{(i)})\ge 0$ by the assumption (i.e., it is lower bounded by zero), it can be shown that for all j

$$\begin{aligned} \big (L_{\theta }({\mathbf {x}}^{(j)})-\kappa \varDelta ({\mathbf {x}}^{(i)},{\mathbf {x}}^{(j)})\big )_+\le L_{\theta }({\mathbf {x}}^{(i)}). \end{aligned}$$

Hence, by the definition of $\widehat{L}_{\theta }({\mathbf {x}})$ and taking the maximum over j on the left hand side, we have

$$\begin{aligned} \widehat{L}_{\theta }({\mathbf {x}}^{(i)})\le L_{\theta }({\mathbf {x}}^{(i)}) \end{aligned}$$

On the other hand, we have

$$\begin{aligned} \widehat{L}_{\theta }({\mathbf {x}}^{(i)}) \ge L_{\theta }({\mathbf {x}}^{(i)}) \end{aligned}$$

because $\widehat{L}_{\theta }({\mathbf {x}}) \ge \big (L_\theta ({\mathbf {x}}^{(i)})-\kappa \varDelta ({\mathbf {x}},{\mathbf {x}}^{(i)})\big )_+$ for any ${\mathbf {x}}$, and it is true in particular for ${\mathbf {x}}={\mathbf {x}}^{(i)}$. This shows $\widehat{L}_{\theta }({\mathbf {x}}^{(i)}) = L_{\theta }({\mathbf {x}}^{(i)})$.

Similarly, one can prove $\widetilde{L}_\theta ({\mathbf {x}}^{(i)})=L_\theta ({\mathbf {x}}^{(i)})$. To show this, we have

$$\begin{aligned} L_\theta ({\mathbf {x}}^{(j)})+\kappa \varDelta ({\mathbf {x}}^{(i)},{\mathbf {x}}^{(j)})\ge L_\theta ({\mathbf {x}}^{(i)}) \end{aligned}$$

by the Lipschitz continuity of $L_\theta$. By taking the minimum over j, we have

$$\begin{aligned} \widetilde{L}_\theta ({\mathbf {x}}^{(i)}) \ge L_\theta ({\mathbf {x}}^{(i)}). \end{aligned}$$

On the other hand, we have $\widetilde{L}_\theta ({\mathbf {x}}^{(i)})\le L_\theta ({\mathbf {x}}^{(i)})$ by the definition of $\widetilde{L}_\theta ({\mathbf {x}}^{(i)})$. Combining these two inequalities shows that $\widetilde{L}_\theta ({\mathbf {x}}^{(i)})=L_\theta ({\mathbf {x}}^{(i)})$.

Now we can prove for any function $L_\theta \in {\mathcal {F}}_\kappa$, there exist $\widehat{L}_\theta$ and $\widetilde{L}_\theta$ both of which attain the same value of $S_{n,m}$ as $L_\theta$, since $S_{n,m}$ only depends on the values of $L_\theta$ on the data points $\{{\mathbf {x}}^{(i)}\}$. In particular, this shows that any global minimum in ${\mathcal {F}}_\kappa$ of $S_{n,m}$ can also be attained by the corresponding functions of the form (10). By setting $l^*_i=\widehat{L}_{\theta ^*}({\mathbf {x}}^{(i)})=\widetilde{L}_{\theta ^*}({\mathbf {x}}^{(i)})$ for $i=1,\ldots ,n+m$, this completes the proof. $\square$

Finally, we prove Corollary 2 that bounds $L_\theta$ with $\widehat{L}_\theta ({\mathbf {x}})$ and $\widetilde{L}_\theta ({\mathbf {x}})$ constructed above.

Proof

By the Lipschitz continuity, we have

$$\begin{aligned} L_\theta ({\mathbf {x}}^{(i)}) - \kappa \varDelta ({\mathbf {x}},{\mathbf {x}}^{(i)}) \le L_\theta ({\mathbf {x}}) \end{aligned}$$

Since $L_\theta ({\mathbf {x}})\ge 0$, it follows that

$$\begin{aligned} \big (L_\theta ({\mathbf {x}}^{(i)}) - \kappa \varDelta ({\mathbf {x}},{\mathbf {x}}^{(i)})\big )_+ \le L_\theta ({\mathbf {x}}) \end{aligned}$$

Taking the maximum over i on the left hand side, we obtain

$$\begin{aligned} \widehat{L}_\theta ({\mathbf {x}}) \le L_\theta ({\mathbf {x}}) \end{aligned}$$

This proves the lower bound.

Similarly, we have by Lipschitz continuity

$$\begin{aligned} L_\theta ({\mathbf {x}}) \le \kappa \varDelta ({\mathbf {x}},{\mathbf {x}}^{(i)}) + L_\theta ({\mathbf {x}}^{(i)}) \end{aligned}$$

which, by taking the minimum over i on the left hand side, leads to

$$\begin{aligned} \widetilde{L}_\theta ({\mathbf {x}}) \le L_\theta ({\mathbf {x}}) \end{aligned}$$

This shows the upper bound. $\square$