Skip to main content
Log in

The Forward–Backward Algorithm and the Normal Problem

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

The forward–backward splitting technique is a popular method for solving monotone inclusions that have applications in optimization. In this paper, we explore the behaviour of the algorithm when the inclusion problem has no solution. We present a new formula to define the normal solutions using the forward–backward operator. We also provide a formula for the range of the displacement map of the forward–backward operator. Several examples illustrate our theory.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. We point out that the assumption A is firmly nonexpansive can be relaxed to A is cocoercive (see Remark 3.1).

  2. For general conditions on strong convergence of the forward–backward algorithm we refer the reader to [13].

  3. Let \({n\in {\mathbb N}}\). The positive orthant in \(\mathbb {R}^n\) is \(\mathbb {R}^n_{+}=[0,+\infty [^n\) and the strictly positive orthant in \(\mathbb {R}^n\) is \(\mathbb {R}^n_{++}=]0,+\infty [^n\). Likewise, we define the negative orthant and the strictly negative orthant \(\mathbb {R}^n_{-}\) and \(\mathbb {R}^n_{--}\), respectively.

  4. Let \(B:X\rightrightarrows X\). Then and (see [27, Equation (10)]).

  5. Recall that \(A:X\rightarrow X\) is cocoercive if \((\exists \alpha >0)\) such that \(\alpha A\) is firmly nonexpansive.

  6. For detailed discussion and examples of paramonotone operators we refer the reader to [33].

  7. For detailed discussion and examples of \(3^*\) monotone operators we refer the reader to [34].

  8. Let \(h:X\rightarrow ]-\infty ,+\infty ]\) be proper. The set of minimizers of h, \(\left\{ x\in X~:~h(x)=\inf h(X)\right\} \), is denoted by \(\mathrm{argmin}\, h\).

  9. Suppose that \(g:X\rightarrow ]-\infty ,+\infty ]\) is convex, lower semicontinuous and proper. Then \(\mathrm{Prox}_g\) is the Moreau prox operator associated with g defined by \(\mathrm{Prox}_g:X\rightarrow X:x\mapsto (\mathrm{Id}+{\partial }g)^{-1}(x) =\mathrm{argmin}_{y\in X} \left( g(y)+\tfrac{1}{2}||x-y||^2\right) \).

  10. Let C be a nonempty closed and convex subset of X. We use \(d_C\) to denote the distance from the set C defined by \(d_C:X\rightarrow [0,+\infty [ :x\mapsto \min _{c\in C}||x-c||=||x-P_C x||\).

  11. For convenience we shall use \(v_{\mathrm{FB}}\) and \(v_{\mathrm{DR}}\) to denote \(v_{T_{{}{\mathrm{FB}}}}\) and \(v_{T_{{}{\mathrm{DR}}}}\), respectively.

  12. Let C be a subset of H. We use \(\mathrm{ri}C\) to denote the interior of C with respect to the affine hull of C.

  13. For detailed discussion on the algebra of nearly convex sets, we refer the reader to [42, Section 3].

  14. For detailed discussion on the properties of nearly equal and nearly convex sets, we refer the reader to [43].

  15. Let \(B:X\rightrightarrows X\). Then, B is an affine relation if \(\mathrm{gra}B\) is an affine subspace of \(X\times X\).

  16. Let \(f:X\rightarrow ]-\infty ,+\infty ]\) be convex, lower semicontinuous and proper. We use \(f^*\) to denote the convex conjugate (a.k.a. Fenchel conjugate) of f, defined by \(f^*:X\rightarrow ]-\infty ,+\infty ]:x\mapsto \sup _{u\in X}(\langle x,u\rangle -f(x))\).

  17. Let C be a nonempty, closed and convex subset of X. The recession cone of C is \(\mathrm{rec}C:=\{x\in X~:~x+C\subseteq C\}\), and the polar cone of C is \({C^{\ominus }:=\big \{{u\in X}~:~{\sup _{c\in C}\langle c,u\rangle \le 0}\big \}, } \)

  18. Suppose that U is a closed affine subspace of X. We use \(\mathrm{par}U\) to denote the parallel space of U defined by \(\mathrm{par}U:=U-U\).

References

  1. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, Berlin (2011)

    Book  MATH  Google Scholar 

  2. Borwein, J.M., Vanderwerff, J.G.: Convex Functions. Cambridge University Press, Cambridge (2010)

    MATH  Google Scholar 

  3. Brezis, H.: Operateurs Maximaux Monotones et Semi-Groupes de Contractions dans les Espaces de Hilbert. North-Holland/Elsevier, Amsterdam (1973)

    MATH  Google Scholar 

  4. Burachik, R.S., Iusem, A.N.: Set-Valued Mappings and Enlargements of Monotone Operators. Springer-Verlag, Berlin (2008)

    MATH  Google Scholar 

  5. Simons, S.: Minimax and Monotonicity. Springer-Verlag, Berlin (1998)

    Book  MATH  Google Scholar 

  6. Simons, S.: From Hahn-Banach to Monotonicity. Springer-Verlag, Berlin (2008)

    MATH  Google Scholar 

  7. Zeidler, E.: Nonlinear Functional Analysis and Its Applications II/A: Linear Monotone Operators. Springer-Verlag, Berlin (1990)

    Book  MATH  Google Scholar 

  8. Zeidler, E.: Nonlinear Functional Analysis and Its Applications II/B: Nonlinear Monotone Operators. Springer-Verlag, Berlin (1990)

    Book  MATH  Google Scholar 

  9. Zeidler, E.: Nonlinear Functional Analysis and Its Applications I: Fixed Point Theorems. Springer-Verlag, Berlin (1993)

    MATH  Google Scholar 

  10. Tseng, P.: Applications of a splitting algorithm to decomposition in convex programming and variational inequalities. SIAM J. Control Optim. 29, 119–138 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  11. Lemaire, B.: Which fixed point does the iteration method select? In: Lecture Notes in Economics and Mathematical Systems, Vol 452, pp. 154–167. (1979)

  12. Combettes, P.L.: Solving monotone inclusions via compositions of nonexpansive averaged operators. Optimization 53, 475–504 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  13. Attouch, H., Briceño-Arias, L.M., Combettes, P.L.: A parallel splitting method for coupled monotone inclusions. SIAM J. Control Optim. 48, 3246–3270 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  14. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Model. Simul. 4, 1168–1200 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  15. Combettes, P.L., Dũng, Đ., Vũ, B.C.: Dualization of signal recovery problems. Set-Valued Var. Anal. 18, 373–404 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  16. Bauschke, H.H., Hare, W.L., Moursi, W.M.: Generalized solutions for the sum of two maximally monotone operators. SIAM J. Control Optim. 52, 1034–1047 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  17. Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Springer-Verlag, Berlin (2009). corrected 3rd printing

    MATH  Google Scholar 

  18. Bruck, R.E., Reich, S.: Nonexpansive projections and resolvents of accretive operators in Banach spaces. Houst. J. Math. 3, 459–470 (1977)

    MathSciNet  MATH  Google Scholar 

  19. Baillon, J.B., Bruck, R.E., Reich, S.: On the asymptotic behavior of nonexpansive mappings and semigroups in Banach spaces. Houst. J. Math. 4, 1–9 (1978)

    MathSciNet  MATH  Google Scholar 

  20. Pazy, A.: Asymptotic behavior of contractions in Hilbert space. Israel J. Math. 9, 235–240 (1971)

    Article  MathSciNet  MATH  Google Scholar 

  21. Bauschke, H.H., Dao, M.N., Moursi, W.M.: On Fejér monotone sequences and nonexpansive mappings. Linear Nonlinear Anal. 1, 287–295 (2015)

    MathSciNet  MATH  Google Scholar 

  22. Bauschke, H.H., Moursi, W.M.: The Douglas-Rachford algorithm for two (not necessarily intersecting) affine subspace. SIAM J. Optim. 26, 968–985 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  23. Goebel, K., Reich, S.: Uniform Convexity, Hyperbolic Geometry, and Nonexpansive Mappings. Marcel Dekker, New York (1984)

    MATH  Google Scholar 

  24. Attouch, H., Théra, M.: A general duality principle for the sum of two operators. J. Convex Anal. 3, 1–24 (1996)

    MathSciNet  MATH  Google Scholar 

  25. Svaiter, B.F.: On weak convergence of the Douglas-Rachford method. SIAM J. Control Optim. 49, 280–287 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  26. Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  27. Bauschke, H.H., Boţ, R.I., Hare, W.L., Moursi, W.M.: Attouch-Théra duality revisited: paramonotonicity and operator splitting. J. Approx. Theory 164, 1065–1084 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  28. Bauschke, H.H., Wang, X., Yao, L.: Rectangularity and paramonotonicity of maximally monotone operators. Optimization 63, 487–504 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  29. Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  30. Baillon, J.B., Haddad, G.: Quelques propriétés des opérateurs angle-bornés et n-cycliquement monotones. Israel J. Math. 26, 137–150 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  31. Rockafellar, R.T.: On the maximal monotonicity of subdifferential mappings. Pac. J. Math. 33, 209–216 (1970)

    Article  MathSciNet  MATH  Google Scholar 

  32. Gossez, J.-P.: Opérateurs monotones non linéaires dans les espaces de Banach non réflexifs. J. Math. Anal. Appl. 34, 371–395 (1971)

    Article  MathSciNet  MATH  Google Scholar 

  33. Iusem, A.N.: On some properties of paramonotone operators. J. Convex Anal. 5, 269–278 (1998)

    MathSciNet  MATH  Google Scholar 

  34. Brezis, H., Haraux, A.: Image d’une somme d’opérateurs monotones et applications. Israel J. Math. 23, 165–186 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  35. Burachik, R.S., Jeyakumar, V.: A new geometric condition for Fenchel’s duality in infinite dimensional spaces. J. Convex Anal. 12, 279–290 (2005)

    MathSciNet  MATH  Google Scholar 

  36. Moreau, J.-J.: Proximité et dualité dans un espace hilbertien. Bull. Soc. Math. Fr. 93, 273–299 (1965)

    Article  MATH  Google Scholar 

  37. Zarantonello, E.H.: Projections on convex sets in Hilbert space and spectral theory. In: Zarantonello, E.H. (ed.) Contributions to Nonlinear Functional Analysis, pp. 237–424. Academic Press, New York (1971)

    Chapter  Google Scholar 

  38. Eckstein, J.: Splitting Methods for Monotone Operators with Applications to Parallel Optimization, Ph.D. Thesis, MIT (1989)

  39. Bauschke, H.H., Hare, W.L., Moursi, W.M.: On the range of the Douglas-Rachford operator. Math. Oper. Res. 41, 884–897 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  40. Bauschke, H.H., Borwein, J.M.: Dykstra’s alternating projection algorithm for two sets. J. Approx. Theory 79, 418–443 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  41. Bauschke, H.H., Moffat, S.M., Wang, X.: Firmly nonexpansive mappings and maximally monotone operators: correspondence and duality. Set-Valued Var. Anal. 20, 131–153 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  42. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)

    Book  MATH  Google Scholar 

  43. Bauschke, H.H., Moffat, S.M., Wang, X.: Near equality, near convexity, sums of maximally monotone operators, and averages of firmly nonexpansive mappings. Math. Program. (Ser. B) 139, 55–70 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  44. Bauschke, H.H., Lukens, B., Moursi, W.M.: Affine nonexpansive operators, Attouch–Théra duality and the Douglas–Rachford algorithm, to appear in Set-Valued Var. Anal. arXiv:1603.09418 [math.OC]

  45. Baillon, J.B.: Quelques propriétés de convergence asymptotique pour les contractions impaires. In: Comptes rendus de l’Académie des Sciences 238, Aii, pp. A587–A590 (1976)

  46. Bauschke, H.H., Deutsch, F., Hundal, H., Park, S.-H.: Accelerating the convergence of the method of alternating projections. Trans. Am. Math. Soc. 355, 3433–3461 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  47. Conway, J.B.: A Course in Functional Analysis. Springer-Verlag, Berlin (1990)

    MATH  Google Scholar 

  48. Pierra, G.: Decomposition through formalization in a product space. Math. Program. 28, 96–115 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  49. Combettes, P.L., Vũ, B.C.: Variable metric forward-backward splitting with applications to monotone inclusions in duality. Optimization 63, 1289–1318 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  50. Knopp, K.: Infinite Sequences and Series. Dover, New York (1956)

    MATH  Google Scholar 

  51. Moffat, S.M., Moursi, W.M., Wang, X.: Nearly convex sets: fine properties and domains or ranges of subdifferentials of convex functions. Math. Program. (Ser. A) 160, 193–223 (2016)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The author thanks Heinz Bauschke for his constructive comments and support. The author also thanks two anonymous referees for their careful refereeing of the paper.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Walaa M. Moursi.

Additional information

Communicated by Michel Théra.

Appendix

Appendix

Proof of Example 5.1

The claim about firm nonexpansiveness of A follows from, e.g. [37, Equation 1.6 on page 241] or [23, Section 3] and maximal monotonicity of B follows from Fact 3.2(i) applied to \(f^*\). Using Fact 3.2(ii) and [42, Example on page 218], we see that \(\mathrm{dom}{\partial }f=\mathrm{ran}({\partial }f)^{-1}= \mathrm{ran}{\partial }f^* =\mathrm{ran} B= \big \{{(\xi _1,\xi _2)}~:~{\xi _1>0, \xi _2\in \mathbb {R}}\big \} \) \( \cup \big \{{(0,\xi _2)}~:~{|\xi _2|\ge 1}\big \}\). Note that in view of Theorem 5.1(i) we have \( \big \{{(\xi _1,\xi _2)}~:~{\xi _1>0, \xi _2\in \mathbb {R}}\big \}=\mathrm{ri}(\mathrm{ran}A+\mathrm{ran}B) \) \(\subseteq \mathrm{ran}(\mathrm{Id}-T)\subseteq \mathrm{cl}{(\mathrm{ran}A+\mathrm{ran}B)} =\big \{{(\xi _1,\xi _2)}~:~{\xi _1\ge 0, \xi _2\in \mathbb {R}}\big \}\). Therefore, we only need to check the points in \(\big \{{(0,\beta )}~:~{\beta \in \mathbb {R}}\big \}\). To proceed further we recall that (see [51, Example 6.5])

$$\begin{aligned} {\partial }f(\xi _1,\xi _2)= {\left\{ \begin{array}{ll} \varnothing , &{}\text {if }\xi _{1}<0;\\ \varnothing , &{} \text {if } \xi _1 =0 \text{ and } |\xi _2|<1;\\ \mathbb {R}_{-}\times \left\{ 1\right\} , &{} \text {if } \xi _1=0 \text{ and } \xi _2\ge 1; \\ \mathbb {R}_{-}\times \left\{ -1\right\} , &{} \text {if } \xi _1=0 \text { and }\xi _2\le -1; \\ \mathrm{conv}\left\{ (-\tfrac{1}{2}{\xi _1}^{-{1}/{2}},0),(0,1)\right\} , &{} \text {if } {\xi _2}= 1-\sqrt{\xi _1} \text{ and } 0<\xi _1<1;\\ \mathrm{conv}\left\{ (-\tfrac{1}{2}{\xi _1}^{-{1}/{2}},0),(0,-1)\right\} , &{} \text {if } {-\xi _2}= 1-\sqrt{\xi _1} \text{ and } 0<\xi _1<1;\\ (-\tfrac{1}{2}{\xi _1}^{-\tfrac{1}{2}},0), &{} \text {if } 0<\xi _1<1 \text{ and } 1-\sqrt{\xi _1}>|\xi _2|;\\ (0,1), &{} \text {if }0<\xi _{1}<1 \text { and } \xi _2> 1-\sqrt{\xi _1};\\ (0,-1), &{} \text {if }0<\xi _{1}<1 \text { and } -\xi _2> 1-\sqrt{\xi _1};\\ \mathrm{conv}\left\{ (-\tfrac{1}{2},0),(0,1),(0,-1)\right\} , &{} \text {if } \xi _1=1 \text{ and } \xi _2=0; \\ \mathrm{conv}\left\{ (0,1),(0,-1)\right\} , &{} \text{ if } \xi _1>1 \text { and } \xi _2=0;\\ (0,1), &{} \text {if }\xi _{1}>1 \text { and } \xi _2>0;\\ (0,-1), &{} \text {if }\xi _{1}>1 \text { and } -\xi _2 > 0. \end{array}\right. }\nonumber \\ \end{aligned}$$
(28)

Let \(\beta \in \mathbb {R}\). In view of Proposition 4.1 and Fact 3.2(ii) we have

$$\begin{aligned} (0,\beta )\in \mathrm{ran}(\mathrm{Id}-T_{{}{\mathrm{FB}}})&\Leftrightarrow (\exists (\xi _1,\xi _2)\in \mathbb {R}^2)~ (0,\beta )\in P_{\mathbb {R}^2_+} (\xi _1,\xi _2) + {\partial }f^* (\xi _1,\xi _2-\beta )\end{aligned}$$
(29a)
$$\begin{aligned}&\quad =P_{\mathbb {R}^2_+} (\xi _1,\xi _2)+ ({\partial }f)^{-1} (\xi _1,\xi _2-\beta )\end{aligned}$$
(29b)
$$\begin{aligned}&\Leftrightarrow (\exists (\xi _1,\xi _2){\in } \mathbb {R}^2)~ (0,\beta ){-}P_{\mathbb {R}_+^2} (\xi _1,\xi _2) {\in } ({\partial }f)^{-1} (\xi _1,\xi _2{-}\beta )\end{aligned}$$
(29c)
$$\begin{aligned}&\Leftrightarrow (\exists (\xi _1,\xi _2){\in } \mathbb {R}^2)~(\xi _1,\xi _2-\beta ){\in } {\partial }f \left( (0,\beta )-P_{\mathbb {R}^2_+} (\xi _1,\xi _2)\right) . \end{aligned}$$
(29d)

We argue by cases using (28) and Example 29.

Case 1: :

\(\xi _1\ge 0\) and \(\xi _2\ge 0\). Then \((0,\beta )\in \mathrm{ran}(\mathrm{Id}-T_{{}{\mathrm{FB}}})\) \(\Leftrightarrow (\exists (\xi _1,\xi _2)\in \mathbb {R}^2)~ (\xi _1,\xi _2-\beta )\in {\partial }f ( (0,\beta )-P_{\mathbb {R}_+^2} (\xi _1,\xi _2))\) \( ={\partial }f (-\xi _1,\beta -\xi _2))\) \(\Leftrightarrow \) [\((\exists (\xi _1,\xi _2)\in \mathbb {R}^2)~\xi _1=0, \xi _2-\beta =1 \text { and } \beta -\xi _2\ge 1\) or \(\xi _1=0, \xi _2-\beta =-1 \text { and } \beta - \xi _2\le -1\)], which is impossible.

Case 2: :

\(\xi _1\le 0\) and \(\xi _2\le 0\). Then \((0,\beta )\in \mathrm{ran}(\mathrm{Id}-T_{{}{\mathrm{FB}}})\) \(\Leftrightarrow (\exists (\xi _1,\xi _2)\in \mathbb {R}^2)~ (\xi _1,\xi _2-\beta )\in {\partial }f ( (0,\beta )-P_{\mathbb {R}_+^2} (\xi _1,\xi _2))\) \( ={\partial }f (0,\beta ))\) \(\Leftrightarrow \) [\((\exists (\xi _1,\xi _2)\in \mathbb {R}^2)~ \xi _1\le 0, \xi _2-\beta =1 \text { and } \beta \ge 1\) or \(\xi _1\le 0, \xi _2-\beta =-1 \text { and } \beta \le -1\) ] \(\Leftrightarrow \) [ \((\exists (\xi _1,\xi _2)\) \(\in \mathbb {R}^2)~ \xi _1\le 0, \xi _2=\beta +1\ge 2 \) or \(\xi _1\le 0, \xi _2=\beta -1\le -2\)]. Since \(\xi _2\le 0\) we conclude that \(\beta \le -1\).

Case 3: :

\(\xi _1> 0\) and \(\xi _2 < 0\). Then \((0,\beta )\in \mathrm{ran}(\mathrm{Id}-T_{{}{\mathrm{FB}}})\) \(\Leftrightarrow (\exists (\xi _1,\xi _2)\in \mathbb {R}^2)~ (\xi _1,\xi _2-\beta )\in {\partial }f ( (0,\beta )-P_{\mathbb {R}_+^2} (\xi _1,\xi _2)) \) \( ={\partial }f (-\xi _1,\beta )\) \(\Rightarrow \) [\( \xi _1>0\) and by (28) \(-\xi _1>0\)] which is impossible.

Case 4: :

\(\xi _1< 0\) and \(\xi _2>0\). Then, \((0,\beta )\in \mathrm{ran}(\mathrm{Id}-T_{{}{\mathrm{FB}}})\) \(\Leftrightarrow (\exists (\xi _1,\xi _2)\in \mathbb {R}^2)~ (\xi _1,\xi _2-\beta )\in {\partial }f ( (0,\beta )-P_{\mathbb {R}_+^2} (\xi _1,\xi _2)) \) \( ={\partial }f (0,\beta -\xi _2)\) \(\Leftrightarrow \)[\(\xi _1<0, \xi _2-\beta =1 \text { and }\beta -\xi _2\ge 1 \text { or } \xi _1<0, \xi _2-\beta =-1 \text { and }\beta -\xi _2\le -1\)], which never occurs.

Altogether we conclude that \(\mathrm{ran}(\mathrm{Id}-T_{{}{\mathrm{FB}}})= \big \{{(\xi _1,\xi _2)}~:~{\xi _1>0, \xi _2\in \mathbb {R}}\big \} \cup \big \{{(0,\xi _2)}~:~{\xi _2\le -1}\big \}\), as claimed. \(\square \)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moursi, W.M. The Forward–Backward Algorithm and the Normal Problem. J Optim Theory Appl 176, 605–624 (2018). https://doi.org/10.1007/s10957-017-1113-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-017-1113-4

Keywords

Mathematics Subject Classification

Navigation