Skip to main content
SpringerLink
Log in

Douglas–Rachford splitting for nonconvex optimization with application to nonconvex feasibility problems

  • Full Length Paper
  • Series A
  • Published:
Mathematical Programming Submit manuscript

Abstract

We adapt the Douglas–Rachford (DR) splitting method to solve nonconvex feasibility problems by studying this method for a class of nonconvex optimization problem. While the convergence properties of the method for convex problems have been well studied, far less is known in the nonconvex setting. In this paper, for the direct adaptation of the method to minimize the sum of a proper closed function g and a smooth function f with a Lipschitz continuous gradient, we show that if the step-size parameter is smaller than a computable threshold and the sequence generated has a cluster point, then it gives a stationary point of the optimization problem. Convergence of the whole sequence and a local convergence rate are also established under the additional assumption that f and g are semi-algebraic. We also give simple sufficient conditions guaranteeing the boundedness of the sequence generated. We then apply our nonconvex DR splitting method to finding a point in the intersection of a closed convex set C and a general closed set D by minimizing the squared distance to C subject to D. We show that if either set is bounded and the step-size parameter is smaller than a computable threshold, then the sequence generated from the DR splitting method is actually bounded. Consequently, the sequence generated will have cluster points that are stationary for an optimization problem, and the whole sequence is convergent under an additional assumption that C and D are semi-algebraic. We achieve these results based on a new merit function constructed particularly for the DR splitting method. Our preliminary numerical results indicate that our DR splitting method usually outperforms the alternating projection method in finding a sparse solution of a linear system, in terms of both the solution quality and the number of iterations taken.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. We note that the assumption where f has a Lipschitz continuous gradient is commonly used in the literature of first-order methods; see for example, [5, Section 5].

  2. We note that, a more general and informative statement that applies to the square distance function of a possibly nonconvex but prox-regular set can be found in [29, Theorem 1.3].

  3. We note that there are also another analytical examples constructed in [9, 20], where the authors showed that the classical DR splitting method need not to be convergent. For simplicity, we do not discuss it here.

  4. We also solved a couple instances using directly a small \(\gamma < \gamma _0\) in the DR splitting method. The sequence generated tends to get stuck at stationary points that are not global minimizers.

  5. We report \(\frac{1}{2} d_C^2(z^t)\) for DR splitting, and \(\frac{1}{2} d_C^2(x^t)\) for alternating projection.

  6. The two thresholds are different and hence \(\mathrm{succ} + \mathrm{fail}\) is not necessarily 50. We set different thresholds so as to see if it is easy to determine whether the method has got stuck at stationary points that are not global minimizers.

References

  1. Aragón, A.F., Borwein, J.M.: Global convergence of a non-convex Douglas–Rachford iteration. J. Global Optim. 57, 1–17 (2012)

    MathSciNet  MATH  Google Scholar 

  2. Aragón, A.F., Borwein, J.M., Tam, M.K.: Douglas–Rachford feasibility methods for matrix completion problems. ANZIAM J. 55, 299–326 (2014)

    MathSciNet  MATH  Google Scholar 

  3. Aragón, A.F., Borwein, J.M., Tam, M.K.: Recent results on Douglas–Rachford methods for combinatorial optimization problems. J. Optim. Theory Appl. 163, 1–30 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  4. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems. An approach based on the Kurdyka–Lojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  5. Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods. Math. Program. 137, 91–129 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bauschke, H.H., Borwein, J.M.: On the convergence of von Neumann’s alternating projection algorithm for two sets. Set-Valued Anal. 1, 185–212 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  7. Bauschke, H.H., Borwein, J.M.: On projection algorithms for solving convex feasibility problems. SIAM Rev. 38, 367–426 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  8. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, Berlin (2011)

    Book  MATH  Google Scholar 

  9. Bauschke, H.H., Noll, D.: On the local convergence of the Douglas–Rachford algorithm. Arch. Math. 102, 589–600 (2014)

  10. Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17, 1205–1223 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  11. Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradients of stratifiable functions. SIAM J. Optim. 18, 556–572 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  12. Borwein, J.M., Li, G., Yao, L.J.: Analysis of the convergence rate for the cyclic projection algorithm applied to basic semialgebraic convex sets. SIAM J. Optim. 24, 498–527 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  13. Combettes, P.L., Pesquet, J.-C.: A Douglas–Rachford splitting approach to nonsmooth convex variational signal recovery. IEEE J. Sel. Top. Signal Proces. 1, 564–574 (2007)

    Article  Google Scholar 

  14. Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two or three space variables. Trans. Am. Math. Soc. 82, 421–439 (1956)

    Article  MathSciNet  MATH  Google Scholar 

  15. Eckstein, J., Bertsekas, D.P.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  16. Gandy, S., Recht, B., Yamada, I.: Tensor completion and low-\(n\)-rank tensor recovery via convex optimization. Inverse Probl. 27, 025010 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  17. Gong, P., Zhang, C., Lu, Z., Huang, J., Ye, J.: A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems. In: The 30th International Conference on Machine Learning (ICML 2013)

  18. He, B., Yuan, X.: On the \(O(1/n)\) convergence rate of the Douglas–Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  19. Hesse, R., Luke, D.R.: Nonconvex notions of regularity and convergence of fundamental algorithms for feasibility problems. SIAM J. Optim. 23, 2397–2419 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  20. Hesse, R., Luke, D.R., Neumann, P.: Alternating projections and Douglas–Rachford for sparse affine feasibility. IEEE Trans. Signal. Proces. 62, 4868–4881 (2014)

    Article  MathSciNet  Google Scholar 

  21. Lewis, A.S., Luke, D.R., Malick, J.: Local convergence for alternating and averaged nonconvex projections. Found. Comput. Math. 9, 485–513 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  22. Lewis, A.S., Malick, J.: Alternating projections on manifolds. Math. Oper. Res. 33, 216–234 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  23. Li, G., Mordukhovich, B.S., Pham, T.S.: New fractional error bounds for polynomial systems with applications to Hölderian stability in optimization and spectral theory of tensors. Math. Program. 153, 333–362 (2015)

  24. Lions, P.-L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  25. Lu, Z., Zhang, Y.: Sparse approximation via penalty decomposition methods. SIAM J. Optim. 23, 2448–2478 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  26. Luke, D.R.: Finding best approximation pairs relative to a convex and a prox-regular set in Hilbert space. SIAM J. Optim. 19, 714–739 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  27. Patrinos, P., Stella, L., Bemporad, A.: Douglas–Rachford splitting: complexity estimates and accelerated variants. In: Proceedings of the 2014 IEEE 53rd Annual Conference on Decision and Control (CDC), pp. 4234–4239. IEEE, Los Angeles, CA (2014)

  28. Phan, H.: Linear convergence of the Douglas–Rachford method for two closed sets. Optimization. arXiv:1401.6509

  29. Poliquin, R.A., Rockafellar, R.T., Thibault, L.: Local differentiability of distance functions. Trans. Am. Math. Soc. 352, 5231–5249 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  30. Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Springer, Berlin (1998)

    Book  MATH  Google Scholar 

  31. Sun, D., Toh, K.-C., Yang, L.: A convergent 3-block semi-proximal alternating direction method of multipliers for conic programming with 4-type of constraints. SIAM J. Optim. 25, 882–915 (2015)

  32. Zeng, J., Lin, S., Wang, Y., Xu, Z.: \(L_{1/2}\) regularization: convergence of iterative half thresholding algorithm. IEEE Trans. Signal Process. 62, 2317–2329 (2014)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

We would like to thank the two anonymous referees for their comments that helped improve the manuscript.

Author information

Authors and Affiliations

  1. Department of Applied Mathematics, University of New South Wales, Sydney, 2052, Australia

    Guoyin Li

  2. Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong

    Ting Kei Pong

Authors
  1. Guoyin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ting Kei Pong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ting Kei Pong.

Additional information

Guoyin Li was partially supported by a research Grant from Australian Research Council. Ting Kei Pong was supported partly by a research Grant from Hong Kong Polytechnic University.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, G., Pong, T.K. Douglas–Rachford splitting for nonconvex optimization with application to nonconvex feasibility problems. Math. Program. 159, 371–401 (2016). https://doi.org/10.1007/s10107-015-0963-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-015-0963-5

Mathematics Subject Classification

Search

Navigation