Skip to main content
Log in

Greedy expansions in convex optimization

  • Published:
Proceedings of the Steklov Institute of Mathematics Aims and scope Submit manuscript

Abstract

This paper is a follow-up to the author’s previous paper on convex optimization. In that paper we began the process of adjusting greedy-type algorithms from nonlinear approximation for finding sparse solutions of convex optimization problems. We modified there the three most popular greedy algorithms in nonlinear approximation in Banach spaces-Weak Chebyshev Greedy Algorithm, Weak Greedy Algorithm with Free Relaxation, and Weak Relaxed Greedy Algorithm-for solving convex optimization problems. We continue to study sparse approximate solutions to convex optimization problems. It is known that in many engineering applications researchers are interested in an approximate solution of an optimization problem as a linear combination of elements from a given system of elements. There is an increasing interest in building such sparse approximate solutions using different greedy-type algorithms. In this paper we concentrate on greedy algorithms that provide expansions, which means that the approximant at the mth iteration is equal to the sum of the approximant from the previous, (m − 1)th, iteration and one element from the dictionary with an appropriate coefficient. The problem of greedy expansions of elements of a Banach space is well studied in nonlinear approximation theory. At first glance the setting of a problem of expansion of a given element and the setting of the problem of expansion in an optimization problem are very different. However, it turns out that the same technique can be used for solving both problems. We show how the technique developed in nonlinear approximation theory, in particular, the greedy expansions technique, can be adjusted for finding a sparse solution of an optimization problem given by an expansion with respect to a given dictionary.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. N. K. Bari, Trigonometric Series (Fizmatgiz, Moscow, 1961); Engl. transl.: A Treatise on Trigonometric Series (Pergamon, Oxford, 1964), Vols. I, II.

    Google Scholar 

  2. J. M. Borwein and A. S. Lewis, Convex Analysis and Nonlinear Optimization. Theory and Examples (Springer, New York, 2006).

    Book  MATH  Google Scholar 

  3. V. Chandrasekaran, B. Recht, P. A. Parrilo, and A. S. Willsky, “The convex algebraic geometry of linear inverse problems,” in Communication, Control, and Computing: Proc. 48th Annu. Allerton Conf. (IEEE Press, Piscataway, NJ, 2010), pp. 699–703.

    Google Scholar 

  4. R. A. DeVore, “Nonlinear approximation,” Acta Numerica 7, 51–150 (1998).

    Article  MathSciNet  Google Scholar 

  5. P. Habala, P. Hájek, and V. Zizler, Introduction to Banach Spaces (Matfyzpress, Prague, 1996), Vol. 1.

    MATH  Google Scholar 

  6. A. Nemirovski, Optimization. II: Numerical Methods for Nonlinear Continuous Optimization (Technion-Isr. Inst. Technol., Haifa, 1999).

    Google Scholar 

  7. Yu. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course (Kluwer, Boston, 2004).

    Book  Google Scholar 

  8. S. Shalev-Shwartz, N. Srebro, and T. Zhang, “Trading accuracy for sparsity in optimization problems with sparsity constraints,” SIAM J. Optim. 20(6), 2807–2832 (2010).

    Article  MATH  MathSciNet  Google Scholar 

  9. V. N. Temlyakov, “Greedy algorithms and m-term approximation with regard to redundant dictionaries,” J. Approx. Theory 98, 117–145 (1999).

    Article  MATH  MathSciNet  Google Scholar 

  10. V. N. Temlyakov, “Weak greedy algorithms,” Adv. Comput. Math. 12(2–3), 213–227 (2000).

    Article  MATH  MathSciNet  Google Scholar 

  11. V. N. Temlyakov, “Greedy-type approximation in Banach spaces and applications,” Constr. Approx. 21(2), 257–292 (2005).

    Article  MATH  MathSciNet  Google Scholar 

  12. V. N. Temlyakov, “Greedy approximation,” Acta Numerica 17, 235–409 (2008).

    Article  MATH  MathSciNet  Google Scholar 

  13. V. Temlyakov, Greedy Approximation (Cambridge Univ. Press, Cambridge, 2011).

    Book  MATH  Google Scholar 

  14. V. N. Temlyakov, “Greedy approximation in convex optimization,” IMI Preprint No. 03 (Univ. South Carolina, Columbia, SC, 2012); arXiv: 1206.0392v1 [stat.ML].

    Google Scholar 

  15. V. N. Temlyakov, “Greedy expansions in convex optimization,” IMI Preprint No. 05 (Univ. South Carolina, Columbia, SC, 2012); arXiv: 1206.0393v1 [stat.ML].

    Google Scholar 

  16. A. Tewari, P. Ravikumar, and I. S. Dhillon, “Greedy algorithms for structurally constrained high dimensional problems,” Adv. Neural Inf. Process. Syst. 24, 882–890 (2011).

    Google Scholar 

  17. T. Zhang, “Sequential greedy approximation for certain convex optimization problems,” IEEE Trans. Inf. Theory 49(3), 682–691 (2003).

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to V. N. Temlyakov.

Additional information

Published in Russian in Trudy Matematicheskogo Instituta imeni V.A. Steklova, 2014, Vol. 284, pp. 252–270.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Temlyakov, V.N. Greedy expansions in convex optimization. Proc. Steklov Inst. Math. 284, 244–262 (2014). https://doi.org/10.1134/S0081543814010180

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0081543814010180

Keywords

Navigation