Abstract
This paper is a follow-up to the author’s previous paper on convex optimization. In that paper we began the process of adjusting greedy-type algorithms from nonlinear approximation for finding sparse solutions of convex optimization problems. We modified there the three most popular greedy algorithms in nonlinear approximation in Banach spaces-Weak Chebyshev Greedy Algorithm, Weak Greedy Algorithm with Free Relaxation, and Weak Relaxed Greedy Algorithm-for solving convex optimization problems. We continue to study sparse approximate solutions to convex optimization problems. It is known that in many engineering applications researchers are interested in an approximate solution of an optimization problem as a linear combination of elements from a given system of elements. There is an increasing interest in building such sparse approximate solutions using different greedy-type algorithms. In this paper we concentrate on greedy algorithms that provide expansions, which means that the approximant at the mth iteration is equal to the sum of the approximant from the previous, (m − 1)th, iteration and one element from the dictionary with an appropriate coefficient. The problem of greedy expansions of elements of a Banach space is well studied in nonlinear approximation theory. At first glance the setting of a problem of expansion of a given element and the setting of the problem of expansion in an optimization problem are very different. However, it turns out that the same technique can be used for solving both problems. We show how the technique developed in nonlinear approximation theory, in particular, the greedy expansions technique, can be adjusted for finding a sparse solution of an optimization problem given by an expansion with respect to a given dictionary.
Similar content being viewed by others
References
N. K. Bari, Trigonometric Series (Fizmatgiz, Moscow, 1961); Engl. transl.: A Treatise on Trigonometric Series (Pergamon, Oxford, 1964), Vols. I, II.
J. M. Borwein and A. S. Lewis, Convex Analysis and Nonlinear Optimization. Theory and Examples (Springer, New York, 2006).
V. Chandrasekaran, B. Recht, P. A. Parrilo, and A. S. Willsky, “The convex algebraic geometry of linear inverse problems,” in Communication, Control, and Computing: Proc. 48th Annu. Allerton Conf. (IEEE Press, Piscataway, NJ, 2010), pp. 699–703.
R. A. DeVore, “Nonlinear approximation,” Acta Numerica 7, 51–150 (1998).
P. Habala, P. Hájek, and V. Zizler, Introduction to Banach Spaces (Matfyzpress, Prague, 1996), Vol. 1.
A. Nemirovski, Optimization. II: Numerical Methods for Nonlinear Continuous Optimization (Technion-Isr. Inst. Technol., Haifa, 1999).
Yu. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course (Kluwer, Boston, 2004).
S. Shalev-Shwartz, N. Srebro, and T. Zhang, “Trading accuracy for sparsity in optimization problems with sparsity constraints,” SIAM J. Optim. 20(6), 2807–2832 (2010).
V. N. Temlyakov, “Greedy algorithms and m-term approximation with regard to redundant dictionaries,” J. Approx. Theory 98, 117–145 (1999).
V. N. Temlyakov, “Weak greedy algorithms,” Adv. Comput. Math. 12(2–3), 213–227 (2000).
V. N. Temlyakov, “Greedy-type approximation in Banach spaces and applications,” Constr. Approx. 21(2), 257–292 (2005).
V. N. Temlyakov, “Greedy approximation,” Acta Numerica 17, 235–409 (2008).
V. Temlyakov, Greedy Approximation (Cambridge Univ. Press, Cambridge, 2011).
V. N. Temlyakov, “Greedy approximation in convex optimization,” IMI Preprint No. 03 (Univ. South Carolina, Columbia, SC, 2012); arXiv: 1206.0392v1 [stat.ML].
V. N. Temlyakov, “Greedy expansions in convex optimization,” IMI Preprint No. 05 (Univ. South Carolina, Columbia, SC, 2012); arXiv: 1206.0393v1 [stat.ML].
A. Tewari, P. Ravikumar, and I. S. Dhillon, “Greedy algorithms for structurally constrained high dimensional problems,” Adv. Neural Inf. Process. Syst. 24, 882–890 (2011).
T. Zhang, “Sequential greedy approximation for certain convex optimization problems,” IEEE Trans. Inf. Theory 49(3), 682–691 (2003).
Author information
Authors and Affiliations
Corresponding author
Additional information
Published in Russian in Trudy Matematicheskogo Instituta imeni V.A. Steklova, 2014, Vol. 284, pp. 252–270.
Rights and permissions
About this article
Cite this article
Temlyakov, V.N. Greedy expansions in convex optimization. Proc. Steklov Inst. Math. 284, 244–262 (2014). https://doi.org/10.1134/S0081543814010180
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S0081543814010180