Hui Han Chin
ABSTRACT
We study theoretical runtime guarantees for a class of optimization problems that occur in a wide variety of inference problems. These problems are motivated by the LASSO framework and have applications in machine learning and computer vision. Our work shows a close connection between these problems and core questions in algorithmic graph theory. While this connection demonstrates the difficulties of obtaining runtime guarantees, it also suggests an approach of using techniques originally developed for graph algorithms.
We then show that most of these problems can be formulated as a grouped least squares problem, and give efficient algorithms for this formulation. Our algorithms rely on routines for solving quadratic minimization problems, which in turn are equivalent to solving linear systems. Some preliminary experimental work on image processing tasks are also presented.
- S. Arora, E. Hazan, and S. Kale. The multiplicative weights update method: a meta algorithm and applications. Technical report, Princeton University, 2005. 5Google Scholar
- H. Avron, G. Shklarski, and S. Toledo. On element SDD approximability. CoRR, abs/0911.0547, 2009. 2.1Google Scholar
- S. R. Becker, E. J. Candes, and M. Grant. Templates for convex cone problems with applications to sparse signal recovery. MATHEMATICAL PROGRAMMING COMPUTATION, 3:165, 2011. 1, 4.4Google ScholarCross Ref
- G. E. Blelloch, A. Gupta, I. Koutis, G. L. Miller, R. Peng, and K. Tangwongsan. Near linear-work parallel SDD solvers, low-diameter decomposition, and low-stretch subgraphs. In Proceedings of the 23rd ACM symposium on Parallelism in algorithms and architectures, SPAA '11, pages 13--22, New York, NY, USA, 2011. ACM. 2.1 Google ScholarDigital Library
- E. G. Boman, B. Hendrickson, and S. A. Vavasis. Solving elliptic finite element systems in near-linear time with support preconditioners. CoRR, cs.NA/0407022, 2004. 2.1Google Scholar
- S. Boyd and L. Vandenberghe. Convex Optimization. Camebridge University Press, 2004. 1, 4.1, C, C.1 Google ScholarDigital Library
- J. Candes, E. Compressive sampling. Proceedings of the International Congress of Mathematicians, 2006. 1, 1Google Scholar
- T. Chan and J. Shen. Image Processing And Analysis: Variational, Pde, Wavelet, And Stochastic Methods. SIAM, 2005. 3.1, 6.1 Google ScholarDigital Library
- P. Christiano, J. A. Kelner, A. Madry, D. Spielman, and S.-H. Teng. Electrical Flows, Laplacian Systems, and Faster Approximation of Maximum Flow in Undirected Graphs. In Proceedings of the 43rd ACM Symposium on Theory of Computing (STOC), 2011. 1, 2.1, 4.2, 5, 5, 5, 7, B, B Google ScholarDigital Library
- E. Cohen. Polylog-time and near-linear work approximation scheme for undirected shortest paths. J. ACM, 47(1):132--166, 2000. 1 Google ScholarDigital Library
- D. Coppersmith and S. Winograd. Matrix multiplication via arithmetical progressions. J. Symbolic Computation, 9:251--280, 1990. 2.1 Google ScholarDigital Library
- S. I. Daitch and D. A. Spielman. Support-graph preconditioners for 2-dimensional trusses. CoRR, abs/cs/0703119, 2007. 2.1Google Scholar
- S. I. Daitch and D. A. Spielman. Faster approximate lossy generalized flow via interior point algorithms. In Proceedings of the 40th annual ACM symposium on Theory of computing, STOC '08, pages 451--460, New York, NY, USA, 2008. ACM. 1, 5, A, C Google ScholarDigital Library
- J. Darbon and M. Sigelle. Image restoration with discrete constrained total variation part i: Fast and exact optimization. Journal of Mathematical Imaging and Vision, 26(3):261--276, 2006. 4.2 Google ScholarDigital Library
- Y. Freund and R. E. Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior, 29(1-2):79--103, October 1999. 5Google ScholarCross Ref
- A. V. Goldberg and S. Rao. Beyond the flow decomposition barrier. J. ACM, 45:783--797, September 1998. 4.2 Google ScholarDigital Library
- D. Goldfarb and W. Yin. Second-order cone programming methods for total variation-based image restoration. SIAM J. Sci. Comput, 27:622--645, 2004. 4.1, C Google ScholarDigital Library
- T. Goldstein and S. Osher. The split bregman method for l1-regularized problems. SIAM J. Img. Sci., 2:323--343, April 2009. 6.1, 1 Google ScholarDigital Library
- T. Hocking, J.-P. Vert, F. Bach, and A. Joulin. Clusterpath: an algorithm for clustering using convex fusion penalties. In L. Getoor and T. Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning (ICML-11), ICML '11, pages 745--752, New York, NY, USA, June 2011. ACM. 3.4Google Scholar
- D. S. Johnson and C. C. McGeoch, editors. Network Flows and Matching: First DIMACS Implementation Challenge. American Mathematical Society, Boston, MA, USA, 1993. 1 Google ScholarDigital Library
- J. A. Kelner, G. L. Miller, and R. Peng. Faster approximate multicommodity flow using quadratically coupled flows. In Proceedings of the 44th symposium on Theory of Computing, STOC '12, pages 1--18, New York, NY, USA, 2012. ACM. 2.1, 3.2 Google ScholarDigital Library
- V. Kolmogorov and R. Zabih. What energy functions can be minimized via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26:65--81, 2004. 4.2 Google ScholarDigital Library
- I. Koutis and G. Miller. The combinatorial multigrid solver. Conference Talk, March 2009. 6Google Scholar
- I. Koutis and G. L. Miller. A linear work, O(n1/6) time, parallel algorithm for solving planar Laplacians. In Proc. 18th ACM-SIAM Symposium on Discrete Algorithms (SODA 2007), 2007. 3.1 Google ScholarDigital Library
- I. Koutis, G. L. Miller, and R. Peng. Approaching optimality for solving sdd linear systems. In Proceedings of the 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, FOCS '10, pages 235--244, Washington, DC, USA, 2010. IEEE Computer Society. 2.1 Google ScholarDigital Library
- I. Koutis, G. L. Miller, and R. Peng. A nearly-m log n time solver for sdd linear systems. In Proceedings of the 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science, FOCS '11, pages 590--598, Washington, DC, USA, 2011. IEEE Computer Society. 2.1 Google ScholarDigital Library
- I. Koutis, G. L. Miller, and D. Tolliver. Combinatorial preconditioners and multilevel solvers for problems in computer vision and image processing. In International Symposium of Visual Computing, pages 1067--1078, 2009. 6 Google ScholarDigital Library
- S. Kullback and R. A. Leibler. On information and sufficiency. Ann. Math. Statist., 22(1):79--86, 1951. 5Google ScholarCross Ref
- N. Littlestone and M. K. Warmuth. The weighted majority algorithm. Inf. Comput., 108(2):212--261, Feb. 1994. 5 Google ScholarDigital Library
- C. A. Micchelli, L. Shen, and Y. Xu. Proximity algorithms for image models: denoising. Inverse Problems, 27(4):045009, 2011. 6.1, 1Google ScholarCross Ref
- K. Mulmuley, U. V. Vazirani, and V. V. Vazirani. Matching is as easy as matrix inversion. Combinatorica, 7(1):105--113, Jan. 1987. A Google ScholarDigital Library
- D. Mumford and J. Shah. Optimal approximations by piecewise smooth functions and associated variational problems. Communications on Pure and Applied Mathematics, 42:577--685, 1989. 2, 7Google ScholarCross Ref
- Y. NESTEROV. Gradient methods for minimizing composite objective function. CORE Discussion Papers 2007076, Universital' catholique de Louvain, Center for Operations Research and Econometrics (CORE), 2007. 1, 4.4Google Scholar
- P. Perez, M. Gangnet, and A. Blake. Poisson image editing. ACM Transactions on Graphics (SIGGRAPH'03), 22(3):313--318, 2003. 3.3, 6.2 Google ScholarDigital Library
- L. Rosasco, E. De, V. A. Caponnetto, M. Piana, and A. Verri. Are loss functions all the same. Neural Computation, 15, 2004. 7 Google ScholarDigital Library
- L. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithm. Physica D, 1(60):259--268, 1992. 1, 1, 6.1 Google ScholarDigital Library
- D. A. Spielman and S.-H. Teng. Nearly-linear time algorithms for graph partitioning, graph sparsification, and solving linear systems. In Proceedings of the 36th Annual ACM Symposium on Theory of Computing, pages 81--90, June 2004. 2.1 Google ScholarDigital Library
- D. A. Spielman and S.-H. Teng. Nearly-linear time algorithms for preconditioning and solving symmetric, diagonally dominant linear systems. CoRR, abs/cs/0607105, 2006. 1, 2.1Google Scholar
- V. Strassen. Gaussian elimination is not optimal. Numerische Mathematik, 13:354--356, 1969. 2.1Google ScholarDigital Library
- R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society (Series B), 58:267--288, 1996. 1Google ScholarCross Ref
- R. Tibshirani, M. Saunders, S. Rosset, J. Zhu, and K. Knight. Sparsity and smoothness via the fused lasso. Journal of the Royal Statistical Society Series B, 67(1):91--108, 2005. 2Google ScholarCross Ref
- V. Vassilevska Williams. Breaking the Coppersmith-Winograd barrier. In Proceedings of the 44th symposium on Theory of Computing, STOC '12, 2012. 2.1Google Scholar
- B. Wohlberg and P. Rodriguez. An iteratively reweighted norm algorithm for minimization of total variation functionals. Signal Processing Letters, IEEE, 14(12):948--951, dec. 2007. 4.3Google ScholarCross Ref
- M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society, Series B, 68:49--67, 2006. 2Google ScholarCross Ref
Index Terms
- Runtime guarantees for regression problems
Recommendations
A unified framework for bivariate clustering and regression problems via mixed-integer linear programming
AbstractClustering and regression are two of the most important problems in data analysis and machine learning. Recently, mixed-integer linear programs (MILPs) have been presented in the literature to solve these problems. By modelling the ...
Using locally weighted learning to improve SMOreg for regression
PRICAI'06: Proceedings of the 9th Pacific Rim international conference on Artificial intelligenceShevade et al.[1] are successful in extending some improved ideas to Smola and Scholkopf's SMO algorithm[2] for solving regression problems, simply named SMOreg. In this paper, we use SMOreg in exactly the same way as linear regression(LR) is used in ...
Algorithms for Generalized Clusterwise Linear Regression
Clusterwise linear regression CLR, a clustering problem intertwined with regression, finds clusters of entities such that the overall sum of squared errors from regressions performed over these clusters is minimized, where each cluster may have ...
Comments