Optimality Conditions for Sparse Quadratic Optimization Problem

Abstract

Sparse models are preferred in machine learning problems because of their computational interpretability and it is seen in many applications such as in Google Page Rank, classification and regression problems, in the method of Principal Component Analysis (PCA) that finds the most important features and further applications in graphical models. In this study, we derive optimality conditions for the quadratic problem which has cardinality constraint imposing sparse solution. Our Quadratic model is a special application of ensemble pruning model in ensemble learning algorithms. Here, we refer to our previous study on this application in ensemble selection for clustering problems. The quadratic model proposed in this study optimizes trade-off between accuracy and diversity of discriminant functions (classifiers) simultaneously so that the best candidates of ensemble are selected for prediction step. The selection of the best classifiers in the ensemble is crucial for the overall performance of ensemble learning algorithms since redundant/outlier solutions in the ensemble library will decrease the overall prediction accuracy. In order to eliminate such candidates, both accuracy and diversity are taken into account when selecting the best subset of the ensemble. The cardinality constraint is further relaxed by considering various approximations such as $l_1$-norm regularization and student t-log likelihood approximations. Under these considerations and approximations, we build optimality criteria for our quadratic optimization problem with a cardinality constraint.