Elsevier

Pattern Recognition Letters

Volume 39, 1 April 2014, Pages 52-64
Pattern Recognition Letters

A framework for selection and fusion of pattern classifiers in multimedia recognition

https://doi.org/10.1016/j.patrec.2013.07.014Get rights and content

Highlights

  • Description and learning techniques are studied in four applications.

  • A new framework for selection and fusion of classifiers is created.

  • A new strategy for selecting classifiers based on diversity measures is shown.

  • Comparable results to well-known techniques from the literature are achieved with fewer classifiers.

Abstract

The frequent growth of visual data, either by countless monitoring video cameras wherever we go or the popularization of mobile devices that allow each person to create and edit their own images and videos have contributed enormously to the so-called “big-data revolution”. This shear amount of visual data gives rise to a Pandora box of new visual classification problems never imagined before. Image and video classification tasks have been inserted in different and complex applications and the use of machine learning-based solutions has become the most popular approach for several applications. Notwithstanding, there is no silver bullet that solves all the problems, i.e., it is not possible to characterize all images of different domains with the same description method nor is it possible to use the same learning method to achieve good results in any kind of application. In this work, we aim at proposing a framework for classifier selection and fusion. Our method seeks to combine image characterization and learning methods by means of a meta-learning approach responsible for assessing which methods contribute more towards the solution of a given problem. The framework uses a strategy of classifier selection which pinpoints the less correlated, yet effective, classifiers through a series of diversity measures analysis. The experiments show that the proposed approach achieves comparable results to well-known algorithms from the literature on four different applications but using less learning and description methods as well as not incurring in the curse of dimensionality and normalization problems common to some fusion techniques. Furthermore, our approach is able to achieve effective classification results using very reduced training sets. The proposed method is also amenable to continuous learning and flexible enough for implementation in highly-parallel architectures.

Introduction

The ever growing presence of sensors in our daily lives led us to the so called big-data revolution and within this shear amount of data, visual data is of particular interest. Citing a recent The New York Times article (Brooks, 2013) “The philosophy of data”, surely the rising philosophy of the day, is data-ism in which everyone wants to take advantage of data as it is the holy grail of contemporaneity. But, amidst such a massive amount of data, the question is how to process such data to actually come out with useful conclusions?

Visual data is of particular interest in this revolution. The explosion of visual data makes us face many new challenges unthinkable two decades ago. Image and video classification tasks have been inserted in different and complex applications (e.g., data categorization in search, biometric recognition, and document indexing through visual content, object recognition, etc.) and the use of machine learning-based solutions has become the most popular approach for several applications. However, there is no silver bullet that solves all the problems which means that it is not possible to characterize all images of different domains with the same description method nor is it possible to use the same learning method to achieve good results in any kind of application.

In this work, we investigate the combination of several learning methods and image descriptors aiming at creating more effective classifiers. We propose a framework for automatically combining the most discriminative classifiers using the support vector machine technique (SVM), as well as exploring the use of diversity measures to select the less-correlated, yet effective, classifiers. We have performed experiments that demonstrate that the proposed framework for classifier fusion yields comparable results to the traditional fusion approaches but using less learning and description methods as well as not incurring in the curse of dimensionality problems, which are common to some fusion techniques. Another major advantage of the proposed method is that it yields good classification results using small training examples being more robust to the small sample size problem common in many classification techniques (Bishop, 2006).

This paper extends upon two previous conference papers of ours (Faria et al., 2012a, Faria et al., 2012b) and differs regarding the following aspects: (i) the related work section was thoroughly extended; (ii) a new approach for classifier selection is introduced; (iii) other diversity measures have been included as additional criteria for selecting the most appropriate classifiers to be combined; (iv) additional classification problems have been considered; (v) more experiments for comparing the proposed framework with other state-of-the-art ensemble approaches were performed. It is also worth mentioning that the framework proposed in Faria et al., 2012a, Faria et al., 2012b were used as baselines in our experiments.

We organized the remainder of this paper in five sections. Section 2 presents related work and background concepts necessary for the understanding of this paper. Section 3 describes the steps of the proposed framework for classifier fusion and for selecting the most appropriate classifiers based on diversity measures. Section 4 shows the experimental protocol we devised to validate our work, while Section 5 discusses the results. Finally, Section 6 concludes the paper and points out future research directions.

Section snippets

Related work

This section presents related work on image classification and information fusion (Section 2.1), as well as introduces background concepts necessary for the full understanding of this work (Section 2.2).

The classifier fusion framework

This section presents our framework for classifier fusion and selection.

Experimental protocol

In this section, we present the experimental setup we used in this work.

Experiments and discussion

This section discusses the results regarding the effectiveness and efficiency of the proposed framework using four different datasets (Caltech, Coffee, Freefoto, and Fruits). Before going to the results, we present an experiment showing that diversity measures can provide different opinions and the research possibilities this might open.

Final remarks and future work

This paper presented a framework for fusion and selection of classifiers using a diversity measure analysis and meta-learning on top of classifiers outcomes. Particularly in this paper, we have used the support vector machine technique (SVM) but other learning methods could be used as well. We also compared several different learning methods and image descriptors in four different classification problems (scene/object classification and crop/produce recognition) showing the proposed method is

Acknowledgments

The authors are grateful to CAPES (Grants 1260-12-0), CNPq (Grants 304352/2012-8, 484254/2012-0 and 306580/2012-8), FAPESP (Grants 2010/14910-0, 2012/18768-0 and 2010/05647-4), and Microsoft Research for the financial support.

References (71)

  • J.A. Benediktsson et al.

    Parallel consensual neural networks

    IEEE Trans. Neural Networks

    (1997)
  • C.M. Bishop

    Pattern Recognition and Machine Learning (Information Science and Statistics)

    (2006)
  • Boser, B.E., Guyon, I.M., Vapnik, V.N. 1992. A training algorithm for optimal margin classifiers. In: Workshop on...
  • L. Breiman

    Bagging predictors

    Mach. Learn.

    (1996)
  • L. Breiman

    Random forests

    Mach. Learn.

    (2001)
  • Brennan, R.L., Prediger, D.J. 1981. Coefficient kappa: some uses, misuses, and alternatives. Educ. Psychol. Meas. 41...
  • Brooks, D. 2013. The philosophy of data. In: The New York Times, A23, February,...
  • C.-C. Chang et al.

    LIBSVM: a library for support vector machines

    ACM Trans. Intell. Syst. Technol.

    (2011)
  • N. Cristianini et al.

    An Introduction to Support Vector Machines and Other Kernel-based Learning Methods

    (2000)
  • J. Deng et al.

    What does classifying more than 10,000 image categories tell us?

  • dos Santos, J.A., Penatti, O.A.B., Torres, R. da S., 2010. Evaluating the potential of texture and color descriptors...
  • dos Santos, J.A., Faria, F.A., Torres, R. da S., Rocha, A., Gosselin, P.-H., Philipp-Foliguet, S., Falcao, A. 2012....
  • J.A. dos Santos et al.

    Interactive multiscale classification of high-resolution remote sensing images, selected topics in applied earth observations and remote sensing

    IEEE J. PP

    (2013)
  • Duin, R.P.W., Pekalska, E. 2005. Open issues in pattern recognition. In: Int. Conf. on Computer Recognition Systems...
  • S. Džeroski et al.

    Is combining classifiers with stacking better than selecting the best one?

    Mach. Learn.

    (2004)
  • Faria, F.A., Calumby, R.T., Torres, R. da S., 2011. RECOD at ImageCLEF 2011: Medical modality classification using...
  • Faria, F.A., dos Santos, J.A., Torres, R. da S., Rocha, A., Falcão, A.X. 2012a. Automatic fusion of region-based...
  • Faria, F.A., dos Santos, J.A., Rocha, A., Torres, R. da S., 2012b. Automatic Classifier Fusion for Produce Recognition....
  • Fei-fei, L., Fergus, R., Perona, P. 2004. Learning generative visual models from few training examples: an incremental...
  • Fernando, B., Fromont, E., Muselet, D., Sebban, M. 2012. Discriminative feature fusion for image classification. In:...
  • Freund, Y., Schapire, R.E. 1996. Experiments with a new boosting algorithm. In: Int. Conf. on Machine Learning, pp....
  • J. Friedman et al.

    The Elements of Statistical Learning

    (2001)
  • Gehler, P., Nowozin, S. 2009. On feature combination for multiclass object classification. In: Intl. Conf. on Computer...
  • L. Guigues et al.

    Scale-sets image analysis

    Int. J. Comput. Vision

    (2006)
  • I. Hampshire et al.

    The Meta-Pi network: building distributed knowledge representations for robust multisource pattern recognition

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1992)
  • Cited by (42)

    • An ensemble classifier for vibration-based quality monitoring

      2022, Mechanical Systems and Signal Processing
      Citation Excerpt :

      Since these two approaches are complementary to each other, the best solution would be their combination. However, there is limited literature that addresses both approaches [51]. In this paper, an information-based framework has been proposed to select proper classifiers and then combine them to achieve the highest possible accuracy.

    • Pusion - A Generic and Automated Framework for Decision Fusion

      2023, Proceedings - International Conference on Data Engineering
    View all citing articles on Scopus
    View full text