Skip to main content
SpringerLink
Log in

An adaptive ensemble-based system for face recognition in person re-identification

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Recognizing individuals of interest from faces captured with video cameras raises several challenges linked to changes in capture conditions (e.g., variation in illumination and pose). Moreover, in person re-identification applications, the facial models needed for matching are typically designed a priori, with a limited amount of reference samples captured under constrained temporal and spatial conditions. Tracking can, however, be used to regroup the system responses linked to a facial trajectory (facial captures from a person) for robust spatio-temporal recognition, and to update facial models over time using operational data. In this paper, an adaptive ensemble-based system is proposed for spatio-temporal face recognition (FR). Given a diverse set of facial captures in a trajectory of a target individual, an ensemble of 2-class classifiers is designed. A pool of ARTMAP classifiers is generated using a dynamic PSO-based learning strategy, and classifiers are selected and combined using Boolean combination. To train classifiers, target samples are combined with a set of reference non-target samples selected from the cohort and universal models using One-Sided Selection. During operations, facial trajectories are captured, and each individual-specific ensemble of the system seeks to detect target individuals, and possibly self-update their facial models. To update an ensemble, a learn-and-combine strategy is employed to avoid knowledge corruption, and a memory management strategy based on Kullback–Leibler divergence allows to rank and select stored validation samples over time to bound the system’s memory consumption. Spatio-temporal fusion is performed by accumulating classifier predictions over a time window, and a second threshold allows to self-update facial models. The proposed systems were validated with videos from the Face in Action and COX-S2V datasets, that feature both abrupt and gradual patterns of change. At the transaction level, results show that the proposed system allows to increase AUC accuracy by about 3 % for scenarios with abrupt changes, and by about 5 % with gradual changes. Subject-based analysis reveals the difficulties of face recognition with different poses, affecting more significantly the lamb- and goat-like individuals. Compared to reference spatio-temporal fusion approaches, results show that the proposed accumulation scheme produces the highest discrimination.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. A facial trajectory is defined as a set of facial captures (produced by face segmentation) that correspond to a same high-quality track of an individual across consecutive frames.

  2. For simplicity of notation, the k has been omitted from all design data blocks, e.g., \(D_k^t \equiv D^t\).

References

  1. Barry, M., Granger, E.: Face recognition in video using a what-and-where fusion neural network. In: IEEE International Conference on Neural Networks - Conference Proceedings, pp. 2256–2261. Orlando, United states (2007)

  2. Best-Rowden, L., Klare, B., Klontz, J., Jain, A.: Video-to-video face matching: Establishing a baseline for unconstrained face recognition. In: Biometrics: Theory, Applications and Systems (BTAS), 2013 IEEE Sixth International Conference on, pp. 1–8 (2013)

  3. Burghouts, G., Schutte, K., Bouma, H., Hollander, R.: Selection of negative samples and two-stage combination of multiple features for action detection in thousands of videos. Mach. Vision Appl. 25(1), 85–98 (2014). doi:10.1007/s00138-013-0514-0

    Article  MATH  Google Scholar 

  4. Chen, L.F., Liao, H.Y., Lin, J.C.: Person identification using facial motion. In: Image Processing. Proceedings. 2001 International Conference on, vol. 2, pp. 677–680 (2001)

  5. Chen, Y.C., Patel, V.M., Phillips, P.J., Chellappa, R.: Dictionary-based face and person recognition from unconstrained video. IEEE Trans. Image Process, 1–14 (2014) (preprint)

  6. Cohen, I., Cozman, F.G., Sebe, N., Cirelo, M.C., Huang, T.S.: Semisupervised learning of classifiers: Theory, algorithms, and their application to human–computer interaction. IEEE Trans. Pattern Anal. Mach. Intell. 26(12), 1553–1567 (2004)

  7. Connolly, J.F., Granger, E., Sabourin, R.: Evolution of heterogeneous ensembles through dynamic particle swarm optimization for video-based face recognition. Pattern Recognit. 45(7), 2460–2477 (2012)

    Article  Google Scholar 

  8. DelaTorre, M., Granger, E., Radtke, P.V.W., Sabourin, R., Gorodnichy, D.O.: Incremental update of biometric models in face-based video surveillance. In: Proc. IJCNN, pp. 1–8. Brisbane, Australia (2012)

  9. DelaTorre, M., Granger, E., Radtke, P.V.W., Sabourin, R., Gorodnichy, D.O.: Partially-supervised learning from facial trajectories for face recognition in video surveillance. Inf. Fusion 24, 31–53 (2015). doi:10.1016/j.inffus.2014.05.006

    Article  Google Scholar 

  10. DelaTorre, M., Granger, E., Sabourin, R., Gorodnichy, D.O.: An individual-specific strategy for management of reference data in adaptive ensembles for face re-identification. In: IET (ed.) 5THh I. C. on Imaging for Crime Detection and Prevention (ICDP), pp. 1–7. London (2013)

  11. DelaTorre, M., Radtke, P.V.W., Granger, E., Sabourin, R., Gorodnichy, D.O.: A comparison of adaptive matchers for screening of faces in video surveillance. In: Symposium on Computational Intelligence for Security and Defence Applications, pp. 1–8. Ottawa, Canada (2012)

  12. Despiegel, V., Gentric, S., Fondeur, J.: Border control: From technical to operational evaluation. In: International Biometric Performance Testing Conference. Gaithersburg (2012)

  13. Dewan, M., Granger, E., Roli, F., Sabourin, R., Marciallis, G.: A comparison of adaptive appearance methods for tracking faces in video surveillance. In: International Conference on Imaging for Crime Detection and Prevention (ICDP), pp. 1–8. London (2013)

  14. Didaci, L., Roli, F.: Using co-training and self-training in semi-supervised multiple classifier systems. In: LNCS (including LNAI and LNB), LNCS, vol. 4109, pp. 522–530. Hong Kong (2006)

  15. Doddington, G., Liggett, W., Martin, A., Przybocki, M., Reynolds, D.: Sheep, goats, lambs and wolves: A statistical analysis of speaker performance. In: International conference on spoken language processing, pp. 1351–1354 (1998)

  16. Ekenel, H.K., Stallkamp, J., Stiefelhagen, R.: A video-based door monitoring system using local appearance-based face models. Comput. Vision Image Underst. 114(5), 596–608 (2010)

    Article  Google Scholar 

  17. El Gayar, N., Shaban, S.A., Hamdy, S.: Face recognition with semi-supervised learning and multiple classifiers. In: Proc. of WSEAS Int. Conf. on Computational Intelligence, Man-Machine Systems and Cybernetics, pp. 296–301. USA (2006)

  18. Fawcett, T.: An introduction to roc analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006)

    Article  MathSciNet  Google Scholar 

  19. Fischer, M., Ekenel, H., Stiefelhagen, R.: Person re-identification in tv series using robust face recognition and user feedback. Multimed. Tools Appl. 55(1), 83–104 (2011)

    Article  Google Scholar 

  20. Franco, A., Maio, D., Maltoni, D.: Incremental template updating for face recognition in home environments. Pattern Recogn. 43(8), 903–2891 (2010)

    Article  Google Scholar 

  21. Goh, R., Liu, L., Liu, X., Chen, T.: The CMU Face In Action Database. In: Analysis and Modelling of Faces and Gestures, pp. 255–263. Carnegie Mellon University (2005)

  22. Gorodnichy, D.: Video-based framework for face recognition in video. In: Proceedings. The 2nd Canadian Conference on Computer and Robot Vision, pp. 330–8. Piscataway (2005)

  23. Hewitt, R., Belongie, S.: Active learning in face recognition: Using tracking to build a face model. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p. 157. New York (2006)

  24. Huang, Z., Shan, S., Wang, R., Zhang, H., Lao, S., Kuerban, A., Chen, X.: A benchmark and comparative study of video-based face recognition on cox face database. Accepted for publication, IEEE Transactions on Image Processing (2015)

  25. Jain, A.K., Ross, A.: Learning user-specific parameters in a multibiometric system. In: Int. Conf. on Image Processing, pp. 57–60 (2002)

  26. Kamgar-Parsi, B., Lawson, W., Kamgar-Parsi, B.: Toward development of a face recognition system for watchlist surveillance. IEEE Trans. PAMI 33(10), 1925–1937 (2011)

    Article  Google Scholar 

  27. Khreich, W., Granger, E., Miri, A., Sabourin, R.: A comparison of techniques for on-line incremental learning of hmm parameters in anomaly detection. In: Proceedings of the 2009 IEEE Symposium on Computational Intelligence in Security and Defense Applications, pp. 2732–2752 (2010)

  28. Khreich, W., Granger, E., Miri, A., Sabourin, R.: Iterative Boolean Combination of classifiers in the ROC space: An application to anomaly detection with HMMs. Pattern Recogn. 43(8), 2732–2752 (2010)

    Article  MATH  Google Scholar 

  29. Klare, B., Jain, A.: On a taxonomy of facial features. In: International Conference on Biometrics: Theory Applications and Systems (BTAS), pp. 1–8 (2010)

  30. Kubat, M., Matwin, S.: Addressing the curse of imbalanced training sets: One-sided selection. In: Proceedings of the Fourteenth International Conference on Machine Learning, pp. 179–186. Morgan Kaufmann (1997)

  31. Kuncheva, L.: Combining Pattern Classifiers: Methods and Algorithms. Wiley (2004)

  32. Lee, K.C., Ho, J., Yang, M.H., Kriegman, D.: Video-based face recognition using probabilistic appearance manifolds. In: Proc. in CVPR, vol. 1, pp. I–313, I–320 (2003)

  33. Lee, K.C., Ho, J., Yang, M.H., Kriegman, D.: Visual tracking and recognition using probabilistic appearance manifolds. Comput. Vision Image Underst. 99, 303–331 (2005)

    Article  Google Scholar 

  34. Li, F., Wechsler, H.: Open set face recognition using transduction. IEEE Trans. PAMI 27(11), 1686–1697 (2005)

    Article  Google Scholar 

  35. Lim, C.P., Harrison, R.F.: Probabilistic fuzzy artmap: An autonomous neural network architecture for bayesian probability estimation. In: Artificial Neural Networks, Fourth Int. Conf. on, pp. 148–153 (1995)

  36. Lim, C.P., Harrison, R.F.: An incremental adaptive network for on-line supervised learning and probability estimation. Neural Netw. 10(5), 925–939 (1997)

    Article  Google Scholar 

  37. Liu, X., Cheng, T.: Video-based face recognition using adaptive Hidden Markov Models. In: Proceedings 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 340–345. Los Alamitos (2003)

  38. Lu, K., Ding, Z., Zhao, J., Wu, Y.: A novel semi-supervised face recognition for video. In: Proc. of the International Conference on Intelligent Control and Information Processing, pp. 313–316 (2010)

  39. Lu, Z., Wu, X., Bongard, J.: Active learning with adaptive heterogeneous ensembles. In: Proceedings of the 2009 Ninth IEEE International Conference on Data Mining (ICDM 2009), pp. 327–36. Piscataway (2009)

  40. Matta, F., Dugelay, J.L.: Video face recognition: A physiological and behavioural multimodal approach. In: Proceedings—International Conference on Image Processing, ICIP, vol. 4, pp. IV497–IV500. San Antonio (2006)

  41. Matta, F., Dugelay, J.L.: Person recognition using facial video information: a state of the art. J. Visual Lang. Comput. 20(3), 180–187 (2009)

    Article  Google Scholar 

  42. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Tr. PAMI 24(7), 971–987 (2002)

    Article  Google Scholar 

  43. Okada, K., Kite, L., von der Malsburg, C.: An adaptive person recognition system. In: Proceedings 10th IEEE International Workshop on Robot and Human Interactive Communication, pp. 436–41. Piscataway (2001)

  44. Pagano, C., Granger, E., Sabourin, R., Gorodnichy, D.O.: Detector ensembles for face recognition in video surveillance. In: IJCNN, pp. 1–8. Brisbane (2012)

  45. Polikar, R., Udpa, L., Udpa, S.S., Honavar, V.: Learn++: An Incremental Learning Algorithm for MLP Networks. IEEE Trans. SMC 31(4), 497–508 (2001)

    Google Scholar 

  46. Rattani, A.: Adaptive biometric system based on template update procedures. Ph.D. thesis, University of Cagliari (2010)

  47. Rattani, A., Freni, B., Marcialis, G.L., Roli, F.: Template update methods in adaptive biometric systems: A critical review. In: Lecture Notes in Computer Science (included Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 5558, pp. 847–856. Alghero (2009)

  48. Rattani, A., Marcialis, G., Roli, F.: Capturing large intra-class variations of biometric data by template co-updating. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–6. Piscataway (2008)

  49. Roli, F., Didaci, L., Marcialis, G.: Template co-update in multimodal biometric systems. In: International Conference on Biometrics, vol. 4642, pp. 1194–202. Seoul (2007)

  50. Roli, F., Didaci, L., Marcialis, G.L.: Adaptive biometric systems that can improve with use. In: Govindaraju, N.R.V. (ed.) Advances in Biometrics: Sensors, Systems and Algorithms, pp. 447–471. Springer (2008)

  51. Roli, F., Marcialis, G.L.: Semi-supervised pca-based face recognition using self-training. In: JIAPR - Int. Workshop on Structural and Syntactical Pat. Rec. and Statistical Techniques in Pat. Rec., vol. 4109, pp. 560–568. Springer, Hong Kong (2006)

  52. Ross, D.A., Lim, J., Lin, R.S., Yang, M.H.: Incremental learning for robust visual tracking. Int. J. Comput. Vision (2008) (Special issue: Learning for vision)

  53. Satta, R.: Appearance descriptors for person re-identification: A comprehensive review. arXiv:1307.5748v1 (2013)

  54. Singh, R., Vatsa, M., Ross, A., Noore, A.: Biometric classifier update using online learning: A case study in near infrared face verification. Image Vision Comput. 28, 1098–1105 (2010)

    Article  Google Scholar 

  55. Tax, D., Duin, R.: Growing a multi-class classifier with a reject option. Pattern Recogn. 29(10), 1565–1570 (2008)

    Article  Google Scholar 

  56. Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cogn. Neurosci. 3(1), 71–86 (1991)

    Article  Google Scholar 

  57. Viola, P., Jones, M.: Robust real-time face detection. Int. J. Comput. Vision 2(57), 137–154 (2004)

    Article  Google Scholar 

  58. Yilmaz, A., Javed, O., Shah, M.: Object tracking: A survey. ACM Comput. Surv. 38(4) (2006)

  59. Yu, G., Zhang, G., Domeniconi, C., Yu, Z., YouZ, J.: Semi-supervised classification based on random subspace dimensionality reduction. Pattern Recogn. 45(3), 1119–1135 (2012)

    Article  MATH  Google Scholar 

  60. Zenobi, G., Cunningham, P.: Using diversity in preparing ensembles of classifiers based on different feature subsets to minimize generalization error. In: Raedt, L., Flach, P. (eds.) Machine Learning: ECML 2001. Lecture Notes in Computer Science, vol. 2167, pp. 576–587. Springer, Berlin, Heidelberg (2001)

    Chapter  Google Scholar 

  61. Zhang, Y., Martnez, A.M.: A weighted probabilistic approach to face recognition from multiple images and video sequences. Image Vision Comput. 24, 626–638 (2006)

    Article  Google Scholar 

  62. Zhou, S., Chellappa, R., Moghaddam, B.: Visual tracking and recognition using appearance-adaptive models in particle filters. Image Process. IEEE Trans. 13(11), 1491–1506 (2004)

    Article  Google Scholar 

Download references

Acknowledgments

This work was partially supported by the Natural Sciences and Engineering Research Council of Canada, and the Defence Research and Development Canada’s Centre for Security Science Public Security Technical Program (project PSTP-03-401BIOM). This work was also supported by the Program for the Improvement of the Professoriate of the Secretariat of Public Education, Mexico, the Mexican National Council for Science and Technology, and the University Center of Los Valles, University of Guadalajara, Ameca, Mexico.

Author information

Authors and Affiliations

  1. Laboratoire d’imagerie de vision et d’intelligence artificielle, École de technologie supérieure, Université du Québec, Montreal, Canada

    Miguel De-la-Torre, Eric Granger & Robert Sabourin

  2. Science and Engineering Directorate, Canada Border Services Agency, Ottawa, Canada

    Dmitry O. Gorodnichy

Authors
  1. Miguel De-la-Torre
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric Granger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Robert Sabourin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dmitry O. Gorodnichy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miguel De-la-Torre.

Update tables

Update tables

Tables 11 and 12 present the correct and incorrect update trajectories used by the system for self-update. The tables correspond to the scenarios with abrupt and gradual changes, respectively.

Table 11 Update table for the system with correct (bold) and incorrect update trajectories in the Left and Right update trajectories
Full size table
Table 12 Update table for the system with correct (bold) and incorrect update trajectories in the Left and Right update trajectories
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

De-la-Torre, M., Granger, E., Sabourin, R. et al. An adaptive ensemble-based system for face recognition in person re-identification. Machine Vision and Applications 26, 741–773 (2015). https://doi.org/10.1007/s00138-015-0697-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-015-0697-7

Keywords

Search

Navigation