Skip to main content
SpringerLink
Log in

Local Alignments for Fine-Grained Categorization

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

The aim of this paper is fine-grained categorization without human interaction. Different from prior work, which relies on detectors for specific object parts, we propose to localize distinctive details by roughly aligning the objects using just the overall shape. Then, one may proceed to the classification by examining the corresponding regions of the alignments. More specifically, the alignments are used to transfer part annotations from training images to unseen images (supervised alignment), or to blindly yet consistently segment the object in a number of regions (unsupervised alignment). We further argue that for the distinction of sub-classes, distribution-based features like color Fisher vectors are better suited for describing localized appearance of fine-grained categories than popular matching oriented shape-sensitive features, like HOG. They allow capturing the subtle local differences between subclasses, while at the same time being robust to misalignments between distinctive details. We evaluate the local alignments on the CUB-2011 and on the Stanford Dogs datasets, composed of 200 and 120, visually very hard to distinguish bird and dog species. In our experiments we study and show the benefit of the color Fisher vector parameterization, the influence of the alignment partitioning, and the significance of object segmentation on fine-grained categorization. We, furthermore, show that by using object detectors as voters to generate object confidence saliency maps, we arrive at fully unsupervised, yet highly accurate fine-grained categorization. The proposed local alignments set a new state-of-the-art on both the fine-grained birds and dogs datasets, even without any human intervention. What is more, the local alignments reveal what appearance details are most decisive per fine-grained object category.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. http://www.allaboutbirds.org/guide/forsters_tern/id.

  2. http://en.wikipedia.org/wiki/Great_Grey_Shrike.

  3. https://en.wikipedia.org/wiki/Parapatric_speciation.

  4. http://www.allaboutbirds.org/guide/loggerhead_shrike/id.

References

  • Alexe, B., Deselaers, T., & Ferrari, V. (2012). Measuring the objectness of image windows. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11), 2189–2202.

    Article  Google Scholar 

  • Arbelaez, P. Hariharan, B. Gu, C. Gupta, S. Bourdev, L. & Malik, J. (2012). Semantic segmentation using regions and parts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012 (pp. 3378-3385). IEEE.

  • Azizpour, H. & Laptev, I. (2012). Object detection using strongly-supervised deformable part models. In Proceedings of the European conference on Computer Vision, (pp. 836–849).

  • Bay, H., Ess, A., Tuytelaars, T., & Van Gool, L. (2008). Speeded-up robust features (surf). Vision and Image Understanding: Computer, 110(3), 346–359.

    Article  Google Scholar 

  • Berg, T. & Belhumeur, P. N. (2013). POOF: Part-based one-vs.-one features for fine-grained categorization, face verification, and attribute estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (pp. 955–962). IEEE.

  • Biederman, I. (1987). Recognition-by-components: A theory of human image understanding. Psychological Review, 94(2), 115.

    Article  Google Scholar 

  • Bo, L. Ren, X. & Fox, D. (2010) Kernel descriptors for visual recognition. In Proceedings of the Neural Information Processing Systems.

  • Bourdev, L. & Malik, J. (2009). Poselets: Body part detectors trained using 3D human pose annotations. In IEEE International Conference on Computer Vision, (pp. 1365–1372). IEEE.

  • Branson, S. Wah, C. Schroff, F. Babenko, B. Welinder, P. Perona, P. & Belongie, S. (2010). Visual recognition with humans in the loop. In Proceedings of the European Conference on Computer Vision.

  • Branson, S. Perona, P. & Belongie, S. (2011). Strong supervision from weak annotation: Interactive training of deformable part models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

  • Branson, S. Van Horn, G. Wah, C. Perona, P. & Belongie, S. (2014). The ignorant led by the blind: A hybrid human—machine vision system for fine-grained categorization. International Journal of Computer Vision, 1–27.

  • Carreira, J. (2012). CPMC: Automatic object segmentation using constrained parametric min-cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7), 1312–1328.

    Article  Google Scholar 

  • Chai, Y. Lempitsky, V. & Zisserman, A. (2011). BiCoS: A bi-level co-segmentation method for image classification. In IEEE International Conference on Computer Vision, (pp. 2579–2586). IEEE.

  • Chai, Y. Rahtu, E. Lempitsky, V. Van Gool, L. & Zisserman, A. (2012). TriCoS: A tri-level class-discriminative co-segmentation method for image classification. In Proceedings of the European Conference on Computer Vision.

  • Chai, Y. Lempitsky, V. & Zisserman, A. (2013). Symbiotic segmentation and part localization for fine-grained categorization. In IEEE International Conference on Computer Vision (ICCV). IEEE.

  • Cinbis, R. G. Verbeek, J. & Schmid, C. (2013). Segmentation driven object detection with fisher vectors. In IEEE International Conference on Computer Vision.

  • Dalal, N. & Triggs, B. (2005). Histograms of oriented gradients for human detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition.

  • Darwin, C. (1859) On the Origin of Species by Means of Natural Selection, or the Preservation of Favoured Races in the Struggle for Life.

  • Deng, J. Dong, W. Socher, R. Li, L.-J. Li, K. & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Donahue, J. Jia, Y. Vinyals, O. Hoffman, J. Zhang, N. Tzeng, E. & Darrell, T. (2013). DeCAF: A deep convolutional activation feature for generic visual recognition. Technical report. arXiv:1310.1531.

  • Duan, K. Parikh, D. Crandall, D. & Grauman, K. (2012). Discovering localized attributes for fine-grained recognition. In Proceedings of the IEEE Conference on Vision and Pattern Recognition.

  • Everingham, M. Van Gool, L. Williams, C. K. I. Winn, J. & Zisserman, A. (2007). The PASCAL Visual Object Classes Challenge (VOC2007) Results. http://www.pascal-network.org/challenges/VOC/voc2007 2007

  • Farrell, R. Oza, O. Zhang, N. Morariu, V. I. Darrell, T. & Davis, L. S. (2011). Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance. In Proceedings of the IEEE International Conference on Computer Vision.

  • Felzenszwalb, P. F., & Huttenlocher, D. P. (2004). Efficient graph-based image segmentation. International Journal of Computer Vision, 59(2), 167–181.

    Article  Google Scholar 

  • Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9), 1627–1645.

    Article  Google Scholar 

  • Gavves, E. Snoek, C. G. M. & Smeulders, A. W. M. (2012). Convex reduction of high-dimensional kernels for visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Gavves, E. Fernando, B. Snoek, C. G. M. Smeulders, A. W. M. & Tuytelaars, T. (2013). Fine-grained categorization by alignments. In Proceedings of the IEEE International Conference on Computer Vision.

  • Gosselin, P. H. Murray, N. Jégou, H. & Perronnin, F. (2013). Inria+Xerox@FGcomp: Boosting the fisher vector for fine-grained classification. Research Report RR-8431, INRIA.

  • Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 1254–1259.

    Article  Google Scholar 

  • Jia, Y. Vinyals, O. & Darrell, T. (2013). Pooling-Invariant Image Feature Learning. Technical report. arXiv:1302.5056.

  • Khosla, A. Jayadevaprakash, N. Yao, B. & Fei-Fei, L. (2011). Novel dataset for fine-grained image categorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Lazebnik, S. Schmid, C. & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition.

  • Liu, J. Kanazawa, A. Jacobs, D. & Belhumeur, P. (2012). Dog breed classification using part localization. In Proceedings of the European Conference on Computer Vision.

  • Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.

    Article  Google Scholar 

  • Maji, S. Berg, A. C. & Malik, J. (2008). Classification using intersection kernel support vector machines is efficient. In Proceedings of the IEEE Conference on Vision and Pattern Recognition.

  • Maji, S. Kannala, J. Rahtu, E. Blaschko, M. & Vedaldi, A. (2013). Fine-grained visual classification of aircraft. Technical report.

  • Manén, S. Guillaumin, M. & Van Gool, L. (2013). Prime object proposals with randomized prim’s algorithm. In Proceedings of the IEEE International Conference on Computer Vision.

  • Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., et al. (2005). A comparison of affine region detectors. International Journal of Computer Vision, 65, 43–72.

    Article  Google Scholar 

  • Nilsback, M.E. & Zisserman, A. (2008). Automated flower classification over a large number of classes. In ICVGIP.

  • Parikh, D. & Grauman, K. (2011). Interactive discovery of task-specific nameable attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Parkhi, O. M. Vedaldi, A. Zisserman, A. & Jawahar, C. V. (2012). Cats and dogs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Perdóch, M. Chum, O. & Matas, J. (2009). Efficient representation of local geometry for large scale object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Perronnin, F. Sanchez, J. & Mensink, T. (2010). Improving the fisher kernel for large-scale image classification. In Procedings of the European Conference Computer Vision.

  • Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., & Boyes-Braem, P. (1976). Basic objects in natural categories. Cognitive Psychology, 8, 382–439.

    Article  Google Scholar 

  • Rother, C. Kolmogorov, V. & Blake, A. (2004). Interactive foreground extraction using iterated graph cuts. In ACM Transactions on Graphics: Grabcut. ACM

  • Sanchez, J. Perronnin, F. & Akata, Z. (2011). Fisher vectors for fine-grained visual categorization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition.

  • Shalev-Shwartz, S. Singer, Y. & Srebro, N. (2007) Pegasos: Primal estimated sub-gradient solver for svm. In Proceedings of the International Conference on Machine Learning.

  • Swain, M. J., & Ballard, D. H. (1991). Color indexing. International Journal of Computer Vision, 7, 11–32.

    Article  Google Scholar 

  • Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013). Selective search for object recognition. International Journal of Computer Vision, 104, 154–171.

    Article  Google Scholar 

  • van de Sande, K. E. A. Gevers, T. & Snoek, C. G. M. (2010). Evaluating color descriptors for object and scene recognition. In IEEE Transactions on Pattern Analysis and Machine Intelligence.

  • Vedaldi, A. & Fulkerson, B. (2010). VLFeat: An open and portable library of computer vision algorithms. In Proceedings of the International Conference on Multimedia. ACM

  • Vedaldi, A. Gulshan, V. Varma, M. & Zisserman, A. (2009). Multiple kernels for object detection. In Proceedings of the International Conference on Vision.

  • Wah, C. Branson, S. Perona, P. & Belongie, S. (2011a). Multiclass recognition and part localization with humans in the loop. In Proceedings of the IEEE International Conference on Computer Vision.

  • Wah, C. Branson, S. Welinder, P. Perona, P. & Belongie, S. (2011b). The Caltech-UCSD Birds-200-2011 Dataset. Technical report.

  • Xie, L. Tian, Q. Yan, B. & Zhang, S. (2013). Hierarcical part matching for fine-grained visual categorization. In Proceedings of the IEEE Conference on Computer Vision.

  • Yang, S. Bo, L. Wang, J. & Shapiro, L. (2012). Unsupervised template learning for fine-grained object recognition. In Proceedings of the Neural Information Processing Systems.

  • Yao, B. Khosla, A. & Fei-Fei, L. (2011). Combining randomization and discrimination for fine-grained image categorization. In Proceedings of the IEEE Conference on Vision and Pattern Recognition.

  • Yao, B. Bradski, G. & Fei-Fei, L. (2012). A codebook-free and annotation-free approach for fine-grained image categorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Zhang, N. Farrell, R. & Darrell, T. (2012). Pose pooling kernels for sub-category recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Zhang, N. Farrell, R. Iandola, F. & Darrell, T. (2013). Deformable part descriptors for fine-grained recognition and attribute prediction. In Proceedings of the IEEE Conference on Computer Vision.

Download references

Acknowledgments

The projects IMPact BeeldCanon, AXES, STW STORY, COGNIMUND, and the Dutch national program COMMIT support this research.

Author information

Authors and Affiliations

  1. University of Amsterdam, Amsterdam, Netherlands

    Efstratios Gavves, Cees G. M. Snoek & Arnold W. M. Smeulders

  2. KU Leuven, ESAT PSI, iMinds, Leuven, Belgium

    Basura Fernando & Tinne Tuytelaars

Authors
  1. Efstratios Gavves
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Basura Fernando
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cees G. M. Snoek
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arnold W. M. Smeulders
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tinne Tuytelaars
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Efstratios Gavves.

Additional information

Communicated by Florent Perronnin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gavves, E., Fernando, B., Snoek, C.G.M. et al. Local Alignments for Fine-Grained Categorization. Int J Comput Vis 111, 191–212 (2015). https://doi.org/10.1007/s11263-014-0741-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-014-0741-5

Keywords

Search

Navigation