Skip to main content
SpringerLink
Log in

A knowledge-based semantic approach for image collection summarization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With the advent of digital cameras, the number of digital images is on the increase. As a result, image collection summarization systems are proposed to provide users with a condense set of summary images as a representative set to the original high volume image set. In this paper, a semantic knowledge-based approach for image collection summarization is presented. Despite ontology and knowledge-based systems have been applied in other areas of image retrieval and image annotation, most of the current image summarization systems make use of visual or numeric metrics for conducting the summarization. Also, some image summarization systems jointly model visual data of images together with their accompanying textual or social information, while these side data are not available out of the context of web or social images. The main motivation of using ontology approach in this study is its ability to improve the result of computer vision tasks by the additional knowledge which it provides to the system. We defined a set of ontology based features to measure the amount of semantic information contained in each image. A semantic similarity graph was made based on semantic similarities. Summary images were then selected based on graph centrality on the similarity graph. Experimental results showed that the proposed approach worked well and outperformed the current image summarization systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Abdollahpour Z, Samani ZR, Moghaddam ME (2015) Image classification using ontology based improved visual words. In: 2015 23rd Iranian conference on electrical engineering, IEEE, pp 694–698

  2. Bannour H, Hudelot C (2014) Building and using fuzzy multimedia ontologies for semantic image annotation. Multimed Tools Appl 72(3):2107–2141

    Article  Google Scholar 

  3. Barrilero M et al (2011) Innetwork content based image recommendation system for Contentaware Networks. In: IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)

  4. Berg TL, Forsyth D (2007) Automatic ranking of iconic images. University of California, Berkeley, Tech Rep

  5. Brin S, Page L (2012) Reprint of: the anatomy of a large-scale hypertextual web search engine. Comput Netw 56(18):3825–3833

    Article  Google Scholar 

  6. Camargo JE, González FA (2016) Multimodal latent topic analysis for image collection summarization. Inf Sci 328:270–287

    Article  Google Scholar 

  7. Chatfield K et al (2011) The devil is in the details: an evaluation of recent feature encoding methods

  8. Clough P, Joho H, Sanderson M (2005) Automatically organising images using concept hierarchies. In: Proceedings of the multimedia workshop running at ACM SIGIR conference. Sheffield.

  9. Crandall DJ et al (2009) Mapping the world’s photos. In: Proceedings of the 18th international conference on world wide web

  10. Das D, Martins AF (2007) A survey on automatic text summarization. Literature survey for the language and statistics II course at CMU 4:192–195

  11. Delest M, Don A, Benois-Pineau J (2006) DAG-based visual interfaces for navigation in indexed video content. Multimed Tools Appl 31(1):51–72

    Article  Google Scholar 

  12. Deng J, Berg AC, Fei-Fei L (2011) Hierarchical semantic indexing for large scale image retrieval. in Computer Vision and Pattern Recognition (CVPR), 2011 I.E. Conference on. IEEE

  13. Deng J et al (2009) Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition, 2009. CVPR 2009

  14. Elleuch N, Ammar AB, Alimi AM (2015) A generic framework for semantic video indexing based on visual concepts/contexts detection. Multimed Tools Appl 74(4):1397–1421

    Article  Google Scholar 

  15. Fan J, Gao Y, Luo H (2008) Integrating concept ontology and multitask learning to achieve more effective classifier training for multilevel image annotation. Image Process IEEE Trans 17(3):407–426

    Article  MathSciNet  Google Scholar 

  16. Fan J, Yuli G et al (2008) A novel approach to enable semantic and visual image summarization for exploratory image search. In: Proceedings of the 1st ACM international conference on multimedia information retrieval

  17. Fang H et al (2015) Topic aspect-oriented summarization via group selection. Neurocomputing 149:1613–1619

    Article  Google Scholar 

  18. Forsati R, Shamsfard M (2016) Symbiosis of evolutionary and combinatorial ontology mapping approaches. Inf Sci

  19. Gehler P, Nowozin S (2009) On feature combination for multiclass object classification. In: Computer Vision, 2009 I.E. 12th International Conference on. IEEE

  20. Griffin G, Perona P (2008) Learning and using taxonomies for fast visual categorization. In: Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE

  21. Jaffe A et al (2006) Generating summaries and visualization for large collections of geo-referenced photographs. In: Proceedings of the 8th ACM international workshop on multimedia information retrieval

  22. Jeong J-W et al (2012) Visual summarization of the social image collection using image attractiveness learned from social behaviors. Multimedia and expo (ICME), 2012 I.E. International Conference on 538–543

  23. Jia Y et al (2008) Finding image exemplars using fast sparse affinity propagation. In: Proceedings of the 16th ACM international conference on multimedia

  24. Jiao B et al (2012) Visually summarizing web pages through internal and external images. IEEE Trans Multimed 14(6):1673–1683

    Article  Google Scholar 

  25. Jing Y, Baluja S (2008) Visualrank: applying pagerank to large-scale image search. Pattern Anal Mach Intell IEEE Trans 30(11):1877–1890

    Article  Google Scholar 

  26. Jing Y, Baluja S, Rowley H (2007) Canonical image selection from the web. In: Proceedings of the 6th ACM international conference on image and video retrieval

  27. Kennedy LS, Chang S-F, Kozintsev IV (2006) To search or to label?: predicting the performance of search-based automatic image classifiers. In: Proceedings of the 8th ACM international workshop on multimedia information retrieval

  28. Kennedy LS, Naaman M (2008) Generating diverse and representative image search results for landmarks. In: Proceedings of the 17th international conference on world wide web

  29. Khosla A et al (2013) Large-scale video summarization using web-image priors. In: Computer Vision and Pattern Recognition (CVPR), 2013 I.E. conference on. IEEE

  30. Kim G, Sigal L, Xing EP (2014) Joint summarization of large-scale collections of web images and videos for storyline reconstruction. In: Computer Vision and Pattern Recognition (CVPR), 2014 I.E. Conference on. IEEE

  31. Koller, T.G.a.D. Discriminative Learning of Relaxed Hierarchy for Large-scale Visual Recognition. in in Computer Vision (ICCV), 2011 I.E. International Conference on. 2011.

  32. Latha TKK (2015) Optimization of sparse dictionary model for multimodal image summarization using firefly algorithm. Int J Appl Eng Res 10(55):1896–1901

    Google Scholar 

  33. Lemos FD et al (2012) Towards a context-aware photo recommender system. In: Context-aware recommender system workshops

  34. Li C, Feng Z, Han Y (2015) Image attribute learning with ontology guided fused lasso. Multimed Tools Appl 1–15

  35. Li L, Jiang S, Huang Q (2012) Learning hierarchical semantic description via mixed-norm regularization for image understanding. IEEE Trans Multimed 14(5):1401–1413

    Article  Google Scholar 

  36. Li Y, Merialdo B (2010) VERT: automatic evaluation of video summaries. In: Proceedings of the international conference on multimedia

  37. Li M, Zhao C, Tang J (2013) Hybrid image summarization by hypergraph partition. Neurocomputing 119:41–48

    Article  Google Scholar 

  38. Li L-J et al (2010) Building and using a semantivisual image hierarchy. In: Computer Vision and Pattern Recognition (CVPR), 2010 I.E. Conference on. IEEE

  39. Lin C-Y, Hovy E (2003) Automatic evaluation of summaries using n-gram co-occurrence statistics. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1. Association for Computational Linguistics

  40. Lowe, D.G., Distinctive image features from scale-invariant keypoints. International journal of computer vision, 2005: p. 91--110.

  41. Marszałek M, Schmid C (2008) Constructing category hierarchies for visual recognition, in Computer Vision–ECCV 2008. Springer 479–491

  42. Mei S et al (2015) Video summarization via minimum sparse reconstruction. Pattern Recogn 48(2):522–533

    Article  Google Scholar 

  43. Naci SU et al (2008) The COST292 experimental framework for rushes summarization task in TRECVID 2008. In: Proceedings of the 2nd ACM TRECVID video summarization workshop. ACM

  44. Newman M (2010) Networks: an introduction. Oxford University Press, Oxford

    Book  MATH  Google Scholar 

  45. Nister D, Stewenius H (2006) Scalable recognition with a vocabulary tree. In: Computer Vision and Pattern Recognition, 2006 I.E. Computer Society Conference on. IEEE

  46. Over P, Smeaton AF, Awad G (2008) The TRECVid 2008 BBC rushes summarization evaluation. In: Proceedings of the 2nd ACM TRECVid video summarization workshop. ACM

  47. Palmer S, Rosch E, Chase P (1981) Canonical perspective and the perception. Atten Perform 4:135–151

    Google Scholar 

  48. Pang Y, Qiang H et al (2011) Summarizing tourist destinations by mining user-generated travelogues and photos. Comput Vis Image Underst 115(3):352–363

    Article  Google Scholar 

  49. Poole D (2014) Linear algebra: a modern introduction. Cengage Learning

  50. Qian G, Sural S, Pramanik S (2002) A comparative analysis of two distance measures in color image databases. In: Image Processing. 2002. Proceedings. 2002 International Conference on. IEEE

  51. Qian G et al (2004) Similarity between Euclidean and cosine angle distance for nearest neighbor queries. In: Proceedings of the 2004 ACM symposium on applied computing. ACM

  52. Raguram R, Lazebnik S (2008) Computing iconic summaries of general visual concepts. In: Computer vision and pattern recognition workshops, 2008. CVPRW’08. IEEE Computer Society Conference on

  53. Rudinac S, Larson M, Hanjalic A (2013) Learning crowdsourced user preferences for visual summarization of image collections. Multimed IEEE Trans 15(6):1231–1243

    Article  Google Scholar 

  54. Russell BC et al (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1–3):157–173

    Article  Google Scholar 

  55. Schadd FC, Roos N (2012) Coupling of wordnet entries for ontology mapping using virtual documents. In: Proceedings of the ISWC

  56. Shen X, Tian X (2015) Multi-modal and multi-scale photo collection summarization. Multimed Tools Appl 1–15

  57. Simon I, Noah S, Seitz SM (2007) Scene summarization for online image collections. In: IEEE 11th International Conference on Computer Vision (ICCV 2007)

  58. Tschiatschek S et al (2014) Learning mixtures of submodular functions for image collection summarization. In: Advances in neural information processing systems

  59. van Leuken RH et al (2009) Visual diversification of image search results. In: Proceedings of the 18th international conference on world wide web

  60. Verma N et al (2012) Learning hierarchical similarity metrics. In: Computer Vision and Pattern Recognition (CVPR), 2012 I.E. Conference on

  61. Wang J, Jia L, Hua X-S (2011) Interactive browsing via diversified visual summarization for image search results. Multimedia Systems 17(5):379–391

    Article  Google Scholar 

  62. Xu H et al (2011) Hybrid image summarization. In: Proceedings of the 19th ACM international conference on multimedia

  63. Yang L, Adviser-Johnstone JK (2011) Mining canonical views from internet image collections

  64. Yang Y, Chen S-C (2012) Disaster image filtering and summarization based on multi-layered affinity propagation. In: IEEE International Symposium on Multimedia (ISM), 2012

  65. Yang YH et al (2008) ContextSeer: context search and recommendation at query time for shared consumer photos. In: Proceedings of the 16th ACM international conference on multimedia

  66. Yang C et al (2013) Image collection summarization via dictionary learning for sparse representation. Pattern Recogn 46:948–961

    Article  Google Scholar 

  67. Yu H et al (2014) A joint optimization model for image summarization based on image content and tags. In: Twenty-eighth AAAI conference on artificial intelligence

  68. Zhang L, Lin F, Zhang B (2001). Support vector machine learning for image retrieval. in Image Processing, 2001. Proceedings. 2001 International Conference on. IEEE

  69. Zhao B, Li F, Xing EP (2011) Large-scale category structure aware image categorization. Adv Neur Inf Process Syst 1251–1259

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Shahid Beheshti University; GC, Tehran, Iran

    Zahra Riahi Samani & Mohsen Ebrahimi Moghaddam

Authors
  1. Zahra Riahi Samani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohsen Ebrahimi Moghaddam
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zahra Riahi Samani.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Samani, Z.R., Moghaddam, M.E. A knowledge-based semantic approach for image collection summarization. Multimed Tools Appl 76, 11917–11939 (2017). https://doi.org/10.1007/s11042-016-3840-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3840-1

Keywords

Search

Navigation