Skip to main content
SpringerLink
Log in

Saliency detection on sampled images for tag ranking

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Image saliency contributes to rank the unordered tags extracted from social media, but the existing saliency detection methods can hardly efficiently handle massive images in tag ranking. In this paper, we focus on improving the efficiency of saliency detection methods by applying them on the sampled images with suitable resolutions. We extensively investigate the influence of image resolution to saliency detection performance of the typical methods, and summarize a sampling strategy for different categories of salient object detection methods. Furthermore, we validate the effectiveness of the sampling strategy by applying the salient object detection methods on the sampled images with the selected resolutions in tag ranking. The experimental results show that sampling can significantly improve the efficiency of the existing salient object detection methods without obvious loss in effectiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Sang, J., Xu, C.: Right buddy makes the difference: An early exploration of social relation analysis in multimedia applications. In: ACM International Conference on Multimedia, ACM, pp. 19–28 (2012)

  2. Zhang, H., Yang, Y., Luan, H., Yang, S., Chua, T.S.: Start from scratch: towards automatically identifying, modeling, and naming visual attributes. In: ACM International Conference on Multimedia, ACM, pp. 187–196 (2014)

  3. Zheng, L., Wang, S., Guo, P., Liang, H., Tian, Q.: Tensor index for large scale image retrieval. Multimed. Syst. 21(6), 569–579 (2015)

    Article  Google Scholar 

  4. Cheng, Z., Shen, J.: On very large scale test collection for landmark image search benchmarking. Signal Process. 124, 13–26 (2016)

    Article  Google Scholar 

  5. Corridoni, J.M., Del Bimbo, A., Pala, P.: Image retrieval by color semantics. Multimed. Syst. 7(3), 175–183 (1999)

    Article  Google Scholar 

  6. Xu, X., Geng, W., Ju, R., Yang, Y., Ren, T., Wu, G.: Obsir: object-based stereo image retrieval. In: IEEE International Conference on Multimedia and Expo, IEEE, pp. 1–6 (2014)

  7. Cao, W., Liu, N., Kong, Q., Feng, H.: Content-based image retrieval using high-dimensional information geometry. Sci. China Inf. Sci. 57(7), 1–11 (2014)

    MathSciNet  MATH  Google Scholar 

  8. Zhu, L., Shen, J., Xie, L.: Unsupervised visual hashing with semantic assistant for content-based image retrieval. IEEE Trans. Knowl. Data Eng. 29(2), 472–486 (2016)

    Article  Google Scholar 

  9. Zhao, N., Hong, R., Wang, M., Hu, X., Chua, T.S.: Searching for recent celebrity images in microblog platform. In: ACM International Conference on Multimedia, ACM, pp. 841–844 (2014)

  10. Zhang, H., Shang, X., Luan, H., Wang, M., Chua, T.S.: Learning from collective intelligence: Feature learning using social images and tags. ACM Trans. Multimed. Comput. Commun. Appl. 13, 1–23 (2016)

    Article  Google Scholar 

  11. Sang, J., Xu, C., Liu, J.: User-aware image tag refinement via ternary semantic analysis. IEEE Trans. Multimed. 14(3), 883–895 (2012)

    Article  Google Scholar 

  12. Zhong, S.H., Liu, Y., Liu, Y.: Bilinear deep learning for image classification. In: ACM International Conference on Multimedia, ACM, pp. 343–352 (2011)

  13. Tang, J., Zha, Z.J., Tao, D., Chua, T.S.: Semantic-gap-oriented active learning for multilabel image annotation. IEEE Trans. Image Process. 21(4), 2354–2360 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  14. Gao, Z., Zhang, L.F., Chen, M.Y., Hauptmann, A., Zhang, H., Cai, A.N.: Enhanced and hierarchical structure algorithm for data imbalance problem in semantic extraction under massive video dataset. Multimed. Tools Appl. 68(3), 641–657 (2014)

    Article  Google Scholar 

  15. Zhang, H., Shang, X., Yang, W., Xu, H., Luan, H., Chua, T.S.: Online collaborative learning for open-vocabulary visual classifiers. In: IEEE International Conference on Computer Vision and Pattern Recognition, ACM (2016)

  16. Liu, A.A., Su, Y.T., Nie, W.Z., Kankanhalli, M.: Hierarchical clustering multi-task learning for joint human action grouping and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(1), 102–114 (2017)

    Article  Google Scholar 

  17. Sang, J., Xu, C.: Browse by chunks: topic mining and organizing on web-scale social media. ACM Trans. Multimed. Comput. Commun. Appl. 7(1), 30 (2011)

    Google Scholar 

  18. Wang, M., Ni, B., Hua, X.S., Chua, T.S.: Assistive tagging: a survey of multimedia tagging with human–computer joint exploration. ACM Comput. Surv. 44(4), 25 (2012)

    Article  Google Scholar 

  19. Gao, Y., Wang, M., Zha, Z.J., Shen, J., Li, X., Wu, X.: Visual–textual joint relevance learning for tag-based social image search. IEEE Trans. Image Process. 22(1), 363–376 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  20. Liu, D., Hua, X.S., Yang, L., Wang, M., Zhang, H.J.: Tag ranking. In: International World Wide Web Conference, ACM, pp. 351–360 (2009)

  21. Li, X., Snoek, C.G., Worring, M.: Learning tag relevance by neighbor voting for social image retrieval. In: ACM International Conference on Multimedia Information Retrieval, ACM, pp. 180–187 (2008)

  22. Feng, S., Lang, C., Xu, D.: Beyond tag relevance: integrating visual attention model and multi-instance learning for tag saliency ranking. In: ACM International Conference on Image and Video Retrieval, ACM, pp. 288–295 (2010)

  23. Nguyen, T.V., Xu, M., Gao, G., Kankanhalli, M., Tian, Q., Yan, S.: Static saliency vs. dynamic saliency: a comparative study. In: ACM International Conference on Multimedia, ACM, pp. 987–996 (2013)

  24. Jian, M., Lam, K.M., Dong, J., Shen, L.: Visual-patch-attention-aware saliency detection. IEEE Trans. Cybernet. 45(8), 1575–1586 (2015)

    Article  Google Scholar 

  25. Zhong, S.H., Liu, Y., Ng, T.Y., Liu, Y.: Perception-oriented video saliency detection via spatio-temporal attention analysis. Neurocomputing 207, 178–188 (2016)

    Article  Google Scholar 

  26. Ren, T., Liu, Y., Wu, G.: Image retargeting based on global energy optimization. In: IEEE International Conference on Multimedia and Expo, IEEE, pp. 406–409 (2009)

  27. Du, H., Liu, Z., Jiang, J., Shen, L.: Stretchability-aware block scaling for image retargeting. J. Vis. Commun. Image Represent. 24(4), 499–508 (2013)

    Article  Google Scholar 

  28. Wei, Y., Xia, W., Lin, M., Huang, J., Ni, B., Dong, J., Zhao, Y., Yan, S.: Hcp: a flexible cnn framework for multi-label image classification. IEEE Trans. Pattern Anal. Mach. Intell. 38(9), 1901–1907 (2015)

    Article  Google Scholar 

  29. Lu, Y., Lai, Z., Fan, Z., Cui, J., Zhu, Q.: Manifold discriminant regression learning for image classification. Neurocomputing 166, 475–486 (2015)

    Article  Google Scholar 

  30. Bao, B.K., Liu, G., Xu, C., Yan, S.: Inductive robust principal component analysis. IEEE Trans. Image Process. 21(8), 3794–3800 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  31. Kuang, H., Chong, Y., Li, Q., Zheng, C.: Mutualcascade method for pedestrian detection. Neurocomputing 137, 127–135 (2014)

    Article  Google Scholar 

  32. Cheng, M.M., Mitra, N.J., Huang, X., Torr, P.H.S., Hu, S.M.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)

    Article  Google Scholar 

  33. Wang, W., Lang, C., Feng, S.: Contextualizing tag ranking and saliency detection for social images. In: International Conference on Multimedia Modeling, Springer, pp. 428–435 (2013)

  34. Cao, Y., Kang, K., Zhang, S., Zhang, J., Wang, Z.: Automatic tag saliency ranking for stereo images. Neurocomputing 172, 9–18 (2016)

    Article  Google Scholar 

  35. Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.H.: Saliency detection via graph-based manifold ranking. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp. 3166–3173 (2013)

  36. Jiang, B., Zhang, L., Lu, H., Yang, C., Yang, M.H.: Saliency detection via absorbing markov chain. In: IEEE International Conference on Computer Vision, IEEE, 1665–1672 (2013)

  37. Zhu, W., Liang, S., Wei, Y., Sun, J.: Saliency optimization from robust background detection. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2814–2821 (2014)

  38. Zhang, J., Sclaroff, S., Lin, Z., Shen, X., Price, B., Mĕch, R.: Minimum barrier salient object detection at 80 fps. In: IEEE International Conference on Computer Vision, IEEE (2015)

  39. Hou, X., Zhang, L.: Saliency detection: a spectral residual approach. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE (2007)

  40. Guo, J., Ren, T., Bei, J.: A comprehensive study of salient object detection on diverse image resolutions. In: National Conference on Multimedia Technology, CCF (2016)

  41. Zhuang, J., Hoi, S.C.: A two-view learning approach for image tag ranking. In: ACM International Conference on Web Search and Data Mining, ACM, pp. 625–634 (2011)

  42. Tang, J., Li, M., Li, Z., Zhao, C.: Tag ranking based on salient region graph propagation. Multimed. Syst. 21(3), 267–275 (2015)

    Article  Google Scholar 

  43. Feng, S., Lang, C., Liu, H., Huang, X.: Adaptive all-season image tag ranking by saliency-driven image pre-classification. J. Vis. Commun. Image Represent. 24(7), 1031–1039 (2013)

    Article  Google Scholar 

  44. Achanta, R., Hemami, S., Estrada, F., Süsstrunk, S.: Frequency-tuned salient region detection. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp. 1597–1604 (2009)

  45. Achanta, R., Süsstrunk, S.: Saliency detection using maximum symmetric surround. In: IEEE International Conference on Image Processing, IEEE, pp. 2653–2656 (2010)

  46. Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)

    Article  Google Scholar 

  47. Ju, R., Liu, Y., Ren, T., Ge, L., Wu, G.: Depth-aware salient object detection using anisotropic center-surround difference. Signal Process. Image Commun. 38, 115–126 (2015)

    Article  Google Scholar 

  48. Guo, J., Ren, T., Bei, J.: Salient object detection for rgb-d image via saliency evolution. In: IEEE International Conference on Multimedia and Expo, IEEE (2016)

  49. Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., Zheng, Y.: NUS-WIDE: a real-world web image database from national university of Singapore. In: ACM International Conference on Image and Video Retrieval, ACM, p. 48 (2009)

  50. Margolin, R., Zelnik-Manor, L., Tal, A.: How to evaluate foreground maps. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp. 248–255 (2014)

  51. Zhang, M.L., Zhou, Z.H.: Adapting RBF neural networks to multi-instance learning. Neural Process. Lett. 23(1), 1–26 (2006)

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous reviews for their helpful suggestion. This work is supported by the National Science Foundation of China (61321491, 61202320), Research Fund of the State Key Laboratory for Novel Software Technology at Nanjing University (ZZKT2016B09), and Collaborative Innovation Center of Novel Software Technology and Industrialization.

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China

    Jingfan Guo & Tongwei Ren

  2. Software Institute, Nanjing University, Nanjing, China

    Lei Huang & Jia Bei

Authors
  1. Jingfan Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tongwei Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jia Bei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tongwei Ren.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, J., Ren, T., Huang, L. et al. Saliency detection on sampled images for tag ranking. Multimedia Systems 25, 35–47 (2019). https://doi.org/10.1007/s00530-017-0546-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-017-0546-9

Keywords

Search

Navigation