Skip to main content
SpringerLink
Log in

Automatic content based image retrieval using semantic analysis

  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

We present a new text-to-image re-ranking approach for improving the relevancy rate in searches. In particular, we focus on the fundamental semantic gap that exists between the low-level visual features of the image and high-level textual queries by dynamically maintaining a connected hierarchy in the form of a concept database. For each textual query, we take the results from popular search engines as an initial retrieval, followed by a semantic analysis to map the textual query to higher level concepts. In order to do this, we design a two-layer scoring system which can identify the relationship between the query and the concepts automatically. We then calculate the image feature vectors and compare them with the classifier for each related concept. An image is relevant only when it is related to the query both semantically and content-wise. The second feature of this work is that we loosen the requirement for query accuracy from the user, which makes it possible to perform well on users’ queries containing less relevant information. Thirdly, the concept database can be dynamically maintained to satisfy the variations in user queries, which eliminates the need for human labor in building a sophisticated initial concept database. We designed our experiment using complex queries (based on five scenarios) to demonstrate how our retrieval results are a significant improvement over those obtained from current state-of-the-art image search engines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. The image data can be downloaded from: http://labelme.csail.mit.edu/

References

  • Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.G. (2007). Dbpedia: A nucleus for a web of open data. Proc. International Semantic Web Conference (ISWC) (pp. 11–15).

  • Banko, M., Cafarella, M.J., Soderland, S., Broadhead, M., Etzioni, O. (2007). In “Open information extraction from the web,”: Proc. International Joint Conference on Artificial Intelligence (IJCAI).

  • Benitez, A.B., & Chang, S.-F. (2003). Image classification using multimedia knowledge networks. Proc. IEEE International Conference on Image Processing (ICIP) (pp. 613–616).

  • Blei, D., Ng, A., Jordan, M. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

    MATH  Google Scholar 

  • Bosch, A., Zisserman, A., Muñoz, X. (2007). Image classification using random forests and ferns. Proc. International Conference Computer Vision (ICCV).

  • Carbonell, J.G., et al. (1997). Translingual information retrieval: A comparative evaluation. Proc. int’l joint conf. artificial intelligence. Morgan Kaufmann, 708–715.

  • Carneiro, G., Chan, A.B., Moreno, P.J., Vasconcelos, N. (2006). Supervised learning of semantic classes for image annotation and retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(3), 394–410.

    Article  Google Scholar 

  • Carson, C., Belongie, S., Greenspan, H., Malik, J. (2002). Blobworld: image segmentation using expectation-maximization and its application to image querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(8), 1026–1038.

    Article  Google Scholar 

  • Chen, Y., & Wang, J.Z. (2002). A region-based fuzzy feature matching approach to content-based image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9), 252–1267.

    Google Scholar 

  • Chen, Y.X., & Wang, J.Z. (2004). Image categorization by learning and reasoning with regions. Journal of Machine Learning Research, 5, 913–939 .

    Google Scholar 

  • Cowling, P.I., Remde, S.M., Hartley, P., Stewart, W., Stock-Brooks, J., Woolley, T. (2010). C-Link: concept linkage in knowledge repositories. AAAI Spring Symposium Series.

  • Dy, J.G., Brodley, C.E., Kak, A., Broderick, L.S., Aisen, A.M. (2003). Unsupervised feature selection applied to content-based retrieval of lung images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(3), 373–378.

    Article  Google Scholar 

  • Feng, Y., & Lapata, M. (2008). Automatic image annotation using auxiliary text information. Proc. ACL (pp. 272–280). Columbus, Ohio, USA.

  • Fergus, R., Perona, P., Zisserman, A. (2004). A visual category filter for Google images. Proc. International Conference on Computer Vision ECCV (pp. 242–256).

  • Gao, Y., & Fan, J.P. (2006). Incorporating concept ontology to enable probabilistic concept reasoning for multi-level image annotation. Proc. 8th ACM international workshop on Multimedia information retrieval (SIGMM) (pp. 79–88).

  • Gupta, A., Rafatirad, S., Gao, M., Jain, R. (2009). Medialife: from images to a life chronicle. Proc. 35th SIGMOD international conference on Management of data (SIGMOD).

  • Hauptmann, A.G. (2004). Towards a large scale concept ontology for broadcast video. Proc. ACM International Conference on Image and Video Retrieval (CIVR) (pp. 674–675).

  • Hsu, W., Kennedy, L., Chang, S.-F. (2007). Novel reranking methods for visual search. IEEE Multimedia, 14(3), 14–22.

    Article  Google Scholar 

  • Hull, D. (1993). Using statistical testing in the evaluation of retrieval experiments. Proc. 16nd annual international ACM SIGIR (pp. 329–338).

  • Lavrenko, V., Manmatha, R., Jeon, J. (2003). A model for learning the semantics of pictures. Proc. 17th Annual Conference on Neural Information Processing Systems (NIPS).

  • Li, J., & Wang, J.Z. (2008). Real-time computerized annotation of pictures. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(6), 985–1002.

    Article  Google Scholar 

  • Lin, W.-H., Jin, R., Hauptmann, A. (2003). Web image retrieval re-ranking with relevance model. Proc. IEEE Web Intelligence Consortium (WIC).

  • Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. Journal of Computer Vision, 2(60), 91–110.

    Article  Google Scholar 

  • Natsev, A., Naphade, M.R., Tesic, J. (2005). Learning the semantics of multimedia queries and concepts from a small number of examples. Proc. ACM Int’l Conf. Multimedia (598–607). ACM Press.

  • Natsev, A., Rastogi, R., Shim, K. (2004). WALRUS: a similarity retrieval algorithm for image databases. IEEE Transactions on Knowledge and Data Engineering, 16(3), 316.

    Article  Google Scholar 

  • Rasiwasia, N., Costa Pereira, J., Coviello, E., Doyle, G., Lanckriet, G.R.G., Levy, R., Vasconcelos, N. (2010). A new approach to cross-modal multimedia retrieval. ACM Proceedings of the 15th international conference on multimedia. Florence, Italy.

  • Rasiwasia, N., Moreno, P., Vasconcelos, N. (2007). Bridging the gap: query by semantic example. IEEE Transactions on Multimedia, 9(5), 923–938.

    Article  Google Scholar 

  • Santos, E. Jr, Nguyen, H., Brown, S.M. (2001). Kavanah: an active user interface information retrieval agent technology (pp. 412–423). Japan: Maebashi.

  • Santos, E. Jr, Santos, E., Nguyen, H., Pan, L., Korah, J., Huadong, X., Yu, F., Li, D. (2008). Analyst-ready large scale real time retrieval tool for e-governance. E-Government Diffusion, Policy and Impact: Advanced Issues and Practices, IGI Global, ISBN: 978-1-60566-130-8.

  • Schroff, F., Criminisi, A., Zisserman, A. (2007). Harvesting image databases from the web. Proc. IEEE 11th International Conference Computer Vision (ICCV) (pp. 14–21).

  • Sebe, N., & Lew, M.S. (2003). Comparing salient points detectors. Journal of Pattern Recognition Letters, 24(1–3), 89–96.

    Article  MATH  Google Scholar 

  • Snoek, C.G. M., & Smeulders, A.W.M. (2010). Visual-concept search solved?Computer, 43(6), 76–78.

    Article  Google Scholar 

  • Taneva, B., Kacimi, M., Weikum, G. (2010). Gathering and ranking photos of named entities with high precision, high recall, and diversity. Proc. ACM International Conference on Web Search and Data Mining (WSDM).

  • Tao, D., Tang, X., Li, X., Wu, X. (2006). Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence (pp. 1088–1099).

  • Tsai, Y.H. (2009). Salient points reduction for content-based image retrieval. World Academy of Science, Engineering and Technology (WASET) (vol. 49).

  • Üstün, B., Melssen, W.J., L.Buydens, M.C. Facilitating the application of support vector regression by using a universal pearson VII function based kernel. Chemometrics Intel. Lab. Syst (pp. 29–40). Chemometrics Intel.

  • Veltkamp, R.C., & Tanase, M. (2000). Content-based image retrieval systems: A survey. Technique. Report No. UU-CS-2000-34.

  • Vu, K., Hua, K.A., Tavanapong, W. (2003). Image retrieval based on regions of interest. IEEE Transactions on Knowledge and Data Engineering, 15(4), 1045–1049.

    Article  Google Scholar 

  • Wang, J.Z., Li, J., Wiederhold, G. (2001). SIMPLIcity: semantics–sensitive integrated matching for picture libraries. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(9), 947–963.

    Article  Google Scholar 

  • Wang, X., Qiu, S., Liu, K., Tang, X. (2013). In Web image re-ranking using query-specific semantic signatures. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI).

  • Wang, C.H., Zhang, L., Zhang, H.J. (2008). Learning to reduce the semantic gap in web image retrieval and annotation. Proc. 31st annual international ACM SIGIR (pp. 355–362).

  • Yan, R., Hauptmann, A., Jin, R. (2003). Multimedia search with pseudo-relevance feedback. Proc. Int’l Conf. Image and Video Retrieval, LNCS 2728 (pp. 238–247). Springer.

  • Zhao, R., & Grosky, W.I. (2002). Negotiating the semantic gap: from feature maps to semantic landscapes. Journal of Pattern Recognition, 35(3), 593–600.

    Article  MATH  Google Scholar 

  • Zhou, Z-H., Sun, Y-Y., Li, Y-F. (2009). Multi-instance learning by treating instances as non-i.i.d. samples. Proceedings of the 26th annual international conference on machine learning (pp. 1249–1256).

  • Zhou, Z-H., & Zhang, M-L. (2007). Multi-instance multi-label learning with application to scene classification. Advances in Neural Information Processing Systems 19 (NIPS’06). Cambridge: MIT press (pp. 1609–1616).

Download references

Acknowledgments

This work was supported in part by AFOSR Grant No. FA9550-07-1-0050. The authors would like to thank John Korah and Fei Yu for their helpful discussions.

Author information

Authors and Affiliations

  1. Thayer school of Engineering, Dartmouth College, Hanover, USA

    Eugene Santos Jr. & Qi Gu

Authors
  1. Eugene Santos Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qi Gu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Santos Jr., E., Gu, Q. Automatic content based image retrieval using semantic analysis. J Intell Inf Syst 43, 247–269 (2014). https://doi.org/10.1007/s10844-014-0321-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-014-0321-8

Keywords

Search

Navigation