Elsevier

Neurocomputing

Volume 195, 26 June 2016, Pages 40-49
Neurocomputing

3D object retrieval with multi-feature collaboration and bipartite graph matching

https://doi.org/10.1016/j.neucom.2015.09.118Get rights and content

Abstract

In this paper, we propose a novel 3D object retrieval with features collaboration and bipartite graph matching strategies. We explored the essential characters of 3D object in a view-based retrieval framework, which extracts complement descriptors from both the contour and the interior region of 3D object effectively. Specifically, a greedy bipartite graph matching algorithm is employed. With the bipartite graph matching and feature concatenation, significant performance improvement is achieved in the 3D object retrieval task. The proposed method is evaluated by the third party on the data set comprising more than 500 3D objects and achieves the best performance for SHREC’15 challenge.

Introduction

With the rapid development of 3D technologies, computer graphics hardware and networks, 3D objects have been widely explored in plenty of applications [23], [24], especially in architecture design [1], movie production, 3-D graphics, and the medical industry, which leads to the eager requirement of effective and efficient 3D object retrieval. Based on different data type adopted, 3D object retrieval methods can be roughly categorized into two groups [8], [13]: 3D models based [2], [3] and multiple views based [4], [5].

In 3D model-based methods, each 3D object is represented by a virtual 3D model with geometry-based methods. To describe the information of 3D models, 3D objects are described with model-based features, such as low-level feature (e.g. the volumetric descriptor [25], the surface distribution [26] and surface geometry [27], [28], [29]). With the 3D model data, 3D model-based methods can preserve the global spatial information of 3D objects; while in some cases when we want to search the objects in the world, 3D model information is not available. For example, when the tourist finds some interesting things and wants to find similar ones in the dataset, it is hard to obtain the model information but just take several pictures. Some method [17] employs a set of 2D images to constructs 3D model, but it is both time consuming and fine sampling. Due to the expensive computational costs and the limitation of obtaining explicit 3D information, the poor performance of reconstructions methods often results in low-quality 3D models, which restricts the development of model-based 3D parsing methods in some practical applications.

Different from the 3D model-based methods, the view-based 3D object retrieval methods use a group of images from different directions for 3D object representation [30], [31] and the matching between two 3D objects is accomplished via multiple-view matching. These views may be captured with a static camera array or without such camera array constraint. Such view-based methods release the restriction of 3D model, and the ubiquity of mobile devices with cameras makes it convenient to capture real objects images. Besides, online multiview data of 3D objects have become increasingly available on websites, which facilitates the practical application for view-based method. Due to convenient obtainment and bargain price of equipment, plenty of researchers pitch into multi-view based 3D object retrieval methods [6], [7], [8], recently. On the one hand, View-based retrieval may learn nutrition from large quantity studies of visual parsing techniques, like search [9], [10], segmentation [11] and tracking [12] etc. On the other hand, it’s greatly flexible to represent a 3D object by a set of 2D views. For location-based mobile applications, view based methods also provide new search opportunities with the help of cameras. Compared with model-based methods, view-based methods is more discriminative for 3D objects, which can lead to better object retrieval performance[32], [33].

Here, we focus on the recent progress in view-based 3D object retrieval, which has been widely used in CAD applications. For example, We first survey the key technologies and challenges in view-based 3D object retrieval and then discuss the state-of-the-art methods and future research directions in the field. Different from general classifier [38], an ideal view-based 3D object retrieval should has the abilities of exploring effective strategy to conduct multi-view matching and estimate the relevance among different 3D objects. Aiming at utilizing the collaborated features for multiple views and conducting multi-view matching and estimating the relevance among different 3D objects, this paper proposes a novel 3D object retrieval with features collaboration and bipartite graph matching strategies, and the main contributions of this paper are summarized as follows:

  • Inspired by the essential characters of 3D object, a view-based retrieval framework with multi-feature collaboration and bipartite graph matching is proposed, which extracts complement descriptors from both the contour and the interior region of 3D object effectively.

  • A Greedy Search (GS) algorithm is proposed to calculate the similarity of query object and object, and three bipartite graphs are employed to obtain the optimal match of each bipartite graph pair.

Our proposed method participated the SHREC’15 challenge and achieved the state-of-the-art performance in 0.6174 (top 10 precision). The SHREC’15 challenge is designed for 3D object retrieval which is modeled by multi-view to simulate the true environment.

The remainder of this paper is organized as follows. Related work is reviewed in Section 2. The detail descriptions of the proposed method are presented in Section 3. Extensive experimental results are reported in Section 4. Section 5 concludes the paper.

Section snippets

Related work

Due to convenient obtainment and bargain price of equipment, plenty of researchers pitch into multi-view based 3D object retrieval methods [6], [7], recently. On the one hand, View-based retrieval may learn nutrition from large quantity studies of visual parsing techniques, like search [9], [10], segmentation [11] and tracking [12] etc. On the other hand, it is greatly flexible to represent a 3D object by a set of 2D views. And abundant information can be fetched by multi-view object

Multi-feature collaboration and bipartite graph matching based 3D retrieval

In this section, the proposed view-based 3D object retrieval method is introduced. Firstly three descriptors are extracted. Secondly three bipartite graph are constructed on each descriptor between two objects. At last, combine the three bipartite graphs to one descriptor which is used to calculate the similarity of the two objects.

Experimental results

In this section, extensive experiment results are presented to evaluate the proposed multi-feature collaboration and bipartite graph matching based 3D retrieval.

Conclusion

In this paper, we present a view-based 3D model retrieval algorithm on multi-feature by bipartite graph matching. The proposed method extracts three descriptors and combine them through bipartite graph. The user feedback information is effectively explored to achieve better performance. We compare our method on the SHREC’15 data set with other methods. Experimental results and comparison show that the proposed method outperforms the other methods for 3D model retrieval.

Acknowledgments

We would like to acknowledge the editors and reviewers, whose valuable comments greatly improved the manuscript. This work was supported in part by the Major State Basic Research Development Program of China (973 Program 2015CB351804) and the National Natural Science Foundation of China under Grant nos. 61572155 and 61272386.

Yan Zhang received the B.S. and M.S. degrees from Department of Mathematics and School of Computer Science and Technology, Harbin Institute of Technology (HIT), Harbin, China, in 2007 and 2009, respectively. He is now working towards the Ph.D. degree at School of Computer Science and Technology, HIT.

References (38)

  • Yue Gao et al.

    View-based 3-D object retrieval: challenges and approaches

    IEEE Multimedia Mag.

    (2014)
  • Z. Stejic et al.

    Relevance feedback-based image retrieval interface incorporating region and feature saliency patterns as visualizable image similarity criteria

    IEEE Trans. Ind. Electron.

    (2003)
  • T. Takeguchi et al.

    Using depth aspect images for robust and efficient search of multiple objects in optomechatronic sensing

    IEEE Trans. Ind. Electron.

    (2005)
  • H. Zhuang et al.

    Multichannel pulse-coupled-neural network-based color image segmentation for object detection

    IEEE Trans. Ind. Electron.

    (2012)
  • C.-Y. Chang et al.

    Real-time visual tracking and measurement to control fast dynamics of overhead cranes

    IEEE Trans. Ind. Electron.

    (2012)
  • Yue Gao et al.

    View-Based 3-D Object Retrieval

    (2014)
  • M. Steinbach, G. Karypis, V. Kumar, A comparison of document clustering techniques, In: Proceedings of the KDD Workshop...
  • W.S. Xiao. Graph Theory and Its Algorithms,...
  • S. Avidan et al.

    Trajectory triangulation: 3D reconstruction of moving points from a monocular image sequence

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2000)
  • Cited by (27)

    • Rotation Prediction Based Representative View Locating Framework for 3D Object Recognition

      2022, CAD Computer Aided Design
      Citation Excerpt :

      At the same time, analyzing 3D object features has become a core issue. In recent years, most 3D object recognition technologies use convolutional neural networks (CNN) based methods to extract features from multi-view rendered images, voxels or point clouds of 3D objects [1–27]. So far, the feature performance of multi-view rendered images is superior to those of voxels and point clouds [7,10,18].

    • Discrete artificial electric field algorithm for high-order graph matching

      2020, Applied Soft Computing Journal
      Citation Excerpt :

      Graph matching is a complex NP-hard combinatorial optimization problem and aims to maximize the matching accuracy of the feature sets between two or more graphs. The first order graph matching methods only establish the correspondence between the node to node features using their local descriptor [6,7]. These methods fail in the presence of non-discriminative local appearances, repeated patterns, textures, and because of the ignorance of spatial relationships.

    • High-order graph matching based on ant colony optimization

      2019, Neurocomputing
      Citation Excerpt :

      This function measures the similarity of the features between the graphs (Fig. 1). First-order methods mainly focus on the isolated node relation based on local feature descriptors [12,13]. Such methods cannot work well on the graphs in which many nodes have the similar features.

    View all citing articles on Scopus

    Yan Zhang received the B.S. and M.S. degrees from Department of Mathematics and School of Computer Science and Technology, Harbin Institute of Technology (HIT), Harbin, China, in 2007 and 2009, respectively. He is now working towards the Ph.D. degree at School of Computer Science and Technology, HIT.

    Feng Jiang received the B.S., M.S., and Ph.D. degrees in computer science from Harbin Institute of Technology (HIT), Harbin, China, in 2001, 2003, and 2008, respectively. He is now an Associated Professor in the Department of Computer Science, HIT and a visiting scholar in the School of Electrical Engineering, Princeton University. His research interests include computer vision, pattern recognition and image and video processing.

    Seungmin Rho received his M.S. and Ph.D. degrees in Computer Science from Ajou University, Korea, in Computer Science from Ajou University, Korea, in 2003 and 2008, respectively. In 2008–2009, he was a Postdoctoral Research Fellow at the Computer Music Lab of the School of Computer Science in Carnegie Mellon University. In 2009–2011, he had been working as a Research Professor at School of Electrical Engineering in Korea University. In 2012, he was an assistant professor at Division of Information and Communication in Baekseok University. Dr. Rho is currently a faculty of Department of Multimedia at Sungkyul University. His current research interests include database, big data analysis, music retrieval, multimedia systems, machine learning, knowledge management as well as computational intelligence.

    Shaohui Liu received the B.S., M.S., and Ph.D. degrees in computer science from Harbin Institute of Technology (HIT), Harbin, China, in 2000, 2002, and 2007, respectively. He is now an Associated Professor in the Department of Computer Science, HIT and his research interests include data compression, pattern recognition and image and video processing.

    Debin Zhao received the B.S., M.S., and Ph.D. degrees in computer science from Harbin Institute of Technology (HIT), Harbin, China, in 1985, 1988, and 1998, respectively.

    He is now a Professor in the Department of Computer Science, HIT. He has published over 200 technical articles in refereed journals and conference proceedings in the areas of image and video coding, video processing, video streaming and transmission, and pattern recognition.

    Rongrong Ji is currectly wokring at Xiamen University since 2013, where he directs the Intelligent Multimedia Technology Laboratory and serves as a Dean Assistant in the School of Information Science and Technology. Before that, he used to be a Postdoc research fellow in the Department of Electrical Engineering, Columbia University from 2010 to 2013, worked with Professor Shih-Fu Chang. He obtained his Ph.D. degree in computer science from Harbin Institute of Technology, under supervision of Professor Hongxun Yao. He had been a visiting student at University of Texas of San Antonio worked with Professor Qi Tian, and a research assistant at Peking University worked with Professor Wen Gao in 2010, a research intern at Microsoft Research Asia, worked with Dr. Xing Xie from 2007 to 2008.

    View full text