Xinghai Sun
Lei Zhang
ABSTRACT
Because of the popularity of touch-screen devices, it has become a highly desirable feature to retrieve images from a huge repository by matching with a hand-drawn sketch. Although searching images via keywords or an example image has been successfully launched in some commercial search engines of billions of images, it is still very challenging for both academia and industry to develop a sketch-based image retrieval system on a billion-level database. In this work, we systematically study this problem and try to build a system to support query-by-sketch for two billion images. The raw edge pixel and Chamfer matching are selected as the basic representation and matching in this system, owning to the superior performance compared with other methods in extensive experiments. To get a more compact feature and a faster matching, a vector-like Chamfer feature pair is introduced, based on which the complex matching is reformulated as the crossover dot-product of feature pairs. Based on this new formulation, a compact shape code is developed to represent each image/sketch by projecting the Chamfer features to a linear subspace followed by a non-linear source coding. Finally, the multi-probe Kmedoids-LSH is leveraged to index database images, and the compact shape codes are further used for fast reranking. Extensive experiments show the effectiveness of the proposed features and algorithms in building such a sketch-based image search system.
- S. Belongie, J. Malik, and J. Puzicha. Shape context: A new descriptor for shape matching and object recognition. NIPS, 2001.Google Scholar
- G. Borgefors. Hierarchical chamfer matching: A parametric edge matching algorithm. PAMI, 1988. Google ScholarDigital Library
- Y. Cao, C. Wang, L. Zhang, and L. Zhang. Edgel index for large-scale sketch-based image search. In CVPR, 2011. Google ScholarDigital Library
- Y. Cao, H. Wang, C. Wang, Z. Li, L. Zhang, and L. Zhang. Mindfinder: Interactive sketch-based image search on millions of images. In ACM MM, 2010. Google ScholarDigital Library
- M. Eitz, J. Hays, and M. Alexa. How do humans sketch objects? SIGGRAPH, 2012. Google ScholarDigital Library
- M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa. An evaluation of descriptors for large-scale image retrieval from sketched feature lines. Computers & Graphics, 2010. Google ScholarDigital Library
- M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa. Sketch-based image retrieval: Benchmark and bag-of-features descriptors. TVCG, 2011. Google ScholarDigital Library
- R. Hu, M. Barnard, and J. Collomosse. Gradient field descriptor for sketch based retrieval and localization. In ICIP, 2010.Google ScholarCross Ref
- H. Jégou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. PAMI, 2011.Google ScholarDigital Library
- H. Jégou, R. Tavenard, M. Douze, and L. Amsaleg. Searching in one billion vectors: re-rank with source coding. In ICASSP, 2011.Google ScholarCross Ref
- L. Paulevé, H. Jégou, and L. Amsaleg. Locality sensitive hashing: A comparison of hash function types and querying mechanisms. Pattern Recognition Letters, 2010. Google ScholarDigital Library
- C. Siagian and L. Itti. Rapid biologically-inspired scene classification using features shared with visual attention. PAMI, 2007. Google ScholarDigital Library
- B. Stenger, A. Thayananthan, P. Torr, and R. Cipolla. Model-based hand tracking using a hierarchical bayesian filter. PAMI, 2006. Google ScholarDigital Library
- Z. Sun, C. Wang, L. Zhang, and L. Zhang. Query-adaptive shape topic mining for hand-drawn sketch recognition. In ACM MM, 2012. Google ScholarDigital Library
- A. Torralba, R. Fergus, and W. T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. PAMI, 2008. Google ScholarDigital Library
- K.-Y. Tseng, Y.-L. Lin, C.-Y. Hsiu, and W. H. Hsu. Sketch-based image retrieval on mobile devices using compact hash bits. In ACM MM, 2012. Google ScholarDigital Library
- C. Wang, F. Jing, L. Zhang, and H.-J. Zhang. Scalable search-based image annotation of personal images. In ACM MIR, 2006. Google ScholarDigital Library
Index Terms
- Indexing billions of images for sketch-based retrieval
Recommendations
Sketch-based Image Retrieval using Generative Adversarial Networks
MM '17: Proceedings of the 25th ACM international conference on MultimediaFor sketch-based image retrieval (SBIR), we propose a generative adversarial network trained on a large number of sketches and their corresponding real images. To imitate human search process, we attempt to match candidate images with theimaginary image ...
Sketch4Image: a novel framework for sketch-based image retrieval based on product quantization with coding residuals
Sketch-based Image Retrieval (SBIR) is one important branch of Content-based Image Retrieval (CBIR). SBIR means dealing with retrieval using simple edge or contour images. However, SBIR is more difficult than CBIR due to the lack of visual information, ...
Sketch-based image retrieval using keyshapes
Although sketch based image retrieval (SBIR) is still a young research area, there are many applications capable of exploiting this retrieval paradigm, such as web searching and pattern detection. Moreover, nowadays drawing a simple sketch query turns ...
Comments