Omar Seddati
Stéphane Dupont
ABSTRACT
Freehand sketches are a simple and powerful tool for communication. They are easily recognized across cultures and suitable for various applications. In this paper, we use deep convolutional neural networks (ConvNets) to address sketch-based image retrieval (SBIR). We first train our ConvNets on sketch and image object recognition in a large scale benchmark for SBIR (the sketchy database). We then conduct a comprehensive study of ConvNets features for SBIR, using a kNN similarity search paradigm in the ConvNet feature space. In contrast to recent SBIR works, we propose a new architecture the quadruplet networks which enhance ConvNet features for SBIR. This new architecture enables ConvNets to extract more robust global and local features. We evaluate our approach on three large scale datasets. Our quadruplet networks outperform previous state-of-the-art on two of them by a significant margin and gives competitive results on the third. Our system achieves a recall of 42.16% (at k=1) for the sketchy database (more than 5% improvement), a Kendal score of 43.28Τb on the TU-Berlin SBIR benchmark (close to 6Τb improvement) and a mean average precision (MAP) of 32.16% on Flickr15k (a category level SBIR benchmark).
- Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. 2014. Neural codes for image retrieval. In European conference on computer vision. Springer, 584--599.Google Scholar
Cross Ref
- Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. 2014. Neural codes for image retrieval. In European conference on computer vision. Springer, 584--599.Google ScholarCross Ref
- Serge Belongie, Jitendra Malik, and Jan Puzicha. 2002. Shape matching and object recognition using shape contexts. IEEE transactions on pattern analysis and machine intelligence 24, 4 (2002), 509--522. Google ScholarDigital Library
- Sreyasee Das Bhattacharjee, Junsong Yuan, Weixiang Hong, and Xiang Ruan. 2016. Query Adaptive Instance Search using Object Sketches. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, 1306--1315. Google ScholarDigital Library
- Konstantinos Bozas and Ebroul Izquierdo. 2012. Large scale sketch based image retrieval using patch hashing. In International Symposium on Visual Computing. Springer, 210--219.Google ScholarCross Ref
- Tu Bui and John Collomosse. 2015. Scalable sketch-based image retrieval using color gradient features. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1--8. Google ScholarDigital Library
- Tu Bui, Leonardo Ribeiro, Moacir Ponti, and John Collomosse. 2016. Generali- sation and Sharing in Triplet Convnets for Sketch based Visual Search. arXiv preprint arXiv:1611.05301 (2016).Google Scholar
- Xiaochun Cao, Hua Zhang, Si Liu, Xiaojie Guo, and Liang Lin. 2013. Sym-fish: A symmetry-aware flip invariant sketch histogram shape descriptor. In Proceedings of the IEEE International Conference on Computer Vision. 313--320. Google ScholarDigital Library
- Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. 2011. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop.Google Scholar
- Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, Vol. 1. IEEE, 886--893. Google ScholarDigital Library
- Aishwarya Deore and BL Gunjal. 2016. Advanced Sketch Based Image Retrieval System Using Object Boundary Selection Algorithm. (2016).Google Scholar
- Mathias Eitz, James Hays, and Marc Alexa. 2012. How do humans sketch objects? ACM Trans. Graph. 31, 4 (2012), 44--1. Google ScholarDigital Library
- Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2010. An evaluation of descriptors for large-scale image retrieval from sketched feature lines. Computers & Graphics 34, 5 (2010), 482--498. Google ScholarDigital Library
- Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2011. Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE transactions on visualization and computer graphics 17, 11 (2011), 1624--1636. Google ScholarDigital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.Google ScholarCross Ref
- Rui Hu and John Collomosse. 2013. A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Computer Vision and Image Understanding 117, 7 (2013), 790--806. Google ScholarDigital Library
- Cheng Jin, Zheming Wang, Tianhao Zhang, Qinen Zhu, and Yuejie Zhang. 2015. A Novel Visual-Region-Descriptor-based Approach to Sketch-based Image Re- trieval. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. ACM, 267--274. Google ScholarDigital Library
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classifica- tion with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105. Google ScholarDigital Library
- Joseph J LaViola Jr and Robert C Zeleznik. 2007. MathPad 2: a system for the creation and exploration of mathematical sketches. In ACM SIGGRAPH 2007 courses. ACM, 46. Google ScholarDigital Library
- Qiang Li, Yahong Han, and Jianwu Dang. 2016. Sketch4Image: a novel framework for sketch-based image retrieval based on product quantization with coding residuals. Multimedia Tools and Applications 75, 5 (2016), 2419--2434. Google ScholarDigital Library
- Yi Li, Yi-Zhe Song, and Shaogang Gong. 2013. Sketch Recognition by Ensemble Matching of Structured Features.. In BMVC, Vol. 1. 2.Google Scholar
- David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 2 (2004), 91--110. Google ScholarDigital Library
- Jianwei Niu, Jun Ma, Jie Lu, Xuefeng Liu, and Zeyu Zhu. 2017. M-SBIR: An Improved Sketch-Based Image Retrieval Method Using Visual Word Mapping. In International Conference on Multimedia Modeling. Springer, 257--268.Google Scholar
- Tom Y Ouyang and Randall Davis. 2011. ChemInk: a natural real-time recognition system for chemical drawings. In Proceedings of the 16th international conference on Intelligent user interfaces. ACM, 267--276. Google ScholarDigital Library
- Yonggang Qi, Yi-Zhe Song, Tao Xiang, Honggang Zhang, Timothy Hospedales, Yi Li, and Jun Guo. 2015. Making better use of edges via perceptual grouping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1856--1865.Google ScholarCross Ref
- Yonggang Qi, Yi-Zhe Song, Honggang Zhang, and Jun Liu. 2016. Sketch-based image retrieval via Siamese convolutional neural network. In Image Processing (ICIP), 2016 IEEE International Conference on. IEEE, 2460--2464.Google ScholarCross Ref
- Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG) 35, 4 (2016), 119. Google ScholarDigital Library
- Ravi Kiran Sarvadevabhatla and R Venkatesh Babu. 2015. Freehand sketch recognition using deep features. arXiv preprint arXiv:1502.00254 (2015).Google Scholar
- Ravi Kiran Sarvadevabhatla, Jogendra Kundu, and others. 2016. Enabling My Robot To Play Pictionary: Recurrent Neural Networks For Sketch Recognition. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, 247--251. Google ScholarDigital Library
- Rosália G Schneider and Tinne Tuytelaars. 2014. Sketch classification and classification-driven analysis using fisher vectors. ACM Transactions on Graphics (TOG) 33, 6 (2014), 174. Google ScholarDigital Library
- Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 815--823.Google ScholarCross Ref
- Omar Seddati, Stephane Dupont, and Saïd Mahmoudi. 2015. Deepsketch: deep convolutional neural networks for sketch recognition and similarity search. In Content-Based Multimedia Indexing (CBMI), 2015 13th International Workshop on. IEEE, 1--6.Google ScholarCross Ref
- Omar Seddati, Stephane Dupont, and Saïd Mahmoudi. 2016. DeepSketch 2: Deep convolutional neural networks for partial sketch recognition. In Content-Based Multimedia Indexing (CBMI), 2016 14th International Workshop on. IEEE, 1--6.Google ScholarCross Ref
- Omar Seddati, Stéphane Dupont, and Saïd Mahmoudi. 2016. DeepSketch2Image: Deep Convolutional Neural Networks for Partial Sketch Recognition and Image Retrieval. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, 739--741. Google ScholarDigital Library
- Eli Shechtman and Michal Irani. 2007. Matching local self-similarities across images and videos. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on. IEEE, 1--8.Google ScholarCross Ref
- Josef Sivic, Andrew Zisserman, and others. 2003. Video google: A text retrieval approach to object matching in videos.. In iccv, Vol. 2. 1470--1477. Google ScholarDigital Library
- Ivan E Sutherland. 1964. Sketchpad a man-machine graphical communication system. Transactions of the Society for Computer Simulation 2, 5 (1964), R--3.Google Scholar
- B Szántó, P Pozsegovics, Z Vámossy, and Sz Sergyan. 2011. Sketch4match Content-based image retrieval system using sketches. In Applied Machine Intelligence and Informatics (SAMI), 2011 IEEE 9th International Symposium on. IEEE, 183--188.Google ScholarCross Ref
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition. 1--9.Google ScholarCross Ref
- Ji Wan, Dayong Wang, Steven Chu Hong Hoi, Pengcheng Wu, Jianke Zhu, Yongdong Zhang, and Jintao Li. 2014. Deep learning for content-based image retrieval: A comprehensive study. In Proceedings of the 22nd ACM international conference on Multimedia. ACM, 157--166. Google ScholarDigital Library
- Yuxin Wang, Miao Yu, Qi Jia, and He Guo. 2011. Query by sketch: An asymmetric sketch-vs-image retrieval system. In Image and Signal Processing (CISP), 2011 4th International Congress on, Vol. 3. IEEE, 1368--1372.Google ScholarCross Ref
- Yongxin Yang and Timothy M Hospedales. 2015. Deep neural networks for sketch recognition. arXiv preprint arXiv: 1501.07873 (2015).Google Scholar
- Kemal Tugrul Yesilbek, Cansu Sen, Serike Cakmak, and T Metin Sezgin. 2015. SVM-based sketch recognition: which hyperparameter interval to try?. In Proceedings of the workshop on Sketch-Based Interfaces and Modeling. Eurographics Association, 117--121. Google ScholarDigital Library
- Qian Yu, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales. 2015. Sketch-a-net that beats humans. arXiv preprint arXiv:1501.07873 (2015).Google Scholar
- Hua Zhang, Si Liu, Changqing Zhang, Wenqi Ren, Rui Wang, and Xiaochun Cao. 2016. Sketchnet: Sketch classification with web images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1105--1113.Google ScholarCross Ref
- Rong Zhou, Liuli Chen, and Liqing Zhang. 2012. Sketch-based image retrieval on a large scale database. In Proceedings of the 20th ACM international conference on Multimedia. ACM, 973--976. Google ScholarDigital Library
Index Terms
- Quadruplet Networks for Sketch-Based Image Retrieval
Recommendations
Sketch-based Image Retrieval using Generative Adversarial Networks
MM '17: Proceedings of the 25th ACM international conference on MultimediaFor sketch-based image retrieval (SBIR), we propose a generative adversarial network trained on a large number of sketches and their corresponding real images. To imitate human search process, we attempt to match candidate images with theimaginary image ...
DeepSketch 3
Freehand sketches are a simple and powerful tool for communication. They are easily recognized across cultures and suitable for various applications. In this paper, we use deep convolutional neural networks (ConvNets), state-of-the-art in the field of ...
Sketch-based image retrieval using keyshapes
Although sketch based image retrieval (SBIR) is still a young research area, there are many applications capable of exploiting this retrieval paradigm, such as web searching and pattern detection. Moreover, nowadays drawing a simple sketch query turns ...
Comments