Abstract
Tactile and visual measurements are two classes of sensing modalities which frequently occur in manufacturing industry and robotics. Their matching problem is highly interesting in many practical scenarios since they provide different properties about objects. This chapter investigates the visual–tactile cross-modal matching problem which is formulated as retrieving the relevant sample in an unlabeled gallery visual dataset in response to the tactile query sample. Such a problem exhibits nontrivial challenges that there does not exist sample-to-sample-pairing relation between tactile and visual modalities, which exhibit significantly different characteristics. To this end, a dictionary learning model is designed, which can simultaneously learn the projection subspace and the latent common dictionary for the visual and tactile measurements. In addition, an optimization algorithm is developed to effectively solve the common dictionary learning problem. Based on the obtained solution, the visual–tactile cross-modal matching algorithm can be easily developed. Finally, experimental validations are performed on the PHAC-2 datasets to show the effectiveness of the proposed visual–tactile cross-modal matching framework and method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chu, V., McMahon, I., Riano, L., McDonald, C.G., He, Q., Perez-Tejada, J.M., Arrigo, M., Darrell, T., Kuchenbecker, K.J.: Robotic learning of haptic adjectives through physical interaction. Robot Auton. Syst. 63, 279–292 (2015)
Internet shopping and buying behavior of college students: Dh, L., Am, F., D, L. Serv. Mark. Q. 627, 123–138 (2006)
Gao, Y., Hendricks, L.A., Kuchenbecker, K.J., Darrell, T.: Deep learning for tactile understanding from visual and haptic data. IEEE International Conference on Robotics and Automation (ICRA) 2016, 536–543 (2016)
Huang, D.A., Wang, Y.C.F.: Coupled dictionary and feature space learning with applications to cross-domain image synthesis and recognition. In: Proceedings IEEE International Conference on Computer Vision, pp. 2496–2503 (2013)
Kim, J., Forsythe, S.: Adoption of sensory enabling technology for online apparel shopping. Eur. J. Mark. 43, 1101–1120 (2009)
Klatzky, R.L., Lederman, S.J., Matula, D.E.: Haptic exploration in the presence of vision. J. Exp. Psychol. Hum. Percept. Perform. 726–743 (1993)
Lampert, C., Krömer, O.: Weakly-paired maximum covariance analysis for multimodal dimensionality reduction and transfer learning. Comput. Vis.-ECCV 2010, 566–579 (2010)
Liu, H., Guo, D., Sun, F.: Object recognition using tactile measurements: kernel sparse coding methods. IEEE Trans. Instrum. Meas. 65(3), 656–665 (2016)
Liu, H., Yu, Y., Sun, F., Gu, J.: Visual-tactile fusion for object recognition. IEEE Trans. Autom. Sci. Eng. 14(2), 996–1008 (2017)
Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online learning for matrix factorization and sparse coding. J. Mach. Learn. Res. 11(Jan), 19–60 (2010)
Mandal, D., Biswas, S.: Generalized coupled dictionary learning approach with applications to cross-modal matching. IEEE Trans. Image Process. 25(8), 3826–3837 (2016)
Peck, J., Childers, T.L.: Individual differences in haptic information processing: the need for touch scale. J. Consum. Res. 430–442 (2003)
Rasiwasia, N., Mahajan, D., Mahadevan, V., Aggarwal, G.: Cluster canonical correlation analysis. In: Proceedings International Conference on Artificial Intelligence Statistics, pp. 823–831 (2014)
Shekhar, S., Patel, V.M., Nguyen, H.V., Chellappa, R.: Coupled projections for adaptation of dictionaries. TIP (2016)
Wang, K., Yin, Q., Wang, W., Wu, S., Wang, L.: A comprehensive survey on cross-modal retrieval. arXiv preprint arXiv:1607.06215 (2016)
Wen, Z., Yin, W.: A feasible method for optimization with orthogonality constraints. Math. Program. 1–38 (2013)
Zhang, L., Zhang, D.: Evolutionary cost-sensitive discriminative learning with application to vision and olfaction. IEEE Trans. Instrum. Meas. 66(2), 198–211 (2017)
Zhuang, Y., Wang, Y., Wu, F., Zhang, Y., Lu, W.: Supervised coupled dictionary learning with group structures for multi-modal retrieval. In: AAAI, pp. 1070–1076 (2013)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Liu, H., Sun, F. (2018). Visual–Tactile Cross-Modal Matching Using Common Dictionary Learning. In: Robotic Tactile Perception and Understanding. Springer, Singapore. https://doi.org/10.1007/978-981-10-6171-4_9
Download citation
DOI: https://doi.org/10.1007/978-981-10-6171-4_9
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-6170-7
Online ISBN: 978-981-10-6171-4
eBook Packages: Computer ScienceComputer Science (R0)