Hostname: page-component-848d4c4894-m9kch Total loading time: 0 Render date: 2024-06-03T14:10:39.827Z Has data issue: false hasContentIssue false
  • English
  • Français

A novel tactile sensor with multimodal vision and tactile units for multifunctional robot interaction

Published online by Cambridge University Press:  05 March 2024

Pengwen Xiong ,
Yuxuan Huang ,
Yifan Yin Open the ORCID record for Yifan Yin [Opens in a new window] ,
Yu Zhang  and
Aiguo Song
Pengwen Xiong
Affiliation:
School of Advanced Manufacturing, Nanchang University, Nanchang, China Robotics Institute, Nanchang University, Nanchang, China
Yuxuan Huang
Affiliation:
School of Advanced Manufacturing, Nanchang University, Nanchang, China Robotics Institute, Nanchang University, Nanchang, China
Yifan Yin
Affiliation:
School of Advanced Manufacturing, Nanchang University, Nanchang, China Robotics Institute, Nanchang University, Nanchang, China
Yu Zhang*
Affiliation:
School of Advanced Manufacturing, Nanchang University, Nanchang, China Robotics Institute, Nanchang University, Nanchang, China
Aiguo Song
Affiliation:
School of Instrument Science and Engineering, Southeast University, Nanjing, China
*
Corresponding author: Yu Zhang; Email: zhangyu74@ncu.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

Robots with multi-sensors always have a problem of weak pairing among different modals of the collected information produced by multi-sensors, which leads to a bad perception performance during robot interaction. To solve this problem, this paper proposes a Force Vision Sight (FVSight) sensor, which utilizes a distributed flexible tactile sensing array integrated with a vision unit. This innovative approach aims to enhance the overall perceptual capabilities for object recognition. The core idea is using one perceptual layer to trigger both tactile images and force-tactile arrays. It allows the two heterogeneous tactile modal information to be consistent in the temporal and spatial dimensions, thus solving the problem of weak pairing between visual and tactile data. Two experiments are specially designed, namely object classification and slip detection. A dataset containing 27 objects with deep presses and shallow presses is collected for classification, and then 20 slip experiments on three objects are conducted. The determination of slip and stationary state is accurately obtained by covariance operation on the tactile data. The experimental results show the reliability of generated multimodal data and the effectiveness of our proposed FVSight sensor.

Keywords

Multimodalweak pairingvisual-tactile sensorobject perceptionslip detection
Type
Research Article
Information
Robotica , Volume 42 , Issue 5 , May 2024 , pp. 1420 - 1435
Copyright
© The Author(s), 2024. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Zhang, S., Chen, Z., Gao, Y. and Wan, W., “Hardware technology of vision-based tactile sensor: A review,” IEEE Sens. J. 22(22), 21410–21427 (2022).CrossRefGoogle Scholar
Liao, J., Xiong, P., Liu, X., Li, Z. and Song, A., “Enhancing robotic tactile exploration with multi-receptive graph convolutional networks,” IEEE Trans. Ind. Electron. DOI: 10.1109/TIE.2023.3323695, early access.Google Scholar
Wang, W., Qiu, W., Yang, H., Wu, H., Shi, G., Chen, Z., Lu, K., Xiang, K. and Ju, B., “An improved capacitive sensor for detecting the micro-clearance of spherical joints,” Sensors 19(12), 2694 (2019).CrossRefGoogle ScholarPubMed
Li, N., Yin, Z., Zhang, W., Xing, C., Peng, T., Meng, B., Yang, J. and Peng, Z., “A triboelectric-inductive hybrid tactile sensor for highly accurate object recognition,” Nano Energy 96, 107063 (2022).CrossRefGoogle Scholar
Presti, D. L., Massaroni, C., D’Abbraccio, J., Massari, L., M. Caponero, U. G. Longo, D. Formica, C. M. Oddo and E. Schena, “Wearable system based on flexible FBG for respiratory and cardiac monitoring,” IEEE Sens. J. 19(17), 7391–7398 (2019).CrossRefGoogle Scholar
Li, W., Alomainy, A., Vitanov, I., Noh, Y., Qi, P. and Althoefer, K., “F-TOUCH sensor: Concurrent geometry perception and multi-axis force measurement,” IEEE Sens. J. 21(4), 4300–4309 (2021).CrossRefGoogle Scholar
Zhang, S., Shan, J., Fang, B. and Sun, F., “Soft robotic finger embedded with visual sensor for bending perception,” Robotica 39(3), 378–390 (2021).CrossRefGoogle Scholar
Yuan, W., Dong, S. and Adelson, E. H., “GelSight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors 17(12), 2762 (2017).CrossRefGoogle ScholarPubMed
Yuan, W., Li, R., Srinivasan, M. A. and Adelson, E. H., “Measurement of Shear and Slip with a GelSight Tactile Sensor,” IEEE International Conference on Robotics and Automation (ICRA) (2015) pp. 304–311.Google Scholar
Yuan, W., Zhu, C., Owens, A., Srinivasan, M. A. and Adelson, E. H., “Shape-independent Hardness Estimation using Deep Learning and a GelSight Tactile Sensor,” 2017 IEEE International Conference on Robotics and Automation (ICRA) (2017) pp. 951–958.Google Scholar
Wang, S., She, Y., Romero, B. and Adelson, E., “GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger,” 2021 IEEE International Conference on Robotics and Automation (ICRA) (2021) pp. 6468–6475.Google Scholar
Gomes, D. F., Lin, Z. and Luo, S., “GelTip: A Finger-shaped Optical Tactile Sensor for Robotic Manipulation,” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2020) pp. 9903–9909.Google Scholar
Donlon, E., Dong, S., Liu, M., Li, J., Adelson, E. and Rodriguez, A., “GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger,” 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2018) pp. 1927–1934.Google Scholar
Taylor, I. H., Dong, S. and Rodriguez, A., “GelSlim 3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Finger,” 2022 International Conference on Robotics and Automation (ICRA) (2022) pp. 10781–10787.Google Scholar
Luo, S., Yuan, W., Adelson, E., Cohn, A. G. and Fuentes, R., “ViTac: Feature Sharing Between Vision and Tactile Sensing for Cloth Texture Recognition,” 2018 IEEE International Conference on Robotics and Automation (ICRA) (2018) pp. 2722–2727.Google Scholar
Wei, F., Zhao, J., Shan, C. and Yuan, Z., “Alignment and Multi-Scale Fusion for Visual-Tactile Object Recognition,” 2022 International Joint Conference on Neural Networks (IJCNN) (2022) pp. 1–8.Google Scholar
Xiong, P. and Yin, Y., “FVSight: A Novel Multimodal Tactile Sensor for Robotic Object Perception,” 2022 IEEE International Conference on Networking, Sensing and Control (ICNSC) (2022) pp. 1–6.Google Scholar
Yamaguchi, A. and Atkeson, C. G., “Recent progress in tactile sensing and sensors for robotic manipulation: Can we turn tactile sensing into vision?,” Adv. Robotics 33(14), 661–673 (2019).CrossRefGoogle Scholar
Wang, Y., Wu, X., Mei, D., Zhu, L. and Chen, J., “Flexible tactile sensor array for distributed tactile sensing and slip detection in robotic hand grasping,” Sens. Actuat. A: Phys. 297, 111512 (2019).CrossRefGoogle Scholar
Sferrazza, C. and D’Andrea, R., “Sim-to-real for high-resolution optical tactile sensing: From images to three-dimensional contact force distributions,” Soft Robot. 9(5), 926–937 (2022).CrossRefGoogle ScholarPubMed
Chaudhury, A. N., Man, T., Yuan, W. and Atkeson, C. G., “Using collocated vision and tactile sensors for visual servoing and localization,” IEEE Robot. Autom. Lett. 7(2), 3427–3434 (2022).CrossRefGoogle Scholar
Zhong, C., Zhao, S., Liu, Y., Li, Z., Kan, Z. and Feng, Y., “A flexible wearable e-skin sensing system for robotic teleoperation,” Robotica 41(3), 1025–1038 (2023).CrossRefGoogle Scholar
Sun, F., Liu, C., Huang, W. and Zhang, J., “Object classification and grasp planning using visual and tactile sensing,” IEEE Trans. Syst. 46(7), 969–979 (2016).Google Scholar
Yuan, W., Wang, S., Dong, S. and Adelson, E., “Connecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) pp. 4494–4502.Google Scholar
Sankar, S., Brown, A. A., Balamurugan, D., H. Nguyen, M. Iskarous, T. Simcox, D. Kumar, A. Nakagawa and N. Thakor, “Texture Discrimination Using a Flexible Tactile Sensor Array on a Soft Biomimetic Finger,” In: IEEE SENSORS (2019) pp. 1–4.Google Scholar
Younes, H., Ibrahim, A., Rizk, M. and Valle, M., “Data Oriented Approximate K-Nearest Neighbor Classifier for Touch Modality Recognition,” 2019 15th Conference on Ph.D Research in Microelectronics and Electronics (PRIME) (2019) pp. 241–244.Google Scholar
Chen, Z., Zuo, W., Hu, Q. and Lin, L., “Kernel sparse representation for time series classification,” Inform. Sci. 292, 15–26 (2015).CrossRefGoogle Scholar
Liu, H., Guo, D. and Sun, F., “Object recognition using tactile measurements: Kernel sparse coding methods,” IEEE Trans. Instrum. Meas. 65(3), 656–665 (2016).CrossRefGoogle Scholar
Xiong, P., He, K., Song, A. and Liu, X., “Robotic haptic adjective perception based on coupled sparse coding,” Sci. China Inform. Sci. 66(2), 129201 (2023).CrossRefGoogle Scholar
Xiong, P., Tong, X., Liu, X., Song, A. and Li, Z., “Robotic object perception based on multi-spectral few-shot coupled learning,” IEEE Trans. Syst. Man. Cybern. Syst. 53(10), 6119–6131 (2023).CrossRefGoogle Scholar
Xiong, P., Liao, J., Zhou, M., Song, A. and Liu, X., “Deeply supervised subspace learning for cross-modal material perception of known and unknown objects,” IEEE Trans. Ind. Inform. 19(2), 2259–2268 (2023).CrossRefGoogle Scholar
Sinou, J., Cayer-Berrioz, J. and Berro, H., “Friction-induced vibration of a lubricated mechanical system,” Tribol. Int. 61, 156–168 (2013).CrossRefGoogle Scholar
James, J. W. and Lepora, N. F., “Slip detection for grasp stabilization with a multifingered tactile robot hand,” IEEE Trans. Robot. 37(2), 506–519 (2021).CrossRefGoogle Scholar
James, J. W., Pestell, N. and Lepora, N. F., “Slip detection with a biomimetic tactile sensor,” IEEE Robot. Autom. Lett. 3(4), 3340–3346 (2018).CrossRefGoogle Scholar
Zapata-Impata, B., Gil, P. and Torres, F., “Learning spatio temporal tactile features with a ConvLSTM for the direction of slip detection,” Sensors 19(3), 523 (2019).CrossRefGoogle ScholarPubMed
Li, J., Dong, S. and Adelson, E., “Slip Detection with Combined Tactile and Visual Information,” 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane (2018) pp. 7772–7777.Google Scholar
Stachowsky, M., Hummel, T., Moussa, M. and Abdullah, H. A., “A slip detection and correction strategy for precision robot grasping,” IEEE/ASME Trans. Mechatron. 21(5), 2214–2226 (2016).CrossRefGoogle Scholar