Abstract
Objective
To develop a multimodal deep-learning model for classifying Chinese medicine constitution, i.e., the balanced and unbalanced constitutions, based on inspection of tongue and face images, pulse waves from palpation, and health information from a total of 540 subjects.
Methods
This study data consisted of tongue and face images, pulse waves obtained by palpation, and health information, including personal information, life habits, medical history, and current symptoms, from 540 subjects (202 males and 338 females). Convolutional neural networks, recurrent neural networks, and fully connected neural networks were used to extract deep features from the data. Feature fusion and decision fusion models were constructed for the multimodal data.
Results
The optimal models for tongue and face images, pulse waves and health information were ResNet18, Gate Recurrent Unit, and entity embedding, respectively. Feature fusion was superior to decision fusion. The multimodal analysis revealed that multimodal data compensated for the loss of information from a single mode, resulting in improved classification performance.
Conclusions
Multimodal data fusion can supplement single model information and improve classification performance. Our research underscores the effectiveness of multimodal deep learning technology to identify body constitution for modernizing and improving the intelligent application of Chinese medicine.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ye Y, Wei L. Design and implementation of the traditional Chinese medicine constitution system based on the diagnosis of tongue and consultation. IEEE Access 2021;9:4266–4278.
Wang J, Li YS, Ni C, Zhang HM, Li LR, Wang Q. Cognition research and constitutional classification in Chinese medicine. Am J Chin Med 2011;39:651–660.
Liang X, Wang Q, Jiang ZQ, Li ZQ, Zhang MX, Yang PY, et al. Clinical research linking traditional Chinese medicine constitution types with diseases: a literature review of 1639 observational studies. J Tradit Chin Med 2020;40:690–702.
Wang Q, Bai MH, Yang Y, Liang X, Sun PC, Han JW, et al. Application of TC constitution in lifetime health maintenance. J Tradit Chin Med Sci (Chin) 2018;5:6–15.
Wang J, Li YS, Wang Q. Identification of Chinese medicine constitution in public health services. Chin J Integr Med 2019;25:550–553.
Ardila D, Kiraly AP, Bharadwaj S, Choi, B, Reicher JJ, Peng L, et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nature Med 2019;25:954–961.
Fauw JD, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med 2018;24:1342–1350.
He K, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016;770-778.
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision[C]//2016 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA; 2016:2818–2826.
Szegedy C, Ioffe S, Vanhoucke V, Alemi A. Inception-v4, inception-ResNet and the impact of residual connections on learning[C]//Thirty-First AAAI Conference on Artificial Intelligence. San Francisco, CA, USA;2017:4278–4284.
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale Image recognition. Eprint ArXiv;2014.
Nafi NM, Hsu WH. Addressing class imbalance in image-based plant disease detection: deep generative vs. sampling-based approaches[C]//2020 International Conference on Systems, Signals and Image Processing, Niteroi, Brazil, 2020.
Li HT, Xu B, Wang NY, Liu J. Deep convolutional neural networks for classifying body constitution. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 2016:128–135.
Guo C, Berkhahn F. Entity embeddings of categorical variables. Eprint ArXiv;2016.
Xu Q, Zeng Y, Tang WJ, Peng W, Xia TW, Li ZR, et al. Multi-task Joint learning model for segmenting and classifying tongue images using a deep neural Network. IEEE J Biomed Health Inform 2020;24:2481–2489.
Zhou H, Hu GQ, Zhang XF. Constitution identification of tongue image based on CNN[C]// 2018 11th International Congress on Image and Signal Processing, Biomedical Engineering and Informatics, Beijing, China, 2018.
Li HH, Wen GH, Zeng HB. Natural tongue physique identification using hybrid deep learning methods. Multimed Tools Appl 2019;78:6847–6868.
Deng J, Dong W, Socher R, Li LJ, Li K, Li FF. ImageNet: a large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA,2009;248-255.
Everingham M, Gool LV, Williams CKI, Winn J, Zisserman A. The pascal visual object classes (VOC) challenge. Int J Com Vision 2010;88:303–338.
Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, et al. Microsoft COCO: common objects in context[C]// 2014 13th European Conference on Computer Vision, Switzerland, Zurich, 2014;740-755.
Nie J, Zhang LL, Liu J, Wang YQ. Pulse taking by a piezoelectric film sensor via mode energy ratio analysis helps identify pregnancy status. IEEE J Biomed Health Inform 2021;26:2116–2123.
Zhang Q, Zhou JH, Zhang B. Graph based multichannel feature fusion for wrist pulse diagnosis. IEEE J Biomed Health Inform 2021;25:3732–3743.
Acknowledgment
The authors thank ZHAO Liang at Tianjin Medvalley Technology Co., Ltd., and YIN Hong-nan at Heilongjiang University of Chinese Medicine for their engaged support.
Author information
Authors and Affiliations
Contributions
Gu TY, Yan ZZ and Jiang JH designed the study, performed the research, analyzed data, and wrote the paper.
Corresponding author
Ethics declarations
All authors declare no conflicts of financial interest.
Additional information
Supported by the National Key Research and Development Program of China Under Grant (No. 2018YFC1707704)
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Gu, Ty., Yan, Zz. & Jiang, Jh. Classifying Chinese Medicine Constitution Using Multimodal Deep-Learning Model. Chin. J. Integr. Med. 30, 163–170 (2024). https://doi.org/10.1007/s11655-022-3541-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11655-022-3541-8