Skip to main content

Advertisement

Log in

Sign language : a systematic review on classification and recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Sign language serves as an alternative communication mode for those with limitations in hearing and speaking abilities. In India, an estimated population of around 2.7 million people have hearing or speech impairments. Among this population, a significant majority of 98% use sign language as their primary mode of communication. Unfortunately, the limited availability of human interpreters is a considerable obstacle in recognizing and identifying diverse sign languages. To tackle this issue, the present study aims to undertake a thorough examination of various computational methodologies used in various geographical areas for the purpose of categorizing and identifying discrete sign languages. Among the pool of 587 research papers deemed appropriate for qualitative assessment, a total of 95 studies were specifically focused on the categorization and identification of sign language using artificial intelligence-based techniques. The study examines prior research involving deep learning and machine learning methods for a systematic literature review of Sign Language Recognition (SLR). The categorization is made based on language, and the study investigates several facets, including sign type, signing modes, processing techniques, classification methodologies, and evaluation measures. The study reveals that extensive studies were carried out in Chinese, Arabic, and American sign languages. The findings of this review show that among the machine learning techniques, Support Vector Machine (SVM) exhibited higher performance measures, while Convolutional Neural Networks (CNN) exhibited higher performance among the deep learning techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Data Availability

All data generated or analysed during this study are included in this published article

References

  1. Muradova L, Walker H, Colli F (2020) Climate change communication and public engagement in interpersonal deliberative settings: Evidence from the irish citizens’ assembly. Climate Policy 20(10):1322–1335

    Article  Google Scholar 

  2. Organization WH (2021) World Report on Hearing. https://www.who.int/publications/i/item/world-report-on-hearing

  3. Adeyanju I, Bello O, Adegboye M (2021) Machine learning methods for sign language recognition: A critical review and analysis. Intell Syst Appl 12:200056

    Google Scholar 

  4. Konda PLS, Kondapi A, Jesudoss A (2021) Innovative gesture-based automation system for home appliances. In: Cognitive informatics and soft computing: proceeding of CISC 2020, pp 883–892. Springer

  5. Ravi Kumar R, Mohmmad S, Kothandaraman D, Ramesh D (2022) Static hand gesture recognition for asl using matlab platform. In: Computer communication, networking and IoT: proceedings of 5th ICICC 2021, vol 2, pp 379–392. Springer

  6. El-Alfy E-SM, Luqman H (2022) A comprehensive survey and taxonomy of sign language research. Eng Appl Artif Intell 114:105198

    Article  Google Scholar 

  7. Virrey RA, Liyanage CDS, MbPH Petra, Abas PE (2019) Visual data of facial expressions for automatic pain detection. J Vis Commun Image Represent 61:209–217

    Article  Google Scholar 

  8. Farooq U, Rahim MSM, Sabir N, Hussain A, Abid A (2021) Advances in machine translation for sign language: approaches, limitations, and challenges. Neural Comput Appl 33(21):14357–14399

    Article  Google Scholar 

  9. Subasi A, Khateeb K, Brahimi T, Sarirete A (2020) Human activity recognition using machine learning methods in a smart healthcare environment. In: Innovation in health informatics, pp123–144. Elsevier

  10. Yang J, Wilson JP, Gupta S (2019) Diver gesture recognition using deep learning for underwater human-robot interaction. In: OCEANS 2019 MTS/IEEE SEATTLE, pp 1–5. IEEE

  11. Aloysius N, Geetha M (2020) Understanding vision-based continuous sign language recognition. Multimed Tools Appl 79(31–32):22177–22209

    Article  Google Scholar 

  12. Renjith S, Manazhy R (2023) A comparative analysis of islrs using cnn and vit. In: International conference on iot based control networks and intelligent systems, pp 1–9. Springer

  13. Charan MGKS, Poorna S, Anuraj K, Praneeth CS, Sumanth PS, Gupta CVSP, Srikar K (2022) Sign language recognition using cnn and cgan. In: Inventive systems and control: proceedings of ICISC 2022, pp 489–502. Springer

  14. Renjith S, Manazhy R (2023) Indian sign language recognition: A comparative analysis using cnn and rnn models. In: 2023 International Conference on circuit power and computing technologies (ICCPCT), pp 1573–1576. https://doi.org/10.1109/ICCPCT58313.2023.10245525

  15. Shah SMS, Naqvi HA, Khan JI, Ramzan M, Khan HU et al (2018) Shape based pakistan sign language categorization using statistical features and support vector machines. IEEE Access 6:59242–59252

    Article  Google Scholar 

  16. Raees M, Ullah S, Rahman SU, Rabbi I (2016) Image based recognition of pakistan sign language. J Eng Res 4:1–21

    Article  Google Scholar 

  17. Dewani A, Bhatti S, Memon MA, Arif WA, Arain Q, Zehra SB (2018) Sign language e-learning system for hearing-impaired community of pakistan. Int J Inf Technol 10:225–232

    Google Scholar 

  18. Khan NS, Abid A, Abid K (2020) A novel natural language processing (nlp)-based machine translation model for english to pakistan sign language translation. Cognitive Comput 12:748–765

    Article  Google Scholar 

  19. Imran A, Razzaq A, Baig IA, Hussain A, Shahid S, Rehman T-u (2021) Dataset of pakistan sign language and automatic recognition of hand configuration of urdu alphabet through machine learning. Data in Brief 36:107021

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Shah F, Shah MS, Akram W, Manzoor A, Mahmoud RO, Abdelminaam DS (2021) Sign language recognition using multiple kernel learning: A case study of pakistan sign language. Ieee Access 9:67548–67558

    Article  Google Scholar 

  21. Li J, Yin B, Wang L, Kong D (2014) Chinese sign language animation generation considering context. Multimed Tools Appl 71:469–483

    Article  Google Scholar 

  22. Yao D, Jiang M, Huang Y, Abulizi A, Li H (2017) Study of sign segmentation in the text of chinese sign language. Universal Access in the Information Society 16:725–737

    Article  Google Scholar 

  23. Yang X, Chen X, Cao X, Wei S, Zhang X (2016) Chinese sign language recognition based on an optimized tree-structure framework. IEEE J Biomed Health Inform 21(4):994–1004

    Article  PubMed  Google Scholar 

  24. Yu Y, Chen X, Cao S, Zhang X, Chen X (2019) Exploration of chinese sign language recognition using wearable sensors based on deep belief net. IEEE J Biomed Health Inform 24(5):1310–1320

    Article  PubMed  Google Scholar 

  25. Huang S, Mao C, Tao J, Ye Z (2018) A novel chinese sign language recognition method based on keyframe-centered clips. IEEE Signal Process Lett 25(3):442–446

    Article  ADS  Google Scholar 

  26. Jiang X, Lu M, Wang S-H (2020) An eight-layer convolutional neural network with stochastic pooling, batch normalization and dropout for fingerspelling recognition of chinese sign language. Multimed Tools Appl 79:15697–15715

    Article  Google Scholar 

  27. Xiao Q, Qin M, Yin Y (2020) Skeleton-based chinese sign language recognition and generation for bidirectional communication between deaf and hearing people. Neural Netw 125:41–55

    Article  PubMed  Google Scholar 

  28. Wang F, Li C, Zeng Z, Xu K, Cheng S, Liu Y, Sun S (2021) Cornerstone network with feature extractor: a metric-based few-shot model for chinese natural sign language. Appl Intell 51:7139–7150

    Article  Google Scholar 

  29. Gao L, Li H, Liu Z, Liu Z, Wan L, Feng W (2021) Rnn-transducer based chinese sign language recognition. Neurocomputing 434:45–54

    Article  Google Scholar 

  30. Huang S, Ye Z (2021) Boundary-adaptive encoder with attention method for chinese sign language recognition. IEEE Access 9:70948–70960

    Article  Google Scholar 

  31. Tolba MF, Samir A, Aboul-Ela M (2013) Arabic sign language continuous sentences recognition using pcnn and graph matching. Neural Comput Appl 23:999–1010

    Article  Google Scholar 

  32. Mohandes M, Deriche M, Johar U, Ilyas S (2012) A signer-independent arabic sign language recognition system using face detection, geometric features, and a hidden markov model. Comput Electr Eng 38(2):422–433

    Article  Google Scholar 

  33. Mohandes M, Deriche M, Liu J (2014) Image-based and sensor-based approaches to arabic sign language recognition. IEEE Trans Hum-Mach Syst 44(4):551–557

    Article  Google Scholar 

  34. Samir Elons A, Abull-ela M, Tolba MF (2013) Neutralizing lighting non-homogeneity and background size in pcnn image signature for arabic sign language recognition. Neural Comput Appl 22(Suppl 1):47–53

    Article  Google Scholar 

  35. Ibrahim NB, Selim MM, Zayed HH (2018) An automatic arabic sign language recognition system (arslrs). Journal of King Saud University-Computer and Information Sciences 30(4):470–477

    Article  Google Scholar 

  36. Hassan M, Assaleh K, Shanableh T (2019) Multiple proposals for continuous arabic sign language recognition. Sensing and Imaging 20:1–23

    Article  ADS  Google Scholar 

  37. Deriche M, Aliyu SO, Mohandes M (2019) An intelligent arabic sign language recognition system using a pair of lmcs with gmm based classification. IEEE Sens J 19(18):8067–8078

    Article  ADS  Google Scholar 

  38. Hisham B, Hamouda A (2021) Arabic sign language recognition using ada-boosting based on a leap motion controller. Int J Inf Technol 13:1221–1234

    Google Scholar 

  39. Elatawy SM, Hawa DM, Ewees AA, Saad AM (2020) Recognition system for alphabet arabic sign language using neutrosophic and fuzzy c-means. Educ Inf Technol 25:5601–5616

    Article  Google Scholar 

  40. Bencherif MA, Algabri M, Mekhtiche MA, Faisal M, Alsulaiman M, Mathkour H, Al-Hammadi M, Ghaleb H (2021) Arabic sign language recognition system using 2d hands and body skeleton data. IEEE Access 9:59612–59627

    Article  Google Scholar 

  41. Ansari ZA, Harit G (2016) Nearest neighbour classification of indian sign language gestures using kinect camera. Sadhana 41:161–182

    Article  MathSciNet  Google Scholar 

  42. Raheja J, Mishra A, Chaudhary A (2016) Indian sign language recognition using svm. Pattern Recognit Image Anal 26:434–441

    Article  Google Scholar 

  43. Kishore P, Kumar DA, Sastry ACS, Kumar EK (2018) Motionlets matching with adaptive kernels for 3-d indian sign language recognition. IEEE Sens J 18(8):3327–3337

    Article  ADS  Google Scholar 

  44. Joy J, Balakrishnan K, Sreeraj M (2019) Signquiz: a quiz based tool for learning fingerspelled signs in indian sign language using aslr. IEEE Access 7:28363–28371

    Article  Google Scholar 

  45. Kumar EK, Kishore P, Kumar DA, Kumar MTK (2021) Early estimation model for 3d-discrete indian sign language recognition using graph matching. Journal of King Saud University-Computer and Information Sciences 33(7):852–864

    Article  Google Scholar 

  46. Gupta R, Kumar A (2020) Indian sign language recognition using wearable sensors and multi-label classification. Comput Electr Eng 90:106898

    Article  Google Scholar 

  47. Raghuveera T, Deepthi R, Mangalashri R, Akshaya R (2020) A depth-based indian sign language recognition using microsoft kinect. Sādhanā 45:1–13

    Article  Google Scholar 

  48. Joy J, Balakrishnan K, Madhavankutty S (2021) A novel web based dictionary framework for indian sign language. SN Comput Sci 2:1–7

    Article  Google Scholar 

  49. Oz C, Leu MC (2011) American sign language word recognition with a sensory glove using artificial neural networks. Eng Appl Artif Intell 24(7):1204–1213

    Article  Google Scholar 

  50. Wu J, Sun L, Jafari R (2016) A wearable system for recognizing american sign language in real-time using imu and surface emg sensors. IEEE J Biomed Health Inform 20(5):1281–1290

    Article  PubMed  Google Scholar 

  51. Aly W, Aly S, Almotairi S (2019) User-independent american sign language alphabet recognition based on depth image and pcanet features. IEEE Access 7:123138–123150

    Article  Google Scholar 

  52. Tao W, Leu MC, Yin Z (2018) American sign language alphabet recognition using convolutional neural networks with multiview augmentation and inference fusion. Eng Appl Artif iIntell 76:202–213

    Article  Google Scholar 

  53. Beena M, Namboodiri A, Thottungal R (2020) Hybrid approaches of convolutional network and support vector machine for american sign language prediction. Multimed Tools Appl 79:4027–4040

    Article  Google Scholar 

  54. Lee CK, Ng KK, Chen C-H, Lau HC, Chung S, Tsoi T (2021) American sign language recognition and training method with recurrent neural network. Expert Syst Appl 167:114403

    Article  Google Scholar 

  55. Rahman MM, Islam MS, Rahman MH, Sassi R, Rivolta MW, Aktaruzzaman M (2019) A new benchmark on american sign language recognition using convolutional neural network. In: 2019 International conference on sustainable technologies for industry 4.0 (STI), pp 1–6. IEEE

  56. Mahdikhanlou K, Ebrahimnezhad H (2020) Multimodal 3d american sign language recognition for static alphabet and numbers using hand joints and shape coding. Multimed Tools Appl 79(31–32):22235–22259

    Article  Google Scholar 

  57. Jain V, Jain A, Chauhan A, Kotla SS, Gautam A (2021) American sign language recognition using support vector machine and convolutional neural network. Int J Inf Technol 13:1193–1200

    Google Scholar 

  58. Sharma S, Kumar K (2021) Asl-3dcnn: American sign language recognition technique using 3-d convolutional neural networks. Multimed Tools Appl 80(17):26319–26331

    Article  Google Scholar 

  59. Islam S, Mousumi SSS, Rabby ASA, Hossain SA, Abujar S (2018) A potent model to recognize bangla sign language digits using convolutional neural network. Procedia Comput Sci 143:611–618

    Article  Google Scholar 

  60. Rahaman MA, Jasim M, Ali MH, Hasanuzzaman M (2020) Bangla language modeling algorithm for automatic recognition of hand-sign-spelled bangla sign language. Front Comput Sci 14:1–20

    Article  Google Scholar 

  61. Alam MS, Tanvir M, Saha DK, Das SK (2021) Two dimensional convolutional neural network approach for real-time bangla sign language characters recognition and translation. SN Comput Sci 2:1–13

    Article  Google Scholar 

  62. Nihal RA, Rahman S, Broti NM, Deowan SA (2021) Bangla sign alphabet recognition with zero-shot and transfer learning. Pattern Recognit Lett 150:84–93

    Article  ADS  Google Scholar 

  63. Almeida SGM, Guimarães FG, Ramírez JA (2014) Feature extraction in brazilian sign language recognition based on phonological structure and using rgb-d sensors. Expert Syst Appl 41(16):7259–7271

    Article  Google Scholar 

  64. Cerna LR, Cardenas EE, Miranda DG, Menotti D, Camara-Chavez G (2021) A multimodal libras-ufop brazilian sign language dataset of minimal pairs using a microsoft kinect sensor. Expert Syst Appl 167:114179

    Article  Google Scholar 

  65. Rezende TM, Almeida SGM, Guimarães FG (2021) Development and validation of a brazilian sign language database for human gesture recognition. Neural Comput Appl 33(16):10449–10467

    Article  Google Scholar 

  66. Eryiğit C, Köse H, Kelepir M, Eryiğit G (2016) Building machine-readable knowledge representations for turkish sign language generation. Knowl Based Syst 108:179–194

    Article  Google Scholar 

  67. Sincan OM, Keles HY (2020) Autsl: A large scale multi-modal turkish sign language dataset and baseline methods. IEEE Access 8:181340–181355

    Article  Google Scholar 

  68. Katılmış Z, Karakuzu C (2021) Elm based two-handed dynamic turkish sign language (tsl) word recognition. Expert Syst Appl 182:115213

    Article  Google Scholar 

  69. Karami A, Zanj B, Sarkaleh AK (2011) Persian sign language (psl) recognition using wavelet transform and neural networks. Expert Syst Appl 38(3):2661–2667

    Article  Google Scholar 

  70. Zadghorban M, Nahvi M (2018) An algorithm on sign words extraction and recognition of continuous persian sign language based on motion and shape features of hands. Pattern Anal Appl 21:323–335

    Article  MathSciNet  Google Scholar 

  71. Khomami SA, Shamekhi S (2021) Persian sign language recognition using imu and surface emg sensors. Measurement 168:108471

    Article  Google Scholar 

  72. Tze FWH, Kin KTT et al (2011) Feature extraction from 2d gesture trajectory in malaysian sign language recognition. In: 2011 4th International Conference on Mechatronics (ICOM), pp 1–6. IEEE

  73. Shukor AZ, Miskon MF, Jamaluddin MH, Ali F, Asyraf MF, Bahar MB et al (2015) A new data glove approach for malaysian sign language detection. Procedia Comput Sci 76:60–67

    Article  Google Scholar 

  74. Abdullah A, Abdul-Kadir NA, Harun FKC (2020) An optimization of imu sensors-based approach for malaysian sign language recognition. In: 2020 6th International conference on computing engineering and design (ICCED), pp 1–4. IEEE

  75. Alrubayi AH, Ahmed MA, Zaidan A, Albahri AS, Zaidan B, Albahri OS, Alamoodi AH, Alazab M (2021) A pattern recognition model for static gestures in malaysian sign language based on machine learning techniques. Comput ElectR Eng 95:107383

    Article  Google Scholar 

  76. Sosa-Jiménez CO, Ríos-Figueroa HV, Rechy-Ramírez EJ, Marin-Hernandez A, González-Cosío ALS (2017) Real-time mexican sign language recognition. In: 2017 IEEE international autumn meeting on power, electronics and computing (ROPEC), pp 1–6. IEEE

  77. Espejel-Cabrera J, Cervantes J, García-Lamont F, Castilla JSR, Jalili LD (2021) Mexican sign language segmentation using color based neuronal networks to detect the individual skin color. Expert Syst Appl 183:115295

    Article  Google Scholar 

  78. Mejía-Peréz K, Córdova-Esparza D-M, Terven J, Herrera-Navarro A-M, García-Ramírez T, Ramírez-Pedraza A (2022) Automatic recognition of mexican sign language using a depth camera and recurrent neural networks. Appl Sci 12(11):5523

    Article  Google Scholar 

  79. D RAJ R, JASUJA A (2018) British sign language recognition using hog. In: 2018 IEEE international students’ conference on electrical, electronics and computer science (SCEECS), pp 1–4. IEEE

  80. Young A, Napier J, Oram R (2019) The translated deaf self, ontological (in) security and deaf culture. The Translator 25(4):349–368

    Article  Google Scholar 

  81. Belk RA, Pilling M, Rogers KD, Lovell K, Young A (2016) The theoretical and practical determination of clinical cut-offs for the british sign language versions of phq-9 and gad-7. BMC Psychiatry 16:1–12

    Article  Google Scholar 

  82. Admasu YF, Raimond K (2010) Ethiopian sign language recognition using artificial neural network. In: 2010 10th International conference on intelligent systems design and applications, pp 995–1000. IEEE

  83. Kouremenos D, Ntalianis K, Kollias S (2018) A novel rule based machine translation scheme from greek to greek sign language: Production of different types of large corpora and language models evaluation. Comput Speech Lang 51:110–135

    Article  Google Scholar 

  84. Nakjai P, Katanyukul T (2019) Hand sign recognition for thai finger spelling: An application of convolution neural network. J Signal Process Syst 91:131–146

    Article  Google Scholar 

  85. Vega AR, Vasquez A, Amador W, Rojas A (2018) Deep learning for the recognition of facial expression in the colombian sign language. Ann Phys Rehabil Med 61:96

    Article  Google Scholar 

  86. Fernando P, Wimalaratne P (2016) Sign language translation approach to sinhalese language. GSTF J Comput (JoC) 5:1–9

    Article  CAS  Google Scholar 

  87. Naz N, Sajid H, Ali S, Hasan O, Ehsan MK (2023) Signgraph: An efficient and accurate pose-based graph convolution approach toward sign language recognition. IEEE Access 11:19135–19147

    Article  Google Scholar 

  88. Al-Barham M, Alsharkawi A, Al-Yaman M, Al-Fetyani M, Elnagar A, SaAleek AA, Al-Odat M (2023) RGB Arabic Alphabets Sign Language Dataset. arXiv https://doi.org/10.48550/ARXIV.2301.11932

  89. Li D, Rodriguez C, Yu X, Li H (2020) Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 1459–1469

  90. Zhang J, Zhou W, Xie C, Pu J, Li H (2016) Chinese sign language recognition with adaptive hmm. In: 2016 IEEE international conference on multimedia and expo (ICME), pp 1–6. IEEE

  91. Sridhar A, Ganesan RG, Kumar P, Khapra M (2020) Include: A large scale dataset for indian sign language recognition. In: Proceedings of the 28th ACM international conference on multimedia, pp 1366–1375

  92. Podder KK, Chowdhury M, Mahbub ZB, Kadir M (2020) Bangla sign language alphabet recognition using transfer learning based convolutional neural network. Bangladesh J Sci Res, pp 31–33

  93. Azar SG, Seyedarabi H (2020) Trajectory-based recognition of dynamic persian sign language using hidden markov model. Comput Speech Lang 61:101053

    Article  Google Scholar 

  94. Johari RT, Ramli R, Zulkoffli Z, Saibani N (2023) Mywsl: Malaysian words sign language dataset. Data in Brief, pp 109338

  95. Martínez-Sánchez V, Villalón-Turrubiates I, Cervantes-Álvarez F, Hernández-Mejía C (2023) Exploring a novel mexican sign language lexicon video dataset. Multimodal Technol Interact 7(8):83

    Article  Google Scholar 

  96. Albanie S, Varol G, Momeni L, Bull H, Afouras T, Chowdhury H, Fox N, Woll B, Cooper R, McParland A et al (2021) Bbc-oxford british sign language dataset. arXiv:2111.03635

  97. Bahia NK, Rani R (2023) Multi-level taxonomy review for sign language recognition: Emphasis on indian sign language. ACM Trans Asian Low-Resour Lang Inf Process 22(1):1–39

    Article  Google Scholar 

  98. Amorim CC, Zanchettin C (2021) Asl-skeleton3d and asl-phono: Two novel datasets for the american sign language

Download references

Acknowledgements

Dr Sumi Suresh M S carefully reviewed the paper, correcting both grammatical and technical errors to improve its overall quality.

Funding

The authors declare that no funding is provided for the preparation of the manuscript.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to S Renjith.

Ethics declarations

Competing interest

The authors declare that they have no conflict of interest.

Ethical and informed consent for data used

By submitting this manuscript, the authors affirm that appropriate informed consent was obtained for the use of the data in this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Renjith, S., Manazhy, R. Sign language : a systematic review on classification and recognition. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-18583-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-18583-4

Keywords

Navigation