Artificial Intelligence–Enabled Gesture‐Language‐Recognition Feedback System Using Strain‐Sensor‐Arrays‐Based Smart Glove

Wearable smart glove of gesture language provides a novel strategy for the hearing‐impaired people to commutate with the world. Current commercialized solutions of gesture language are limited by the full extent of human interaction beyond operation dexterity, sensory feedback, and the huge cost of fabrication. Herein, a low‐cost, high‐efficient gesture‐language‐recognition feedback system combined with the strain‐sensor arrays and machine‐learning technology is proposed. The strain‐sensor arrays integrated with 3D‐printed glove can extract both spatial and temporal information about the finger's movement. The smart glove achieves gesture‐language recognition using machine learning with an accuracy of over 99%. Integrating with multidimensional manipulation, visual feedback and artificial intelligence (AI)‐based gesture‐language recognition, the smart system can accurately recognize complex gestures and provide real‐time feedback to users. The smart glove system can not only provide an efficient way for hearing‐impaired persons to communicate with the outside world, but also benefit industries in multiple fields such as entertainment, home healthcare, sports training, and the medical industry.


Introduction
Gesture language has been as a visual language for hearing or speech-impaired people to commutate with outside world executed through the movement of the hands and facial expressions, in accordance with its grammatical rules. It is noted that gesture language is not as prevalent as the speaking language and difficult for non-signers to understand without prior learning, causing many inconveniences in their daily lives and studies with the hearing or speech-impaired people. Gesture-language interpretation is of substantial significance in bridging the gap between the hearing or speech impaired and the general public. Thus, constructing an efficient and accurate gesture-recognition system is an urgent need to make the freely communicate with the hearing/speak-impaired people and the normal people.
Generally, visual images/videos, inertial sensors, and surface electromyography (EMG) electrodes are conventional strategies to reconstruct hand gesture information toward sign-language recognition. [1] Vision-based gesture-recognition system mainly involves image preprocessing, detection and segmentation, feature extraction, and classification recognition. [2][3][4] This type of gesture recognition requires strong theoretical base knowledge to design and extract distinguishing features based on empirical knowledge. Moreover, it always takes a lot of time to perform manual feature extraction. In addition, the expensive equipment and obviously dependent on light also limit its wide application. Compared to vision-based gesture recognition, sensor-based gesture-recognition system shows higher recognition accuracy due to independent of background and lighting conditions. [5] Various sensors are constructed for gesture-recognition methods such as myoelectric sensors, accelerometers, gyroscopes, and EMG electrode, etc. [6,7] However, the reported sensors can't accurately detect weak signals from the small motion stroke of certain sign language, resulting in lower accuracy. In addition, poor user experience and high cost also restrict its mainstream application.
With the aid of recent advances in flexible electronics, flexible sensor-based human-machine interfaces have recently offered new ways of sensing human hand movements and gesture recognition. [8][9][10][11] Compared to traditional gesture-recognition methods, the interface based on the flexible sensor has unique advantages such as comfortable experience, facile processing and easy signal acquisition. [12][13][14][15][16][17] There are great developments in gesture-sensing interfaces based on different mechanisms and materials. [18][19][20][21][22] Su et al. successfully implemented real-time monitoring of human gestures using piezoresistive pressure sensors and cooperating with a decision tree algorithm to explicitly classify different Arabic digital gestures. [23] Liu et al. achieved multidirectional curvature through novel designs including a rose strain gauge for monitoring complex surfaces and a nonadhesive filmlike curvature sensor to monitor finger joint flexion activity. [24] Pu et al. reported a novel human-machine intuitive motion sensing and controlled protocol by developing a fingerworn TENG sensor to quantitatively detect finger joint flexion and extension, velocity, and direction. [25] Gu et al. utilized a highly stretchable transparent wearable soft ionic e-skin (iSkin) system integrating with 10 ionic sensors covering the metacarpophalangeal and proximal interphalangeal joints of each finger to detect joint proprioception and gesture monitoring. [26] It was noted that most of smart gloves mainly focused on the construction of the high-performance sensors to accurately sense the motion and gesture. Thus, the smart glovesintegrated system with sensing and recognition of gesture language became extremely important since gesture in-time recognition was a great need in the fact application in our society.
Artificial intelligence (AI) techniques will highly amplify the intelligence of wearable electronics, and provide more reliable yet simpler solutions to more problems and resonating tasks. As a subfield of machine-learning technology, deep learning has shown its great potential in image processing, speech recognition, human activity recognition, and so on. Hence, combining multiple high-performance sensors with AI technology has also become a trend in the development of smart gloves devices. Zhu et al. proposed a haptic feedback smart glove combining multifunction sensors with machine-learning techniques and achieved accuracy of 96%. [27] Sun et al. achieved high-resolution continuous finger-motion tracking through triboelectricities sensors and gesture/object recognition through AI analysis. [28] Zhu' team also proposed a modular soft glove with multimodal sensing and feedback functions to realize real-time detection, intelligent target recognition, and enhanced feedback. [29] Wen et al. reported an AI-supported sign-language recognition and communication system, which realized the recognition of 50 words and 20 sentences with an average accuracy of 86.67%. [30] Despite great achievement of sensing and recognition of gesture language, the reported smart gloves were still limited by the low recognition rate, poor biocompatible property, especially lack of the real-time feedback function. [31][32][33] Herein, we developed a smart glove system with sensingrecognition-feedback capabilities of gesture language, providing an effective new strategy for the hearing-impaired persons to commutate with the world. The silk-fibroin (SF)-based strain sensor showed excellent performance of high sensitivity, fast response time, good stability, and facile processing. The strain-sensor array with 10 sensor nodes with 3D glove can extract both spatial and temporal information about the finger movement. Combined with the deep learning technology, the smart glove achieves gesture-language recognition with an accuracy of over 99%. Integrating with multidimensional manipulation, visual feedback, and AI-based gesture-language recognition, the smart system can accurately recognize complex gestures and provide real-time feedback to the users.

Scheme of the Smart Gesture-Recognition System
The smart glove system based on SF-hydrogel sensors array for hand-motion capture analysis and intelligent algorithms to design a gesture recognition is shown in Figure 1. The system mainly consisted of 3D-printed glove, SF-hydrogel strain-sensor arrays, data-acquisition card, signal-processing module, and gesture-recognition interface (Figure 1b). The 3D-printed glove shell integrated with 10 strain-sensor arrays which was used to sense the movement information of each finger. The dataacquisition card was connected to the SF-hydrogel sensors to measure the electrical signals from the finger motion. Then, the signal processing would transform the electrical signals to data signals to combine with the recognition processing process. And then, a gesture-language-recognition process was formed based on the machine-learning technology and the data signals. Finally, the gesture information was timely shown in the gesture-recognition display interface ( Figure 1b).

Preparation and Characterization of SF-Hydrogel-Based Strain Sensor
The strain sensors utilized SF-hydrogel as a sensing layer due to its unique advantages of flexibility, good biocompatible, facile fabrication process, and excellent conductive property. SF-hydrogel-based sensors showed high performance and wide applications. [34,35] In this work, the SF-hydrogel preparation process is shown in Figure 2a(i-iii). In briefly, the natural silk cocoons were first sheared, degummed, and dissolved. And the obtained solution was dialyzed to obtain a pure SF solution.
To enhance the mechanical properties of the silk proteins, the acrylamide (AM), N-methylenebisacrylamide (MBA), and ammonium persulfate were introduced into the silk solution to crosslink multiple silk proteins to form a network structure. [36][37][38][39][40][41][42][43] Calcium nitrate was also added to improve the conductivity of SF-hydrogel. The sandwich sensor structure was used to construct strain sensor including the SF-hydrogel sensing layer, Copper electrodes and very high bond (VHB) tape encapsulation layer. The schematic structure is shown in Figure 2a(iv). The SFhydrogel could be bent as well as twisted and be stretched to 600% of its original length ( Figure S1, Supporting Information), demonstrating its good flexibility and excellent mechanical properties.
The composition and structure of SF were also investigated by Fourier-transform infrared spectroscopy (FTIR) and scanning electron microscope (SEM). The SEM images and FTIR spectra of the SF-hydrogel are shown in Figure S2, Supporting Information. The obtained hydrogels had uniform structure, which didn't have too many pores similar to the he reported hydrogels. [44,45] Thus, this hydrogel showed high stretching ability since the pores could cause the mechanical properties of the hydrogels to deteriorate. The FTIR of the hydrogels showed three characteristic structures peaks at 3278, 1620, and 1523 cm À1 , which belonged to the stretching vibration of the OH, the C═C, and the C═O double bonds, respectively. [46] The hydrogel is mainly composed of C, N, O, S, and Ca elements as shown in Figures S3 and S4, Supporting Information.
Afterward, the properties of the different sizes of SF-hydrogels were investigated ( Figure S5, Supporting Information). The stretching property obviously depended on the fabrication condition and size of SF-hydrogels. The hydrogel with the size of 20 Â 5 Â 2 mm showed better properties than that of the other films. In the following section, the SF-hydrogel with 20 Â 5 Â 2 mm was used as sensing layer. In addition, the SF-hydrogel showed excellent self-healing and adhesion properties ( Figure S6, Supporting Information), which was one of the important factors to use in the smart glove.
To demonstrate its potential application, the performance of SF-hydrogel strain sensor was characterized in detail, including sensitivity, hysteresis, repeatability, stability, and detection range. The gauge factor (GF) was used to define the sensitivity where ΔR is the change in resistance relative to the nominal resistance R 0 at zero strain, and ε is the strain of the sensor. The GF increased with increasing strain from 1% to 600% (Figure 2b(i)). The maximum value of GF of SF-hydrogel sensor could reach 7.1 at 600% strain, which was higher than most of the previously reported hydrogel-based strain sensors. [47][48][49] The comparing performances of the proposed SF-based strain sensor and other state-of-the-art resistive strain sensors are added in Table S1, Supporting Information. The obtained SF-based sensor only shows higher sensitivity, but also has very short response time and good durability in the high tensile range. The SFhydrogel sensor exhibited good repeatability at both low strain and high strain (Figure 2b(ii). Moreover, the sensor also showed a highly repeatable response over the frequency range of 0.0625-1 Hz (Figure 2b(iii)), indicating that SF-hydrogel sensor had a fast response to tensile strain. In addition, the response curves during stretch and release largely overlapped in the 0%-100% strain range, as well as the stretch and release response curves in the low 0%-1% and 0%-10% strain ranges ( Figure S7, Supporting Information). To verify the stability of the sensor, the hystereses in the 0%-100% strain of the sensors that are freshly prepared, after 12 h, were measured ( Figure S8, Supporting Information). The hysteresis curves of sensors after 12 h of loading the strain and unloading strain almost overlapped. The previous results confirmed that the SF-hydrogel sensor was almost unaffected by the external environment, further indicating its good stability. These results all indicated that the SF-hydrogel sensor achieved a low hysteresis, which might be attributed to the hybrid hydrogel-polymer structure with a more uniform resistance during stretch distribution. Moreover, the SF-based sensor also showed fast response. The strain response and releasing time was about 80 and 90 ms at 50% strain (Figure 2b(v), respectively. The strain-sensing time was defined as the time required the relative resistance change Figure 1. Schematics of the artificial intelligence (AI)-based smart glove. a) AI-based wearable human-machine interface (HMI) system, i) current mainstream wearable HMI system, ii) AI-based smart glove, iii) silk-fibroin (SF)-hydrogel strain-sensor structure for smart glove, and iv) HMI application for hearing-impaired patient communication. b) Schematic diagram of an AI-based smart glove system.
www.advancedsciencenews.com www.advintellsyst.com to increase from 10% to 90% of the peak value, while the releasing time vice versa. Obviously, the response time could be shorter or compared with most previously reported hydrogel sensors. [50,51] The sensor also exhibited good overall stability over 2000 repetitions of testing with no significant performance degradation (Figure 2b(vi)). The ΔR/R 0 increased with increasing stretch ratio (0%-100% step stretch), while the signal could maintain when the strain kept a fixed value, indicating good stability and accuracy of the sensor ( Figure S9, Supporting Information). In addition, the SF-hydrogel strain sensor could detect human motion ( Figure S10, Supporting Information) and different sounds ( Figure S11, Supporting Information), further confirming its high sensitivity and excellent flexibility. These results proved that the sensor obtained a wide detection range, high sensitivity, and good stability, showing potential applications in the detection of human motion.

Characterization of Smart Glove Based on Strain Sensors
The smart glove mainly included 3D-printed shell and strain-sensor array. The 3D-printed shell part of the smart glove mainly included the finger part with three knuckles and the palm part (Figure 3a). The finger part was connected www.advancedsciencenews.com www.advintellsyst.com to the palm part with an elastic transparent wire to form the complete glove shell (Figure 3a(iii)). Afterward, the strainsensor array with 10 sensor nodes (Figure 3a(iv)) was assembled with the glove shell to form the complete smart glove (Figure 3a(v)). To ensure accurately detect different motions of different fingers, the noninterference of the sensors adhered in different fingers was first investigated. The sensors adhered to different fingers showed corresponding fast response when finger bending. The sensor arrays showed nondependent properties among each sensor (Figure 3b(i)), indicating excellent noninterference properties. In addition, to evaluate the output performance of wearing the gloves at different angles, the performance changes was tested when 10 fingers were bent at 45°as well as 90° (Figure 3b(ii)). There was no significant difference in the ΔR/R 0 values for each sensor. The previous results confirmed the good agreement of the array sensors (Figure 3b(iii)).

Artificial Intelligence Enabled of Gesture Recognition
With the rapid development of AI technology, more devices and systems have been integrated to realize intelligent decisionmaking, status recognition, and automation control. Through training an end-to-end neural network, devices become able to learn more representative features of raw signal data, thus enabling more advanced functionalities. [52][53][54] Herein, to achieve smart glove with intelligent sensing recognition and feedback functions, the deep-learning method was introduced to analyze and extract all features of the 10-strain-sensor signals of gesture. The signal-acquisition module included smart glove ( Figure S12, Supporting Information), a 10-channel signal-conditioning module circuit board, an national instruments (NI) data-acquisition card, and a LabVIEW display interface (Figure 4a). The MCP6002 amplification circuit ( Figure S13, Supporting Information) was used for the secondary amplification of the signal. The optical photo of the circuit board of the 10-channel signal regulation module is shown in Figure S14, Supporting Information. The processed voltage signal was then acquired by the NI-acquisition card and the 10-channel data was displayed by the LabVIEW display interface. The final data was acquired and saved as a data set for later data training and testing. During the gesture commutation, some electrical signals for the strain sensor adhered to gloves could be obtained. Different gestures including of the digits from "0" to "9" (Figure 4c) and eight conventional words and phrases (Figure 4d) were investigated. The distinct signals from different gestures can be generated originating from varying contact points and different bending angles of fingers. The real-time output signals of strain-sensor array for 18 different gestures are shown in Figure 4c,d. Table S2, Supporting Information, shows the relationship between the signal variation of the 10 sensor arrays and the different digital gestures. And Table S3, Supporting Information, shows the relationship between the signal variation of the sensor arrays and the eight everyday communication gestures. It was seen that all of them showed unique signal characteristics.
The schematic of the glove-recognition feedback module is shown in Figure 5a. The system collected the gesture signals in the signal-acquisition module and trained the intelligent algorithm to construct the recognition model to eventually . Smart glove-based gesture-recognition data characterization. a) Gesture-recognition data-acquisition process. b) Sensor signals for each finger. c) Ten-channel signal plot corresponding to "0-9" digital signals. d) Ten-channel signal plot corresponding to 8 commonly used words.    The test physical picture of gesture-recognition system based on smart glove. c) A typical example set of the normalized signal maps of the sensors, rows contain 18 gesture targets, and columns contain 10 sensor channels corresponding to color bars on a scale of 0-1. d) Interface diagram and example signal diagram for 18 gestures with real-time-recognition feedback. a-r) The hand signals "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "you", "good", "I love you", "name", "please", "sit", "love", and "I".
www.advancedsciencenews.com www.advintellsyst.com achieve real-time recognition and feedback. Convolutional neural networks (CNN) with 1D structure (1D-CNN) was usually used for data feature extraction and automatic recognition. The 1D-CNN is a very effective method to derive interesting features from shorter (fixed length) segments of the overall dataset where the positions of the features within the segment are not of high relevance. Thus, this method is an effective way to analyze of time sequences of sensor data. Herein, the 1D-CNN was used to analyze the output signals from the different gestures (Figure 5b). The parameters used to construct the CNN model are labeled in detail as shown in Table S4, Supporting Information. This machine-learning method was proved as an effective method to solve complex classification problems, extracting features from time-domain data of sensor signals, and identifying different samples with high accuracy. [55][56][57][58][59][60][61][62][63][64][65] As a scheme comparison, visual geometry group (VGG) network structure was also used to test a control group a control group. The dataset including 18 gestures is shown in Figure 5c. After undergoing the feature extraction and classification of CNN, the classification performance of the developed 1D-CNN model with clear boundaries among these 18 classes with less overlap is shown in Figure 5d. These results proved the effectiveness of proposed 1D-CNN model for feature classification.
In the training sample collection process, each of the 10 sensors had 1500 data points recorded per sample to train the model for recognition. For each gesture, 250 samples were collected for training (80%) and testing (20%). For the large sample size of the collected data, principal component analysis method was used for feature extraction and data dimensionality reduction. The linear function was used for the kernel functions. Confusion matrix of the model showed that both methods benefited to achieve target-recognition accuracy of over 90% (single run) with 4500 training samples. Their loss function plots and accuracy function plots are shown in Figure S15, Supporting Information. The VGG structure in the CNN neural network approach only achieved 93.222% recognition accuracy (Figure 5e). In contrast, the method based on the 1D-CNN structure achieved a recognition accuracy of 99.333% (Figure 5f ). Thus, in the following section, the 1D-CNN structure was used to realize an in-time recognition and feedback system.

Visual Feedback Interface System for Gesture
Real-time recognition and feedback of gesture are urgently needed for the hearing-impaired persons to communicate with the outside world. The proposed gesture recognition and feedback system mainly included strain-sensors-adhered glove for finger-motion capture, the printed circuit board for signals preprocessing, NI-acquisition card connected PC for data collection, deep-learning-based analytics for signals recognition, and display interface (Figure 6a). The volunteer wore the aforementioned smart glove system to demonstrate the gesture, and the signalacquisition system collected the electrical signals from the sensor sensing gesture. The real-time demonstration interface utilized a neural network algorithm to recognize the final gesture and displayed the feedback signals ( Figure 6b).
As the proposed smart glove system was equipped with 10 strain sensors, the normalized the signals collected by the 10 sensors and converted them into a 10 Â 18 signal map (Figure 6c). The 18 rows of signals from top to bottom correspond to each of the 18 gestures mentioned earlier, and the 10 columns of signals from left to right correspond to each of the 10 sensors. During the process of real-time gesture recognition, different sensors recorded the signals of different gestures and perform 0/1 preprocessing (Figure 6d), judging them as 1 when there was a change in the sensor output and 0 when there was no change. Taking the gesture "0" as an example, the principle of real-time recognition and feedback was explained as follows: when the volunteer indicated the gesture "0," all the sensors had signals, and the real-time gestures we collected were first judged to be 0/1. The judgment results could be seen in the diagram, showing that the gesture made by the volunteer was gesture "0". The principle of the remaining 17 gestures had similar process as that of gesture "0". The Video S1, Supporting Information detailedly describes the recognition and feedback processing for different gestures. It was obvious that the obtained smart glove system could in-time sense, recognize, and feedback on gestures, proving an effective bridge for the hearing-impaired persons.

Conclusions
In summary, we developed a smart glove system with sensing-recognition-feedback capabilities of gesture language. The high-performance strain-sensor arrays were designed based on SF-hydrogels to use as hand-motion capture analysis. The strain-sensor array with 10 sensor nodes with 3D glove could extract both spatial and temporal information about the finger movement. Combined with the machine learning, the smart glove achieved gesture-language recognition with an accuracy of over 99% under 1D-CNN. More importantly, integrating with multidimensional manipulation, visual feedback, and AI-based gesture-language recognition, the smart system could accurately recognize complex gestures and provide real-time feedback to the users. This work showed great prospects for the development of the digital human in future.

Experimental Section
Materials: Cocoons were purchased from local farmers in Nantong, China. Sodium bicarbonate (NaHCO 3 ) and ammonium persulfate ((NH 4 ) 2 S 2 O 8 ) were purchased from Sinopharm Chemical Reagent Co., Ltd., in Shanghai, China. Lithium bromide (LiBr) was purchased from Bittering Fine Chemical Co. and AM was purchased from Aladdin. The dialysis bags were purchased from Rongsheng Biotechnology, China. The double-adhesive tape used in this study was purchased from 3 M. All reagents in this study were used in the original state of purchase without any improvement.
Preparation and Characterization of the SF-Hydrogel: The fabrication process mainly included following steps. First, the cut cocoons were poured into the solution (0.1 mol L À1 NaHCO 3 ) and heated at 100°C for 30 min to remove the SF-hydrogel. The filament was washed using distilled water. After degumming was completed, the degummed filament fibers were dried to obtain degummed filament fibers. Afterward, the 1.5 g of degummed filament fibers was slowly added to 10 mL of 9.5 mol L À1 LiBr solution and dissolved at 60°C for 4 h. The mixed solution was poured into a dialysis bag (12 000 kDa, Viskase) and placed in distilled water at 4°C for 72 h. The supernatant of the bag was taken into a 1.5 mL centrifuge www.advancedsciencenews.com www.advintellsyst.com tube and centrifuged three times at 900r min À1 for 20 min to form silk solution. The 3 g of AM, 0.01 g of ammonium persulfate, and 0.005 g of MBA were added to 7 mL of deionized water and stirred with a magnetic stirrer at 100r min À1 for 20 min to obtain a mixed solution. And then, 2 mL of silk protein solution with 0.6 g of calcium nitrate was added to the previous mixed solution and still stir until the solution is homogeneous. Finally, PTFE mold was added into the homogeneous solution and heated at 80°C for 40 min to obtain silk protein hydrogel. Preparation of the SF-Hydrogel Strain Sensor: The SF-hydrogel strainsensor utilized sandwich structure. The sensing layer size of the hydrogel was 20 Â 5 Â 2 mm. The copper electrodes were connected with sensing layer. And then VHB tape (VHB 4905, 3 M) with 25 Â 10 mm was used to encapsulate the silk hydrogel as well as the electrodes to obtain the SF-hydrogel sensor.
Characterization of the Materials and Sensors: The chemical structure of the hydrogels was analyzed by FTIR (Nicolet 8700, USA) in the wave number range 4000-500 cm À1 . The morphological characteristics of the hydrogels were measured with a field-emission SEM (S-4800, Hitachi, Japan) at 200 kV. Tensile tests were performed using a tensile testing machine (SHIMADZU Corporation, model AGS-X, 100 N, Japan) with a constant speed of 100 mm min À1 . In the uniaxial tensile test, the hydrogel samples were cut into strips of different lengths but with a width of 5 mm and a thickness of 2 mm. The electrical characteristics of the single strain transducer were tested under different conditions at room temperature by means of an LCR-TH2838 with an over PWS-Motor-1SV-0.1R-250 uniaxial translation stage.
Construction of the Smart Glove System: The smart glove shell was fabricated by 3D-printed method using resin material. The 3D model was adjusted to the size of the user's hand. The glove shell was mainly used to fuse the fingers and the flexible sensors. The finger section and palm section were held in place by transparent elastic threads, depending on the row of thread holes. After the glove was attached, the double-sided tape was applied to the back of the second as well as the third knuckle. Since both the double-sided tape and the VHB tape had good adhesion, the SF-hydrogel sensor could be well fixed to the back of the finger and could be easily stretched when the fingers were bent. The assembled glove was connected to the 10-channel signal-conditioning module circuit board, NI-acquisition card and LabVIEW multichannel data-acquisition program and display interface. The 1D-CNN framework was constructed to achieve finger-motion signal-acquisition and real-time feedback for gesture recognition.

Supporting Information
Supporting Information is available from the Wiley Online Library or from the author.