Surface Texture Recognition by Deep Learning‐Enhanced Tactile Sensing

Tactile perception is a primary sensing channel for both humans and robots to be conscious of the surface properties of an object. Due to the unique functionalities of mechanoreceptors in human skin, humans can easily distinguish materials with different surface characteristics (e.g., compressibility, roughness, etc.) by simply pressing and sliding the fingertip over the samples. However, how to achieve such delicate texture recognition for robots remains an open challenge due to the lack of skin‐comparable tactile sensing systems and smart pattern recognition algorithms. Herein, a novel texture recognition method is proposed by designing an arc‐shaped soft tactile sensor and a bidirectional long short‐term memory (LSTM) model with the attention mechanism. By using the proposed method, a respective recognition accuracy of 97% for Braille characters and 99% for 60 types of fabrics have been achieved, revealing the effectiveness of our method in surface texture recognition and the potential benefit to various applications, such as Braille reading for visually impaired people and defect detection in the textile industry.


Introduction
Surface texture, a combination of roughness (nano-and microroughness), waviness (macroroughness), lay, and flaw, provides important features for humans to discriminate different materials, and the automation of surface texture recognition has been of increasing interest in various areas, such as textile manufacturing, [1] defect detection, [2,3] and robotic manipulation. [4,5] Traditionally, surface textures are inspected by devices such as roughometers and microscopes, which are bulky and time-consuming to use. With the recent advances in computer vision, vision-based methods [3,6,7] have gained increasing popularity in texture recognition due to their high spatial resolution and excellent recognition efficiency. However, these methods rely on the indirect visual effects of the surface texture rather than directly measuring the mechanical properties of the surface, thus lacking the key tactile cues for perceiving the material, such as thermal conductivity, compressibility, roughness, and so on. To overcome this limitation, involving tactile sensing in the surface texture recognition would be beneficial.
Recently, tremendous efforts have been made to develop artificial tactile sensors based on different working mechanisms, including piezoelectricity, [8][9][10] piezoresistance, [11][12][13][14][15][16] capacitance, [17,18] strain gauges, [19] optics, [20][21][22] and magnetics, [23][24][25] within which machine learning technologies have played a critical role to extract useful information from raw tactile data for pattern recognition. Mami et al. [10] reported a tactile sensor system that is based on a piezoelectric film for Braille characters recognition, where a 91.45% recognition accuracy was achieved by using the support vector machine (SVM) and multilayer perceptron (MLP). To further improve the recognition accuracy, researchers have recently developed tactile sensor arrays using a wide variety of advanced materials. [11,13,17,18] However, when reading continuous lines of Braille texts, most of these sensors have to discontinuously press each individual Braille character and use each of the six taxels in the sensor array to check whether a dot is raised or not in a six-dot Braille cell. This leads to a high device complexity and low reading efficiency. Therefore, a more compact tactile sensor and a more efficient Braille recognition method are required. In addition to Braille recognition, tactile sensors have also been widely used for fabric recognition by using machine learning algorithms. Florian et al. [19] reported a texture recognition method based on a 3-axis force sensor, in which the MLP was used for classifying textures of ten types of papers and a recognition accuracy of around 60% was achieved. By using a magnetostrictive tactile sensor and the extreme learning machine (ELM), Zheng et al. [23] achieved a recognition accuracy of 93% for five types of fabrics. More recently, a high recognition accuracy of 99% was achieved for 12 types of fabrics by Chun et al., [12] in which the SVM was used for DOI: 10.1002/aisy.202100076 Tactile perception is a primary sensing channel for both humans and robots to be conscious of the surface properties of an object. Due to the unique functionalities of mechanoreceptors in human skin, humans can easily distinguish materials with different surface characteristics (e.g., compressibility, roughness, etc.) by simply pressing and sliding the fingertip over the samples. However, how to achieve such delicate texture recognition for robots remains an open challenge due to the lack of skin-comparable tactile sensing systems and smart pattern recognition algorithms. Herein, a novel texture recognition method is proposed by designing an arc-shaped soft tactile sensor and a bidirectional long short-term memory (LSTM) model with the attention mechanism. By using the proposed method, a respective recognition accuracy of 97% for Braille characters and 99% for 60 types of fabrics have been achieved, revealing the effectiveness of our method in surface texture recognition and the potential benefit to various applications, such as Braille reading for visually impaired people and defect detection in the textile industry.
classification based on the tactile outputs of a piezoresistive film. However, most of these methods require manual feature selection or extraction to reduce the data dimension (see Table 1), which could be tedious in practical applications.
Here, we propose a novel texture recognition method by designing an arc-shaped soft tactile sensor (built upon our previous work [26] ) and an attention-based long short-term memory (LSTM) model, which can efficiently recognize both Braille characters and fabrics with high accuracy. Different from traditional tactile sensors that can only characterize the surface texture with a 1D feature, such as resistance, capacitance, or light intensity, our proposed sensor is magnetic-based and thus can provide 3D feature outputs in terms of the 3-axis magnetic flux densities, which could convey richer contact information in both normal and shear directions during sliding. [27] Moreover, the sequential outputs of the sensor can be directly fed into the proposed LSTM model with no need for handcrafting features due to the special architecture of recurrent neural networks, which is simple and efficient to implement. By using the proposed method, we have achieved a recognition accuracy of 99% for 60 types of fabrics. A real-time recognition accuracy of 97% is also achieved for Braille characters by natural and continuous sliding (rather than discontinuous pressing of existing tactile sensors [11,13,17,18] ), which is comparable with that of proficient human readers. [28] 2. Results and Discussions

Arc-shaped Tactile Sensor Design
The tactile sensor is composed of a flexible magnetic film (mixture of polydimethylsiloxane [PDMS] and neodymium [NdFeB] powders) and a Hall sensor embedded on the printed circuit board, with a silicone elastomer sheet (Ecoflex 00-50) sandwiched in between, as shown in Figure 1A (also see Figure 3B). When an external force is applied to the flexible magnetic film, the change in magnetic flux densities due to the deformation of the flexible film would be sensed by the Hall sensor. Here, the magnetic film is magnetized sinusoidally (in Halbach arrays) with multiple alternate north-south poles, so that the magnetic field is strengthened on the one side (strong side, toward the Hall sensor) and canceled to nearly zero on the other side (weak side, toward the object to be touched). [29] At the same time, we align the centers of the magnetic pole and the Hall sensor to obtain the largest measurement range of the magnetic flux density along the z axis (B z ) as well as a symmetric measurement range of the magnetic flux density along the x axis (B x ) simultaneously; B z and B x will change in proportion to the sensor deformation along the normal direction (i.e., the z axis) and the shear direction (i.e., the x axis), respectively. Such an arrangement is important for the sensor to "feel" the texture of a material via tactile interaction, where more details can be found in Section 2.2 and 2.3.
As shown in Figure 1A, we design our sensor as an arc-shaped tip (with radius %6 mm) similar to the human fingerpad (with radius %8 mm). Compared with our previous design of a planar sensor structure, [26] such a curved "magnetic skin" could deform more easily to comply with a variety of surface textures (e.g., knitted fabrics and embossed dots patterns of Braille characters) under a small contact force due to the reduced contact area, making the sensor sensitive to the difference among similar textures. At the same time, the curved sensor shape also results in a lower friction force between the sensor and the contact surface during sliding, which could reduce the tactile sensor's wear and test cost and thereby extend its lifespan.

Braille Character Recognition
Braille is an efficient and significant tool for people who are visually impaired to communicate with the world. Braille characters are small rectangular blocks including six dots (either raised or flat) in two columns, and there are in total 63 different patterns (except the "all flat" case). Usually, it will take years of professional training to the level of fluently reading a book in Braille texts. [30] To assist Braille learning, two types of methodologies are available to assist visually impaired people, which use visual sensors or tactile sensors to perceive and recognize Braille dot modes. [11,31] Compared with the visual sensor, the tactile sensor is more compact in size, more robust to disturbance (e.g., the change in environmental illumination), regarded as an alternative, and potentially more attractive solution to Braille character recognition. However, when performing the recognition task, existing tactile sensors must discontinuously press each individual character one by one. And for each character, they use one taxel to check whether a dot is raised or flat, and in total they need Piezoresistive sensor array [11] Pressing RDF 99 Strain gauges [19] 1D Sliding 10 types of paper MLP PSD %60 Magnetostrictive tactile sensor [23] 5 types of fabrics ELM PSD 93 Piezoelectric film [9] 5 types of fabrics MLP FFT 98.7 Piezoresistive film [12] 12 six taxels for a character with six dots. In this way, these solutions have a high device complexity but low reading efficiency. By designing a soft magnetic film that has continuous and smooth magnetic field distribution in 3D space, our tactile sensor is capable of a continuous way of recognizing Braille characters via natural sliding, which can significantly improve the recognition efficiency. The experimental setup for Braille recognition is shown in Figure 2, where the tactile sensor is installed at the endpoint of a robot hand. When the sensor slides over the Braille characters with different embossed patterns, the flexible magnet tip deforms accordingly, and the measured magnetic flux densities (B x , B y , and B z ) along the x, y, and z directions also change in a particular manner. Unlike the "reading by pressing" method that only takes into account the normal force, the 3D tactile signal sequence (B x , B y , and B z ) involves both normal and shear contact information, which is fed into a neural network to estimate the characters touched in a real-time manner as how humans do. In particular, the magnitude variations of B x and B y are correlated with the position of the raised dots of each Braille character in the column and row directions, respectively, while B z reflects the overall pattern of the raised dots.
As shown in Figure 1B, we designed a bidirectional LSTM model with the attention mechanism for Braille character recognition. The inputs of the LSTM network are sequential tactile signals in terms of magnetic flux densities when sliding over the Braille text, and the outputs of the network are the probabilities (or scores) that the block under the sensor represents each of the 29 characters in the Braille alphabet, including 26 English letters and three special symbols. The character with the highest score will be chosen as the final estimation result based on a majority voting. An attention layer is also introduced to help the LSTM network learn how to focus on a particular part of the tactile signals that can well distinguish characters with similar readings (such as the letters "l" and "s" as shown in Figure 1C). After training on around 900 instances of Braille characters, the proposed model achieves a real-time recognition accuracy of 97% tested on the Braille poem Dreams with the sensor mounted on the robot ( Figure 2 and Movie S1, Supporting Information), which is comparable with that of proficient human readers. The corresponding confusion matrix is shown in Figure S2, Supporting Information, where only five Braille characters were mislabeled among 163 instances of the test set.
To further test its potential in helping blind people in the future, we also wear the tactile sensor (with "legs" for support) on the fingertip of a sighted person for Braille character recognition and trained another LSTM model on 680 instances of Braille characters with different finger poses and reading speed. The real-time recognition accuracy on the fingertip is 78% (when the correct answer is the highest scored label of the Softmax layer) and 90% (when the correct answer is within the top three highest scored labels of the Softmax layer), with an average reading speed of 15 mm s À1 (see Movie S2, Supporting Information). The recognition accuracy on the fingertip is lower than that on the robot because the sensor reading is sensitive to the variations of finger poses and reading speed, while the current training set is not large enough to cover all such variations. Moreover, the alignment variation may result in similar sensor outputs for B) The architecture of the bidirectional LSTM neural network with attention mechanism, which is used for predicting Braille characters slid over by the tactile sensor. C) Sensor responses (in terms of the change of magnetic flux densities B x , B y , and B z ) when sliding the tactile sensor over the Braille characters "l", "s", "r", "o," and "e" by the robot, where letters "l" and "s" have similar data patterns because of their similar dots modes, and letter "r" has a much different (wider) pattern compared with the former two due to the increased number of dots in the second row. Letters "o" and "e" also have similar tactile signals, and the former might be mistaken for the latter if the single dot in the third row of character "o" is omitted by the sensor due to misalignment.
www.advancedsciencenews.com www.advintellsyst.com different characters, which increases the difficulty in recognition. For example, the letter "o" would be wrongly recognized as letter "e" if the sensor leaves out the third row of the Braille character "o" due to too much upward shift during reading (see Figure 1C and Movie S2, Supporting Information). Nonetheless, it is possible to further improve the accuracy of Braille character recognition on the fingertip. For example, we could increase the size of the training set by including Braille characters perceived with different finger poses and moving speed to completely reflect the reading habits of different users. We could also use the semantic association strategy in the prediction process (e.g., by taking into account natural language priors), by which characters with similar sensor outputs can be correctly recognized according to the context.

Fabric Recognition
In addition to the capability of recognizing Braille characters, the proposed method can also discriminate 60 types of fabrics of different softness, roughness, and friction (see Figure S1, Supporting Information). The experimental setup for fabric recognition is shown in Figure 3A, where the tactile sensor is mounted on the tip of a robotic arm, below which is a fabric sample (corduroy) glued on a fixed board. A stereo view and a sectional view (schematic illustration) of the tactile sensor are shown on the left and right side of Figure 3B, respectively. As shown in Figure 3C, the sensor outputs change continuously as the sensor slides over the corduroy sample. In the beginning (t 0 ), the sensor does not touch anything and thereby outputs the initial magnetic flux densities at no load. Then, the magnetic flux density along the z axis (B z ) increases from 1760 to 4100 uT and keeps constant afterward as the sensor is pressed on the fabric (t 0 -t 1 ) and held still (t 1 -t 2 ) subsequently. When the sensor starts to slide at t 2 , the magnetic flux density along the x axis (B x ) decreases rapidly from À90 to À3300 uT (t 2 -t 3 ) and then rises to À1000 uT (t 3 -t 4 ) because the contact condition between the sensor surface and the fabric sample changed from the static friction stage (t 2 -t 3 ) to the initial slipping stage (t 3 -t 4 ). Similar responses are also observed for B z . Then, the sliding friction (i.e., complete slipping) occurs at t 4 , and the B z and B x readings fluctuate periodically as the sensor slides over the parallel ridges (i.e., raised lines) of the corduroy sample.
To further investigate the relationship between the change in magnetic flux densities and the surface properties of fabric samples, we compare the sensor responses of three other fabrics, as shown in Figure 4. It is found that the outputs of the tactile sensor (in terms of 3-axis magnetic flux densities) are directly related to the fabric's surface properties. In particular, the increasing magnitude (h 1 ) of the magnetic flux density along the z axis (B z ) during pressing is proportional to the hardness of the fabric, while the decreasing magnitude (h 2 ) of the magnetic flux density along the x axis (B x ) during initial sliding is proportional to the friction of the surface. This is intuitive to understand because the stiffer the material is, the larger the deformation of the magnetic film along the z axis (and therewith B z ) would be when the sensor is pressed on the material surface with a fixed depth (1 mm here). Similarly, the more frictional the surface is, the larger the lateral deformation of the magnetic film along the x axis (and therewith www.advancedsciencenews.com www.advintellsyst.com B x ) would be when sliding the sensor on the materials. Moreover, we observe that the oscillation magnitude (h 3 ) of B z is proportional to the roughness of the surface; this is because the rougher the surface is, the larger the vibration of the magnetic film alone the z axis (and therewith B z ) would be when sliding the sensor on the material surface. In summary, the larger the hardness, friction, or roughness of the material is, the larger the h 1 , h 2 or h 3 of the sensor responses would be, respectively. This is consistent with the actual surface conditions of the three fabric samples, i.e., fabric #13 has a surface of moderate hardness but with the lowest roughness and friction; fabric #15 has the lowest hardness but with the highest roughness and friction, and fabric #43 has the highest hardness and a moderate roughness and friction.
Taking advantage of the rich information conveyed in the tactile data in terms of surface hardness, friction, and roughness, the proposed LSTM model successfully discriminates 60 types  www.advancedsciencenews.com www.advintellsyst.com of fabrics with a recognition accuracy of 99% on the test set (20% of the total data), where only one instance of fabric #5 was mislabeled as fabric #13 among 120 instances of the test set. This indicates that the proposed method is efficient in recognizing fabrics with different surface textures.

Discussion
LSTM is a powerful tool to process sequential data (e.g., the tactile data generated in this work) in the field of deep learning. Before evaluating the performance of the proposed LSTM model for Braille recognition, we have also tried dynamic time warping (DTW), which is a popular method for measuring similarity between two temporal sequences that may vary in speed. However, we observe that the recognition accuracy with using DTW is only 63%, which is much lower than that with using LSTM (97%). This is mainly because that DTW is distancebased and thereby has a poorer generalization ability than the learning-based LSTM. At the same time, the attention mechanism of the proposed LSTM model also plays an important role in improving the recognition accuracy. To be specific, an improvement of 5% (from 92% to 97%) was achieved after introducing the attention layer in the LSTM for Braille recognition. This can be attributed to the nature of the attention mechanism that learnable attention values (or weights) can be assigned to the hidden state at each time step of the input sequence, enabling the LSTM to focus on a particular part of the tactile information and ignore the useless one. Such property is important for distinguishing Braille characters with similar readings (e.g., the letters "l" and "s" as shown in Figure 1C) that cannot be correctly discerned by vanilla LSTM. More details of the architecture of the LSTM unit can be found in Figure S4, Supporting Information.
Here, we only demonstrate the efficacy of the proposed texture recognition method on two types of materials, i.e., Braille characters and fabrics, though our proposed approach could be easily extended to recognize more materials by retraining the LSTM model. In addition, we only demonstrate the sensor's capability of recognizing Braille characters for sighted people, while it could be potentially used to help blind people read Braille in the future by further optimizing the sensor design and recognition algorithms.

Conclusion
In this work, we present a novel texture recognition method via deep learning-enhanced tactile sensing. The outputs of the tactile sensor are directly related to the material's surface properties in terms of softness, friction, and roughness, thus providing rich tactile information for the proposed LSTM model to discriminate different textures. We demonstrate the efficacy of the proposed method using two tasks, i.e., Braille characters recognition and fabric recognition, with respective recognition accuracies of 97% and 99% achieved, illustrating that our method is effective in texture recognition and could be beneficial to relevant applications such as Braille reading for visually impaired people, defect detection in the textile industry, and so on.

Experimental section
Sensor Fabrication: The tactile sensor is composed of three layers, where the top layer is a thin flexible magnetic film (E % 2 MPa and thickness, 0.5 mm); the middle layer is an arc-shaped elastomer (E % 83 kPa and radius, 6 mm), and the bottom layer is a printed circuit board (thickness, 1.6 mm) with a Hall sensor (MLX90393, Melexis) embedded on it. The flexible magnet is a mixture of PDMS (SYLGARD 184, Dow Corning) and NdFeB magnetic powders with a weight ratio of 1:3, and it was fabricated using the master mold technique. After 2 h of curing at 80 C, the flexible magnet was magnetized under a strong rotating magnetic field generated by the electromagnet. The elastomer sandwiched between the flexible magnet and magnetic sensors on printed circuit board (PCB) is Ecoflex 00-50 (Smooth-on Inc.), which was directly cast on the PCB. To bond the flexible magnetic film onto the elastomer, an intermediate prepolymerized mixture of PDMS and Ecoflex 00-50 was prepared. After the centers of the magnetic pole and the Hall sensor are perfectly aligned by adjusting the position of the magnetic film, the entire structure was heated to cure the prepolymerized mixture and keep the alignment permanently.
Real-Time Braille Character Recognition Setup: The sensor was mounted at the end of the robotic arm (Universal Robot 10e) through a 3D-printed connector, and it slid over the Braille characters at a pressing depth of 0.7 mm and the speed of 2 mm s À1 . Here, we limited the moving speed of the robot to 2 mm s À1 to prevent the sensor from being broken, and the reading speed is improved to 15 mm s À1 (or %1 character per 0.8 s) on the human fingertip, which is close to that of proficient human readers (%one character per 0.4 s for the fastest and 3.8 s for the slowest [28] ). This results in the length of the LSTM input being around 54 on the robot hand (with a sampling rate of 10 Hz) and 10 on the human fingertip (with a sampling rate of 16 Hz). When the sensor was put on the human fingertip, a ring-like connector with four "legs" was designed to keep the pressing depth (or contact force) the same (%0.7 mm) during sliding, where each leg is 2.5 mm in diameter and 4.5 mm in length. The 3D-printed Braille board is in A4 size, on which all raised dots are 1 mm in height and 1.5 mm in diameter. The distance between two dots of each character is 2.5 mm in both horizontal and vertical directions, and the distance between two characters is 12 mm in the same row and 15 mm in two adjacent rows. To accomplish real-time Braille recognition, we first segment the valid signals from real-time tactile measurements when sliding the sensor over a character (e.g., from p 1 to p 1 0 for letter "d" as shown in Figure S5, Supporting Information). This was achieved by checking the change of B z , whose value would increase from zero at the start point p 1 and decreased back to zero at the end point p 1 0 . When the sensor slides over the end point p 1 0 , the tactile data from p 1 to p 1 0 are immediately fed into the pretrained LSTM model for real-time Braille recognition. For Braille character recognition on the robot, there are 902 Braille characters (including 26 English letters and three special symbols) in total for training the neural network and 163 characters for testing in real time. For Braille character recognition on the human fingertip, there are 678 characters in the training set (collected with different finger poses and reading speed) and 68 characters in the test set. The training set (with random Braille alphabets and symbols) and test set (with characters in a Braille poem) for Braille character recognition are completely different in the aforementioned demonstrations. The dataset and codes for Braille recognition are available at: https://github.com/ JJHu1993/braille.
Fabric Recognition Setup: The experimental setup for fabric recognition is similar to that for Braille character recognition, where the Braille board is replaced with fabric samples in the size of 100 mm Â 100 mm. During data collection, the tactile sensor was first pressed on the fabric sample with a pressing depth of 1 mm after the initial contact, and then slid along the sample surface for 30 mm at a speed of 3 mm s À1 . The aforementioned process is repeated ten times per fabric sample by the robot, and thereby 600 instances of fabrics are obtained in total, where 80% of the data are used as the training set and the rest 20% are used as the test set. The dataset and codes for fabric recognition are available at: https://github.com/JJHu1993/fabric. www.advancedsciencenews.com www.advintellsyst.com