Artificial Intelligence of Things (AIoT) Enabled Virtual Shop Applications Using Self‐Powered Sensor Enhanced Soft Robotic Manipulator

Abstract Rapid advancements of artificial intelligence of things (AIoT) technology pave the way for developing a digital‐twin‐based remote interactive system for advanced robotic‐enabled industrial automation and virtual shopping. The embedded multifunctional perception system is urged for better interaction and user experience. To realize such a system, a smart soft robotic manipulator is presented that consists of a triboelectric nanogenerator tactile (T‐TENG) and length (L‐TENG) sensor, as well as a poly(vinylidene fluoride) (PVDF) pyroelectric temperature sensor. With the aid of machine learning (ML) for data processing, the fusion of the T‐TENG and L‐TENG sensors can realize the automatic recognition of the grasped objects with the accuracy of 97.143% for 28 different shapes of objects, while the temperature distribution can also be obtained through the pyroelectric sensor. By leveraging the IoT and artificial intelligence (AI) analytics, a digital‐twin‐based virtual shop is successfully implemented to provide the users with real‐time feedback about the details of the product. In general, by offering a more immersive experience in human–machine interactions, the proposed remote interactive system shows the great potential of being the advanced human–machine interface for the applications of the unmanned working space.

. The structure and the working mechanism of the pneumatic actuator.                  Supplementary Note 1. The current limit of the resolution for the TENG sensors and the influence of the resolution on the machine learning result.

Supplementary Note 2. Humidity effects on the T-TENG sensor output.
Other Supplementary Material for this manuscript includes the following: Movie S1. Online virtual shop application.
Movie S2. Shape-size-temperature fused sensory system.                Optical waveguides [7] LED, photodiode, elastomer    Note: "Correct/Total" means the correct times in total numbers testing. Note: "Correct/Total" means the correct times in total numbers testing.

Supplementary Notes
Supplementary Note 1. The current limit of the resolution for the TENG sensors and the influence of the resolution on the machine learning result.
The size of the L-TENG and T-TENG sensors is determined by the available space with a certain design or size of the manipulator structure. For the L-TENG sensor, the larger the size of the manipulator structure or more reserved space for the sensor, the larger gear that can be installed, thus reserving more space on the gear surface for more gear teeth and an improved detecting resolution due to the greater number of teeth under a certain arc of rotation. Besides, with a specific size of the L-TENG sensor or the gear, we can further utilize advanced fabrication processes, such as MEMS process, micro-machining, etc., to increase the teeth density and improve the resolution. Considering the relatively small size of the current manipulator design and the low cost of the 3D-printing fabrication process, the current resolution of the L-TENG sensor is acceptable as one feature of the object size for further machine learning. For the T-TENG sensor, the resolution is determined by the gap between adjacent electrodes and the total number of electrodes within a certain area of the tactile panel.
Commonly, for such a grating structure, more grating electrodes with a smaller gap will result in a higher resolution, and many high-resolution grating solutions based on triboelectric have been proposed before. [16,17] However, in our design, more grating electrodes also mean more channels of the output signal and more complex data to be used for machine learning. Though more features and information can be extracted to increase the recognition accuracy due to the increasing channel numbers, when the accuracy has reached a certain level, the cost and benefit will not be proportional. In our case, the 10 mm level resolution of the T-TENG sensors already able to contribute a high accuracy of 97.143% when fused with the L-TENG sensor data, which is acceptable for the low cost and energy-saving purpose with a minimalistic design.
Additional test have been added to see the influence of the sensor resolution on the machine result. Due to the current limitation of the 3D-printing fabrication process we used and the specific manipulator size, the resolution of the L-TENG sensor is relatively difficult to be improved at this stage. However, it is easy for us to increase the number of short electrodes from three to four in the T-TENG sensors to improve the resolution from 10 mm to ~6.67 mm, and see the machine learning result. In the experiment, we integrated one pneumatic finger with the L-TENG sensor and the T-TENG sensor and test its recognition performance on 5 objects: apple, orange, big box, long can, and short can. Figure S13(a) shows that when the number of the short electrodes is three, the classification accuracy can reach 92%, and can be further improved to 94% when the number of the short electrodes increases to four depicted in Figure   S13(b), proving that the enhanced resolution and the increased data/channel number will benefit the machine learning result.

Supplementary Note 2. Humidity effects on the T-TENG sensor output.
The main structure of the L-TENG sensor is packaged with the PLA material as shown in Figure   1(a-i), while the T-TENG sensor is directly exposed to the surrounding air and the contact surface of the objects. So the moisture in the air or on the surface of the object will have a greater impact on the outputs of the T-TENG sensor. Actually, the contact position detecting mechanism of the T-TENG sensor is based on the voltage ratio between the short electrodes.
Although the output of triboelectricity will become smaller under high humidity, the degree of change of all short electrodes' outputs is the same, so the ratio of each other will not change, and the position detection function will not be affected. However, for the long electrode which detects the contact area based on the output voltage amplitude, the detecting ability will be influenced.
To further evaluate the humidity effects on the T-TENG sensor output, we have done the extra experiments under different humidity conditions as shown in Figure S14. The normal relative humidity in the surrounding air is about 70% -80% RH. It is clear that the output voltage amplitude of the T-TENG sensor declines gradually when the humidity increase from 65% to 85%, and drops sharply when the relative humidity reaches 95%. Because we use the voltage spectrum generated during the process of grasping objects directly for machine learning, the variation in voltage amplitude under different humidity will inevitably affect the results of machine learning. However, if we collect the data under different humidity and combine them into a more generalized data set, i.e. the category of each object contains the data that the object was captured under different humidity conditions, the influence of humidity on accuracy will be avoided to a certain extent.