Soft-smart robotic end effectors with sensing, actuation, and gripping capabilities

Soft and smart robotic end effectors with integrated sensing, actuation, and gripping capabilities are important for autonomous and intelligent grasping and manipulation of difficult-to-handle and delicate materials. Grasping and actuation are challenging to achieve if using only one opto-mechanical tactile sensor. It is highly desirable to equip these useful sensors with multimodal actuation and gripping functionalities. Current electroadhesive (EA) grippers, however, cannot differentiate object size and shape, nor can they grasp concave or convex objects. In this paper, we present TacEA, an integration of a pneumatically actuated visio-tactile (TacTip) sensor and a stretchable EA pad, resulting in a monolithic soft-smart robotic end effector with concomitant sensing, actuation, and gripping capabilities. This soft composite-materials device delivers the first soft tactile sensor with actuation and gripping capability and the first EA end effector that can sort different 2D object sizes and shapes with one touch, and which can actively grasp flat, concave and convex objects. The soft-smart TacEA is expected to widen the capabilities of current tactile sensors and increase EA end effector use in material handling and in processing and assembly lines.


Introduction
Human hands can easily sense and differentiate different surface textures and temperatures, and grasp and manipulate objects of a variety of sizes, weights, and shapes. These capabilities are realized by an efficient integration of sensing, actuation, and gripping functionalities. Various manufacturing automation applications, especially in the context of 'Industry 4.0', require controllable, safe, reliable, and robust robotic grasping and manipulation technologies closer to human capabilities. To this end, various robotic gripping [1,2] and tactile sensing [3][4][5] technologies have been pursued and implemented.
Tactile sensors are critical elements to equip robotics and automation systems with a sense of the environment, to perform complex manipulations [6,7], and are important for human-robot interactions [8]. Many kinds of tactile sensors have been investigated, including those based on piezoresistive, capacitive, optical, and piezoelectric technologies [9][10][11]. One promising technology employs visio-tactile information, with examples including the Gelforce [12], GelSight [13], and TacTip [14] sensors. These sensors use camera sensors to translate mechanical deformations, containing structural, tactile force, and movement information, into image data, resulting in a high bandwidth (only limited by cameras), low cost, and parallel tactile sensing mechanism [15]. The TacTip sensor is especially interesting because it exploits a soft skin structure that is inspired by the human fingertip and which, through local lever action of papillae-like pins, transforms small normal deformations into precise lateral image features [15,16] (figure 1(a) top). TacTip sensors have been used for shape recognition [17], edge detection [18], tumour detection [19], and force sensing [20] applications.
Robotic grippers are essential components to enable autonomous material handling applications. Under the growing paradigm shift of conventional and rigid to soft robotics, soft grippers are starting to replace conventional rigid grippers for safe grasping and manipulating of difficult-to-handle and delicate objects. Soft gripping technologies can be divided into three main types: actuation (e.g. pneumatic grippers [1]), controlled stiffness (e.g. granular jamming and shape memory grippers), and controlled adhesion (e.g. gecko grippers [2]). Soft grippers realized by controlled adhesion, including dry adhesion and electroadhesion, can manipulate very fragile objects. Dry adhesives, inspired by the feet of animals such as the gecko, exploit micro van der Waals forces [2], and can be used to grasp a broad range of clean and smooth surfaces. They, however, have poor grasping performance on dusty surfaces due to insufficient self-cleaning capabilities, and on low surface-energy materials, and are not electrically controllable. Electroadhesion (EA) [21] employs high electric fields to induce polarization in a surface and to generate electrostatic attraction forces between the EA pad and a surface [22]. EA is cost-effective, versatile, and readily electrically controllable, making it suitable for many soft robotic and gripping applications [23,24]. Although EA generates modest gripping forces, it has many advantages compared to other grasping methods, including enhanced adaptability (it can be used on a wide range of surfaces and environments), gentle/flexible handling (it can be used to pick up delicate objects), reduced complexity (both in terms of mechanical structure and control system), and low energy consumption [25,26].
Controlled adhesion requires close contact with an object to generate sufficient gripping force. It is therefore highly desirable to equip these technologies with actuation and sensing capabilities, thereby enabling them to recognise and conform to different surface morphologies and to grip more effectively. Guo et al proposed a morphologically adaptive EA [27] in order to sort different materials and grip objects with a range of surface curvatures. Extending this principle, the PneuEA gripper [28], a combination of a soft pneumatic gripper and a soft EA end effector, was able to determine object contact using two touch sensors and to grasp flat and convex objects. Subsequently the self-sensing EA composite gripper [25] employed a combined dielectric elastomer actuator and EA design that could determine contact (using intrinsic capacitive sensing) and could grasp flat and concave objects. Currently there is no EA end effector that can grasp both concave and convex objects.
To overcome the limitations of both EA grippers and the passive TacTip tactile sensors, here we present TacEA, an integration of a pneumatically actuated TacTip sensor and a stretchable EA end effector, resulting in a monolithic softsmart robotic end effector with concomitant sensing, actuation, and gripping capabilities. This work demonstrates three significant advances: (1) the first TacTip visio-tactile sensor with actuation and gripping capability, (2) the first EA end effector that can sort different 2D object sizes and shapes with one touch, and (3) the first EA end effector that can actively grasp both flat, concave, and convex objects. The TacEA concept is expected to widen the capabilities of current tactile sensors and to increase the use of EA end effectors in material handling and processing and assembly lines. This paper is organized as follows. The TacEA concept design, working principle, and fabrication details are presented in section 2. In section 3, experimental characterization of the multimodal sensing, actuation, and gripping capabilities of TacEA device is described and discussed. As a potential industry application study, an intelligent TacEA material handling system is introduced and implemented. Further discussions are presented in section 4. Finally, significant research outputs and future work are summarized in section 5. We aim to design and develop a soft-smart multimodal robotic end effector that can sense the environment, actively change its shape to either conform to different shapes or tune its stiffness whilst actuating, and grasp objects with different sizes and shapes. To this end, the soft-smart TacEA robotic end effector consists of an inflatable visio-tactile sensor (TacTip) and a soft and stretchable EA pad (see figures 1 and 2). As a result, TacEA is not only able to sense different shapes and sizes both in its non-inflated and inflated states, thereby demonstrating exteroceptive sensing (figure 1(a)), but is also able to estimate its own deformations against different inflation pressures, thereby demonstrating proprioceptive sensing ( figure 1(b)). These sensing capabilities are achieved by visual observation of pin movements on the inside of the TacEA sensor skin in relation to object interactions and pressure-driven expansion of the skin. Static and dynamic displacement of the skin can be detected by camera observations of the skin. Due to the inflation of the soft and stretchable TacEA skin and inclusion of electroadhesion within the sensor surface, the device is able to grasp flat, concave, and convex materials (figure 1(c)).

TacEA fabrication
The TacEA device fabrication consists of manufacturing a TacTip sensor skin and integration of an EA pad. This includes the following five steps: Moulding black pins. Black pins (a mixture of Ecoflex 30 and black dye (Smooth-On Inc., USA)) were first moulded onto a 0.5 mm thick circular black rubber sheet (J-Flex, UK) using a laser-cut mould. The pins were distributed uniformly and the distance between adjacent pins was 3 mm, with the diameter and height of each pin being 1.5 mm and 3 mm respectively. The Ecoflex was degassed for 20 min. Excess resin was removed from above the mould, leaving a small gap at the top of each pin to enable the moulding of pin-tips in next step. The mould was placed in an oven to cure at 50°C for four hours.
Moulding white pin-tips. A mixture of Ecoflex 30 and white pigments (Sennelier, Inc., France) was squeegeed on to the top of the mould, filling in the top of each black pin. After curing (50°C for four hours) the sensor skin was then peeled from the acrylic mould.
EA pad fabrication. An electrically conductive, soft, and stretchable EA pad (J-flex, UK) was cut (Cricut) into a symmetrical EA electrode pattern, with electrode width and space of both 5 mm and effective electrode diameter of 125 mm.
Housing. A simple cylindrical tube was 3D printed from ABS to act as the sensor frame and pneumatic chamber. A USB camera (640 pixels×480 pixels, 30 fps, 0.01 Lux, ELP webcam 960P, Ailipu Technology Co., Ltd, China), a pneumatic tube (Heritage Pet Products Ltd, UK), and electroluminescent Panel (EL Panel Tape, UK) were mounted to one end.
Integration of EA with TacTip. The EA pad was bonded to the TacTip sensor skin using a layer of VHB 4905 (3M, USA) adhesive tape. A second layer of VHB adhesive was placed to cover the electrodes. Talcum powder was lightly dusted onto the VHB surface to remove its inherent stickiness. Finally, the EA pad was bonded to the other end of the housing using silicone adhesive (Sil-Poxy). The 3D exploded view of the complete TacEA end effector is shown in figure 2.

Results and discussions
We demonstrate and characterize the functionality of the TacEA soft-smart multimodal robotic end effector in terms of exteroceptive and proprioceptive sensing, pneumatic actuation, and shape adaptive grasping capability. 3.1. Exteroceptive and proprioceptive sensing 3.1.1. Exteroceptive sensing. Exteroceptive sensors are defined as sensors that acquire information from the environment. Examples include tactile sensors that can inform deformations and forces [20]. The TacEA end effector is not only able to indicate contact and sense different 2D sizes and shapes but also its own deformation. We define 2D size and shape sensing as the detection of an object's top surface. As presented in table 1, the TacEA end effector can sense and differentiate square (50 mm×50 mm, 4.59 g), circle (diameter of 56 mm, 4.93 g), and rectangular (40 mm×62 mm, 5.09 g) objects (laser-cut plywood plates of 5mm thickness). We employed the image processing procedure defined in supplementary materials and figure S1 is available online at stacks.iop.org/SMS/28/055034/mmedia to detect an object and calculate its size as the TacEA was lightly pushed onto each object. Table 1 shows less than 6.73% relative difference between estimated object surface area and actual object surface area, and a maximum of 3.41% relative difference between the non-inflated and inflated measured data. The relative difference is defined here as (max−min)/min×100%. The proposed algorithm therefore enabled the TacEA end effector to effectively detect different object sizes and shapes both in its non-inflated and inflated states. To ensure maximally safe interactions between the device and objects, it can be inflated, thereby minimising potential contact between the object and the rigid housing of the TacEA.
3.1.2. Proprioceptive sensing. Proprioceptive sensors are defined as sensors that measure values internal to the system. Examples include soft foams that can detect their own deformations [29]. We inflated the TacEA skin to pressures of 0.1, 0.15, 0.2, and 0.25 bar. The mechanical deformations of the TacEA skin, measured by the curvature of the surface, are presented in figure 3(a). The physical deformations are shown in figure 3(b). The sums of the white pixels at different curvatures are also shown in figure 3(a). This value decreases with increasing the air pressure and curvature of the TacEA skin. This indicates that the TacEA device can sense its own deformations by correlating the change of its curvature and measured sum of white pixels. We define the curvature Cur as the reciprocal of the radius R of the curve: Cur=1/R. TacEA's exteroceptive and proprioceptive sensing capabilities were not totally independent. Here we separated these modalities temporally. We used proprioception to measure and set pre-touch inflation and exteroception to detect subsequent touch interaction.

Pneumatic actuation
The TacEA end effector can be made to actively change its shape by increasing or reducing the air pressure within the housing, with respect to atmospheric pressure. We characterize the TacEA actuation performance by measuring its stiffness change at different internal air pressures, as shown in figure 4. A linear rail (X-LSQ150B-E01, Zaber Technologies Inc., USA) was used to move the inflated end effector downward to touch an inline miniature s-beam load cell (Applied Measurements Ltd, UK, accuracy of ±0.05%). Acknowledging that deformation of the inflated skin under a small-area load is complex and we therefore define a firstapproximation stiffness value for the overall TacEA device, Sti, as the measured force F divided by displacement, x D (5 mm used here, see figure 4(b)):  4(a)). It should be noted that the stiffness may change if we vary the contact area between the metal cylinder tip (diameter of 27 mm) connected to the load cell and the TacEA skin. The stiffness study was inspired by the work by McInroe et al [30]. Using the same set up shown in figure 4(b), we investigated the relationship between the sum of white pixels and force magnitude to show the force sensing capability of the TacEA end effector. We inflated the end effector to a pressure of 0. Five tests were repeated for each force magnitude. As the force magnitude increased, the sum of white pixels decreased monotonically ( figure 4(c)). The pin deformations under 0 and 6 N are shown in figure 4(d).

Shape adaptive EA grasping and quick release
When a high voltage (4.8 kV used here) is applied between the electrodes of the EA pad, counter charges are induced at the surface of any object it touches due to mainly electric polarization or induction, causing attractive forces between the object and the EA end effector. Air can be sucked out or inflated into the TacEA end effector to enable surface conformability to both concave and convex shapes. Figure 5 presents four examples of this adaptability. The TacEA gripper was able to grasp and lift a convex object with cur- Electroadhesion is a complicated and dynamic electrostatic attractive effect [21,22,[25][26][27][28]. The dynamic behaviour means that it takes time (more than seconds) for the maximum EA force to be generated, and it can take a similar time for decay of the force (and object de-adhesion) when the EA system is turned off. This is due to the residual charges that remain (trapped) in the EA system [22,[25][26][27][28]. The dynamic nature of the electroadhesion phenomenon leads to the challenge of speeding up the electroadhesion chucking and de-chucking process. Various de-electroadhesion methods have been proposed and implemented to accelerate the de-chucking process. These methods can be classified into mechanical and electrical methods. Mechanical solutions include the use of vibrations, pegs, and air jets [21,31]. Electrical solutions include advanced voltage control methods and the associated electronics, including oscillating release waveforms and adaptive release voltages [32], reversion of the polarity voltage [33], varying polarity reversion [34]. Gao et al proposed a rapid release method by exploiting the resonant vibration of an embedded dielectric elastomer actuator [35].
In this study, we employed the pneumatic actuation described in section 3.2 as an alternative solution to facilitate fast EA release. We applied 4.8 kV to the TacEA gripper to hold a 50 mm×50 mm×5 mm plywood plate (4.59 g) for 10 s ( figure 6(a)). It is shown in the supplementary video 3 that the plate would release 25 s after the voltage was removed ( figure 6(b)). By rapidly inflating the TacEA to approx. 0.27 bar we were able to release the plate within one second.

Intelligent TacEA material handling system design and implementation
An autonomous, intelligent, non-inflatable TacEA material handling system was developed to sort and lift different object sizes and shapes. The schematic diagram of the system can be shown in figure 7. A bipolar 5 kV high voltage power supply (Advanced Energy Industries, Inc., USA) was used to activate the TacEA EA end effector. An NI USB-6343 X Series DAQ device (National Instruments, UK) was used to control the output voltage of the high voltage amplifier. A 2-DOF linear stage, comprising a horizontal linear rail connected to the vertical linear rail, was employed to enable movement of the end effector in two axes. The image acquisition was implemented through the USB camera module. All program codes were realized through MATLAB. Grasping of objects with different shapes and sizes is demonstrated in the supplementary video 4.
The control flow chart showing integrated movement, object detection and handling is shown in figure 8. Firstly, the TacEA end effector was moved down until contact had been identified, with the EA pad turned off and the image processing process initiated. As the TacEA skin was lightly pushed into the object, skin deformations were detected by the camera. When the area of the touch exceeded a threshold U, the real-time image processing algorithm started to classify the object being touched. Then the object size and shape were determined. When the object type and size have been determined, EA was turned on and a suitable voltage was applied for a certain period of time (5 s here) to ensure an appropriate amount of adhesive to grip the object. After this, the end effector was moved upwards to pick up the object. The horizontal linear rail was actuated to move the object left-right to the designated destination. When the object was above the designated destination, EA was turned off to release the object and drop it into the appropriate sorting box.

Further discussions
When inflating the TacEA skin, the effective area of the EA, and therefore the attractive force it can generate, changes. The schematic diagram of the geometric parameters of the non-   The autonomous, intelligent, non-inflatable TacEA material handling system: (a) schematic diagram, where 1 is the vertical linear rail, 2 is the horizontal linear rail, 3 is the USB camera module, 4 is the EA end effector, 5 is the object to be grasped, 6 is the object support, 7 is the vibration isolation platform, and 8 is the sorting box and (b) the system prototype.
inflated and inflated states are presented in figure 9(a). The relationship between the obtainable normal EA force and noninflated geometric parameters is approximated as [36]: where C is defined as a dimensionless function of geometric parameters w w s + and , t w s + w is the width of the electrodes, s is the space between electrodes, and t is the thickness of the dielectric cover.
When inflating the TacEA end effector, we assume there is no material property change (such as resistivity) during the deformation. We also assume the deformation of the TacEA skin is uniform, then both where r=73.5 mm. As shown in figure 9(b), the normal EA force per unit area increases with increasing the radius. This manifests that the flatter the TacEA skin the greater the normal EA force per unit area, which agrees with the results published by Germann et al [37]. Since it is impractical to accurately model the EA force if taking environmental and surface texture factors [22] and the dynamic material property change into account, only a limited theoretical is described here. Further and extensive characterization to aid the accurate modelling are required and will be the subject of future work.
There are several limitations associated with the proposed TacEA multifunctional end effector. Firstly, it cannot  measure objects with larger radius than 73.5 mm. To overcome this a finger-rolling strategy [15] and active explorative sensing may help to detect larger objects. Secondly, the EA geometry has not been optimised and therefore the resultant EA forces are relatively small. Further work is needed to optimise electrode geometry, materials, and control strategies to increase adhesion forces.

Conclusions and future work
We have proposed the TacEA concept, a soft-smart robotic end effector with integrated sensing, actuation, and gripping functionalities. The TacEA was implemented using a costeffective and easy to-implement fabrication process, consisting of a deformable TacTip-like visio-tactile sensor and an EA end effector. The TacEA device was able to conduct exteroceptive and proprioceptive sensing, exemplified by detecting different sizes and shapes both in its flat and inflated forms and sensing its own deformations by correlating the change of its curvature with measured sum of white pixels and internal pressure respectively. Completely soft and safer interactions between the robotic end effector and objects can be achieved by inflating the soft membrane. The TacEA was able to lift both flat, concave, and convex objects due to the combined morphology adaptability and EA application. This soft-smart robotic end effector is expected to widen the capabilities of current optical tactile sensors and increase the use of EA in material handling and size/shape sorting assembly lines.
The main contributions of this work include: (1) the development of the concept of a soft-smart robotic end effector with integrated sensing, actuation, and gripping functionalities, (2) the first TacTip-like standalone optomechanical sensor with actuation and gripping capability, (3) the first EA end effector that can sort different 2D object sizes and shapes with one touch, (4) the first EA end effector that can actively grasp both flat, concave, and convex objects, and (5) the TacEA fabrication method and material handling system. Future work will include modelling and optimizing the TacEA design to achieve better performance such as lifting and differentiating heavier weights. In addition, scalability of the TacEA design will be considered.