Plantar Tactile Feedback for Biped Balance and Locomotion on Unknown Terrain ¤

This work introduces a new sensing system for biped robots based on plantar robot skin, which provides not only the resultant forces applied on the ankles but a precise shape of the pressure distribution in the sole together with other extra sensing modalities (temperature, pre-touch and acceleration).Theinformationprovidedbytheplantarrobotskincanbeusedtocomputethecenter of pressure and the ground reaction forces. This information also enables the online construction of the supporting polygon and its preemptive shape before foot landing using the proximity sensors in the robot skin. Two experiments were designed to show the advantages of this new sensing tech- nology for improving balance and walking controllers for biped robots over unknown terrain. bio-mimetic of human vision, computational neuroscience of vision, action understanding, human – robot interaction, active vision, mobile robot navigation and object-oriented software construction.


Introduction
Humanoid robot's biped balance and locomotion is a challenging task due to the naturally unstable dynamics of°oating base systems. Its di±culty increases, even more, when the terrain is not°at and is unstructured. In such conditions, walking controllers must be adapted to the terrain either by mapping the terrain to plan a set of footholds [2][3][4] or by reacting compliantly to the terrain modifying the footholds online. [5][6][7][8][9] Di®erent methods have been developed for mapping the terrain looking for suitable footholds by using cameras, 2 laser scanners 3 or exploratory motions. 4 Following the reactive approach, di®erent methods have been proposed by including Force-Torque (FT) sensors in the ankles or the joints [5][6][7]10,11 or even with contact switches mounted on the sole. 8,9 In this paper, a biologically-motivated approach for terrain sensing is presented by mounting robot skin [12][13][14] on the foot soles without modi¯cations on the foot architecture. The plantar robot skin can be used to estimate all the states that are required for balance and locomotion 15,16 (e.g. Center of Pressure (CoP), Zero Moment Point (ZMP) and Centroidal Moment Pivot (CMP)). Besides, it provides complementary information that ankle FT sensors cannot by exploiting the additional sensing modalities. For example, the spatial distribution of force sensors in the robot skin enables the reconstruction of the footprint's pressure distribution and thus an accurate shape of the supporting polygon. In the following sections, the motivation and details of the plantar skin sensor are presented as well as its capabilities to improve biped balance and locomotion.

Related Work
Humanoid robot balance and walking controllers require feedback of the ground reaction forces to guarantee the stability of the posture and the gait. In most cases, this information comes from FT sensors mounted in the ankles of the robot. 5,6 However, the addition of extra sensors to the robot's feet can improve the performance or help simplify the control laws.
Di®erent sensing technologies have been applied to robot soles to improve the performance and versatility of walking controllers. For example, resistive sensor arrays are mounted on the foot soles of small-size humanoid robots without ankle FT sensors to enable the direct measurement of the CoP. [17][18][19][20] For bigger robot sizes (closer to human size and weight), resistive sensors are combined within the structural layers of the soles to increase the sensing range and endurance. 21,22 Likewise, other tactile sensing principles have been applied as the optical measurement of rubber deformation 23 or the high-speed pressure sensor grid 24 which can acquire the pressure shape of the foothold at a frequency of 1 kHz.
Furthermore, tactile sensors can be combined with other sensing modalities. The perceptual foot developed in Ref. 25 combines the classical ankle FT sensor architecture with a°exible resistive sensor array and an IMU sensor. With this foot architecture, the ZMP and the supporting polygon are acquired at a rate of 100 Hz.
With the addition of contact or pressure sensors on the sole, walking algorithms have been extended to handle unknown terrain conditions 8,19,26 without complex compliant mechanic designs 27 or fast reactive motions. 28 Furthermore, even for point feet robots, the detection of premature contacts on a step trajectory can be used to reset the phase of a Central Pattern Generator (CPG). 9,29 Wider information about the contacts with the terrain can be exploited not only for footstep compliance but to learn and classify the terrain conditions. 20,30 By adding more sensing modalities to the sole, more complex interaction conditions can be detected such as the slipperiness of the foothold. 31

Biological Motivation For Plantar Sensing
The postural equilibrium of a human being results from a complex synergy of sensation and actuation. 32 It is a dynamic phenomenon which keeps the center of gravity oscillating over the standing feet by the application of antagonistic re°exes triggered by di®erent stimuli. 33 This balance depends on a complex fusion of visual, vestibular, proprioceptive and exteroceptive feedback. It is known that none of them is essential for the purpose of balance, but the lack of each of these kinds of sensory feedback produces di®erent changes in the posture and the motion patterns during stance balance. 34 Between the exteroceptive receptors, the cutaneous receptors of the foot sole play an important role for both balance and locomotion. 35 The human foot sole is covered with an endurable layer of glabrous skin that apart of protecting the foot from rough surfaces, provides rich sensorial information about the terrain such as texture, hardness, temperature and pressure distribution on the foothold. The role of its cutaneous receptors in the equilibrium control has been studied for more than a century. 36 Studies reveal that subjects with the soles anesthetized by hypothermia show increased posture sway while standing. For example in Ref. 37, the subjects presented larger amplitude oscillations while closing the eyes (Romberg's test 38 ) than subjects with non-anesthetized soles. A similar result was generated in Ref. 39 when body sway was galvanically induced in the vestibular system of healthy subjects with anesthetized soles. Furthermore, the inhibition of the sense of touch in the soles also induces delays on the compensatory stepping reactions with di®erent e®ects depending on the phase and the direction of the step. 40 The human sole shows various sensitivity to mechanical stimuli in di®erent regions. 35 This is because its mechanoreceptive a®erents are connected to distinct nerves to transport the data to the spinal cord 41 and because within the regions connected to one of these nerves the receptor density is not uniform and their pressure thresholds are mismatched. 42 Furthermore, the sensitivity and thresholds of these a®erents change according to the phase of a footstep during walking. For example, sensitivity is increased during single support phase on the swing foot and reduced considerably right after footfall. 43 In summary, (1) human equilibrium and locomotion rely signi¯cantly on the sense of touch provided by the mechanical a®erents located on the feet, (2) these a®erents are distributed with di®erent sensitivities and thresholds along the sole, (3) the sensitivity and threshold of the a®erents depend on the standing or walking phase with clear di®erences for supporting and swinging leg, (4) the a®erents are connected to three main nerve branches that coordinate di®erent re°exes on muscular groups also depending on the walking phase.

Plantar Robot Skin
In recent years, robotic tactile sensing gained attention for both industrial 44,45 and research-oriented platforms like humanoid robots. 46,47 Robot skin can provide precise information about external contacts with the environment in other parts of the body than the end e®ectors. With some exceptions as the ones mentioned in Sec. 2, most of the tactile sensors have been developed to provide high sensitivity at low and middle force ranges. 48 These technologies have successfully improved human-robot interaction, 46,49 grasping of complex objects [50][51][52] and robot safety in industrial environments. 45,53 Nevertheless, the application proposed in this work requires a skin technology capable of holding the weight of a full-size biped or humanoid robot while resisting the impacts produced during biped locomotion on unknown terrain.
Full-size humanoid robots are designed to work in human environments and to perform tasks with humans. Their mass can be bigger than 100 kg and consequently, the weight of these robots can vary around 980 N without considering payloads. Also, biped walking consists of a succession of single and double support phases. Therefore, the skin sensor required for this application must be capable of holding around 980 N in the area of a foot and still sense variations in the contact pressure.
Due to the heavy load requirement, the suitable robot skins for full-size humanoid robot soles are the resistive matrix arrays 21,22,25 and the capacitive force cell. 54 Some of the skin technologies that have proved capable of resisting high pressure and still sense variations are the Tekscan pressure sensors 55 with pressures larger than 1 MPa, the°exible ROBOSKIN sensors 54 under pressures larger than 150 kPa, and the stretchable PsSi tactile sheet 30 under pressures larger than 150 kPa. Nevertheless, these tactile sensors only provide the pressure shape, and the geometrical mapping over the robot sole must be obtained manually. Even more, they would require protective layers to prevent damage caused by harsh terrain textures.
The robot skin developed at our lab ful¯lls the load requirement. 13 Each skin cell with a surface area of 6.9 cm 2 ( Fig. 1) can hold more than 80 kg and still sense variations in the applied force (operating pressure % 80 MPa). Additionally, it provides four sensing modalities (force, pre-touch, acceleration and temperature) over a uniform spatial distribution that can cover complex curved surfaces. 12 The complete tactile information can be delivered at a rate up to 250 Hz in large scale areas (covering not only the soles) due to new neuromorphic paradigms, such as the even-driven communication architecture. 56 The robot skin also provides the position of every cell on the sole 57 which can be used to reconstruct the pressure shape when the foot is on the ground in real-time.
The skin cells are connected with the neighboring cells conforming a dynamic network where the data packets are sent following optimal communication paths to reduce the latency. 58 This dynamic network con¯guration is valuable for the foot-sole application because the skin must recon¯gure its communication channels in case of hardware damage. Therefore, if a heavy impact is produced between the terrain and the sole, and a skin cell in the sole is damaged, the network recon¯gures itself and continues transporting the data generated by the other cells. Nevertheless, the construction of the skin cells is resilient and can hold hard impacts without receiving damage. Additionally, following the biological principle of the sole skin in human feet, the external silicon layer was thickened for this special application to extend its lifespan. This silicon material shows a high friction coe±cient over di®erent surfaces which improves the stability of the steps, reducing the likelihood of slipping. Two 42-skin-cell patches were created to cover the soles of our robot as shown in Fig. 2. These patches are mounted on a set of 3D printed removable sole covers which make it easy to mount and dismount without making permanent changes on the robot. This mounting mechanism can be easily adapted to di®erent foot sizes and geometries by creating the mountable covers for the robot foot and the skin patches to cover the sole area.

Center of Pressure Estimation
There are di®erent methods to locate the CoP of a robot's foothold with di®erent foot and sensor architectures. 59 For example, using an array of single-axis force sensors distributed on the sole. The normal force sensors in our robot skin ful¯ll that requirement. It covers the whole area of the sole with a uniform distribution of normal force sensors as shown in Fig. 2. Then, for a single sole, the position of the CoP is de¯ned as where, for a group of k skin cells mounted on a robot's sole, Fjz is the vertical ground reaction force measured by the jth skin cell located at p j with respect to the sole coordinate frame O sole as shown in Fig. 3. We denote p as the computed CoP of the foot sole. Furthermore, the uniform spatial distribution and size of the skin cells enables a fast computation of the contact area by counting the cells that detect contact force over an activation threshold F jz > F . All the skin cells that satisfy this condition are called Active Cells. Then, the contact area is with A cell % 691 mm 2 . Fig. 3. Computation of the CoP on the sole using the normal force sensors in the robot skin. F j is the normal force measured by the jth skin cell, p j its position with respect of the sole coordinate frame and p is the computed CoP position. As an example of additional information that can be generated with the robot skin, the contact area A is easily computed.

Supporting Polygon Acquisition
The extra information the robot skin provides enables the direct acquisition of important balance-stability metrics such as the supporting polygon. As depicted in Fig. 5 the construction of the supporting polygon from skin information starts bȳ nding the active cells [ Fig. 5(b)]. Then, for every active cell in the sole, the set of corner points S c ¼ ½ 1 2 . . . 6 , 1 2 R 4 (homogeneous points) of the hexagonal skin cell are transformed to the sole frame O sole as shown in Fig. 4 and added to a cluster set [ Fig. 5(c)] with Lsole T world and Rsole T world being the transformation from the left sole frame O Lsole and right sole frame O Rsole to world frame O world , respectively. And then,¯nd the convex hull wrapping S feet which generate the double support supporting polygon [ Fig. 5(g)].
This procedure can be applied to other sensing modalities to obtain the temperature footprint of a sole or the pre-touch modality to¯nd a preemptive shape of the supporting polygon from a few centimeters before the foot landing [ Fig. 5(e)]. This information is useful to create preemptive controllers for robust walking over unknown terrain.

Experimental Results
In this section, the methods described above are experimentally evaluated. The results obtained using robot skin are compared to similar methods using ankle FT sensors. The controllers were implemented on an 86 kg REEM-C humanoid robot with the skin sole covers shown in Fig. 2.
The measurement of the CoP and the reconstruction of the supporting polygon from raw skin data are shown in Fig. 6. First, we identify the active cells as shown in Fig. 6(a). The cells where the pre-touch sensor is active are marked in red and the cells which detect contact force are marked in blue. With the active cells data, as shown in Fig. 6(b), we compute the CoP of each foot with the method described in Sec. 5. With this data, as shown in Fig. 6(c), we also construct the supporting polygon and the pre-touch polygon for each foot using the method described in Sec. 6. Finally, we combine the feet's polygons to construct the composite supporting polygon and the composite pre-touch polygon as described at the end of Sec. 6.
In the following sections, the skin measurements are used to show how tactile feedback can be used in a balance controller and to re-plan steps when an obstacle is detected under the foot during walking.

ZMP in a balance controller
The online acquisition of the CoP can be used to compute the ZMP feedback for stable balance controllers and dynamically balanced walking controllers. 16 To assess the performance of the skin ZMP acquisition, a bench test was implemented comparing the ZMP position estimated by both the robot skin and ankle FT sensors in a balance controller. The feedback law used for this test is the 2D CP-ZMP regulator controller. 62 2 R is the inverted pendulum frequency at CoM height x z , p 2 R 2 and p d 2 R 2 are the real and desired ZMP, 2 R 2 and d 2 R 2 are the real and desired CP, x 2 R 2 and x d 2 R 2 are the real and desired CoM positions and k , k f and k x are positive gains. The controller was tuned to be compliant to external pushing forces and return to a standing position afterwards. The tuned parameters are listed in Table 1.
The robot was standing on both legs keeping balance and was pushed and pulled by an operator to introduce unknown disturbances on the controller as shown in   Fig. 9 show the comparison of the estimated ZMP with the FT sensors and the skin in the xy plane. Figure 9(a) shows the local ZMP in each foot and the supporting polygon detected for each foot. Figure 9(b) shows the composite ZMP and supporting polygon. The experiment was conducted two times,¯rst using the skin measured ZMP as feedback in the balance controller and then using the ZMP estimated with the FT sensor. In both cases, the gains of the balance controller were the same. The operator Fig. 7. An experimental test of the ZMP estimation using robot skin. The robot is standing executing a CoM-ZMP compliant balance controller and an operator is pushing the upper body to disturb its balance. The ZMP used in the feedback loop was estimated using the robot skin.  (Fig. 7). The ZMP was estimated using two di®erent sensors: the robot's ankle FT sensors (blue), and the robot skin on the sole (red). The experiment was recorded twice,¯rst using the skin ZMP as feedback in the balance controller and then, using the pushed and pulled the robot from the neck to induce compliant motions in the robot body. When the external force was removed, the robot came back to the desired ZMP position located at the center of the supporting polygon. During the experiment, the ZMP was deviated up to 10 cm from the reference (located at 0 cm) by the external forces. However, it never reached the supporting polygon border to prevent breaking the full-sole contact assumption. The most considerable di®erence between the ZMP estimation from the FT sensor and the skin was approximately 1 cm and, although the magnitude of the disturbances was unknown, the balancer controller showed a stable behavior in both cases.

Footstep feasibility check and re-plan
This experiment is designed to examine the advantages the plantar robot skin has over the ankle FT sensors for footstep re-planning over unknown terrain. The robot is intended to walk on a straight line without knowing the geometry of the terrain. The walking controller is designed for°at terrain and does not consider ankle compliance. If an early contact occurs during the swing-leg phase (Fig. 10), the feasibility of stepping on the detected obstacle is evaluated with a simple rule which only uses the information acquired during the impact. If the obstacle is considered as stable, the robot continues walking over it following the planned footsteps. If the obstacle is considered as not safe, the footstep is re-planed to circumvent the obstacle and the robot continues walking as shown in Fig. 11. To check the step feasibility, the following rules are designed with the di®erent sensors: (a) With the FT sensors, the moment produced at the ankle by the collision M s (Fig. 10) is compared to a feasibility threshold of t . This was de¯ned under the assumption that the impact of a°at foot over°at terrain produces only vertical force. Therefore, if M s ! t the step must be re-planed. (b) With the robot skin, the relation A p A s is compared to a feasibility threshold of A , where A p is the area of the supporting polygon detected at the moment of the impact, and A s is the total foot sole area (measured directly from the robot's foot). This rule was designed under the assumption that a safe foothold must cover at least a certain percentage of the sole area to provide enough surface to generate enough friction for stepping on it. Therefore, if A p A s A the step must be re-planed.
Both rules are analogous. However, the FT sensor cannot directly measure the contact area. Thus, it cannot be used for rule b while the plantar robot skin can be used for both. This is one major advantage of having more information about the ground contacts.
The conditions for step re-plan are represented in Fig. 12. For the cases where the contact provides a large°at surface, both rules a and b result in no re-plan Figs. 12(a) and 12(b). If the contact is produced by a small°at obstacle Fig. 12(c), both rules trigger a step re-plan. All these conditions are correctly evaluated by both, the FT sensors and the robot skin. Nevertheless, when premature contact generates a moment, which is not large enough, the FT sensor fails to detect small surfaces and considers them as similar to a big°at obstacle Fig. 12(b). For example, when a small°a t obstacle is touched with the sole right below the FT sensor Fig. 12(d). On the other hand, the robot skin can measure the small obstacle area and trigger a step re-plan.
Right after the¯rst contact, the robot skin can additionally provide the exact distance to sidestep an obstacle. We can obtain the obstacle's width Á o from the bounding box of the supporting polygon and the clearance distance Á c from the geometry of the foot sole. With this information, we can easily re-plan the step as shown in Fig. 11(b). Figure 13 shows the experimental comparison for an obstacle with the shape shown in Fig. 12(d). The FT sensor fails to detect the unsafe foothold and the walking controller tries to step onto the obstacle causing the robot to fall. The robot skin detects correctly the unsafe obstacle and triggers the step re-plan with the exact distance to sidestep the obstacle and continues walking.
The walking test was performed eight times using four di®erent obstacles as shown in Table 2. For each obstacle, the test was run once using only the FT sensor and once using only the robot skin. The thresholds for the feasibility check rules are t for the FT sensor and A for the contact area. The FT sensor failed with obstacles 3 and 4 which provide a small°at area that was assumed to be large enough to support the robot. Obstacle 4 is made with 3 small spheres which are providing round surfaces similar to stepping on pebbles. This three-point contact condition is not detectable by the FT sensor without exploratory motions. Fig. 13. Experimental evaluation of the step feasibility rules using FT sensors and robot skin. On the lefthand side, the robot skin successfully detects the small area of the obstacle and triggers a step re-plan with the exact distance required to sidestep the obstacle. On the right-hand side, the FT sensor fails to detect the unsafe condition and the walking controller tries to step on the obstacle leading the robot to fall.

Conclusion
The plantar robot skin sensor presented in this work enables the direct measurement of the ground contact forces, together with the spatial information related to them. This plantar sensor can be used to complement ankle sensors in adding redundancy, and combined force perception in a similar manner as human feet.
Plantar tactile sensing proved capable of measuring the ZMP and resultant ground reaction forces similar to ankle FT sensors. These measurements can be used as feedback for balance using the same control techniques as with FT sensors. Additionally, plantar robot skin provides geometrical information from the feet's ground interaction forces. Such information is useful to map the pressure distribution of the footprint, and consequently to construct an accurate shape of the supporting polygon. With this information, balance and walking controllers can adapt online the kinematic constraints to ensure stability without the need for extra exploratory motions or complex constraint estimation methods.
Extra sensing modalities provide information about other terrain conditions or even a preemptive shape of the foothold before foot landing. The spatial resolution of tactile sensors can be used for online step re-planing, as shown in the experimental section. With the extra information about the terrain, it becomes feasible to estimate the size and geometry of small obstacles that can be easily side-stepped instead of walked over.

1950036-16
In future works, we will explore the bene¯ts of exploiting the°exible sensitivity, spatial distribution and additional sensing modalities of the plantar robot skin for developing robust balance and dynamic walking controllers for unknown terrain.