Bio-inspired neuromuscular reflex based hopping controller for a segmented robotic leg

It has been shown that human-like hopping can be achieved by muscle reflex control in neuromechanical simulations. However, it is unclear if this concept is applicable and feasible for controlling a real robot. This paper presents a low-cost two-segmented robotic leg design and demonstrates the feasibility and the benefits of the bio-inspired neuromuscular reflex based control for hopping. Simulation models were developed to describe the dynamics of the real robot. Different neuromuscular reflex pathways were investigated with the simulation models. We found that stable hopping can be achieved with both positive muscle force and length feedback, and the hopping height can be controlled by modulating the muscle force feedback gains with the return maps. The force feedback neuromuscular reflex based controller is robust against body mass and ground impedance changes. Finally, we implemented the controller on the real robot to prove the feasibility of the proposed neuromuscular reflex based control idea. This paper demonstrates the neuromuscular reflex based control approach is feasible to implement and capable of achieving stable and robust hopping in a real robot. It provides a promising direction of controlling the legged robot to achieve robust dynamic motion in the future.


Introduction
The legged locomotion found in humans and animals are energetic, versatile, and robust against perturbations. The functionality of the leg can be separated into three locomotor sub-functions, which are stance (axial leg function), swing (rotational leg function), and balance (trunk posture control) [1]. The stance sub-function is to support body weight and create the centre of mass bouncing behavior (e.g. during walking, running, etc). Although human leg structure and the locomotion control are complex, highly simplified template models emphasizing the elastic stance leg function can describe and reproduce some basic characteristics of human walking and running gait [2][3][4]. Hopping can be considered as a primitive motion which focuses on the stance subfunction. A better understanding of how hopping motion is generated and controlled can help us further recognize the basic principles of human locomotion.
In general, the models proposed for explaining the stance leg function can be divided into two levels: mechanical level and neuromuscular level. Mechanical level models simplify the neuromuscular properties of major muscle groups in the stance leg and represent the stance leg function as a mechanical spring. For instance, a constant stiffness prismatic spring used in the spring loaded inverted pendulum (SLIP) template model is one of the simplest representations of the stance leg sub-function for dynamic locomotion (e.g. hopping, walking, running, etc) [2,4]. However, the constant stiffness spring model cannot describe the perturbation behaviors because it is energy conservative. Therefore, a few extended SLIP models were proposed. For instance, the ESLIP [5] model and VSLIP [6] model modulate the spring stiffness and/or rest length during the locomotion to regulate the system energy.
The neuromuscular level models include the muscle properties (e.g. force-length and force-velocity relationship) and neural reflexes (e.g. muscle force, length, Bio-inspired neuromuscular reflex based hopping controller for a segmented robotic leg and velocity reflex). For instance, a two-segmented model with a point mass and a Hill-type extensor muscle was proposed in [7] to explain the functionality of the positive force feedback in bouncing gaits. Recently, this model was used with linear reflex compositions for investigating the sensor-motor maps [8]. A further simplified model which ignores the leg geometric properties was used for demonstrating the role of intrinsic muscle properties and reflexes in generating stable hopping [9, 10]. With multi-segment and multi-muscle complex models, Geyer et al showed that human-like stable and rich bipedal locomotion can be achieved with neural networks emphasizing muscle reflexes [11,12]. The neuromuscular reflex based controller proposed for these complex models have also been implemented on lower limb prostheses [13,14] and exoskeletons [15,16] to assist human walking.
Compared to the mechanical level models, neuromuscular level models provide deeper insights on the potential benefits of muscle properties and the neural reflex control for generating bouncing gaits (e.g. hopping, walking, running, etc). Although the hopping motion has been extensively studied with the simulation models mentioned above, it is unclear if the control concept is applicable and feasible for controlling a real hopping robot because most of the time the physical properties of the leg (e.g. leg inertia, foot-ground collision model, damping in the joint, etc) are ignored in the simplified models. Therefore, in this paper, we focus on investigating if the bio-inspired neuromuscular reflex based controller can generate stable hopping in both the realistic simulation model and the real robot.
Lots of legged robots emphasizing the bouncing leg function were developed during the last few decades. They can be divided into three groups based on the actuator types: (i) serial elastic actuators (SEAs, e.g. [17][18][19]), (ii) soft actuators (e.g. pneumatic actuators [20][21][22]), (iii) quasi-direct-drive (QDD) actuators (e.g. electric motors with low gear ratio gearboxes [23][24][25][26]). The intrinsic impedance in the SEAs and soft actuators can be beneficial for reducing the energetic cost and impact forces. However, it is hard to modulate the intrinsic impedance during the motion. For the QDD actuators, we can control the impedance and emulate different actuator dynamics (e.g. springs, Hill-type muscles, etc) with a virtual model control approach [27]. In addition, QDD actuators can potentially transfer kinetic energy back to electric energy by regenerative braking [23]. Therefore, the QDD actuators were chosen for the robotic leg in this study.
The goal of this paper is to develop a low-cost robotic leg for hopping and demonstrate the benefits of the bio-inspired neuromuscular reflex based controller in both simulation and hardware experiments. In the following section, we first introduce the robot hardware design and simulation environment (section 2). Then the bio-inspired neuromuscular reflex based hopping controller is presented in section 3. Sections 4 and 5 present the design of the experiments and the results of the simulation and hardware experiments which demonstrates the feasibility and benefits of the bio-inspired controller. Section 6 discusses the results and gives insights about this study and future work.

Mechanical design
The single leg robot consists of two brushless direct current (BLDC) motors (E8318-120KV, Hymotor, China) which control the hip and knee joint separately in the sagittal plane (figures 1(A) and (B)). The leg is serially actuated. To minimize the leg moment of inertia, the hip and knee motors are co-axially located at the top of the thigh. Knee joint is coupled with the knee motor shaft by a rope-pulley mechanism (gear-ratio 4:1). In order to avoid high mechanical stiffness and friction in the transmission chain, no gearbox is used for the motors. The direct-drive for the hip and the QDD for the knee actuation ensures the transparency between the motor and the external environment [24]. This makes it possible to achieve relative good torque control performance by motor current sensing (without any force/torque sensors).
Carbon fiber tubes were chosen as the thigh and shank segments to withstand high loads while keeping the weight and the moment of inertia low. All other mechanical parts, except the screws and bearings, were 3D printed with plastic materials (PLA and ABS) to further reduce the robot weight and keep the cost low. The robot hip is fixed on a 1D linear guide rail so that we can focus on the leg extension function for hopping. The total mass of the robot is 2.8 kg. The thigh and shank segment lengths are 0.27 m.
To test if the design is also suitable for bipedal hopping, a bipedal robot was built by connecting two single robotic legs with a trunk (figures 1(C) and (D)). This is important for multi-legged robots because the individual leg function in the robot may be different between both legs which would result in performance deficits (e.g. asymmetric leg function during hopping). Similar to the single leg robot, the trunk motion is also constrained in 1D (up and down movement) by a linear bearing. The total mass of the bipedal robot is 6.2 kg. To keep the maximum knee joint torque to robot mass ratio similar as the single leg robot, the gear-ratio of the bipedal robot leg knee motors (ropepulley mechanism) is designed as 5:1. The rest of the mechanical design of the leg is the same as the single leg robot.

Control system architecture
In the single leg hopping robot, each motor is equipped with an incremental encoder (AMT102-V, CUI, USA) to measure the motor angle. The encoders are used for both low-level current control (motor driver MTVESC50A, Maytech, China) and high-level reflex based control. The motor driver runs low-level field oriented control (FOC) at 20 kHz. A force-sensing resistor is mounted underneath the foot to detect if the robot is in the stance phase or flight phase. An ESP32s microcontroller reads all the sensor data and sends the data to the high-level controller. The high-level controller is implemented in realtime at 1 kHz with Matlab Simulink xPC target (Matlab R2018a, Mathworks, USA). The motor drivers and the microcontrollers are interfaced with the xPC target machine through EtherCAT communication bus at 1 kHz. Except the xPC target machine, which is off-board to the robot, all the components described here are onboard. The overview of the control system architecture is illustrated in figure 2.
The bipedal robot has the same control architecture as the single leg robot. The main difference is the motor driver. In order to generate higher torques in the motors, the motor drivers in the bipedal robot (FSESC6.6, Flipsky, China) are able to deliver about twice the amount of peak current compared to the motor drivers in the single leg robot.

Simulation
We built a physical simulation model in MuJoCo [28] based on the robot CAD design. Each part of the robot was weighted before the assembling. The moment of inertia of every part was calculated based on the measured weight by assuming the density  Bioinspir. Biomim. 15 (2020) 026007 was homogeneous. In order to get realistic footground contact model parameters and joint damping coefficients for the simulation, the real robot experiment hopping height data (range from 0.02 m to 0.12 m) were used to manually tune those parameters. Initially, by assuming a relatively stiff ground and low joint damping coefficients in the simulation, the simulation model showed higher hopping height than the real robot. Then we manually tuned those parameters iteratively till the simulation model exhibited similar hopping behaviors and hopping heights as the real robot. The MuJoCo physical simulation model runs at 10 kHz to ensure stable and accurate simulations. The control rate in the simulation was set to 1 kHz to match the control rate of the real robot.

Muscular model
The Hill-type muscle-tendon unit (MTU) model ( figure 3 [11]), is used in this paper. The MTU consists of a series elasticity (SE), a contractile element (CE), a parallel elasticity (PE), and a buffer elasticity (BE). The generated CE force F ce is computed by the muscle activation (A), maximum isometric force F max , forcelength (f l ) and force-velocity ( f v ) relationships of the CE [7,11]: where negative CE velocity v ce denotes the concentric movement. The width w and residual force factor c define the shape of f l . The eccentric force enhancement N and the shape factor K define the f v . The MTU force F m can be computed as where and l se is the SE length, l slack is the SE rest length, ε ref is the reference strain of the SE, ε pe is the reference strain of the PE, l opt is the optimum length, l min is the BE rest length, ε be is the BE reference strain. All the parameter values are listed in the appendix table A3. They are taken from previous studies [7,11] and adapted to the robot.

Neural reflex
The muscle excitation-contraction coupling (ECC) is modelled as [7]: is the muscle activation, and τ is a time constant. We assume a linear relation between S and the sensory feedback P (i.e. F m , l ce , v ce ): where S 0 is the constant stimulation bias, G is the gain factor for different feedback signals, and ∆t is the sensory feedback time delay. S(t) is saturated in the range of [0, 1]. This linear assumption is a common approach to represent the sensory feedback mechanism [7,8,11]. In the implementation, each sensory feedback P signal (i.e. F m , l ce , v ce ) is normalized. More specifically, S(t) for each individual feedback pathway (i.e. force feedback (FFB), length feedback (LFB), and velocity feedback (VFB)) is computed as: where F n m = F m /F max , l n ce = l ce /l opt , and v n ce = v ce /v max . G F , G L , and G V denote the gain for force, length, and velocity feedback pathway, respectively. Compared to the other approaches [7,9,10], the length and velocity offsets are not taken into account in the feedback pathways because finding the optimal control parameters for a certain motion is not the aim of this paper. Here, we aim at demonstrating the feasibility and potential benefits of the neuromuscular reflex control concept on the robotic hardware system.

Hopping control scheme
The hopping controller is separated into flight, stance, and collision phase. The overview of the control scheme for a single leg is shown in figure 4. The individual leg control scheme of the bipedal robot is the same as the single leg robot.

Stance phase
In the stance phase, the hip motor is set to free (desired current set to 0 A) while the knee motor is controlled as a virtual Hill-type MTU (figures 3 and 4). The virtual MTU only produces knee extension torque during the stance phase. The virtual MTU length l mtc is calculated as where θ k mot , c and r mtc are the robot knee motor angle measured by the encoder, the gear ratio, and the virtual MTC moment arm, respectively. c is 4, which is the ratio between knee joint pulley diameter and the knee motor pulley diameter. r mtc is set to 0.04 m in this study. The muscle activation A is calculated based on the neural reflex controller. Virtual muscle states (i.e. muscle F, l, v) are computed based on the muscular model given l mtc . The knee motor desired current I k mot is calculated and sent to the motor driver based on the motor model given the desired virtual MTC force.

Flight phase
The flight phase controller is used to prepare the leg with an appropriate posture for the next landing. Here, a simple PD position controller with fixed target knee and hip angles is used during the flight phase.
The PD values are manually tuned so that the robot can not only achieve the desired posture before next touch down (TD) but also have low effective joint compliance to avoid high impact forces at TD. In this study, the hip and knee desired joint angles during the flight phase are set to 20 • and 40 • , respectively. The controller switches from stance phase control to flight phase control if the knee angle reaches 40 • or the force sensor underneath the foot detects no contact forces.

Collision phase
The collision phase is defined as a very short time duration t c after TD. Both hip and knee motor are position controlled with relatively low P but high D values to absorb the impact energy during the collision. Due to relatively high P values in the hip and knee motor flight PD position controller, the robot shank starts rebounding/oscillating if the robot lands on stiff ground. The collision phase control can absorb the impact and prevent the undesired oscillations after landing. We set t c as 20 ms because it is much shorter than the muscle reflex time (around 200 ms for hopping) while the shank rebound can still be eliminated.

Implementation
The hopping control scheme was implemented with Matlab Simulink (2018a) in both simulation and the real-time controller for the real robots at 1 kHz control rate. The simulation model foot-ground contact parameters and the joint damping coefficients were manually tuned (see detailed description in section 2.3). The gains for different reflex pathways will be described later for each experiment. The parameters for the PD position controller during the flight phase and the collision phase were tuned by hand. The desired motor currents were saturated due to the motor driver hardware limitation. The maximum motor current was 50 A and 100 A for the single leg robot and the bipedal robot, respectively. A lithium polymer battery was used to delivery high enough peak currents to the motor drivers. Both simulation and the real-time controller have the same parameter values because the simulation model is very close to the hardware setup.

Experiments
This section presents the design of the experiments in both simulations and robot hardware systems. First, in order to show if stable hopping can be achieved with the proposed bio-inspired hopping controller, the return maps of the individual muscle reflexes are computed. Then the robustness of the muscle force feedback is demonstrated by comparing the return maps of different robot model properties and ground properties. Finally, the robot hardware experiments were conducted to verify the simulation model and prove the feasibility of the hardware design and implementation.

Return maps
In order to investigate the influence of the individual muscle feedback pathway (i.e. force, length and velocity) on the hopping motion, we computed return maps for each feedback pathway with brute force in simulation by dropping the robot from 0.005 m to 0.3 m height with a step size of 0.005 m. The hopping height zero is defined as the robot hip height at TD. The range of the feedback gain was chosen as [−5, 5] because the robot can generate both stable hopping and landing (not rebound) motion within this range.

Robustness of the FFB
To explore the robustness of the neuromuscular reflex based controller, we analyzed the effects of parameters of the model and the environment on the return map. For the robot parameters, we increased the robot mass from the original mass m to 1.6 × m with a step size of 0.2 × m. We also investigated the influences of the rope impedance on the hopping performance. We changed the rope from the original rigid configuration (stiffness 8 × 10 7 N m −1 , damping coefficient 500 Ns m −1 ) to stiff (10 6 N m −1 , 5 Ns m −1 ), moderate (10 5 N m −1 , 5 Ns m −1 ) and soft (10 4 N m −1 , 5 Ns m −1 ) configurations. In this case, the rope can be considered as an elastic component in serial of the virtual MTU. For the environment parameters, we decreased the ground impedance from the original impedance (scaled as 1) to 0.4 with a step size of 0.2. Here we focus on how the FFB return map changes. Based on the original FFB return map (shown in figure 5(A)), the robot will not rebound if the gain is smaller than 0.5. And the robot hopping height will be saturated if the gain is larger than 2.5. Therefore, the FFB return map with the gain in [0.5, 2.5] (step size 0.1) was computed for different body mass, ground impedance, and rope stiffness conditions.

Robot hardware demonstration
The bio-inspired neuromuscular reflex based controller (figure 4) was implemented on the real robot. To validate the simulation model, the stable hopping height was compared between the real robot and the simulation model with different FFB gains. The real robot hopping height was measured by a high-speed camera based motion capture system (Qualisys, Sweden). Ten continuous stable hopping data in each FFB gain were used to calculate the mean and the standard deviation of the real robot hopping height. The robot does not have a stable hopping pattern if the FFB gain is smaller than 1.1. The pelvis range of motion is limited by the linear guide length which is 0.4 m. In addition, considering the strength and durability of the 3D printed plastic parts (e.g. knee shafts, motor pulleys, etc), we only did the robot hopping experiments with the FFB gain from 1.1 to 1.8 with a step size of 0.1.

Results
Because the results of the single leg robot and the bipedal robot are very similar, we only present the results of the bipedal robot. In the FFB, the robot will not rebound even with the 0.3 m dropping height if the G F is less than zero.

Return maps
The FFB shows superior features (e.g. smoothness, range of stable hopping height, stability) compared to the LFB and VFB (figure 5). Therefore in the following analysis, we will only focus on the FFB.

Robustness of the FFB
The FFB return maps with different body mass, ground impedance and rope stiffness conditions are shown in figure 6. Compared to the FFB return map in the normal condition ( figure 6(A)), the robot maximum stable hopping height decreases with the increasing body mass or decreasing ground impedance. Compared to different ground impedance and rope stiffness conditions, the return maps in the different body mass conditions are more similar to the normal condition in terms of the stable hopping height and the shape of the map. For instance, the maximum stable hopping height only dropped 0.016 m from the normal condition to the 1.6 times body mass condition. For the different ground impedance conditions, the robot shows stable hopping solutions. However, the maximum stable hopping height drops a lot with decreasing ground impedance compared to the normal condition. The maximum stable hopping height for normal, 0.8 impedance, 0.6 impedance, and 0.4 impedance are 0.26 m, 0.215 m, 0.145 m, and 0.102 m, respectively. The return map in the stiff rope condition is almost the same as the normal condition. It gets a bit unstable in the moderate rope stiffness condition. In the soft rope condition, there is no stable hopping solution in the hopping height from 0 to 0.3 m.

Robot hardware demonstration
Both the single leg and the bipedal robot can achieve stable and robust hopping with appropriate feedback gains (figure 7, see the supplementary video for more details (stacks.iop.org/BB/15/026007/mmedia)). The real robot hopping behavior is very similar to the hopping behavior we observed from the simulation. The comparison between the stable hopping height of the real robot and the simulation model with different FFB gains are shown in figure 8. The hopping height of both simulation and the real robot experiments increases with higher FFB gain. The maximum hopping height difference is 0.008 m at G F = 1.6. The difference between the simulation and experimental hopping height are 0.004 ± 0.002 m (mean± standard deviation). This confirms that the simulation model is valid. Note that the simulation parameters  (i.e. foot-ground contact parameters and joint damping coefficients) were manually tuned to fit the data (see detailed description in section 2.3). The measured motor current and the electric power consumption of the single leg robot and the bipedal robot during hopping at the hopping height of 4 cm are shown in figure 9. The single leg knee motor current is saturated at 50 A (due to the motor driver hardware limitation) during the mid-stance phase. We need to increase the motor driver maximum current and/or the gear ratio if we want to achieve higher hopping height. The hip motor peak current in the bipedal robot reaches around 90 A at the beginning and the end of the flight phase. This is because of relative large P value in the PD control of the hip motor. The knee motor of both the single leg robot and the bipedal robot show regenerative braking at the beginning of stance phase and the beginning of flight phase. The average electric energy consumption for each knee motor is 40 J per hop. This is relatively high because the motor operated in low speed high current (inefficient, high heat loss) condition. It is required to avoid gear boxes which would cause additional moment of inertia and undesired side effects due to friction. The regenerated energy is 30% of the consumption energy in each knee motor. The hip motors do not have regenerative braking phases.

Discussion
In this paper, we presented a low-cost robotic leg design which is capable of demonstrating the bioinspired neuromuscular reflex based hopping controller. Based on the return maps from the simulation results, we found that the stable hopping can be achieved with both positive force and length reflex while the velocity reflex could result in unstable behaviors. The force reflex based control is more stable than the length reflex based control. The robustness of the force reflex based control was investigated by varying the model parameters (body mass and rope stiffness) and the ground impedance in the simulation. The robot hardware experimental results show that the bio-inspired controller is feasible to implement and capable of achieving stable and robust hopping with the proposed low-cost robotic leg.
Recently, it has been more and more popular to use QDD electric motors in legged robots for dynamic locomotion [23][24][25][26]29]. The planetary gearbox with low gear ratio is often used in QDD actuators because the motor direct torque output is small. Here, instead of planetary gearboxes, the rope-pulley mechanism was used in our robotic leg to achieve similar effects as the gearbox. The pulleys used in the robot were 3D printed with plastics. This reduces the mass and the cost of the robot. In addition, it also enables fast prototyping and testing with different gear ratios.
Compared to the robots which use customized high torque density motors (e.g. MIT Cheetah [23] and MIT Cheetah 3 [30], the torque output of the motor used in our robot is relatively low because we are using off the shelf motors and motor drivers. For instance, the max knee torque of our bipedal robot is 40 N m while the max torque of the MIT cheetah 3 actuator is 230 N m). Therefore the robot was built with 3D printed plastic parts and carbon tubes to minimize the weight and the cost. The robot experimental results prove that the motor is capable of delivering enough torque to generate hopping motion. The robot has been tested for more than 1000 hops with different hopping heights (from 2 cm to 12 cm). No mechanical failure occurred. This demonstrates that the robot design is robust and can be used as a test platform for investigating dynamic legged locomotion. Because we used 3D printed parts and off the shelf motors and motor drivers, the total cost of one robotic leg is less than 600 Euros (excluding the battery and the PC for controlling the robot). This low-cost design makes it more accessible for research and education in legged robotics.
Compared to the mechanical spring or springdamper based virtual model control (e.g. [26,[31][32][33]), the proposed bio-inspired control inherits the intrinsic muscle dynamics and neuroreflex properties which can be beneficial for stabilizing the motion [9, 10] and simplifying the high level control by muscle reflexes (FFB, LFB and VFB) [8,11,12]. The return maps of the FFB, LFB and VFB demonstrate that both FFB and LFB can result in stable hopping motion and can be used for controlling the robot hopping height ( figure 5). This is in line with the findings from the simplified point mass simulation models [7,8,10]. These findings also indicate that, for the stance phase control, the neuromuscular reflex properties play a more important role than the leg inertia properties (a point mass model was used in [7,8]) and the leg geometry (a prismatic leg was used in [10]) in shaping the hopping behavior. Further analysis on the change of the FFB return maps with different body masses and ground impedance (figure 6) highlights the robustness of the neuromuscular FFB based hopping controller. Therefore the proposed bioinspired neuromuscular reflex based control approach can potentially be implemented on other legged robots to achieve bouncing gait without too much tuning of the control parameters. Note that the length and velocity offsets were not taken into account in the feedback pathways in this study. In the future, investigating the influences of the offsets on the return maps could help us find better hopping behavior with LFB and VFB.
Adding an elastic component in serial of the actuator can be beneficial in terms of energy efficiency and stability for dynamic legged locomotion [17,34]. However, the results of different rope stiffness (with only FFB, figure 6) show that the additional elastic component in serial of the virtual MTU can lead to unstable hopping if the serial component is too soft. Other control approaches (e.g. combining different reflexes, modulating feedback gains during the stance phase) are required to have a stable hopping in this condition.
In this study, we focused on the neuromuscular reflex based hopping controller during the stance phase. A PD position control was used in the flight phase. This results in high current peaks in the motors at the beginning and the end of flight phase. In future research, the robot flight phase PD control could be replaced by the neuromuscular reflex control to solve this issue. And the current 1D hopping controller can potentially be transferred to 2D hopping (i.e. hopping forward and backward) by tuning the flight phase control. Besides, the MTU model used in this study could potentially be replaced by a simplified model with fewer parameters (e.g. linearized Hill model from [35]) in the future. This could be helpful in optimizing the hopping performance.
Another limitation of this study is that we investigated the hopping with individual reflex pathways (i.e. force, length, and velocity). With a simplified simulation model, it has been shown that the hopping performance (e.g. hopping height, efficiency, robustness, etc) can be improved by linear combinations of different reflexes [8]. In addition, combining the muscle reflex control and the feed forward stimulation could help to generate and stabilize hopping motion [10]. These different combinations could be investigated in the future.