A biohybrid fly-robot interface system that performs active collision avoidance

We have designed a bio-hybrid fly-robot interface (FRI) to study sensorimotor control in insects. The FRI consists of a miniaturized recording platform mounted on a two-wheeled robot and is controlled by the neuronal spiking activity of an identified visual interneuron, the blowfly H1-cell. For a given turning radius of the robot, we found a proportional relationship between the spike rate of the H1-cell and the relative distance of the FRI from the patterned wall of an experimental arena. Under closed-loop conditions during oscillatory forward movements biased towards the wall, collision avoidance manoeuvres were triggered whenever the H1-cell spike rate exceeded a certain threshold value. We also investigated the FRI behaviour in corners of the arena. The ultimate goal is to enable autonomous and energy-efficient manoeuvrings of the FRI within arbitrary visual environments.


Introduction
The interest in using biohybrid robotic systems [1] to study general principles underlying biological sensorimotor control has steadily increased over the last two decades. One example of biological sensorimotor control is the ability to avoid collisions. It is a necessary requirement for efficient locomotor activity in any aquatic, terrestrial or airborne animal. Blowflies, which are amongst the most manoeuvrable flying insects [2,3], mainly use visual cues to estimate relative distance and to perform collision avoidance behaviour in various environments at high speed [4] and with a limited energy budget [5,6]. Vision, however, is the only one modality that supports flight and gaze control in flies [7,8] and is modulated by signals from other sensory systems [9][10][11]. The principles underlying multisensory integration, however, are not yet fully understood. Progress in this area may be useful for technical applications with high energy efficiency requirements, where biological solution evolved on a phylogenetic time scale, could inspire novel engineering approaches to autonomous driving or mobile robotic platforms [12][13][14][15][16][17].
Collision avoidance and escape behaviour based on looming detection have been studied for years [14,[18][19][20]. Basic collision avoidance requires knowl-edge of the distance to obstacles in the environment. By continuously estimating obstacle distance in the brain during forward movements, a threshold of spike rate could be set to trigger a collision avoidance manoeuvre [20].
How does the brain of the blowfly estimate its distance from obstacles? Some predatory insect species, such as praying mantis, possibly dragonfly and robber fly, have a region of binocular overlapping within the visual field, which could provide stereopsis-based distance information [21]. Insects with less overlapping visual fields of their two eyes would have to use alternative strategies, for instance, motion parallax or optic flow [22].
Translational movements produce motion parallax and generate optic flow which contains relative distance information [23]. In blowflies, some of the so-called lobula plate tangential cells (LPTCs) have been suggested to function as matched filters for optic flows generated during translation [24]. In particular, the HSN and HSE cells-the 'Northern' and 'Equatorial' elements of the horizontal system (HS)-have been and still are implicated in the measurement of yaw rotations. The idea that the HSN-and HSE-cell signal relative distance [25,26] is relatively new. The 'Southern' HS-cell, HSS-cell, on the other hand, was previously suspected to sense translation because it does not receive rotation-specific input from the contralateral eye and has a receptive field directed towards the ground where distances are relatively close and translational optic flow dominates [22,27]. Both the HSE-and HSN-cell together with the spiking H1-and H2-cells form a heterolateral network that enables the distinction between rotation and translation [22,28].
Besides vision, several other sensory modalities are involved in blowfly sensorimotor control, including the ocelli-another visual system [10]-and mechanosensory systems [11]. But the neuronal mechanisms that ultimately trigger collision avoidance manoeuvres are not entirely understood as yet. A recent study suggested that the H1-cell spike rate may signal relative distance information if the fly kept the turning radius of its trajectory at a fixed small value [29]-in which case closer distances to objects result in higher activity. The animal could easily achieve a constant turning radius by appropriate activation of steering muscles in its flight motor which control the kinematics of the left and the right wing. So, in principle, there should be a way to initiate collision avoidance manoeuvres whenever the activity of the cell exceeds a certain threshold value.
Here we further study the distance-dependent responses of the H1-cell in Calliphora vicina, to selfmotion induced optic flow. To this end, we fixed the fly to a miniaturized recording platform that was mounted on a two-wheeled mobile robotic platform [30] and monitored the spiking activity of the cell using extracellular recording techniques. Our experiments were performed under open-or closed-loop condition. In open-loop conditions, the trajectory of the robot was pre-programmed, and the H1-cell responses were studied under various kinematic and visual parameters without feeding the signals back to the controls of the robot. During closed-loop experiments, whenever the spike rate of the H1-cell exceeded a certain distance-dependent threshold value as the FRI (fly-robot-interface, figure 1) approached the wall of our experimental arena, a collision avoidance manoeuvre was triggered. Effectively, in closed-loop experiments, the H1-cell used as a visual sensor, the outputs of which was controlling the robot trajectory. At this stage of our studies, we confined ourselves to recordings only from the H1-cell that received its input through the eye facing the wall of the arena.

Experiment setup
The FRI was operating in an illuminated arena with dimensions of 1800 × 500 × 520 mm covered by black fabric on top. The wall on the left side was cladded with a striped pattern (spatial wavelength = 30 mm, pattern contrast = 84%) covering a height of 520 mm. LED light strips (length: 1000 mm) were mounted opposite to the striped pattern at a height of 540 mm. A camera with a fisheye lens (GoPro HERO3+, GoPro Inc., San Mateo, CA, USA) was fixed above the arena at a height of 540 mm for recording the trajectory of the robot.
The FRI firmware was programmed and stored on a microprocessor (Mbed LPC1768, Arm Holdings plc, Cambridge, UK) on board of a two-wheeled robot (m3pi, Pololu Corporation, Las Vegas, NV, USA), which was carrying a miniaturized extracellular recording platform. A blowfly was mounted on the recording platform in the centre of the rotation axis of the robot, approximately 150 mm above the ground. A tungsten electrode was used to record extracellular H1-cell signals which were amplified by a small customised preamplifier with a nominal gain of 10k. The amplified signals were digitised at a rate of 5 kHz by the microprocessor where the neural signals were processed and converted into command signals for controlling the robot when operating under closedloop conditions. The data was also recorded by a data acquisition card (NI USB-6215, National Instruments Corporation, Austin, TX, USA) at a rate of 20 kHz through wire connection and stored on a PC hard disk for off-line analysis.
The FRI was pre-programmed (as the flowchart in figure 2) to move forward in an oscillating movement with a turning radius of 5 cm (figure 3(D)). The speed of the robot was limited by a 30% Pulse Width Modulation (around 30% of its full speed 1 m s −1 : ≈0.3 m s −1 ). The oscillations resulted in an alternating pattern of increased and decreased spiking activity in the H1-cell (figures 3(B) and (C)). In closed-loop conditions, a bias was added on top of the oscillating forward movement, where the firmware added an extra time period of rotating the right wheel, consequently steering the robot to the left and thus towards the wall. A collision avoidance manoeuvre was triggered whenever the spike rate exceeded a threshold value, the level of which was based on the previous characterization of the linear relationship between spike rate and wall distance at a fixed turning radius [29]. The bias mentioned above gradually drove the FRI towards the left wall which allows us to repeatedly monitor the reliability at which collision avoidance manoeuvres were triggered. The sampling frequency of the neural signals was set to 5 kHz on the robot microprocessor. The H1-cell activity was sampled during each turn where the rotating centre was on the same side of the recorded eye, while the spike rates were calculated during each opposite turn. During a collision avoidance manoeuvre, the robot rotated on the spot for 200 ms away from the wall.
The video footage showing the FRI in position was pre-processed by software (Adobe Lightroom, Adobe System Inc, San Jose, CA, USA) to remove fish-eye lens distortion. To obtain the trajectories of the FRI, we used software based on the open source physics framework (Tracker, physlets.org) and Python (OpenCV), which allow us to follow tracking markers on the robot frame by frame.

Blowfly dissection
The dissection of the blowfly, Calliphora vicina, for electrophysiological recordings on the FRI has been previously described (e.g. [29]). In brief: 4 to 11 d old female blowflies were chosen for the experiments. The legs and proboscis of the animal were removed. Open wounds and wing hinges were sealed and blocked with bee wax, respectively. The head of the blowfly was adjusted according to the deep pseudopupil [32] and fixed to a fly holder by using bee wax. The thorax was bent down and fixed with bee wax to the fly holder as well, to expose the back of the head. The cuticle on the Flow chart of the FRI firmware. The robot was pre-programmed to perform an oscillatory forward movement (with a bias towards the wall), while the interrupt service routine (ISR) was running in the background, sampling neural signals and updating spike rates (modified from [31]). Note that: 'turn away from a wall' is a rightward turn with a non-zero radius (part of the oscillatory forward movement), while 'collision-avoidance manoeuvre' is a rightward turn with zero radius, i.e. a turn on the spot. back of the head was cut open. Fat and muscle tissue were removed to provide access to the lobula plate. The dissection took place under optical magnification using a stereomicroscope (Stemi 2000, Zeiss©). Ringer solution was added to the lobula plate from time to time to prevent desiccation (for recipe see [33]).
H1-cell signals were extracellularly recorded using sharp tungsten electrodes (of ~3 MΩ impedance, product code: UEWSHGSE3N1M, FHC Inc., Bowdoin, ME, USA). The signal-to-noise-ratio (SNR) was accepted only if it was no less than 2:1, which was defined by the peak amplitudes of the spikes of the cell divided by the peak amplitudes of the background noise.

FRI wall following and collision avoidance in the experimental area
The trajectory shown in figure 3 was obtained by a frame-by-frame analysis of the video footage using the 'Tracker software'. Movements of the left (port) and right (starboard) markers mounted on the FRI as well as the average trajectory are indicated by red, green, and white lines along the wall of the experimental arena plotted on the first frame of the video sequence. After switching on the on-board microprocessor, the FRI started its oscillatory forward trajectory at a turning  figure 2 for the underlying control algorithm). The video footage can be accessed at: https://youtu.be/g902ZEw_YcQ (modified from [31]). radius of 5 cm. The extra time period added to the rotation of the right wheel (the bias) gradually drove the FRI towards the left wall. As soon as the spike rate of the H1-cell exceeded the pre-set threshold value, the FRI performed a collision avoidance manoeuvre, i.e. turning right on a spot, preventing a collision into the wall. Figure 2 shows the control flow chart underlying this behaviour and figure 3(D) itemizes the different phases, respectively. Example traces of neuronal activities and results of an individual experiment are given in figures 3(A)-(C), where the FRI collision avoidance manoeuvre was triggered 3 times within approximately 12 s.
The experiment was repeated ten times, where the FRI was reaching the end of the arena without collision in 9 out of 10 trials.

Spike rate, wall distance and robot orientation
We computed the mean spike rate across all the average spike rate of the H1-cell during each 400 ms oscillatory turn resulting in stimulation of the neuron with motion in its preferred direction (PD) obtained across the ten trials of the experiments. These data allowed us to quantify the relationship between H1-cell activity, wall distance and orientation of the FRI relative to the wall of the experimental arena. The resulting mean spike rates shown in figure 4 are correspondingly plotted as a function of mean instantaneous wall distance and robot orientations. A flat surface function was fitted to the scattered data points producing a gradient heat map of H1-cell activity. The data indicate that the spike rate is higher when the FRI is closer to the wall and the frontolateral part of the H1-cell receptive field with its high motion sensitivity is oriented towards the visual pattern.
Next, we compared our closed-loop data with the previous results obtained under open-loop conditions which revealed a linear relationship between spike rate and wall distance for the range between 10-25 cm [29]. For a turning radius of 5 cm, we analysed the mean spike rates measured during FRI orientations in the range of −45 to 45° relative to its trajectory, where 0° corresponds to an alignment of the forward direction of the FRI in the forward direction towards the end of the experimental arena. The chosen range of orientations is similar to the FRI orientations pre-programmed for open-loop experiments. Those data were plotted against wall distance, binned into five classes (0-10, 10-20, 20-30, 30-40, 40-50 cm). As shown in figure 5(A), the mean spike rate is proportional to wall distances, but only up to a distance of 35 cm, beyond which the H1-response become distance independent.
The slope of the function obtained under closedloop conditions ( figure 5(A)), which-at a turning radius of Rt = For a turning radius of 5 cm, the H1-cell responses provide a sufficiently broad output range of distancedependent spike rates to use a simple threshold criterion for triggering a collision avoidance manoeuvre of the FRI.

Collision avoidance in corners (right and acute angles)
While the FRI produced a 9/10 success rate when following the wall of our experimental arena, we were wondering whether the system would be capable of a collision-free negotiation of corners. An extra patterned wall was added to the far end of the arena to form a corner. The angle of the corner was adjusted to 90° (right angle) or approximately 60° (acute angle).
The results show that the FRI was able to successfully avoid collisions when engaging a corner of 90° in 10/10 trails using the same control algorithm implemented to follow the wall of our experimental arena. The performance dramatically dropped to 3/10 successful trails when the FRI had to negotiate a corner of 60° (see figure 6).

Collision avoidance performance
The success rate at which our FRI reached the end of our experimental arena without colliding into its wall was 90%, where the only failure was a trail in which the robot scratching the wall, but still reached the end of the arena. In comparison, only 5/12 trails were successful in the previous experiments in which a simple 'bangbang' control algorithm was implemented to steer the FRI [34]. The previous bang-bang algorithm was controlling the time period of the leftwards turns by incrementing a spike counter up to a set value. During turns at a close distance to the wall, the set value was reached earlier, and the FRI trajectory switched to a turn in the opposite direction; i.e. away from the wall. While during pre-programmed rightward turns the time period was always kept the same. In cases where the fly was further away from the wall, the spike rate was lower and it, therefore, took longer to reach the set spike count during the leftwards turn, compare to the fixed time period during the rightwards turn, thus steering the FRI towards the wall [34].
Later, we found that the spike rate of the H1-cell contains distance information, given the turning radius of the FRI was known. The turning radius of the FRI depends on the proportion of the translational and rotational self-motion component of the system: a zero turning radius corresponds to a pure rotation, while an infinite turning radius is equivalent to a pure forward translation [29]. To understand the significance of the turning radius on the spiking activity of the H1-cell, we have to consider that during a pure translation the entire receptive field of the cell is stimulated by visual motion in its anti-preferred direction, resulting in a reduction of its spike rate towards zero. During pure rotation to the left, the receptive field of the H1-cell in the left lobula plate would be exclusively stimulated in its preferred direction, resulting in a strong increase in spike rate. For any turning radius in between zero and infinity, there will be a certain ratio of those parts of the receptive field of the H1-cell that are stimulated in the preferred and the anti-preferred direction which determines the level of spiking in the H1-cell during turns [29].
The new control algorithm took advantage of an active component-the oscillatory movements of the FRI at a known and constant turning radius. This enabled the interpretation of the spike rates as a linear function of wall distance, which not only simplified the computational overhead, but also increased the acc uracy of the control. In the experiments we report here, we have used a simple threshold algorithm to trigger a pre-programmed collision avoidance manoeuvre, consisting of rotation on the spot away from the wall for 0.2 s. In the future, the wall distance could be estimated more accurately from a linear model that takes into account the proportion of motion in the preferred and anti-preferred direction of the H1-cell to predict its activity, based on which the rotation angle of the collision avoidance manoeuvre may be adjusted depending on the current wall distance [29].  [29]. Note the similarity of the slope and plateau of the blue curves in (A) and (B), both obtained for a turning radius of Rt = 5 cm (modified from [31]).
The ultimate goal of our work is operating the FRI in an arbitrary environment without colliding with any obstacles, while studying biological principles underlying multi-sensory locomotor control. We broke down this task into three stages: (i) simple wall following without collision in a controlled visual environment. (ii) the ability of the system to negotiate corners and (iii) avoiding collisions of the system with obstacles along the trajectory of the FRI. After we have now achieved two out of the three stages, we will next attempt to establish collision avoidance capabilities of the FRI in environments other than our experimental arena.

Spike rates and input parameters
The relationship between H1-cell spike rates and wall distances measured under closed-loop conditions was strikingly similar to that observed in previous openloop experiments [29]. The H1-cell spike rate was proportional to wall distance (figure 4) up to a value of 25 (open-loop) and 35 cm (closed-loop), respectively, although the increase of the overall light intensity (due to the non-black floor) might increase the overall spike rates. When increasing distance beyond 25 and 35 cm, in both cases, the spike rates reached certain plateau levels that became distance-independent (figure 5). This is probably due to the limited spatial resolution of the compound eye which effectively acts as a lowpass filter, reducing image contrast as a function of distance [35]. Image contrast is a key parameter for elementary movement detectors (EMDs) which have been suggested to provide directional selective output signals that are integrated on the dendrites of LPTCs [36]. Due to the functional structure of the EMD, its steady state output depends on the square of the image contrast [37]. The expansive nonlinear dependence may, however, be strongly attenuated at larger distances as a result of the filtering properties of the low-resolution compound eye. This could in principle have the beneficial effect that blowflies are not sensitive to visual features in their environment that are further away and do not require immediate behavioural responses.
Why does the relationship between the spike rate of the H1-cell and wall critically depend on turning radius? This finding can be explained when we consider the two extreme cases: at an infinite turning radius the FRI would be performing a pure forward motion which results in an optic flow field dominated by frontto-back motion. During front-to-back motion the H1-cell is inhibited, i.e. its spike rate becomes nearly zero. A turning radius of 0 cm, on the other hand is equivalent to a rotation on the spot. If this is in the counter-clockwise (top view) direction when the FRI is following a wall on the left and the activity of the left H1-cell is recorded, the cell would be strongly excited. Effectively, greater turning radii result in larger areas of the H1-cell receptive field being stimulated in the antipreferred direction which reduces the response. At a turning radius of 15 cm, a point is reached where the response becomes independent of wall distance. This suggests that the ratio between the rotation and the translation component always have reached an excitation to inhibition ratio that is the same for all distances to the wall. For smaller radii excitation dominates with an increasingly steeper slop of the distance dependence. For greater turning radii inhibition takes over, again with a slightly steeper slop of the distancedependence, but an opposite sign ( figure 5(b)).
The receptive field of the H1-cell has an anisotropic motion sensitivity distribution, with highest sensitivities in its frontal/frontolateral parts and lower sensitivities in the lateral and caudal parts. As a result, the orientation of the blowfly within our experimental arena will have inevitably affected the spike rate of the H1-cell during oscillatory forward movements along a wall [38]. This feature may as well have increased the performance of the FRI in our experiments, because nearly all potential collisions would be expected to occur frontally or frontolaterally in case the FRI is moving in an otherwise static environment. The H1-cell is highly sensitive to visual motion in these parts of its receptive field and any optic flow generated by potential obstacles is likely to induce substantial spiking activity which would trigger collision avoidance manoeuvres.
The success rate of the collision avoidance dropped in a 60° corner compared to a straight wall. A limiting factor certainly was that we included the signals of only one H1-cell but did not consider the signals of its contralateral counterpart. Previous research suggested a mutually inhibitory connection between the H1-cells in either part of the visual system [39]. A possible explanation for the relatively high number of collisions with the long wall might be that under acute angle conditions the contralateral H1-cell was more strongly stimulated by the short wall during the oscillatory movements of the FRI than under a 90° angle condition. The stronger excitation of the contralateral H1-cell may have had a lasting inhibitory effect on the spike rate of the ipsilateral H1-cell, which therefore was more unlikely to exceed the threshold value to trigger an avoidance manoeuvre. With recordings from both H1-cells, we could improve the control algorithm to use bilateral distance information, increasing the potential to detect a tapered corridor, and to respond with adequate corrective manoeuvres.
We also identified light intensity as another parameter, affecting performance. Triggering collision avoidance manoeuvres was delayed at the end of the corridor where the light intensity was lower. However, as lower light levels do only cause time constants in photoreceptors and their postsynaptic targets to moderately increase as a result of light adaptation [40] the performance of the system should not drop below an acceptable level.
Another limiting factor of the current FRI is the frequency of 5 kHz at which the on-board microprocessor samples H1-cell spikes. Such a low sampling frequency potentially results in under-sampling of spike peak amplitudes which may, in turn, fail to exceed the spike detection threshold and cause a reduction of the output dynamic range of the H1-cell. In the future, a faster microprocessor will be used to sample spikes at 20 kHz, which might further improve the performance of the FRI.

Active control and potential energy efficiency
There is more than one type of LPTC in the blowfly that is sensitive to horizontal motion. The HSE-cell [38], for instance, receives contralateral inputs from the H1-and H2-cells [41] and is sensitive to front-toback motion which might be more suitable to signal forward movement [42]. Why choosing the H1-cell to close the loop for collision avoidance control?
Energy efficiency could be one of the potential reasons. At the behavioural level, a blowfly would save energy by flying along a straight thrust line rather than on an oscillatory forward movement, because thrust uses less switching between acceleration and deceleration. At the cellular level, more energy will be consumed for a cell that responds preferentially to frontto-back motion (e.g. HSE-cell) to monitor and control a straight thrust trajectory, where the cell would be continuously stimulated. In contrast, a cell which is sensitive to back-to-front optic flow (e.g. H1-cell) may save energy during thrust movements or flying along a trajectory with larger turning radii, as the spike rate of the cell is reduced during front-to-back optic flow [43]. Would there be a favourable balance of power consumption in different environments?
As previous studies suggested [4], a blowfly generates more thrust in open as opposed to confined spaces. If space is limited, they tend to perform a greater number of yaw body saccades. Flying along a trajectory with a large turning radius generates a lower spike rate in the H1-cell as most of its receptive field is stimulated in the anti-preferred direction. Conversely, when engaging on flights with a small turning radius, the receptive field of the cell is mostly stimulated in its preferred direction, resulting in a high spike rate ( figure 5(B)) [29]. A solution would be to transition between larger and smaller turning radii to reduce the average energy consumption [3,44]. For example: instead of exhausting the H1-cell by generating spikes during frequent rapid saccades, the fly could perform larger turning radii in spacious environments to reduce H1-cell spike rates. In narrow environments, on the other hand, it could reduce turning radii to actively sense the distance to close obstacles at an increased spike rate, gathering sufficient information. The movement patterns during active sensing by means of rapid changes of turning radii, has similarities to the silk-moth robot [45] and would provide a bigger spike rate increments if obstacles are close.
A fixed turning radius of 5 cm was chosen in the experiments to obtain a steep linear relationship between spike rate and wall distance. This resulted in comparatively high neuronal activity and presumably a significant amount of energy consumption in both the H1-cells and the robot. In the future, we are planning to implement adaptive turning radii on the FRI which should support its operation in different visual environments and reduce overall energy consumption.

Summary
We have successfully implemented a collision avoidance mechanism on a biohybrid fly-robot-interface based on the spike rate of the identified H1-cell. The FRI was capable of moving along an oscillatory trajectory within our experimental arena and to manoeuvre through a 90° corner at the end of the arena without colliding with its wall. We used the turning radius of the FRI oscillations as an active component for control, which simplified distance estimation based on neuronal activity. For a given turning radius and within a certain range, the spike rate of the cell was linearly related to the distance between the FRI and the patterned wall of the arena. The distance-dependent dynamic output range of the H1-cell was found to be broad enough to reliably trigger collision avoidance manoeuvres when the spike rate exceeds a given threshold value. In the future, the performance of the FRI may be improved by adaptive adjustments of the turning radius to generate more energy-efficient trajectories, by including the signals of the H1-cells recorded from both halves of the fly brain and to enable the system to operate in arbitrary environments.