Low-cost three-dimensional gait analysis system for mice with an infrared depth sensor

Three-dimensional (3D) open-field gait analysis of mice is an essential procedure in genetic and nerve regeneration research. Existing gait analysis systems are generally expensive and may interfere with the natural behaviors of mice because of optical markers and transparent floors. In contrast, the proposed system captures the subjects shape from beneath using a low-cost infrared depth sensor (Microsoft Kinect) and an opaque infrared pass filter. This means that we can track footprints and 3D paw-tip positions without optical markers or a transparent floor, thereby preventing any behavioral changes. Our experimental results suggest with healthy mice that they are more active on opaque floors and spend more time in the center of the open-field, when compared with transparent floors. The proposed system detected footprints with a comparable performance to existing systems, and precisely tracked the 3D paw-tip positions in the depth image coordinates.


Introduction
Rodent gait analysis in an open-field is an essential procedure in genetic and nerve regeneration research (Leroy et al., 2009a;Sheets et al., 2013). For example, the first stage of standard mice phenotyping protocols (SHIRPA (Gailus-Durner et al., 2005) and modified-SHIRPA (Masuya et al., 2005)) include a qualitative scaling of the subject's gait in an open-field. Another example is the widely used rating scales (Basso, Beattie, Bresnahan Locomotor Rating Scale (BBB) (Basso et al., 1995) and the Basso Mouse Scale (BMS) (Basso et al., 2006)) used for regeneration research into spinal cord injury, which focus on the subject's hind limb gait in an open field. These protocols depend on human observations and manipulations. They are, therefore, subjective and difficult to replicate precisely. To solve this problem, we need an automated analysis system that is objective and repeatable. Several researchers have proposed automated analysis systems for open-field tests. However, most of these systems focus on the subject's locomotor activity (Zurn et al., 2005;Tort et al., 2006) or behavior classification (Noldus et al., 2001;Giancardo et al., 2013), and do not directly measure limb movements. Foot tracking systems (Vlamings et al., 2007;Crone et al., 2009) and marker-based motion capture systems (Courtine et al., 2008;Oota et al., 2009) have generally been used to measure limb movements. However, these systems affect the subject, and are therefore unsuitable for observing naturalistic behavior. In existing footprint tracking systems, the subject is placed on a transparent so that their footprints can be measured from underneath. The transparent floor, however, may induce discomfort in the subject and promote acrophobic behaviors (Van Abeelen and Kroes, 1967;Owen et al., 1970). Marker-based motion capture systems may also cause changes in a subject's behavior, for example, they may attempt to remove the attached markers. For these reasons, there is currently no automated gait analysis system that does not induce acrophobia or other behavioral changes. Moreover, most existing footprint-tracking or markerbased motion capture systems are expensive, which is a barrier to gait analysis systems because many researchers have insufficient funding (Dell et al., 2014). The aim of this study was to develop a low-cost gait analysis system that can measure the subject's gait in an open-field apparatus without any effects on its behavior.
Our approach to gait analysis is to observe the subject from underneath using an infrared depth sensor. Because the limbs of mice usually move underneath their bodies, this bottom-view setup can observe richer locomotive information than top-view or side-view setups. This idea was inspired by previous studies that captured subjects in an open-field from underneath a transparent floor (Leroy et al., 2009a,b). In contrast to those studies, the floor of our system is covered with infrared-pass filters so that the subjects behave naturally without being influenced by acrophobia. To observe the subject through the infrared-pass filters, we used a Kinect TM (Microsoft, Redmond, WA) infrared depth sensor. Although, Kinect was originally developed as a human motion sensor (Han et al., 2013), it is sufficiently accurate to measure a three-dimensional (3D) mouse gait. Using the Kinect's depth map, our system can track the subject's footprint and paw-tip movements without any marker equipment. To track the 3D movements of body parts, we used a modified Dijkstra's algorithm called the AGEX (Accumulative Geodesic EXtrema) algorithm (Plagemann et al., 2010). The AGEX algorithm is a data driven algorithm that can robustly detect 3D extrema such as noses, tails and paw-tips without requiring a body model (Baak et al., 2013). Furthermore, a Kinect is cheaper than the sensors used in existing gait system (e.g., high-speed or multiple infrared cameras).
In this study, we conducted two experiments with healthy mice. We first experiment investigated the behavioral differences of the subjects in transparent and opaque (infrared-pass filters installed) conditions. The second experiment evaluated the tracking errors of the two-dimensional (2D) positions of the footprints and the 3D position of the paws.

Related works
A number of systems for measuring rodents' gait and posture changes are currently available. A detailed history of the automated observation of rodents was reviewed in Noldus et al. (2001), Ou-Yang et al. (2011), andDell et al. (2014), so we only list notable research and systems that are closely related to this study (Table 1).
Position tracking is the traditional method of assessing the level of activity of a subject (Zurn et al., 2005;Publicover et al., 2009). Recently, Kinects have been used to track the positions of subjects in open-field settings (Ou-Yang et al., 2011). There are two benefits to using a Kinect for position tracking: (1) it is robust against surrounding light conditions, and (2) it can capture subjects in dark conditions, which is important because mice are more active in darker environments (Valentinuzzi et al., 2000). These advantages mean that a behavioral analysis system that uses a Kinect can be used in all light conditions. 2D body part (head, center of body and tail) tracking systems (Noldus et al., 2001;Clark et al., 2007;Lo et al., 2008) can determine the subject's direction. These systems can estimate the subject's interest in objects or other subjects, and validate the subject's social function.
Footprint tracking systems (Vlamings et al., 2007;Crone et al., 2009;Okamoto et al., 2011) are used to evaluate the subject's locomotive functionality. Footprint frequencies and positions depend on the condition of the subject's joints or muscles, which can be affected by arthritis (Ueno and Yamashita, 2011;Vrinten and Hamers, 2003) or nerve injuries (Vlamings et al., 2007;Neumann et al., 2009). Some systems use a treadmill (Crone et al., 2009) or wheel (Okamoto et al., 2011) to capture images of the subject in a static position, and result in high-resolution images that can distinguish the subject's toes. CatWalk TM measures the pressure map produced by the paws on the ground using an optics-based pressure sensor that detects distortions in the plate. This sensor collects more gait information than other footprint tracking systems and allows the subject to move freely. This system has been shown to be very sensitive and objective when assessing motor impairments in rodents (Vlamings et al., 2007).
For studies that require more detailed information than footprints, marker-present motion capture systems such as the KinemaTracer (Ito, 2008) are used to track the 3D positions of the paws. MotoRater (Zörner et al., 2010) is another marker-present motion capture system, which can measure simple walking, wading, swimming, beam walking, and skilled ladder walking. The subject's body parts are marked with colors, and simultaneously tracked from three sides (bottom, left and right). The subject is placed in a rectangular pathway that is a beam, a ladder, a pool, or a path with a transparent floor to its home cage.
To provide possible improvements over the above systems, we developed a low-cost tracking method for 3D positions of rodent paws and 2D footprint positions in an open-field environment, with no more effect on the subject's behavior than the open-field test.
Our proposed system allows subjects to behave more naturally in the open field using marker-less tracking and infrared measurements. Marker-less tracking avoids marker equipment or coloring, which can change the subject's behavior. The floor of our system was covered with infrared pass filters, which prevented the subject's acrophobia.

Paper organization
The rest of the paper is organized as follows. In Section 2, we explain our method (including the hardware setup and software implementation) and the experimental procedures and evaluation process. Section 3 contains the results of our experiments where we evaluated the accuracy of our system by comparing the result with ground-truth data marked by human mouse-tracking operators. In Section 4, we discuss the properties of our system, possible improvements, and limitations. Section 5 concludes this study.

Animals
Thirteen 8-week-old male C57BL/6J mice (CLEA Japan, Inc., Tokyo, Japan) were used in our experiments. Eight were used for the first experiment to assess the effect of the floor opacity, and the other five were used in the second experiment to evaluate the accuracy of our tracking algorithm. None of the subjects had previously been used in the experiments. The subjects were housed under a 12-h light-dark cycle (lights on at 8:00) with controlled humidity and temperature. Food and water were available ad libitum. The animals were allowed to adapt to the experimental room for at least 16 h before the experiment. All experiments were performed during the light phase of the cycle (10:00-16:00).
All experimental procedures were approved by the local ethics committee established in the Nara Institute of Science and Technology.

Hardware and software environment
The system is composed of an open-field apparatus, a Kinect sensor, and a personal computer (Fig. 1). The open field is a square of 400 mm × 400 mm and the height of the surrounding wall is 320 mm. The Kinect device is fixed 430 mm below the floor so that the entire open-field area can be captured by the device. For the experiment in the opaque conditions, the floor of the open field was covered with tiled infrared-pass filters (FUJIFILM IR-80 (Fuji Film, Tokyo, Japan)), which are commonly used in commercial cameras. Catwalk-XT (Vlamings et al., 2007) Pressure distribution Optical force plate (100 fps) Narrow corridor 2D DigiGait (Crone et al., 2009) Footprint High speed camera (150 fps) Treadmill 2D GAIT (Okamoto et al., 2011) Footprint High speed camera (1200 fps) Running wheel 2D Ou-Yang et al. (Ou-Yang et al., 2011) Center of gravity Depth sensor (20 fps) Open field with a slope 3D TopScan (Lo et al., 2008) Head, center of body and tail Video camera (30 fps) Any 2D EthoVision (Noldus et al., 2001) Head, center of body and tail Video source Any 2D SMART (Clark et al., 2007) Head, center of body and tail Video source Any 2D Motorater (Zörner et al., 2010) Marker position High speed camera Narrow corridor 3D Kinema Tracer (Ito, 2008) Marker position High speed camera (200 fps) Any 3D We used these filters to prevent the subjects from experiencing acrophobia. The depth maps were captured and sent from the Kinect to the personal computer at 30 frames per second (fps). They consisted of 320 × 240 depth pixels, each of which was encoded as a 13-bit integer. The Kinect's measurement may occasionally have lacked image frames, which is depend on the performance of the host computer. In this study, we used a laptop computer that consisted of an Intel(R) Core i5 2.53 GHz (3MB L3 Cache) CPU, 2 GB DDR3 memory, and a 320 GB 5400 RPM hard disk. We found that 29.97 fps was the worst sampling rate recorded by the Kinect (35,975 frames recorded in a 20-min measurement). From this, we can confidently say that the lack of frame were negligible in our proposed tracking algorithm. According to a previous study that investigated the accuracy of the Kinect (Khoshelham, 2011), theoretical values of the local quantization interval (LQI) at the plane 430 mm from the Kinect sensor are approximately 1.4 mm in the x and y directions and 1.0 mm in the z direction (see Appendix A). Here, the x, y and z axes are defined in the left-handed coordinate system shown in Fig. 1. In this study, the Kinect was controlled by the Microsoft software development kit (SDK), which provides device drivers and application program interfaces. The depth information obtained using Microsoft's SDK was recorded to a local drive through the OpenNI framework provided by PrimeSense.

Tracking algorithm
The proposed tracking algorithm consists of four process: preprocessing, feature-point extraction, footprint detection and labeling (Fig. 2a).
During preprocessing, the subject's depth information was extracted from the raw depth map. First, we applied background subtraction to the raw depth map ( Fig. 2b-1). The 3D space of the open field was configured in advance and we subtracted the depth pixels that were not included in the space. Some noise remains after background subtraction, but can be erased by applying a simple dilation and erosion algorithm (Haralick et al., 1987) and by choosing the largest group of pixels ( Fig. 2b-2).
Because the open-field test allows a mouse to move freely, the walking state must be separated from the standing and resting states. To extract the walking frames, we distinguished between the walking and non-walking states as follows. The 2D center of mass c 2 (t) = (c x (t), c y (t)) (mm) at each time t (s) was calculated using where p i (t) = (x i , y i ) (mm) are the 2D positions of the subject's depth pixels, i is the index of the depth pixels, and N is the total number of pixels. The velocityċ(t) (mm/s) in the xy coordinate was derived usinġ where | · | denotes the Euclidian norm. The z element was not included in (2), so that we could ignore the effects of vertical movements such as jumping. Finally, a frame was regarded as being in the walking state when the velocity of the subject was above a threshold value for at least 0.5 s (e.g., 15 frames at 30 fps). We set the threshold value to 2 cm/s in this study. These two threshold parameters were set according to previous works (Leroy et al., 2009a,b). Note that the duration of successive walking states was shorter than that reported in these previous studies, because the size of the open-field was smaller than the apparatus used in the previous studies.
To measure the height of the subject's gait, the depth image coordinates must be transformed to the coordinates on the openfield floor. We placed a flat plate on the open-field area and recorded its depth data prior to the experiments. The floor plane was modeled by a linear equation.
where z floor (x, y) (mm) is the depth given a pair x and y (mm) in realworld coordinates; and A, B and C are constants determined using the RANSAC (RANdom SAmple Consensus) algorithm (Fischler and Bolles, 1981) with the preliminary recorded depth data. In the rest of the paper, p i are low pixels that satisfy where was experimentally set to 5 mm in this study. Each low pixel was temporally labeled by left-front (LF), right-front (RF), left-hind (LH), or right-hind (RH) using c 2 (t) andċ(t) as indicated in Fig. 2b-6. This label is used to distinguish the estimates of the footprints' and paw-tips' positions.
In the feature-point extraction process, a paw tip is estimated as an AGEX feature point ( Fig. 2b-3, red squares) if the first AGEX feature point is farthest from c 3 (t) in the bottom shape of the subject's body in the walking state. The variable c 3 (t) = (c x (t), c y (t), c z (t)) denotes the 3D center of mass at time t. In graph theory, the Dijkstra's algorithm finds the farthest point given a non-directed graph and an initial point in it. The AGEX algorithm sequentially applies the Dijkstra's algorithm to find multiple extrema, calculated as shown in Fig. 3.
In walking state frames, a non-directed geodesic graph G(t) is generated based on an eight-neighbor scheme, which calculates the 3D Euclidean distances from each pixel to its eight neighbors (Fig. 3-1). The eight-neighbor pixels are defined as the upper, lower, right, left, and the four skewed neighboring pixels. If the distance between each neighbors and the selected pixel is smaller than a threshold (ı connect ), then an edge between each pair of neighbors is added to G(t) with a weight equal to its Euclidean distance. ı connect was experimentally set to 3 mm. Denoting the jth AGEX feature point given a graph G(t) by e j (G(t)), the initial AGEX feature point is set to e 0 (G(t)) = c 3 (t), and it is connected to the nearest node of G(t) by an edge with no weight. Below, e j (G(t)) is denoted by e j for simplicity. e j is incrementally calculated until j reaches a constant M = 8, which we determined experimentally so that the result includes all paws. To calculate e j , the Dijkstra's algorithm is applied with e 0 as the starting point, and the farthest node is extracted as e j (Fig. 3-2). Before calculating e j+1 , an edge with no weight between e 0 and e j+1 is added (Fig. 3-3). This operation shrinks the graph and allows us to find another extremum. This process of searching and adding an (1) A non-directed geodesic graph G(t) is generated based on the subject's depth pixel. The red node is the initial AGEX feature point e0(G(t)) = c3(t).
(2) The jth AGEX feature point ( e j ) is calculated using the Dijkstra algorithm.
(3) Before calculating e j , an edge with no weight between e0 and e j is added. (4) Repeat (2) and (3)  edge is repeated until j reaches M (Fig. 3-4). Because the shape of a walking rodent is roughly an ellipse, e 1 and e 2 most likely represent the head and the hip. Hence e j (j = 3, . . ., M) are regarded as candidate points for the limb edge.
In the footprint detection process, the footprint pixels are calculated by fusing the following two types of pixels.
Succeeding low pixels: The pixels that continuously satisfy condition (4) for three or more frames (Fig. 2b-4). AGEX − neighboringpixels: The pixels whose Euclidean distance from any AGEX feature points is below a threshold = 8 (mm), colored in purple in Fig. 2b-3. was experimentally determined so that those pixels cover all the pixels of the footprints (Fig. 2b-3).
Succeeding low pixels encode rough information about the paw shape and include unnecessary pixels (e.g., abdomen or neck) whereas near AGEX pixels are more confident paw locations. Hence, we can determine the footprint pixels by applying the AND operation to succeeding low pixels and AGEX-neighboring pixels, as shown in Fig. 2b-5. Thus, the footprint's position is estimated as the 3D mean of the footprint pixels, similar to (1).
In the labeling process, the estimated positions of the footprints denoted byp foot and pawsp paw are labeled using the footprint pixels positions based on the subject's direction, as shown in Fig. 2b-6. The footprints and paws are labeled according to their positions as leftfront (LF), right-front (RF), left-hind (LH), or right-hind (RH). When there are no corresponding footprint pixels, each paw-tip is labeled according to the nearest paw-tip in terms of the Euclidean distance in the xy coordinate of the previous frame. Note that the labels are initialized every time the size of the footprint pixels is non-zero.

Effect of floor opacity
The first experiment investigated the effect of the floor opacity. A total of eight mice (four in transparent conditions and the other four in opaque conditions) were used. In the opaque floor conditions, infrared-pass filters were pasted under the transparent acrylic plates. In the transparent floor condition, the filters were absent. The experimental procedures were based on the standard operation procedures of the mouse phenotyping platform in RIKEN Japan Mouse Clinic (RIKEN Japan Mouse Clinic, 2008). The illumination intensity of the floor was 90 lux. We placed each subject at one of the four corners of the open field, and be measured its behavior for 15 min. There were no moving objects in sight of the subject, and only the subjects and trained operator were in the experimental room when the measurements were recorded. When each measurement was finished, the open field was cleaned with 80% ethanol and dried for at least 10 min before the next measurement.
The effect was evaluated using two parameters: activity and rate of time spent in the center. The activity was defined as the total distance that a subject traveled in a 1 min time window, that is, where T k is the start time of each sample at a constant interval of 60 s. The rate of time spent in the center area (r k ) was defined as the ratio between the durations spent in center and outer areas.
The center area was defined as a 200 mm × 200 mm square area centered in the open field. We calculated fifteen samples for each subject using b(t) = 1 (c 2 (t) is in the center area) 0 (otherwise) .

Gait-tracking accuracy
The second experiment evaluated the tracking accuracy of the proposed system. Five other mice were released one at a time, into the open field for 20 min, and the positions of their footprints and paws were tracked using the proposed algorithm. Two of the mice were released onto the transparent floor, and the other three were released onto the opaque floor. The other experimental conditions were same as in the first experiment. The tracking results were compared with ground truth data that was generated by human operators (see Appendix B).  Fig. 4 shows the chronological changes in the subjects' activity on a transparent or an opaque floor. There were significant floor effect in the activities on transparent and opaque conditions (F(1, 6) = 9.68, p = 0.021 using two-way ANOVA with repeated measures. The data of each subject passed a normality test (KolmogorovSmirnov test. p > 0.05). In terms of the median of each min, the subjects were more active in opaque conditions, with the exception of the 13th min. Fig. 5 shows the chronological changes in the ratio of time that a subject spent in the center area of a transparent or an opaque floor. There were marginal floor effect in the rations of time spent in the center of the transparent and opaque conditions (F(1, 6) = 4.96, p = 0.067 using two-way ANOVA with repeated measures. The data of each subject passed a normality test (KolmogorovSmirnov test. p > 0.05).

Gait-tracking accuracy
3.2.1. Footprint tracking accuracy Table 2 shows the footprint tracking errors for the transparent and opaque floors. In each set of 300 frames marked by human operators, our system detected 308 of 348 (88.5%) footprints for the transparent floor and 293 of 352 (83.2%) footprints for the opaque floor. No false positive results were detected in either floor setting. The rear paws were tracked better than the front paws, in terms of the detection rate and number of tracking errors. There were no significant differences in the footprint tracking errors for the transparent and opaque floor settings (Student's t test, p = 0.32). Table 3 shows the 3D paw-tracking errors for the transparent and opaque floor settings. For each floor, 300 frames of data were marked by human operators. Fig. 6 shows a representative tracking result on the z-axis. The left rear paw of a subject in the opaque floor setting was tracked for 1 s. There were no significant differences in the paw tip tracking errors of the two floor settings (Student's t test, p = 0.22).

Discussion
One of the advantages of our system is that we can conduct a gait analysis on the mice without introducing any effect on their behavior. Existing 3D gait tracking systems for mice (Ito, 2008;Zörner et al., 2010) require marker equipment and limit the tracking environment. Footprint analysis systems (Okamoto et al., 2011;Crone et al., 2009) do not require any marker equipment and some (Vrinten and Hamers, 2003;Leroy et al., 2009a) allow mice to move more freely. However, these systems use transparent floors without discussing the effect that floor opacity may have on the subject's behavior. The results of our first experiment (Figs. 4 and 5) showed that the floor opacity probably affects behavior. There was a significant effect to the subject's activity when comparing the transparent and opaque floors. Moreover, the subjects on the opaque floor were more active than those on the transparent floor, in terms of the median. Our interpretation of this phenomenon is that the subjects behaved more conservatively when placed on the transparent floor, because of acrophobia. Location differences also support this interpretation. The ratios of time spent in the center were marginally effected with the floor opacity. Additionally, the subjects on the opaque floor spent more time in the median before 13 min, when compared with the subjects on the transparent floor.
These findings confirm that our system can measure a more natural and active behavior in an open-field environment than existing gait analysis systems. This is an important point to consider when evaluating natural gait behavior for applications such as phenotyping. Moreover, for the evaluation of motor function, more active moving is preferred when evaluating motor-impaled mice. Another advantage of this system is that it can acquire additional quantitative 3D information. This system also has no limitations regarding lighting conditions, because it uses a Kinect. This means that a user can easily follow previously standardized procedures (Green et al., 2005;Wahlsten, 2003;RIKEN Japan Mouse Clinic, 2008). This results in a more dependable data collection process than existing footprint systems used in open-field environments. Footprint analysis systems are used as a tool for phenotyping (Leroy et al., 2009b) and for evaluating medicinal effects (Leroy et al., 2009a). However, Oota et al. (2009) reported a phenotype of mice that showed gait abnormality in mice. In their study, the gait patterns in their mice subjects showed no significant differences from normal gait strides, but showed significant differences in limb height variation. This means that our system can potentially have an advantage over other systems, in such a way, that it can detect those possible gait abnormalities. However, our system has some disadvantages when compared with existing systems. First, the performance of the sensor is limited. In the result of the second experiment (Tables 2 and 3), the mean averaged errors (MAE) of the footprint and paw-tip tracking results were small compared with the paw size and stride length (Jacobs et al., 2014). Leroy et al. reported more accurate footprint tracking results, with a MAE of 3.1 mm (Leroy et al., 2009b) (the MAE obtained in this study was approximately 4.3 mm). This difference is probably because the depth map's resolution is lower than that of color images with 2D coordinates. Additionally, the MAE of the 3D paw-tip tracking was approximately 4.7 mm. This can be transformed to approximately ±2 pixels of error in each axis of the depth image coordinates. We believe that improving the depth sensor accuracy will result in more accurate tracking results. The same is true for the temporal resolution. Second, our system availability is confirmed with only healthy mice currently. The footprint and paw-tip tracking algorithm is based on the assumption that the placing foot is lower than other body parts. It can be interfered by abnormal gait that the subjects cannot raise their pelvis from the floor. Even in that case, the AGEX algorithm can extract paw tips as one of the candidate points for the limb edge. For paw-tips tracking of the abnormal gait, a method to distinguish the true paw tips from the candidates is required. In future work, we will confirm the applicability of the proposed algorithm to mice with abnormal gaits such as those with Parkinson's disease. Nevertheless, the system can analyze the gait of subjects that lift their feet. Therefore, it can be used in psychological, electrophysiological, and pharmaceutical research to measure deviations from the normal gait.
Finally, the high cost of current mouse tracking systems is probably why many behavioral analyses of mice are still dependent on human observations (Dell et al., 2014). This simple and low-cost quantitative tracking method can encourage more investigation into behavioral analysis.

Conclusion
In this study, we developed a low-cost 3D gait analysis system that tracks a subject's footprints and paw-tip trajectories in openfield tests. The system observes the subject's behavior in an open field from underneath, using a low-cost depth sensor (Microsoft Kinect TM ). The floor of the open field was covered with infraredpass filters to prevent acrophobic behaviors. We also developed a tracking algorithm that fuses the globally extracted paw-tip candidates from the depth information based on 3D extrema, and locally selects candidates as succeeding low pixels. Our experimental results demonstrate the applicability of our system for healthy mice.
In the future work, we will investigate gait analyses for a model mouse with a motor deficit such as Parkinson's disease, which is a typically important feature in behavioral neuroscience.