A hybrid versatile method for state estimation and feature extraction from the trajectory of animal behavior

Animal behavior is the final and integrated output of the brain activity. Thus, recording and analyzing behavior is critical to understand the underlying brain function. While recording animal behavior has become easier than ever with the development of compact and inexpensive devices, detailed behavioral data analysis requires sufficient previous knowledge and/or high content data such as video images of animal postures, which makes it difficult for most of the animal behavioral data to be efficiently analyzed to understand brain function. Here, we report a versatile method using a hybrid supervised/unsupervised machine learning approach to efficiently estimate behavioral states and to extract important behavioral features only from low-content animal trajectory data. As proof of principle experiments, we analyzed trajectory data of worms, fruit flies, rats, and bats in the laboratories, and penguins and flying seabirds in the wild, which were recorded with various methods and span a wide range of spatiotemporal scales—from mm to 1000 km in space and from sub-seconds to days in time. We estimated several states during behavior and comprehensively extracted characteristic features from a behavioral state and/or a specific experimental condition. Physiological and genetic experiments in worms revealed that the extracted behavioral features reflected specific neural or gene activities. Thus, our method provides a versatile and unbiased way to extract behavioral features from simple trajectory data to understand brain function.

understanding how multiple brain functions affect behavior has been difficult. In order to 28 decipher dynamic brain functions from time-series of behavioral data, we developed a 29 machine learning strategy that extracts distinguishing behavioral features of sensory 30 navigation. We first investigated experience-dependent enhancement of odor avoidance 31 behavior of the nematode Caenorhabditis elegans. We segmented worms' trajectories 32 during olfactory navigation into two behavioral states, analyzed 92 features of the states, 33 and automatically extracted 9 distinguishing features modulated by prior odor 34 experience using a statistical index, the gain ratio. The extracted features included ones 35 previously unidentified, one of which indicated that the prior odor experience lowers 36 worms' behavioral responses to a small increase in odor concentration, causing enhanced 37 odor avoidance. In fact, calcium imaging analysis revealed that the response of ASH 38 nociceptive neurons to a small odor increase was significantly reduced after prior odor 39 experience. In addition, based on extracted features, multiple mutant strains were 40 categorized into several groups that are related to physiological functions of the mutated 41 genes, suggesting a possible estimation of unknown gene function by behavioral features. 42 Furthermore, we also extracted behavioral features modulated by experience in acoustic 43 navigation of bats. Thus, our results demonstrate that, regardless of animal species, 44 sensory modality, and spatio-temporal scale, behavioral features during navigation can be 45 extracted by machine learning analysis, which may lead to the understanding of 46 information processing in the brain. To reveal experience-dependent modulations of olfactory navigation in worms, we 107 extracted prominent features of behavior and their relationships to odor stimuli using 108 machine learning. For this, we segmented the animals' navigation into two behavioral 109 states and, for individual states, calculated the gain ratio, a statistical index used in 110 decision tree analysis to identify features that distinguish the data in different classes 111 (22). We chose this method because it allows us to easily interpret the results of the 112 analysis for the planning of physiological experiment of neural activities. In the present 113 study, we analyzed 92 features of behavior and sensory information from each of ~200 114 behavioral states of wild-type worms either with or without a prior odor experience (25 115 worms per condition) and found that 9 features are modulated in an 116 experience-dependent manner. One of the extracted features, the reduction in behavioral response to a small increase in odor concentration, was consistent with a change in 118 neuronal activity revealed by calcium imaging. In addition, we calculated gain ratios to 119 identify the experience-dependent changes in olfactory navigation of multiple mutant 120 strains and found that the mutants were categorized into several groups based on 121 behavioral features, which reflect physiological functions of the mutated genes. 122 Furthermore, we also identified experience-dependent modulation of behavioral features 123 from bat acoustic navigation. Thus, we propose that machine learning analysis with gain 124 ratios is an efficient strategy to reveal features of animal behavior in general. 125 8 as a turn. A histogram of turn intervals was fitted by two exponentials, suggesting that 164 turn intervals are regulated by two probabilistic mechanisms (25,26). The time of the 165 intersection of the two exponentials was defined as t crit , and turn intervals longer or 166 shorter than the t crit were classified as runs or included in pirouettes, respectively. t crit 167 was calculated for the control (i.e., naive plus mock-treated) condition for wild-type and 168 mutant strains, respectively. In this study, we analyzed features of runs but not of 169 pirouettes except for their duration because pirouettes appear to have little effect on 170 odor avoidance (25). Excel 2010 (Microsoft Corp.) was used for these calculations. The 171 odor concentrations that worms experienced at specific spatio-temporal points were 172 calculated according to the dynamic odor gradient model based on the measured odor 173 concentration (19).  in the x-axis and at 22-cm intervals in the y-axis so that the bat was forced to fly in an 195 S-shaped pattern without passing between chains. With this layout, three naive bats 196 were used: each bat was observed for 12 continuous repeated flights so that 197 echolocation behaviors in unfamiliar and familiar spaces could be compared. In this 198 study, the initial three flights were defined as unfamiliar flights, and the last three flights 199 were defined as familiar flights. For the machine learning analysis of worm olfactory navigation, the following 211 behavioral features were calculated for each run from the centroid coordinates of the 212 worms: start time (RunTime), serial number (RunNum), velocity (V), bearing (B), odor 213 concentration that a worm experienced during run (C), directionality ratio (Dir) (28), 214 run's curvature (called weathervane; WV) (29), run duration (RunDur), and duration of 215 pirouette just before the run (PirDur). Time-differential values were calculated for V 216 (dV), B (dV), and C (dC). For these values, the average (Ave) during a run as well as 217 average values over 2 s at the initiation (Ini) and at the termination (Ter) of a run were 218 also calculated. For Ini and Ter, 0-2 s after the initiation and 2-4 s before the 219 termination of a run were used; we did not use 0-2 seconds before the termination 220 because previous studies revealed that a worm's speed drops largely during this period 221 (25, 26). Although the time windows for worms were set based on the previous studies, 222 the optimal time windows for bats were calculated by machine learning (see below). In 223 addition, to analyze whether these features are independent for each run (in other words, 224 whether any long-term trend among runs across a pirouette exists), we also calculated 225 hysteretic effects (∆) of these run features between successive runs for V, dV, B, dB, C, 226 and dC-in fact, a certain relationship between bearings before and after a pirouette 227 (B_Ini∆Ter) has been reported in salt-taxis (26). For this, Ave, Ini, or Ter of each run 228 feature was subtracted from any of the previous run features. Hysteretic effects were 229 calculated for just one feature for RunDur, PirDur, and WV, which only possess one 230 value per run, and not calculated for RunTime and RunNum. A total of 92 features were 231 calculated by combining all these features. Beeswarm package for R software (The R Project) was used for the scattered plot of 248 data. These parameters are listed in Table 1 for worms and in Table 4 for bats. 249 250

Behavioral classification with gain ratio 251
In order to extract useful features, we calculated the gain ratio used in C4.5 decision tree 252 analysis (30) for each feature. In decision tree analysis, the amount of information, 253 which is acquired when a group of data is divided into sub-groups by a certain feature, 254 is calculated as information gain. In other words, when dividing a group into sub-groups by applying a certain feature, information gain is an index indicating the amount of the 256 increased bias of data in the sub-groups after the division. The information gain was 257 then divided by split info, a degree of division, for normalization to compute the gain 258 ratio. For worm olfactory navigation, we extracted behavioral features that have positive 259 gain ratios in naive versus preexposed worms or in mock-treated versus preexposed 260 worms. Then, we chose the features that were common in both comparisons as "features 261

Calcium imaging 267
Calcium imaging of the worms' ASH neurons was performed according to a previous 268 report (19). Briefly, transgenic strains expressing GCaMP3 (32) and mCherry (33) in 269 ASH sensory neurons under the sra-6 promoter (KDK70034 and KDK70072; 20 ng/µl 270 of sra-6p::GCaMP3, 20 ng/µl of sra-6p::mCherry, 10 ng/µl of lin-44p::GFP, 50 ng/µl 271 of PvuII-cut N2 genomic DNA as a carrier in N2 background) were placed on an NGM 272 agar plate on a robotic microscope system, OSB2 (19). Although these transgenic 273 worms were immobilized with the acetyl choline receptor agonist levamisole (34) for 274 high-throughput data acquisition by simultaneous imaging of multiple worms, the 275 previous study revealed that the ASH activity is essentially unaffected by 276 levamisole-treatment (19). For these worms, a constant gas flow of 8 mL/min was 277 delivered, in which the mixture rate of 2-nonanone gas versus the air was changed to make a temporal gradient of the odor concentration. The temporal change in odor 279 concentration was measured by a custom-made semiconductor sensor before and after 280 the series of calcium imagings on each day. The fluorescence signals of GCaMP3 and 281 mCherry in ASH neurons were divided into two channels using W-View (Hamamatsu, 282 Japan), an image splitting optic, and captured by an EM-CCD camera (ImagEM;

RESULTS 299
A strategy for extracting features of sensory navigation that were modulated by 300 understand changes in information processing in the brain during sensory navigation. 303 However, identifying characteristic changes in behavior and sensory stimulus during 304 navigation has been difficult. It is because sensory stimulus in general changes 305 gradually and continuously during navigation, which may cause gradual or sudden 306 response at some aspects of behavior with certain probabilities. In order to efficiently 307 extract behavioral features of sensory navigation by machine learning, we considered 308 of an animal, whose description requires more detailed spatial and temporal 328 information. 329 330

Sensory information 331
Sensory information is a key factor affecting an animal's behavior. However, because 332 of technical difficulties, it has been included in the analysis of sensory navigation only 333 in a few cases for small model animals (17,19,36). We included the information of 334 odor concentration, which changes dynamically during navigation of worms, as 335 revealed by the direct measurement of odor concentrations in specific spatio-temporal 336 points in a behavioral arena (19). 337 338

Gain ratio 339
To comprehensively examine behavioral features that can be modulated by prior 340 experience, we focused on a statistical index used in machine learning-based 341 classification analysis. In general, classification analysis is the task of classifying new, 342 unlabeled data into appropriate classes using characteristic features and their parameters 343 that have been extracted from the known class-labeled data. In the present study, 344 however, the classification itself was not meaningful because the data were already 345 classified, such as with or without prior experience or wild-type versus mutant strains. 346 distinguishing between the two classes. In other words, behavioral features modulated 348 by prior experience should be able to effectively classify the behavioral data of animals 349 with or without this experience. 350

351
For this purpose, we chose to use gain ratio, the index for decision tree analysis (22). 352 Binary decision tree analysis is performed to split a data set into two sub-groups by 353 automatically selecting the best feature and its parameter that has the largest 354 information gain, the difference of the uncertainty ("information entropy") before and We analyzed the experience-dependent enhancement of odor-avoidance behavior of C. 368 elegans as a model. We have reported that preexposure of worms to the repulsive odor 369 2-nonanone causes enhancement of avoidance behavior to the odor. After 1 h of preexposure, worms migrate farther away from the odor source as a type of 371 non-associative middle-term learning (24). A series of genetic analyses indicated that 372 neuropeptide and dopamine signaling pathways are required for acquisition and 373 execution of the odor memory, respectively (25), suggesting that this non-associative 374 middle-term memory is caused by a circuit-level modulation of neural activity rather 375 than simple sensory sensitization.   (Table 2). Nine features were shared between the two comparisons, 392 suggesting that those features were modulated by the prior experience to cause the enhanced odor avoidance (Table 2 and Fig. 2A). These features are related to run 394 duration (RunDur), temporal differences in bearing (dB_X), odor concentration (C_X), 395 and its temporal difference during runs (dC_X). Modulation of run duration (RunDur; 396  Table 2) have not been revealed previously. Its contribution to the 401 enhancement of avoidance distance, however, are unclear. Differences between the 402 average or the terminal bearing change and the previous initial value (dB_Ave∆Ini or 403 dB_Ter∆Ini) were also extracted (Table 2), while the differences were likely due to the 404 modulation in the previous initial value (∆Ini), not due to the change in hysteretic 405

effects. 406 407
Odor stimuli during runs, which likely drive the worms' odor avoidance behavior, were 408 also found to be modulated in several aspects: the initial, terminal, and average odor 409 concentration (C_Ini, C_Ter, and C_Ave), and the terminal and average odor 410 concentration change (dC_Ter and dC_Ave) ( one possible scenario is that, because the odor-experienced worms were located farther 414 away from the odor source, they sensed a lower concentration of the odor. 415 avoidance behavior depends on dC, rather than C, at least in naive conditions (19), we 419 investigated these features more in detail. We compared ensemble averages of dC/dt 420 that worms sensed during the last 30 s of each run (Fig. 2F). Interestingly, although 421 most of the control (i.e. naive and mock-treated) worms sensed 2-3 µM odor 422 concentrations (Fig. 2D), dC/dt at the end of each run was ± 0.1 nM/s on average (Fig.  423 2F; 0.09 ± 0.72 and -0.09 ± 0.76 nM/s for naive and mock-treated animals, respectively). 424 This result suggests that, to initiate a pirouette, worms respond to a subtle odor 425 concentration change, of which the magnitude is 1/20,000 -1/30,000 of the odor 426 concentration itself per second. Even considering that sensory information is temporally 427 integrated for a few seconds during worm chemosensory navigation (19,37), this value 428 is far lower than the general psychological threshold for sensory signals: The lower 429 threshold of signal change (∆S) is more than 1/100 of the signal intensity (S) (38). and suggest the following model: the worms without prior experience of the odor are 437 highly sensitive to a slight increase in odor concentration during a run, which is a sign 438 of inappropriate movement toward the source of the repulsive odor, and they respond to it by initiating a pirouette. In contrast, the worms with prior odor experience ignore the 440 slight odor increase and continue the run, which leads to a longer run duration (Fig.  441   2G). that worms experience during the odor avoidance assay in the plates (19). Using the 451 OSB2 system, we found that ASH neurons are the major sensory neurons to cause 452 pirouettes upon increases in 2-nonanone concentration (19). However, whether the ASH 453 response is modulated by 2-nonanone experience has not been studied. 454

455
We found that ASH responses were modulated by prior odor experience in a manner 456 consistent with the behavioral modulation. When the worms were stimulated with 5 457 nM/s odor increase, which is the lowest rate of change to cause the threshold-level 458 behavioral response in the previous study (19), ASH neurons in naive as well as 459 mock-treated worms exhibited robust responses ( Fig. 3A and B). However, the ASH 460 responses were significantly reduced in the preexposed animals ( Fig. 3B and C). This 461 result suggests that prior odor experience causes the reduced response to the odor  (Table 3 and Fig. 4). Increases in odor concentration in preexposed worms were 487 interesting because they were observed only in egl-3 mutants but not in any other 488 mutants. 489 490 Mutations in dop-3, which encodes a homolog of the D2-type dopamine receptor, were 491 previously found to affect migratory direction after preexposure (25). This was 492 concluded because the mutants did not exhibit enhanced avoidance distance, although 493 run duration was increase after preexposure, and because run terminal bearing (B_Ter) 494 was worsened (25). These features were extracted in this analysis (Table 3), further 495 supporting the reliability of this analysis. In addition, the averaged directionality ratio 496 (Dir_Ave) was worsened (Table 3 and Fig. 4C). This is also consistent with the idea that 497 migratory direction is worsened in dop-3 mutants after preexposure. Moreover, lowered 498 velocity (V_Ave, V_Ter; Table 3 and Fig. 4B) was also extracted, which may also 499 contribute to the failure in the enhanced odor avoidance. Such multiple abnormal 500 phenotypes are consistent with the fact that dop-3 is expressed in many neurons and 501 involved in the regulation of multiple aspects of behavior (48). 502 503 ocr-2 and osm-9 both encode homologs of TRP-type cation channels, expressed in 504 multiple sensory neurons including ASH neurons and considered to be involved in 505 sensory perception as well as its modulation (43). We found that mutants for these two 506 genes did not exhibit significantly enhanced odor avoidance (Fig. 4A). In addition, 507 these mutants exhibited increases in velocity (V_Ave and V_Ter) after preexposure, which was specific for these two mutants but not observed in other mutants. Some 509 features are specific for each mutant strains, which may reflect the differences in their 510 expression and/or function (43). Mutants of tax-4, encoding a homolog of the 511 cGMP-gated cation channel expressed in different sets of sensory neurons, exhibited a 512 unique pattern of features (Table 3, Fig. 4).  Table 4 and Fig. 5A-C). 527 Interestingly, although velocity (V) itself was modulated by flight experience, 528 acceleration (dV) was not (Fig. 5D), suggesting that bats may determine flight speed 529 before initiating but not during navigation, at least in our experimental conditions. 530 suggest that such higher brain functions during navigation could be extracted by 532 machine learning analysis. 533

DISCUSSION 535
In the present study, we extracted behavioral features that are modulated by experience 536 from olfactory navigation of worms and from acoustic navigation of bats using machine 537 learning analysis. In the case of worm olfactory navigation, we found a neural correlate 538 for one of the newly identified features: The reduced behavioral response to an increase 539 in odor concentration was consistent with the reduced response of ASH nociceptive 540 neurons to a small increase in odor concentration. In addition, we also found that mutant 541 strains can be grouped based on extracted features, which may correspond to the 542 physiological roles of genes in chemotaxis and/or experience-dependent modulation. 543 Furthermore, our machine learning analysis was applied to acoustic navigation of bats 544 to extract the features modulated by prior experience. 545 546

Extraction of behavioral features by machine learning 547
Machine learning has been playing a major role in classifying behavioral data of model 548 animals into several categories (7-13, 50). Instead of such behavioral classification, 549 however, we intended to use a machine learning technique for extracting characteristic 550 features of sensory behavior to decipher information processing in the brain. For that 551 purpose, we first hypothesized that a change in a behavioral feature reflects a change in 552 activity of a functional unit of the brain. Then, we used machine learning to extract 553 behavioral features that differ between two classes of behavior, rather than to categorize prior experience" or "wild-type versus mutant strains") were determined per se and did 556 not need to be categorized. 557

558
In addition, we also included sensory information that worms experienced during the 559 course of behavior in the machine learning analysis. Although small model animals 560 such as C. elegans or Drosophila melanogaster are suitable for machine vision 561 monitoring and subsequent quantitative analysis of behavior, their small size makes it 562 difficult to measure sensory signals they receive during behavior. We have solved this 563 problem by precisely measuring the odor concentration at multiple spatio-temporal 564 points in a paradigm to assess olfactory behavior of worms (19), which allowed us to 565 include the information of odor concentration and its temporal changes (C and dC) into 566 the feature vector for the machine learning analysis. Our machine learning method 567 could also be used for detailed analysis of sensory navigation in the environment where 568 the gradients of chemical signal were also quantitatively monitored (36, 51, 52). For example, because odor concentration (C) and temporal odor concentration change 593 (dC) are both features of sensory stimuli, they were likely candidates for the cause of 594 changes in behavioral response. However, we regard that dC, rather than C itself, is the 595 causal reason because of the following: (1) previous quantitative analysis in the plate 596 assay paradigm revealed that pirouettes and runs are strongly correlated with positive or 597 negative dC, respectively, rather than the value of C; (2) in the OSB2 robotic 598 microscope system, positive or negative dC caused high or low levels of turning, such 599 as pirouettes and runs, respectively (19). For example, the lower C in preexposed shapes, colors, and brightness for visual stimuli, and frequency and intensity for 635 acoustic stimuli) have the most prominent effects on behavior in a particular context. 636 We believe that our method presented herein, which allows extraction of the essential 637 features of information processing for sensory behaviors in the brain in an objective and 638 comprehensive manner, will help to solve this problem. Poor description of behavior 639 with simple indices compared to "big data" on neuronal activity is being recognized as 640 one of the significant problems in modern neuroscience (3-5). We expect that 641 comprehensive and objective feature extraction would increase the wealth of description 642 of behavior, which will provide us clues to understand more of the big data from brain 643 activity monitoring.  (egl-3(ok979)), 18.1 (dop-3(tm1356)), 11.2 (ocr-2(ak47)), 17.8 (osm-9(ky10)), and 6.7 699 (tax-4(p678)). Thick bars represent statistical differences between preexposed worms 700 versus naive and mock-treated worms, suggesting differences caused by the odor 701 preexposure; thin bars represent statistical differences between preexposed worms 702 versus naive or mock-treated worms, which were not caused by the preexposure. (**p <     6" 0" 60" 120" 180" 240" 300" 360" 420" 480" 540" 600" 660" 720"