Skip to main content

Advertisement

SpringerLink
Log in

An automated behavior analysis system for freely moving rodents using depth image

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

A rodent behavior analysis system is presented, capable of automated tracking, pose estimation, and recognition of nine behaviors in freely moving animals. The system tracks three key points on the rodent body (nose, center of body, and base of tail) to estimate its pose and head rotation angle in real time. A support vector machine (SVM)-based model, including label optimization steps, is trained to classify on a frame-by-frame basis: resting, walking, bending, grooming, sniffing, rearing supported, rearing unsupported, micro-movements, and “other” behaviors. Compared to conventional red-green-blue (RGB) camera-based methods, the proposed system operates on 3D depth images provided by the Kinect infrared (IR) camera, enabling stable performance regardless of lighting conditions and animal color contrast with the background. This is particularly beneficial for monitoring nocturnal animals’ behavior. 3D features are designed to be extracted directly from the depth stream and combined with contour-based 2D features to further improve recognition accuracies. The system is validated on three freely behaving rats for 168 min in total. The behavior recognition model achieved a cross-validation accuracy of 86.8% on the rat used for training and accuracies of 82.1 and 83% on the other two “testing” rats. The automated head angle estimation aided by behavior recognition resulted in 0.76 correlation with human expert annotation.

Top view of a rat freely behaving in a standard homecage, captured by Kinect-v2 sensors. The depth image is used for constructing a 3D topography of the animal for pose estimation, behavior recognition, and head angle calculation. Results of the processed data are displayed on the user interface in various forms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Tecott LH, Nestler EJ (2004) Neurobehavioral assessment in the information age. Nat Neurosci 7:462–466

    Article  CAS  Google Scholar 

  2. van Meer PJK, Graham ML, Schuurman H (2015) The safety, efficacy and regulatory triangle in drug development: impact for animal models and the use of animals. Eur J Pharmacol 759:3–13

    Article  Google Scholar 

  3. Baker M (2011) Animal models: inside the minds of mice and men. Nature 475:123–128

    Article  CAS  Google Scholar 

  4. Rat Genome Sequencing Project Consortium (2004) Genome sequence of the Brown Norway rat yields insights into mammalian evolution. Nature 428:493–521

    Article  Google Scholar 

  5. Shi Q, Miyagishima S, Fumino S et al (2010) Development of a cognition system for analyzing rat’s behaviors. In: IEEE Int Conf Robotics and Biomimetics (ROBIO), pp 1399–1401

  6. Shi Q, Ishii H, Konno S et al (2012) Image processing and behavior planning for robot-rat interaction. In: IEEE RAS & EMBS Int. Conf. Biomedical Robotics and Biomechatronics (BioRob), pp 967–973

  7. Jhuang H, Garrote E, Yu X, Khilnani V, Poggio T, Steele AD, Serre T (2010) Automated home-cage behavioural phenotyping of mice. Nat Commun 1(68):1–9

    Article  Google Scholar 

  8. Burgos-Artizzu XP, Dollár P, Lin D et al (2012) Social behavior recognition in continuous videos. In: IEEE Conf. Computer vision and pattern recognition (CVPR 2012), pp 1322–1329

  9. van Dam EA, van der Harst JE, ter Braak CJF et al (2013) An automated system for the recognition of various specific rat behaviours. J Neurosci Methods 218(2):214–224

    Article  Google Scholar 

  10. Patel TP, Gullotti DM, Hernandez P, O'Brien WT, Capehart BP, Morrison B, Bass C, Eberwine JE, Abel T, Meaney DF (2014) An open-source toolbox for automated phenotyping of mice in behavioral tasks. Front Behav Neurosci 8:349

    Article  Google Scholar 

  11. Brodkin J, Frank D, Grippo R, Hausfater M, Gulinello M, Achterholt N, Gutzen C (2014) Validation and implementation of a novel high-throughput behavioral phenotyping instrument for mice. J Neurosci Methods 224:48–57

    Article  Google Scholar 

  12. Lorbach M, Kyriakou EI, Poppe R et al (2017) Learning to recognize rat social behavior: Novel dataset and cross-dataset application. J Neurosci Methods. in press, available online

  13. Ren Z, Noronha A, Ciernia AV et al (2017) Who moved my cheese? Automatic annotation of rodent behaviors with convolutional neural networks. In: IEEE Winter Conf. Applications computer vision (WACV), pp 1277–1286

  14. Crispim-Junior CF, de Azevedo FM, Marino-Neto J (2017) What is my rat doing? Behavior understanding of laboratory animals. Pattern Recogn Lett 94:134–143

    Article  Google Scholar 

  15. Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with Microsoft Kinect sensor: a review. IEEE Trans Cybern 43(5):1318–1334

    Article  Google Scholar 

  16. Hu M, Chen C, Cheng W et al (2015) Real-time human movement retrieval and assessment with Kinect sensor. IEEE Trans Cybern 45(4):742–753

    Article  Google Scholar 

  17. Whitmire E, Latif T, Bozkurt A (2013) Kinect-based system for automated control of terrestrial insect biobots. In: Int. Conf. IEEE EMBC, pp 1470–1473

  18. Wiltschko AB, Johnson MJ, Iurilli G, Peterson RE, Katon JM, Pashkovski SL, Abraira VE, Adams RP, Datta SR (2015) Mapping sub-second structure in mouse behavior. Neuron 88(6):1121–1135

    Article  CAS  Google Scholar 

  19. Lee B, Kiani M, Ghovanloo M (2014) A smart homecage system with 3D tracking for long-term behavioral experiments. In: Int. Conf. IEEE EMBC, pp 2016-2019

  20. Lee B, Kiani M, Ghovanloo M (2015) A smart wirelessly powered homecage for long-term high-throughput behavioral experiments. IEEE Sensors J 14(9):4905–4916

    Article  Google Scholar 

  21. Ou-Yang TH, Tsai ML, Yen CT, Lin TT (2011) An infrared range camera-based approach for three-dimensional locomotion tracking and pose reconstruction in a rodent. J Neurosci Methods 201(1):116–123

    Article  Google Scholar 

  22. Matsumoto J, Urakawa S, Takamura Y, Malcher-Lopes R, Hori E, Tomaz C, Ono T, Nishijo H (2013) A 3D-video-based computerized analysis of social and sexual interactions in rats. PLoS One 8(10):e78460

    Article  CAS  Google Scholar 

  23. Matsumoto J, Nishimaru H, Ono T, Nishijo H (2017) 3D-video-based computerized behavioral analysis for in vivo neuropharmacology and neurophysiology in rodents. In: Philippu A (ed) In Vivo Neuropharmacology and Neurophysiology. Neuromethods, vol 121. Humana Press, New York

    Google Scholar 

  24. Nakamura A, Funaya H, Uezono N, Nakashima K, Ishida Y, Suzuki T, Wakana S, Shibata T (2015) Low-cost three-dimensional gait analysis system for mice with an infrared depth sensor. Neurosci Res 100:55–62

    Article  Google Scholar 

  25. Xu C, Govindarajan LN, Zhang Y, Cheng L (2017) Lie-X: depth image based articulated object pose estimation, tracking, and action recognition on lie groups. Int J Comput Vis (IJCV) 123(3):454–478

    Article  Google Scholar 

  26. Rezaei B, Lowe J, Yee JR et al (2016) Non-contact automatic respiration monitoring in restrained rodents. In: Int. Conf. IEEE EMBC, pp 4946–4950

  27. Rezaei B, Huang X, Yee JR, Ostadabbas S (2017) Long-term non-contact tracking of caged rodents. In: 42nd IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP’17), pp 1952–1956

  28. Monteiro JP, Oliveira HP, Aguirar P, Cardoso JS (2014) A depth-map approach for automatic mice behavior recognition. In: IEEE Int. Conf. Image Processing (ICIP), pp 2261–2265

  29. Hong W, Kennedy A, Burgos-Artizzu XP, Zelikowsky M, Navonne SG, Perona P, Anderson DJ (2015) Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning. PNAS 112(38):E5351–E5360

    Article  CAS  Google Scholar 

  30. Wang Z, Mirbozorgi SA, Ghovanloo M (2015) Towards a Kinect-based behavior recognition and analysis system for small animals. In: IEEE biomed. Circ. Sys. Conf. (BioCAS), pp 683–686

  31. Jia Y, Wang Z, Canales D et al (2016) A wirelessly-powered homecage with animal behavior analysis and closed-loop power control. In: Int Conf IEEE EMBC, pp 6323–6326

  32. Jia Y, Mirbozorgi SA, Wang Z, Ghovanloo M (2017) Position and orientation insensitive wireless power transmission for EnerCage-Homecage system. IEEE Trans Biomed Eng 64(10):2439–2449

    Article  Google Scholar 

  33. Gonzalez RC, Woods RE (2008) Digital image processing, 3rd edn. Prentice Hall, Upper Saddle River

    Google Scholar 

  34. Lorbach M, Poppe R, van Dam EA et al (2015) Automated recognition of social behavior in rats: the role of feature quality. In: Int. Conf. Image analysis processing (ICIAP 2015), pp 565–574

    Chapter  Google Scholar 

  35. Fuzesi T, Daviu N, Cusulin JIW et al (2016) Hypothalamic CRH neurons orchestrate complex behaviours after stress. Nat Commun 7:1–13

    Article  Google Scholar 

  36. Hsu CW, Lin CJ (2002) A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw 13(2):415–425

    Article  Google Scholar 

  37. Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2(2):1–27

    Article  Google Scholar 

  38. Kelland MD, Asdourian D, Kelland DZ (1988) Inhibition and excitation of neck and shoulder muscles during unilateral electrical stimulation of the rat neostriatum. Behav Brain Res 30:1–13

    Article  CAS  Google Scholar 

  39. Chen L, Chan SCY, Yung WH (2002) Rotational behavior and electrophysiological effects induced by GABAB receptor activation in rat globus pallidus. Neuroscience 144(2):417–425

    Article  Google Scholar 

  40. Goutte C, Gaussier E (2005) A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In: 27th European Conf. Advances inform. Retrieval research, pp 345–359

Download references

Acknowledgements

This work was supported in part by the National Institutes of Health award 1R21EB018561 and the National Science Foundation under award ECCS-1407880 and ECCS-1408318. The authors would like to thank Prof. Rainnie and members of his lab at the Emory University Department of Psychiatry for their assistance in conducting the in vivo experiments.

Author information

Authors and Affiliations

  1. GT-Bionics Lab, School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, 30308, USA

    Zheyuan Wang, S. Abdollah Mirbozorgi & Maysam Ghovanloo

Authors
  1. Zheyuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Abdollah Mirbozorgi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maysam Ghovanloo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maysam Ghovanloo.

Ethics declarations

The experiment, which was conducted as part of the evaluation of the EnerCage-HC2 system, was approved by the Institutional Animal Care and Use Committees (IACUC) at Emory University and Georgia Tech.

All procedures performed in studies involving animals were in accordance with the ethical standards of the institution or practice at which the studies were conducted.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Z., Mirbozorgi, S.A. & Ghovanloo, M. An automated behavior analysis system for freely moving rodents using depth image. Med Biol Eng Comput 56, 1807–1821 (2018). https://doi.org/10.1007/s11517-018-1816-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-018-1816-1

Keywords

Search

Navigation