Skip to main content
SpringerLink
Log in

Obtaining three-dimensional trajectory of multiple fish in water tank via video tracking

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Accurately and reliably obtaining the three-dimensional motion data of individuals in fish schools is not only valuable for fish behaviour analysis and hydrodynamics studies but also be helpful in areas such as bio-inspired robot design. Video tracking is the most effective gateway to obtain the quantitative motion data of continuously moving objects. In this paper we propose a method for obtaining the quantitative three-dimensional trajectory of individuals in fish schools. The proposed method works on videos captured by multiple synchronized cameras with the help of a suggested three-camera imaging system. The proposed method follows the master-slave paradigm in which it tracks fish in the master view with the help of the convolutional neural network in the first, and then associates each certain fish in the master view with detections in the slave view by formulating the cross-view data association as the moment-wise linear assignment problems. Experiments have conducted on public datasets to completely evaluate the performance. The proposed method outperforms other state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Andriyenko A, Schindler K (2011) Multi-target tracking by continuous energy minimization. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 1265–1272

  2. Andriyenko A, Schindler K, Roth S (2012) Discrete-continuous optimization for multi-target tracking. In: 2012 IEEE conference on computer vision and pattern recognition (CVPR), pp 1926–1933

  3. Ardekani R, Biyani A, Dalton JE, Saltz JB, Arbeitman MN, Tower J et al. (2013) Three-dimensional tracking and behaviour monitoring of multiple fruit flies. J R Soc Interface 10(78):1–13

    Google Scholar 

  4. Berclaz J, Fleuret F, Turetken E, Fua P (2011) Multiple object tracking using k-shortest paths optimization. IEEE Trans Pattern Anal Mach Intell 33(9):1806–1819

    Article  Google Scholar 

  5. Butail S, Paley DA (2012) Three-dimensional reconstruction of the fast-start swimming kinematics of densely schooling fish. J R Soc Interface 9(66):77–88

    Article  Google Scholar 

  6. Cheng XE, Wang SH, Chen YQ (2016) 3d tracking targets via kinematic model weighted particle filter. In: 2016 IEEE international conference on multimedia and expo (ICME), pp 1–6

  7. Cui J, Liu Y, Xu Y, Zhao H, Zha H (2013) Tracking generic human motion via fusion of low- and high-dimensional approaches. IEEE Trans Syst Man Cybern Syst 43(4):996–1002

    Article  Google Scholar 

  8. Daugman JG (1985) Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. JOSA A 2 (7):1160–1169

    Article  Google Scholar 

  9. Delcourt J, Beco C, Ylieff MY, Caps H, Vandewalle N, Poncin P (2006) Comparing the ethovision 2.3 system and a new computerized multitracking prototype system to measure the swimming behavior in fry fish. Behav Res Methods 38(4):704–710

    Article  Google Scholar 

  10. Dell AI, Bender JA, Branson K, Couzin ID, de Polavieja GG, Noldus LP et al. (2014) Automated image-based tracking and its application in ecology. Trends Ecol Evol 29(7):417–428

    Article  Google Scholar 

  11. Farnebäck G (2003) Two-frame motion estimation based on polynomial expansion. In: Image analysis: 13th Scandinavian conference, vol 2749, pp 363–370

  12. Friedman M, Kandel A (1999) Introduction to pattern recognition: statistical, structural, neural and fuzzy logic approaches, vol 32. World Scientific Publishing Co Inc

  13. Hartley R, Zisserman A (2004) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  14. Jonker R, Volgenant A (1987) A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing 38(4):325–340

    Article  MathSciNet  MATH  Google Scholar 

  15. Kuhn HW (1955) The hungarian method for the assignment problem. Naval Res Logist Quart 2(1–2):83–97

    Article  MathSciNet  MATH  Google Scholar 

  16. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521 (7553):436–444

    Article  Google Scholar 

  17. Li Y, Huang C, Nevatia R (2009) Learning to associate: hybridboosted multi-target tracker for crowded scene. In: IEEE conference on computer vision and pattern recognition, 2009. CVPR 2009, pp 2953–2960

  18. Liu J, Hu H (2010) Biological inspiration: from carangiform fish to multi-joint robotic fish. J Bionic Eng 7(1):35–48

    Article  Google Scholar 

  19. Liu C, Wechsler H (2002) Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. IEEE Trans Image Process 11 (4):467–476

    Article  Google Scholar 

  20. Liu Y, Zhang X, Cui J, Wu C, Aghajan H, Zha H (2010) Visual analysis of child-adult interactive behaviors in video sequences. In: 2010 16th international conference on virtual systems and multimedia, pp 26–33

  21. Liu Y, Cui J, Zhao H, Zha H (2012) Fusion of low-and high-dimensional approaches by trackers sampling for generic human motion tracking. In: Proceedings of the 21st international conference on pattern recognition (ICPR2012), pp 898–901

  22. Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2activity: recognizing complex activities from sensor data. In: IJCAI, pp 1617–1623

  23. Liu Y, Wang S, Qiu Chen Y (2015) Automatic 3d tracking system for large swarm of moving objects. Pattern Recogn

  24. Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016) Recognizing complex activities by a probabilistic interval-based model. In: AAAI, vol 30, pp 1266–1272

  25. Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115

    Article  Google Scholar 

  26. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  27. Milan A, Schindler K, Roth S (2013) Detection- and trajectory-level exclusion in multiple object tracking. In: 2013 IEEE conference on computer vision and pattern recognition (CVPR), pp 3682–3689

  28. Milan A, Roth S, Schindler K (2014) Continuous energy minimization for multitarget tracking. IEEE Trans Pattern Anal Mach Intell 36(1):58–72

    Article  Google Scholar 

  29. Milan A, Schindler K, Roth S (2015) Multi-target tracking by discrete-continuous energy minimization. IEEE Trans Pattern Anal Mach Intell 38(10):2054–2068

    Article  Google Scholar 

  30. Nimkerdphol K, Nakagawa M (2008) Effect of sodium hypochlorite on zebrafish swimming behavior estimated by fractal dimension analysis. J Biosci Bioeng 105 (5):486–492

    Article  Google Scholar 

  31. Pereira P, Oliveira RF (1994) A simple method using a single video camera to determine the three-dimensional position of a fish. Behav Res Methods Instrum Comput 26(4):443–446

    Article  Google Scholar 

  32. Pérez-Escudero A, Vicente-Page J, Hinz RC, Arganda S, de Polavieja GG (2014) idTracker: tracking individuals in a group by automatic identification of unmarked animals. Nat Meth 11(7):743–748

    Article  Google Scholar 

  33. Pirsiavash H, Ramanan D, Fowlkes CC (2011) Globally-optimal greedy algorithms for tracking a variable number of objects. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 1201–1208

  34. Qian ZM, Cheng XE, Chen YQ (2014) Automatically detect and track multiple fish swimming in shallow water with frequent occlusion. PLoS ONE 9(9):e106,506

    Article  Google Scholar 

  35. Rosemberg DB, Braga MM, Rico EP, Loss CM, Córdova S D, Mussulini BHM, Blaser RE, Leite CE, Campos MM, Dias RD (2012) Behavioral effects of taurine pretreatment in zebrafish acutely exposed to ethanol. Neuropharmacology 63(4):613–623

    Article  Google Scholar 

  36. Rosenthal SB, Twomey CR, Hartnett AT, Wu HS, Couzin ID (2015) Revealing the hidden networks of interaction in mobile animal groups allows prediction of complex behavioral contagion. Proc Natl Acad Sci 112(15):4690–4695

    Article  Google Scholar 

  37. Sakakibara J, Nakagawa M, Yoshida M (2004) Stereo-piv study of flow around a maneuvering fish. Exper Fluids 36(2):282–293

    Article  Google Scholar 

  38. Strandburg-Peshkin A, Twomey CR, Bode NWF, Kao AB, Katz Y, Ioannou CC, Rosenthal SB, Torney CJ, Wu HS, Levin SA, Couzin ID (2013) Visual sensory networks and effective information transfer in animal groups. Curr Biol 23(17):R709–R711

    Article  Google Scholar 

  39. Straw A, Branson K, Neumann T, Dickinson M (2011) Multi-camera real-time three-dimensional tracking of multiple flying animals. J R Soc Interface 8 (56):395–409

    Article  Google Scholar 

  40. Tan X, Chen S, Zhou ZH, Zhang F (2006) Face recognition from a single image per person: a survey. Pattern Recogn 39(9):1725–1745

    Article  MATH  Google Scholar 

  41. Tu X, Terzopoulos D (1994) Artificial fishes: physics, locomotion, perception, behavior. In: Proceedings of the 21st annual conference on computer graphics and interactive techniques, pp 43–50

  42. Vedaldi A, Lenc K (2015) MatConvNet – convolutional neural networks for matlab. In: Proceeding of the ACM int. conf. on multimedia, pp 1–4

  43. Voesenek CJ, Pieters RP, van Leeuwen JL (2016) Automated reconstruction of three-dimensional fish motion, forces, and torques. PloS one 11(1):e0146,682

    Article  Google Scholar 

  44. Wu Z, Hristov NI, Hedrick T, Kunz T, Betke M (2009) Tracking a large number of objects from multiple views. In: 2009 IEEE International conference on computer vision (ICCV). IEEE, pp 1546–1553

  45. Wu Z, Fuller N, Theriault D, Betke M (2014) A thermal infrared video benchmark for visual analysis. In: Proceeding of 10th IEEE workshop on perception beyond the visible spectrum (PBVS), pp 1–8

  46. Xu Z, Cheng XE (2017) Zebrafish tracking using convolutional neural networks. Sci Rep 7(42):815

    Google Scholar 

  47. Zhu L, Weng W (2007) Catadioptric stereo-vision system for the real-time monitoring of 3d behavior in aquatic animals. Physiol Behav 91(1):106–119

    Article  Google Scholar 

Download references

Acknowledgements

We thank Shuo Hong Wang at Fudan University for technical help with the experiments.

Author information

Authors and Affiliations

  1. Jingdezhen Ceramic Institute, Jingdezhen, China

    Xi En Cheng, Hui Ying Li, Jing Fang Hu & Ming Lu Chen

  2. School of Computer Science, Fudan University, Shanghai, China

    Xi En Cheng

  3. School of Information Science and Technology, Fudan University, Shanghai, China

    Shan Shan Du

Authors
  1. Xi En Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shan Shan Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hui Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jing Fang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ming Lu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xi En Cheng.

Additional information

The research work presented in this paper is partly supported by Educational Commission of Jiangxi Province, China (Grant No. GJJ160911,GJJ160904) and partly supported by the Science and Technology Program of Jingdezhen City, China (Grant No. 2017GYZD016-02).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheng, X.E., Du, S.S., Li, H.Y. et al. Obtaining three-dimensional trajectory of multiple fish in water tank via video tracking. Multimed Tools Appl 77, 24499–24519 (2018). https://doi.org/10.1007/s11042-018-5755-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5755-5

Keywords

Search

Navigation