Abstract
We present a deep-learning based tracking objects of interest in walking droplet and granular intruder experiments. In a typical walking droplet experiment, a liquid droplet, known as walker, propels itself laterally on the free surface of a vibrating bath of the same liquid. This lateral motion stems from the interaction between the droplet and the wave it generates upon successive bounces off the vibrating liquid surface. A walker can exhibit a highly irregular trajectory over the course of its motion, including rapid acceleration and complex interactions with the other walkers present in the bath. In analogy with the hydrodynamic experiments, the granular matter experiments consist of a vibrating bath of very small solid particles and a larger solid called intruder. Like the fluid droplets, the intruder interacts with and travels the domain due to the waves of the bath but tends to move much slower and much less smoothly than the droplets. When multiple intruders are introduced, they also exhibit complex interactions with each other. We leverage the state-of-the-art object detection model YOLO (You Only Look Once) and the Hungarian Algorithm to accurately extract the trajectory of a walker or intruder in real-time. Our proposed methodology is capable of tracking individual walker(s) or intruder(s) in digital images acquired from a broad spectrum of experimental settings and does not suffer from any identity-switch issues. Thus, the deep learning approach developed in this work could be used to automatize the efficient, fast and accurate extraction of observables of interests in walking droplet, granular intruder experiments and similar particle tracking experiments. Such extraction capabilities are critically enabling for downstream tasks such as building data-driven dynamical models for the coarse-grained dynamics and interactions of the objects of interest.
Similar content being viewed by others
Data availability
The codes, datasets, and results of this study are available in GitHub repository [95]. We also provide detailed step-by-step tutorial on how to adopt our repository for similar problem domains.
References
Couder, Y., Protiere, S., Fort, E., Boudaoud, A.: Dynamical phenomena: walking and orbiting droplets. Nature 437, 208 (2005)
Protiere, S., Boudaoud, A., Couder, Y.: Particle-wave association on a fluid interface. J. Fluid Mech. 554, 85 (2006)
Couder, Y., Fort, E.: Single-particle diffraction and interference at a macroscopic scale. Phys. Rev. Lett. 97, 154101 (2006)
Bush, J.: Quantum mechanics writ large. Proc. Nat. Acad. Sci. 107(41), 17455–17456 (2010)
Harris, D., Moukhtar, J., Fort, E., Couder, Y., Bush, J.: Wavelike statistics from pilot-wave dyanmics in a circular corral. Phys. Rev. E 88, 011001 (2013)
Bush, J.: Pilot-wave hydrodynamics. Ann. Rev. Fluid Mech. 49, 269 (2015)
Bush, J.: The new wave of pilot-wave theory. Phys. Today 68(8), 47 (2015)
Bush, J.W.M., Oza, A.U.: Bouncing droplet dynamics above the Faraday threshold. Rep. Progress Phys. 84, 017001 (2021)
Tambasco, L.D., Pilgram, J.J., Bush, J.W.M.: Bouncing droplet dynamics above the Faraday threshold. Chaos 28, 096107 (2018)
Fort, E., Eddi, A., Boudaoud, A., Moukhtar, J., Couder, Y.: Path-memory induced quantization of classical orbits. Proc. Nat. Acad. Sci. 107, 17515 (2010)
Oza, A., Harris, D., Rosales, R., Bush, J.: Pilot-wave dynamics in a rotating frame: on the emergence of orbital quantization. J. Fluid Mech. 744, 404 (2014)
Oza, A., Wind-Willassen, O., Harris, D., Rosales, R., Bush, J.: Pilot-wave dynamics in a rotating frame: exotic orbits. Phys. Fluids 26, 082101 (2014)
Harris, D., Bush, J.: Droplets walking in a rotating frame: from quantized orbits to multimodal statistics. J. Fluid Mech. 739, 444 (2014)
Tambasco, L.D., Harris, D.M., Oza, A.U., Rosales, R.R., Bush, J.W.M.: The onset of chaos in orbital pilot-wave dynamics. Chaos 26, 103107 (2016)
Oza, A.U., Siéfert, E., Harris, D.M., Molacek, J., Bush, J.W.M.: Orbiting pairs of walking droplets: dynamics and stability. Phys. Rev. F 2, 053601 (2017)
Gilet, T.: Dynamics and statistics of wave-particle interaction in a confined geometry. Phys. Rev. E 90, 052917 (2014)
Bush, J.W.M., Couder, Y., Gilet, T., Milewski, P.A., Nachbin, A.: Introduction to focus issue on hydrodynamic quantum analogs. Chaos 28, 096001 (2018)
Rahman, A., Joshi, Y., Blackmore, D.: Sigma map dynamics and bifurcations. Regul. Chaotic Dyn. 22(6), 740 (2017)
Rahman, A.: Standard map-like models for single and multiple walkers in an annular cavity. Chaos 28, 096102 (2018)
Rahman, A., Blackmore, D.: Interesting bifurcations in walking droplet dynamics. Commun. Nonlinear Sci. Numer. Simul. 90, 105348 (2020)
Durey, M.: Bifurcations and chaos in a Lorenz-like pilot-wave system. Chaos 30, 103115 (2020)
Rahman, A., Blackmore, D.: Walking droplets through the lens of dynamical systems. Mod. Phys. Lett. B 34(34), 2030009 (2020)
Valani, R.N., Slim, A.C., Paganin, D.M., Simula, T.P., Vo, T.: Unsteady dynamics of a classical particle-wave entity. Phys. Rev. E 104, 015106 (2021)
Valani, R.N.: Lorenz-like systems emerging from an integro-differential trajectory equation of a one-dimensional wave-particle entity. Chaos 32(2), 023129 (2022)
Valani, R.N., Slim, A.C.: Pilot-wave dynamics of two identical, in-phase bouncing droplets. Chaos 28, 096114 (2018)
Choueiri, G., Suri, B., Merrin, J., Serbyn, M., Hof, B., Budanur, N.B.: Crises and chaotic scattering in hydrodynamic pilot-wave experiments., Arxiv (2022)
Metcalf, T.H., Knight, J.B., Jaeger, H.M.: Standing wave patterns in shallow beds of vibrated granular material. Phys. A 236(3), 202 (1997)
Melo, F., Umbanhowar, P., Swinney, H.L.: Transition to parametric wave patterns in a vertically oscillated granular layer. Phys. Rev. Lett. 72(1), 172 (1994)
Eshuis, P., Van Der Weele, K., Van Der Meer, D., Bos, R., Lohse, D.: Phase diagram of vertically shaken granular matter. Phys. Fluids 19(12), 123301 (2007)
Kudrolli, A.: Size separation in vibrated granular matter. Rep. Progress Phys. 67(3), 209 (2004). https://doi.org/10.1088/0034-4885/67/3/R01
Knight, J.B., Ehrichs, E.E., Kuperman, V.Y., Flint, J.K., Jaeger, H.M., Nagel, S.R.: Experimental study of granular convection. Phys. Rev. E 54(5), 5726 (1996). https://doi.org/10.1103/PhysRevE.54.5726
Metzger, M.J., Remy, B., Glasser, B.J.: All the Brazil nuts are not on top: vibration induced granular size segregation of binary, ternary and multi-sized mixtures. Powder Technol. 205(1), 42 (2011). https://doi.org/10.1016/j.powtec.2010.08.062
Srikanth, S., Dubey, S.K., Javed, A., Goel, S.: Droplet based microfluidics integrated with machine learning. Sens. Actuators A Phys. 332, 113096 (2021)
Durve, M., Bonaccorso, F., Montessori, A., Lauricella, M., Tiribocchi, A., Succi, S.: A fast and efficient deep learning procedure for tracking droplet motion in dense microfluidic emulsions. Philos. Trans. R. Soc. A 379(2208), 20200400 (2021)
Durve, M., Tiribocchi, A., Bonaccorso, F., Montessori, A., Lauricella, M., Bogdan, M., Guzowski, J., Succi, S.: DropTrack-automatic droplet tracking with YOLOv5 and DeepSORT for microfluidic applications. Phys. Fluids 34(8), 082003–7 (2022)
Durve, M., Bonaccorso, F., Montessori, A., Lauricella, M., Tiribocchi, A., Succi, S.: Tracking droplets in soft granular flows with deep learning techniques. Eur. Phys. J. Plus 136(8), 864 (2021)
Jocher G.: Ultralytics/yolov5: v3.1 - bug fixes and performance improvements. (2020). https://github.com/ultralytics/yolov5https://doi.org/10.5281/zenodo.4154370, Accessed 15 Apr 2023
Wojke, N., Bewley, A., Paulus, D.: in 2017 IEEE international conference on image processing (ICIP) (IEEE, 2017), Simple online and realtime tracking with a deep association metric, pp. 3645–3649 (2017)
Rutkowski, G.P., Azizov, I., Unmann, E., Dudek, M., Grimes, B.A.: Microfluidic droplet detection via region-based and single-pass convolutional neural networks with comparison to conventional image analysis methodologies. Mach. Learn. Appl. 7, 100222 (2022)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6):1137–1149 (2017). https://doi.org/10.1109/tpami.2016.2577031
Valani, R.N., Slim, A.C., Simula, T.P.: Superwalking droplets. Phys. Rev. Lett. 123, 024503 (2019)
Duda, R.O., Hart, P.E.: Use of the hough transformation to detect lines and curves in pictures. Commun. ACM 15(1), 11–15 (1972). https://doi.org/10.1145/361237.361242
Yuen, H., Princen, J., Illingworth, J., Kittler, J.: Comparative study of Hough transform methods for circle finding. Image Vis. Comput. 8(1), 71 (1990)
Atherton, T., Kerbyson, D.: Size invariant circle detection. Image Vis. Comput. 17(11), 795 (1999)
Thapar, S., Garg, S.: Study and implementation of various morphology based image contrast enhancement techniques. Int. J. Comput. Bus. Res. 128, 2229 (2012)
MathWorks. Find edges in 2-D grayscale image - MATLAB edge (2011). https://www.mathworks.com/help/images/ref/edge.html, Accessed 15 April 2023
Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. PAMI–8(6), 679 (1986). https://doi.org/10.1109/TPAMI.1986.4767851
Meyer, F.: Topographic distance and watershed lines. Signal Process. 38(1), 113 (1994). https://doi.org/10.1016/0165-1684(94)90060-4
Friedman, N., Russell, S: Image Segmentation in Video Sequences: A Probabilistic Approach, Image segmentation in video sequences: A probabilistic approach (2013). arxiv:1302.1539
Benraya, I., Benblidia, N.: in 2018 International Conference on Applied Smart Systems (ICASS), Comparison of Background Subtraction methods. pp. 1–5. (2018). https://doi.org/10.1109/ICASS.2018.8652040
Stauffer, C., Grimson, W.E.L.: Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), Adaptive background mixture models for real-time tracking 2, 246 (1999)
Kaewtrakulpong, P., Bowden, R.: An improved adaptive background mixture model for real-time tracking with shadow detection. Springer eBooks, pp 135–144 (2002). https://doi.org/10.1007/978-1-4615-0913-4_11.
Zivkovic, Z.: in Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., Improved adaptive Gaussian mixture model for background subtraction, (2004) 2: 28–31. https://doi.org/10.1109/ICPR.2004.1333992
Zivkovic, Z., van der Heijden, F.: Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognit. Lett. 27(7), 773 (2006)
Grosek, J., Kutz, J.N.: Dynamic mode decomposition for real-time background/foreground separation in video, arXiv preprint arXiv:1404.7592 (2014)
Erichson, N.B., Brunton, S.L., Kutz, J.N.: Compressed dynamic mode decomposition for background modeling. J. Real-Time Image Process. 16(5), 1479 (2019)
Viola, P., Jones, M: in Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001,Rapid object detection using a boosted cascade of simple features, (2001) 1: I–I. https://doi.org/10.1109/CVPR.2001.990517
Seginer, I., Elster, R., Goodrum, J., Rieger, M.: Plant wilt detection by computer-vision tracking of leaf tips. Trans. ASAE 35(5), 1563 (1992). https://doi.org/10.13031/2013.28768
Sites, P.W., Delwiche, M.J.: Computer vision to locate fruit on a tree. Trans. ASAE 31(1), 257 (1988). https://doi.org/10.13031/2013.30697
Zhu, S., Li, C., Rogers, J., Gianni, M., Howard, I., in 2021 27th International Conference on Mechatronics and Machine Vision in Practice (M2VIP), A Real-time Double Emulsion Droplets Detection System using Hough Circle Transform and Color Detection. pp. 36–41. (2021). https://doi.org/10.1109/M2VIP49856.2021.9665023
Kulju, S., Riegger, L., Koltay, P., Mattila, K., Hyväluoma, J.: Fluid flow simulations meet high-speed video: computer vision comparison of droplet dynamics. J. Colloid Interface Sci. 522, 48 (2018)
Zhao, H., Zhou, J., Gu, Y., Benjamin Ho, C.M., Tan, S.H., Gao, Y.: in 2018 IEEE International Conference on Real-time Computing and Robotics (RCAR),Real- Time Computing for Droplet Detection and Recognition. pp. 589–594. (2018). https://doi.org/10.1109/RCAR.2018.8621816
Chong, Z., Tor, S., Gañán-Calvo, A.M.: Automated droplet measurement (ADM): an enhanced video processing software for rapid droplet measurements. Microfluidics Nanofluidics (2016). https://doi.org/10.1007/s10404-016-1722-5
Nakarmi, A.D., Tang, L., Xin, H.: Automated tracking and behavior quantification of laying hens using 3D computer vision and radio frequency identification technologies. Trans. ASABE (2014). https://doi.org/10.13031/trans.57.10505
Ratnayake, M.N., Dyer, A.G., Dorin, A.: Tracking individual honeybees among wildflower clusters with computer vision-facilitated pollinator monitoring. PLOS One 16(2), 1 (2021). https://doi.org/10.1371/journal.pone.0239504
Li, K., Miller, E.D., Chen, M., Kanade, T., Weiss, L.E., Campbell, P.G., in,: 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro. Computer vision tracking of stemness 2008, 847–850 (2008). https://doi.org/10.1109/ISBI.2008.4541129
Szeliski, R.: Computer vision: algorithms and applications, computer vision: algorithms and applications. Springer Nature (2022)
Forsyth, D., Ponce, J.: Computer vision: a modern approach, 2nd edn. Prentice Hall (2011)
Rosenfeld, A.: Computer vision: basic principles. Proc. IEEE 76(8), 863 (1988). https://doi.org/10.1109/5.5961
Zaidi, S.S.A., Ansari, M.S., Aslam, A., Kanwal, N., Asghar, M., Lee, B.: A survey of modern deep learning based object detection models. Digit. Signal Process. (2022). https://doi.org/10.1016/j.dsp.2022.103514
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., Pietikäinen, M.: Deep learning for generic object detection: a survey. Int. J. Comput. Vis. 128(2), 261 (2020)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: in Proceedings of the IEEE conference on computer vision and pattern recognition, You only look once: Unified, real-time object detection, (2016), pp. 779–788
Redmon, J., Farhadi, A.: in Proceedings of the IEEE conference on computer vision and pattern recognition, YOLO9000: better, faster, stronger. pp. 7263–7271 (2017)
Redmon, J., Farhadi, A.: Yolov3: An incremental improvement, arXiv preprint arXiv:1804.02767 (2018)
Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: Optimal speed and accuracy of object detection, arXiv preprint arXiv:2004.10934 (2020)
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: In European conference on computer vision, Microsoft coco: Common objects in context. Springer. pp. 740–755 (2014)
Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 88(2), 303 (2010)
Tzutalin. Labelimg. Free Software: MIT License (2015). https://github.com/tzutalin/labelImg
Jocher, G., Chaurasia, A., Qiu, J.: YOLO by Ultralytics (2023). https://github.com/ultralytics/ultralytics
Pal, S.K., Pramanik, A., Maiti, J., Mitra, P.: Deep learning in multi-object detection and tracking: state of the art. Appl. Intell. 51(9), 6400 (2021)
Milan, A., Schindler, K., Roth, S.: in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Challenges of ground truth evaluation of multi-target tracking, pp. 735–742 (2013)
Ciaparrone, G., Sánchez, F.L., Tabik, S., Troiano, L., Tagliaferri, R., Herrera, F.: Deep learning in video multi-object tracking: a survey. Neurocomputing 381, 61 (2020)
Kuhn, H.W.: The Hungarian method for the assignment problem. Nav. Res. Logist. Q. 2(1–2), 83 (1955)
Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S.J., Brett, M., Wilson, J., Millman, K.J., Mayorov, N., Nelson, A.R.J., Jones, E., Kern, R., Larson, E., Carey, C.J., Polat, İ., Feng, Y., Moore, E.W., VanderPlas, J. J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E.A., Harris, C.R., Archibald, A.M., Ribeiro, A.H., Pedregosa, F., van Mulbregt, P.: SciPy 1.0 contributors, SciPy 1.0: fundamental algorithms for scientific computing in python. Nat. Methods 17, 261 (2020). https://doi.org/10.1038/s41592-019-0686-2
Du, Y., Song, Y., Yang, B., Zhao, Y.: Strongsort: Make deepsort great again, arXiv preprint arXiv:2202.13514 (2022)
Cao, J., Weng, X., Khirodkar, R., Pang, J., Kitani, K.: Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking, arXiv preprint arXiv:2203.14360 (2022)
Maggiolino, G., Ahmad, A., Cao, J., Kitani, K.: Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification, arXiv preprint arXiv:2302.11813 (2023)
Aharon, N., Orfaig, R., Bobrovsky, B.Z.: BoT-SORT: Robust associations multi-pedestrian tracking, arXiv preprint arXiv:2206.14651 (2022)
Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., Wang, X.: in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII, Bytetrack: Multi-object tracking by associating every detection box. Springer. pp. 1–21 (2022)
Broström, M.: Real-time multi-object tracking and segmentation using Yolov8 with StrongSORT and OSNet. Real-time multi-object tracking and segmentation using Yolov8 with StrongSORT and OSNet. https://zenodo.org/record/7629840. https://github.com/mikel-brostrom/yolov8_tracking, Accessed 15 Apr 2023
Brunton, S.L., Proctor, J.L., Kutz, J.N.: Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci. 113(15), 3932 (2016)
Chatterjee, A.: An introduction to the proper orthogonal decomposition, Current science pp. 808–817 (2000)
Kutz, J.N., Brunton, S.L., Brunton, B.W., Proctor, J.L.: Dynamic mode decomposition: data-driven modeling of complex systems, Dynamic mode decomposition: data-driven modeling of complex systems (SIAM, 2016) (2016)
Schmid, P.J.: Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech. 656, 5 (2010)
Kara, E.: Real-time droplet tracking with YOLOv8. https://github.com/erkara/TrackingWalkers-YOLOv8https://doi.org/10.5281/zenodo.7930552
Acknowledgements
The authors acknowledge support from the National Science Foundation AI Institute in Dynamic Systems (grant number 2112085). JNK further acknowledges support from the Air Force Office of Scientific Research (FA9550-19-1-0011).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Kara, E., Zhang, G., Williams, J.J. et al. Deep learning based object tracking in walking droplet and granular intruder experiments. J Real-Time Image Proc 20, 86 (2023). https://doi.org/10.1007/s11554-023-01341-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11554-023-01341-4