Skip to main content
SpringerLink
Log in

View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrums

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13682))

Included in the following conference series:

Abstract

Understanding and forecasting future trajectories of agents are critical for behavior analysis, robot navigation, autonomous cars, and other related applications. Previous methods mostly treat trajectory prediction as time sequence generation. Different from them, this work studies agents’ trajectories in a “vertical” view, i.e., modeling and forecasting trajectories from the spectral domain. Different frequency bands in the trajectory spectrums could hierarchically reflect agents’ motion preferences at different scales. The low-frequency and high-frequency portions could represent their coarse motion trends and fine motion variations, respectively. Accordingly, we propose a hierarchical network V\(^2\)-Net, which contains two sub-networks, to hierarchically model and predict agents’ trajectories with trajectory spectrums. The coarse-level keypoints estimation sub-network first predicts the “minimal” spectrums of agents’ trajectories on several “key” frequency portions. Then the fine-level spectrum interpolation sub-network interpolates the spectrums to reconstruct the final predictions. Experimental results display the competitiveness and superiority of V\(^2\)-Net on both ETH-UCY benchmark and the Stanford Drone Dataset.

C. Wong and B. Xia—Equal contribution.

Codes are available at https://github.com/cocoon2wong/Vertical.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Dataset splits used to train and validate on SDD are the same as [25].

References

  1. Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., Savarese, S.: Social lstm: human trajectory prediction in crowded spaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961–971 (2016)

    Google Scholar 

  2. Alahi, A., et al.: Learning to predict human behavior in crowded scenes. In: Group and Crowd Behavior for Computer Vision, pp. 183–207. Elsevier (2017)

    Google Scholar 

  3. Cao, D., Li, J., Ma, H., Tomizuka, M.: Spectral temporal graph neural network for trajectory prediction. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1839–1845. IEEE (2021)

    Google Scholar 

  4. Cao, D., et al.: Spectral temporal graph neural network for multivariate time-series forecasting. Adv. Neural Inf. Process. Syst. 33, 17766–17778 (2020)

    Google Scholar 

  5. Chai, Y., Sapp, B., Bansal, M., Anguelov, D.: Multipath: multiple probabilistic anchor trajectory hypotheses for behavior prediction. arXiv preprint arXiv:1910.05449 (2019)

  6. Cheng, D., Kou, K.I.: Fft multichannel interpolation and application to image super-resolution. Signal Process. 162, 21–34 (2019)

    Article  Google Scholar 

  7. Choi, C., Malla, S., Patil, A., Choi, J.H.: Drogon: a trajectory prediction model based on intention-conditioned behavior reasoning. arXiv preprint arXiv:1908.00024 (2019)

  8. Cui, H., et al.: Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 2090–2096. IEEE (2019)

    Google Scholar 

  9. Deo, N., Trivedi, M.M.: Convolutional social pooling for vehicle trajectory prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1468–1476 (2018)

    Google Scholar 

  10. Ergezer, H., Leblebicioğlu, K.: Anomaly detection and activity perception using covariance descriptor for trajectories. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9914, pp. 728–742. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48881-3_51

    Chapter  Google Scholar 

  11. Fernando, T., Denman, S., Sridharan, S., Fookes, C.: Soft+ hardwired attention: an lstm framework for human trajectory prediction and abnormal event detection. Neural Netw. 108, 466–478 (2018)

    Article  Google Scholar 

  12. Girase, H., et al.: Loki: Long term and key intentions for trajectory prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9803–9812 (2021)

    Google Scholar 

  13. Giuliari, F., Hasan, I., Cristani, M., Galasso, F.: Transformer networks for trajectory forecasting, pp. 10335–10342 (2021)

    Google Scholar 

  14. Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A.: Social gan: socially acceptable trajectories with generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2255–2264 (2018)

    Google Scholar 

  15. Huang, Y., Bi, H., Li, Z., Mao, T., Wang, Z.: Stgat: modeling spatial-temporal interactions for human trajectory prediction. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 6272–6281 (2019)

    Google Scholar 

  16. Hug, R., Becker, S., Hübner, W., Arens, M.: Particle-based pedestrian path prediction using lstm-mdl models. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 2684–2691. IEEE (2018)

    Google Scholar 

  17. Ivanovic, B., Pavone, M.: The trajectron: probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2375–2384 (2019)

    Google Scholar 

  18. Kaur, K., Jindal, N., Singh, K.: Fractional fourier transform based riesz fractional derivative approach for edge detection and its application in image enhancement. Signal Process. 180, 107852 (2021)

    Article  Google Scholar 

  19. Kim, B., Kang, C.M., Kim, J., Lee, S.H., Chung, C.C., Choi, J.W.: Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network. In: 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 399–404. IEEE (2017)

    Google Scholar 

  20. Komatsu, T., Tyon, K., Saito, T.: 3-d mean-separation-type short-time dft with its application to moving-image denoising. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 2961–2965 (2017)

    Google Scholar 

  21. Kosaraju, V., Sadeghian, A., Martín-Martín, R., Reid, I., Rezatofighi, H., Savarese, S.: Social-bigat: multimodal trajectory forecasting using bicycle-gan and graph attention networks. Adv. Neural Inf. Process. Syst. 32, 137–146 (2019)

    Google Scholar 

  22. Lee, N., Choi, W., Vernaza, P., Choy, C.B., Torr, P.H., Chandraker, M.: Desire: distant future prediction in dynamic scenes with interacting agents. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 336–345 (2017)

    Google Scholar 

  23. Lerner, A., Chrysanthou, Y., Lischinski, D.: Crowds by example. Comput. Graph. Forum 26(3), 655–664 (2007)

    Article  Google Scholar 

  24. Li, S., Zhou, Y., Yi, J., Gall, J.: Spatial-temporal consistency network for low-latency trajectory forecasting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1940–1949 (2021)

    Google Scholar 

  25. Liang, J., Jiang, L., Hauptmann, A.: SimAug: learning robust representations from simulation for trajectory prediction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12358, pp. 275–292. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58601-0_17

    Chapter  Google Scholar 

  26. Liang, J., Jiang, L., Murphy, K., Yu, T., Hauptmann, A.: The garden of forking paths: towards multi-future trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10508–10518 (2020)

    Google Scholar 

  27. Liang, R., Li, Y., Li, X., Zhou, J., Zou, W., et al.: Temporal pyramid network for pedestrian trajectory prediction with multi-supervision. arXiv preprint arXiv:2012.01884 (2020)

  28. Mangalam, K., An, Y., Girase, H., Malik, J.: From goals, waypoints & paths to long term human trajectory forecasting. arXiv preprint arXiv:2012.01526 (2020)

  29. Mangalam, K., et al.: It is not the journey but the destination: endpoint conditioned trajectory prediction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 759–776. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_45

    Chapter  Google Scholar 

  30. Manh, H., Alaghband, G.: Scene-lstm: a model for human trajectory prediction. arXiv preprint arXiv:1808.04018 (2018)

  31. Mao, W., Liu, M., Salzmann, M.: History repeats itself: human motion prediction via motion attention. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12359, pp. 474–489. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58568-6_28

    Chapter  Google Scholar 

  32. Mao, W., Liu, M., Salzmann, M., Li, H.: Learning trajectory dependencies for human motion prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9489–9497 (2019)

    Google Scholar 

  33. Marchetti, F., Becattini, F., Seidenari, L., Bimbo, A.D.: Mantra: memory augmented networks for multiple trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7143–7152 (2020)

    Google Scholar 

  34. Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 935–942. IEEE (2009)

    Google Scholar 

  35. Mohamed, A., Qian, K., Elhoseiny, M., Claudel, C.: Social-stgcnn: a social spatio-temporal graph convolutional neural network for human trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14424–14432 (2020)

    Google Scholar 

  36. Morris, B.T., Trivedi, M.M.: Trajectory learning for activity understanding: unsupervised, multilevel, and long-term adaptive approach. IEEE Trans. Pattern Anal. Mach. Intell. 33(11), 2287–2301 (2011)

    Article  Google Scholar 

  37. Pang, B., Zhao, T., Xie, X., Wu, Y.N.: Trajectory prediction with latent belief energy-based model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11814–11824 (2021)

    Google Scholar 

  38. Pellegrini, S., Ess, A., Schindler, K., Van Gool, L.: You’ll never walk alone: modeling social behavior for multi-target tracking. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 261–268. IEEE (2009)

    Google Scholar 

  39. Phan-Minh, T., Grigore, E.C., Boulton, F.A., Beijbom, O., Wolff, E.M.: Covernet: multimodal behavior prediction using trajectory sets. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14074–14083 (2020)

    Google Scholar 

  40. Rhinehart, N., McAllister, R., Kitani, K., Levine, S.: Precog: prediction conditioned on goals in visual multi-agent settings. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2821–2830 (2019)

    Google Scholar 

  41. Robicquet, A., Sadeghian, A., Alahi, A., Savarese, S.: Learning social etiquette: human trajectory understanding in crowded scenes. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 549–565. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_33

    Chapter  Google Scholar 

  42. Sadeghian, A., Kosaraju, V., Sadeghian, A., Hirose, N., Rezatofighi, H., Savarese, S.: Sophie: an attentive gan for predicting paths compliant to social and physical constraints. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1349–1358 (2019)

    Google Scholar 

  43. Saleh, F., Aliakbarian, S., Salzmann, M., Gould, S.: Artist: autoregressive trajectory inpainting and scoring for tracking. arXiv preprint arXiv:2004.07482 (2020)

  44. Salzmann, T., Ivanovic, B., Chakravarty, P., Pavone, M.: Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12363, pp. 683–700. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58523-5_40

    Chapter  Google Scholar 

  45. Shafiee, N., Padir, T., Elhamifar, E.: Introvert: human trajectory prediction via conditional 3D attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16815–16825 (2021)

    Google Scholar 

  46. Sun, J., Jiang, Q., Lu, C.: Recursive social behavior graph for trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 660–669 (2020)

    Google Scholar 

  47. Tran, H., Le, V., Tran, T.: Goal-driven long-term trajectory prediction. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 796–805 (2021)

    Google Scholar 

  48. Trautman, P., Krause, A.: Unfreezing the robot: navigation in dense, interacting crowds. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 797–803. IEEE (2010)

    Google Scholar 

  49. Vaswani, A., et al.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30, 5998–6008 (2017)

    Google Scholar 

  50. Vemula, A., Muelling, K., Oh, J.: Social attention: modeling attention in human crowds. In: 2018 IEEE international Conference on Robotics and Automation (ICRA), pp. 1–7. IEEE (2018)

    Google Scholar 

  51. Wong, C., Xia, B., Peng, Q., You, X.: Msn: multi-style network for trajectory prediction. arXiv preprint arXiv:2107.00932 (2021)

  52. Xia, B., Wong, C., Peng, Q., Yuan, W., You, X.: Cscnet: contextual semantic consistency network for trajectory prediction in crowded spaces. Pattern Recogn. 126, 108552 (2022)

    Google Scholar 

  53. Xie, D., Shu, T., Todorovic, S., Zhu, S.C.: Learning and inferring “dark matter" and predicting human intents and trajectories in videos. IEEE Trans. Pattern Anal. Mach. Intell. 40(7), 1639–1652 (2017)

    Article  Google Scholar 

  54. Xu, Y., Piao, Z., Gao, S.: Encoding crowd interaction with deep neural network for pedestrian trajectory prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5275–5284 (2018)

    Google Scholar 

  55. Xue, H., Huynh, D.Q., Reynolds, M.: SS-LSTM: a hierarchical lstm model for pedestrian trajectory prediction. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1186–1194. IEEE (2018)

    Google Scholar 

  56. Xue, H., Huynh, D.Q., Reynolds, M.: Scene gated social graph: pedestrian trajectory prediction based on dynamic social graphs and scene constraints. arXiv preprint arXiv:2010.05507 (2020)

  57. Yu, C., Ma, X., Ren, J., Zhao, H., Yi, S.: Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12357, pp. 507–523. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_30

    Chapter  Google Scholar 

  58. Yuan, Y., Weng, X., Ou, Y., Kitani, K.M.: Agentformer: agent-aware transformers for socio-temporal multi-agent forecasting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9813–9823 (2021)

    Google Scholar 

  59. Zhang, P., Ouyang, W., Zhang, P., Xue, J., Zheng, N.: SR-LSTM: state refinement for lstm towards pedestrian trajectory prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12085–12094 (2019)

    Google Scholar 

  60. Zhang, P., Xue, J., Zhang, P., Zheng, N., Ouyang, W.: Social-aware pedestrian trajectory prediction via states refinement lstm. IEEE Trans. Pattern Anal. Mach. Intell. (2020)

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (Grant No. 62172177), and in part by the Fundamental Research Funds for the Central Universities (Grant No. 2021yjsCXCY040).

Author information

Authors and Affiliations

  1. Huazhong University of Science and Technology, Wuhan, China

    Conghao Wong, Beihao Xia, Ziming Hong, Qinmu Peng, Wei Yuan & Xinge You

  2. JD Explore Academy, Beijing, China

    Qiong Cao & Yibo Yang

Authors
  1. Conghao Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Beihao Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ziming Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qinmu Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qiong Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yibo Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xinge You
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xinge You .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wong, C. et al. (2022). View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrums. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13682. Springer, Cham. https://doi.org/10.1007/978-3-031-20047-2_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20047-2_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20046-5

  • Online ISBN: 978-3-031-20047-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation