Skip to main content
SpringerLink
Log in

FDNet: A Deep Learning Approach with Two Parallel Cross Encoding Pathways for Precipitation Nowcasting

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

With the goal of predicting the future rainfall intensity in a local region over a relatively short period time, precipitation nowcasting has been a long-time scientific challenge with great social and economic impact. The radar echo extrapolation approaches for precipitation nowcasting take radar echo images as input, aiming to generate future radar echo images by learning from the historical images. To effectively handle complex and high non-stationary evolution of radar echoes, we propose to decompose the movement into optical flow field motion and morphologic deformation. Following this idea, we introduce Flow-Deformation Network (FDNet), a neural network that models flow and deformation in two parallel cross pathways. The flow encoder captures the optical flow field motion between consecutive images and the deformation encoder distinguishes the change of shape from the translational motion of radar echoes. We evaluate the proposed network architecture on two real-world radar echo datasets. Our model achieves state-of-the-art prediction results compared with recent approaches. To the best of our knowledge, this is the first network architecture with flow and deformation separation to model the evolution of radar echoes for precipitation nowcasting. We believe that the general idea of this work could not only inspire much more effective approaches but also be applied to other similar spatio-temporal prediction tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. Sun J Z, Xue M, Wilson J W, Zawadzki I, Ballard S P, Onvlee-Hooimeyer J, Joe P, Barker D M, Li P W, Golding B, Xu M, Pinto J. Use of NWP for nowcasting convective precipitation: Recent progress and challenges. Bulletin of the American Meteorological Society, 2014, 95(3): 409–426. https://doi.org/10.1175/BAMS-D-11-00263.1.

    Article  Google Scholar 

  2. Bauer P, Thorpe A, Brunet G. The quiet revolution of numerical weather prediction. Nature, 2015, 525(7567): 47–55. https://doi.org/10.1038/nature14956.

    Article  Google Scholar 

  3. Cheung P, Yeung H Y. Application of optical-flow technique to significant convection nowcast for terminal areas in Hong Kong. In Proc. the 3rd WMO International Symposium on Nowcasting and Very Short-Range Forecasting (WSN12), Aug. 2012, pp.6–10.

  4. Bowler N E H, Pierce C E, Seed A. Development of a precipitation nowcasting algorithm based upon optical flow techniques. Journal of Hydrology, 2004, 288(1/2): 74–91. https://doi.org/10.1016/j.jhydrol.2003.11.011.

    Article  Google Scholar 

  5. Germann U, Zawadzki I. Scale-dependence of the predictability of precipitation from continental radar images. Part I: Description of the methodology. Monthly Weather Review, 2002, 130(12): 2859–2873. 10.1175/1520-0493(2002)130<2859:SDOTPO>2.0.CO;2.

  6. Sakaino H. Spatio-temporal image pattern prediction method based on a physical model with time-varying optical flow. IEEE Trans. Geoscience and Remote Sensing, 2013, 51(5): 3023–3036. https://doi.org/10.1109/TGRS.2012.2212201.

    Article  Google Scholar 

  7. Shi X J, Chen Z R, Wang H, Yeung D Y, Wong W K, Woo W C. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Proc. the 28th International Conference on Neural Information Processing Systems, Dec. 2015, pp.802–810. https://doi.org/10.1007/978-3-319-21233-3_6.

  8. Shi X J, Gao Z H, Lausen L, Wang H, Yeung D Y, Wong W K, Woo W C. Deep learning for precipitation nowcasting: A benchmark and a new model. In Proc. the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp.5617–5627.

  9. Wang Y B, Long M S, Wang J M, Gao Z F, Philip S Y. PredRNN: Recurrent neural networks for predictive learning using spatiotemporal LSTMs. In Proc. the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp.879–888.

  10. Villegas R, Yang J M, Hong S, Lin X Y, Lee H. Decomposing motion and content for natural video sequence prediction. In Proc. the 5th International Conference on Learning Representations, Apr. 2017.

  11. Wang Y B, Zhang J J, Zhu H Y, Long M S, Wang J M, Yu P S. Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp.9146–9154. https://doi.org/10.1109/CVPR.2019.00937.

  12. Ilg E, Mayer N, Saikia T, Keuper M, Dosovitskiy A, Brox T. FlowNet 2.0: Evolution of optical flow estimation with deep networks. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp.1647–1655. https://doi.org/10.1109/CVPR.2017.179.

  13. Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. In Proc. the 26th International Conference on Neural Information Processing Systems, Dec. 2012, pp.1097–1105. https://doi.org/10.1145/3065386.

  14. Sutskever I, Vinyals O, Le Q V. Sequence to sequence learning with neural networks. In Proc. the 28th International Conference on Neural Information Processing Systems, Dec. 2014, pp.3104–3112.

  15. Mathieu M, Couprie C, LeCun Y. Deep multi-scale video prediction beyond mean square error. In Proc. the 4th International Conference on Learning Representations, May 2016.

  16. Yu B, Yin H T, Zhu Z X. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In Proc. the 27th International Joint Conference on Artificial Intelligence, Jul. 2018, pp.3634–3640. https://doi.org/10.24963/ijcai.2018/505.

  17. Vondrick C, Pirsiavash H, Torralba A. Generating videos with scene dynamics. In Proc. the 30th International Conference on Neural Information Processing Systems, Dec. 2016, pp.613–621.

  18. Li Y G, Yu R, Shahabi C, Liu Y. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In Proc. the 6th International Conference on Learning Representations, Apr. 30–May 3, 2018.

  19. Ranzato M, Szlam A, Bruna J, Mathieu M, Collobert R, Chopra S. Video (language) modeling: A baseline for generative models of natural videos. arXiv: 1412.6604, 2014. https://arxiv.org/abs/1412.6604, Aug. 2023.

  20. Srivastava N, Mansimov E, Salakhutdinov R. Unsupervised learning of video representations using LSTMs. In Proc. the 32nd International Conference on International Conference on Machine Learning, Jul. 2015, pp.843–852.

  21. De Brabandere B, Jia X, Tuytelaars T, Van Gool L. Dynamic filter networks. In Proc. the 30th International Conference on Neural Information Processing Systems, Dec. 2016, pp.667–675.

  22. Wang Y B, Jiang L, Yang M H, Li L J, Long M S, Li F F. Eidetic 3D LSTM: A model for video prediction and beyond. In Proc. the 2019 International Conference on Machine Learning, May 2019.

  23. Su J H, Byeon W, Kossaifi J, Huang F R, Kautz J, Anandkumar A. Convolutional tensor-train LSTM for spatio-temporal learning. arXiv: 2002.09131, 2020. https://arxiv.org/abs/2002.09131v2, Aug. 2023.

  24. Lin Z H, Li M M, Zheng Z B, Cheng Y Y, Yuan C. Selfattention ConvLSTM for spatiotemporal prediction. In Proc. the 34th AAAI Conference on Artificial Intelligence, Feb. 2020, pp.11531–11538. https://doi.org/10.1609/aaai.v34i07.6819.

  25. Tran Q K, Song S K. Multi-channel weather radar echo extrapolation with convolutional recurrent neural networks. Remote Sensing, 2019, 11(19): 2303. https://doi.org/10.3390/rs11192303.

    Article  Google Scholar 

  26. Dosovitskiy A, Fischer P, Ilg E, Häusser P, Hazirbas C, Golkov V, van der Smagt P, Cremers D, Brox T. FlowNet: Learning optical flow with convolutional networks. In Proc. the 2015 IEEE International Conference on Computer Vision, Dec. 2015, pp.2758–2766. https://doi.org/10.1109/ICCV.2015.316.

  27. Jaderberg M, Simonyan K, Zisserman A, Kavukcuoglu K. Spatial transformer networks. In Proc. the 29th International Conference on Neural Information Processing Systems, Dec. 2015. pp.2017–2025.

  28. Zeiler M D, Taylor G W, Fergus R. Adaptive deconvolutional networks for mid and high level feature learning. In Proc. the 2011 International Conference on Computer Vision, Nov. 2011, pp.2018–2025. https://doi.org/10.1109/ICCV.2011.6126474.

  29. Kingma D P, Ba J. Adam: A method for stochastic optimization. In Proc. the 3rd International Conference on Learning Representations, May 2015.

  30. Maas A L, Hannun A Y, Ng A Y. Rectifier nonlinearities improve neural network acoustic models. In Proc. the 30th International Conference on Machine Learning, June 2013.

  31. Bengio S, Vinyals O, Jaitly N, Shazeer N. Scheduled sampling for sequence prediction with recurrent neural networks. In Proc. the 29th International Conference on Neural Information Processing Systems, Dec. 2015, pp.1171–1179.

  32. Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. the 13th International Conference on Artificial Intelligence and Statistics, May 2010, pp.249–256.

Download references

Author information

Authors and Affiliations

  1. University of Chinese Academy of Sciences, Beijing, 100049, China

    Bi-Ying Yan

  2. Institute of Software, Chinese Academy of Sciences, Beijing, 100190, China

    Bi-Ying Yan & Feng Chen

  3. School of Mathematical Sciences, Peking University, Beijing, 100871, China

    Chao Yang

  4. Peng Cheng Laboratory, Shenzhen, 518052, China

    Chao Yang

  5. Guiyang Academy of Information Technology, Guiyang, 550081, China

    Feng Chen

  6. NTT DATA Corporation, Tokyo, 163-8001, Japan

    Kohei Takeda

  7. NTT DATA Institute of Management Consulting Inc., Tokyo, 163-8001, Japan

    Changjun Wang

Authors
  1. Bi-Ying Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Feng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kohei Takeda
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Changjun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chao Yang.

Supplementary Information

ESM 1

(PDF 188 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, BY., Yang, C., Chen, F. et al. FDNet: A Deep Learning Approach with Two Parallel Cross Encoding Pathways for Precipitation Nowcasting. J. Comput. Sci. Technol. 38, 1002–1020 (2023). https://doi.org/10.1007/s11390-021-1103-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-021-1103-8

Keywords

Search

Navigation