Skip to main content
SpringerLink
Log in

Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12374))

Included in the following conference series:

Abstract

The recent advancement of deep learning techniques has made great progress on hyperspectral image super-resolution (HSI-SR). Yet the development of unsupervised deep networks remains challenging for this task. To this end, we propose a novel coupled unmixing network with a cross-attention mechanism, CUCaNet for short, to enhance the spatial resolution of HSI by means of higher-spatial-resolution multispectral image (MSI). Inspired by coupled spectral unmixing, a two-stream convolutional autoencoder framework is taken as backbone to jointly decompose MS and HS data into a spectrally meaningful basis and corresponding coefficients. CUCaNet is capable of adaptively learning spectral and spatial response functions from HS-MS correspondences by enforcing reasonable consistency assumptions on the networks. Moreover, a cross-attention module is devised to yield more effective spatial-spectral information transfer in networks. Extensive experiments are conducted on three widely-used HS-MS datasets in comparison with state-of-the-art HSI-SR models, demonstrating the superiority of the CUCaNet in the HSI-SR application. Furthermore, the codes and datasets are made available at: https://github.com/danfenghong/ECCV2020_CUCaNet.

Danfeng Hong — Corresponding author.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.cs.columbia.edu/CAVE/databases/multispectral.

  2. 2.

    http://naotoyokoya.com/Download.html.

  3. 3.

    http://landsat.gsfc.nasa.gov/?p=5779.

  4. 4.

    We select the spectral radiance responses of blue-green-red(BGR) bands and BGR-NIR bands for the experiments on Pavia and Chikusei datasets, respectively.

  5. 5.

    http://naotoyokoya.com/Download.html.

  6. 6.

    https://github.com/lanha/SupResPALM.

  7. 7.

    https://github.com/qw245/BlindFuse.

  8. 8.

    https://github.com/alfaiate/HySure.

  9. 9.

    http://see.xidian.edu.cn/faculty/wsdong.

  10. 10.

    https://github.com/marhar19/HSR_via_tensor_decomposition.

  11. 11.

    https://sites.google.com/view/renweidian.

  12. 12.

    https://github.com/aicip/uSDN.

  13. 13.

    https://github.com/XieQi2015/MHF-net.

References

  1. Aiazzi, B., Baronti, S., Selva, M.: Improving component substitution pansharpening through multivariate regression of ms \(+ \) pan data. IEEE Trans. Geosci. Remote Sens. 45(10), 3230–3239 (2007)

    Article  Google Scholar 

  2. Akhtar, N., Shafait, F., Mian, A.: Sparse spatio-spectral representation for hyperspectral image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 63–78. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10584-0_5

    Chapter  Google Scholar 

  3. Barsi, J.A., Lee, K., Kvaran, G., Markham, B.L., Pedelty, J.A.: The spectral response of the landsat-8 operational land imager. Remote Sens. 6(10), 10232–10251 (2014)

    Article  Google Scholar 

  4. Bieniarz, J., Cerra, D., Avbelj, J., Reinartz, P., Müller, R.: Hyperspectral image resolution enhancement based on spectral unmixing and information fusion. In: ISPRS Hannover Workshop 2011 (2011)

    Google Scholar 

  5. Dian, R., Li, S., Fang, L.: Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Trans. Neural Netw. Learn. Syst. 30(9), 2672–2683 (2019)

    Article  MathSciNet  Google Scholar 

  6. Dian, R., Li, S., Guo, A., Fang, L.: Deep hyperspectral image sharpening. IEEE Trans. Neural Netw. Learn. Syst. 29(99), 1–11 (2018)

    MathSciNet  Google Scholar 

  7. Dong, W., et al.: Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans. Image Process. 25(5), 2337–2352 (2016)

    Article  MathSciNet  Google Scholar 

  8. Eismann, M.T.: Resolution enhancement of hyperspectral imagery using maximum a posteriori estimation with a stochastic mixing model. Ph.D. thesis, University of Dayton (2004)

    Google Scholar 

  9. Fu, Y., Zhang, T., Zheng, Y., Zhang, D., Huang, H.: Joint camera spectral sensitivity selection and hyperspectral image recovery. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 788–804 (2018)

    Google Scholar 

  10. Fu, Y., Zhang, T., Zheng, Y., Zhang, D., Huang, H.: Hyperspectral image super-resolution with optimized RGB guidance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11661–11670 (2019)

    Google Scholar 

  11. Gao, L., Hong, D., Yao, J., Zhang, B., Gamba, P., Chanussot, J.: Spectral superresolution of multispectral imagery with joint sparse and low-rank learning. IEEE Trans. Geosci. Remote Sens. (2020). https://doi.org/10.1109/TGRS.2020.3000684

    Article  Google Scholar 

  12. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)

    Google Scholar 

  13. Hong, D., Liu, W., Su, J., Pan, Z., Wang, G.: A novel hierarchical approach for multispectral palmprint recognition. Neurocomputing 151, 511–521 (2015)

    Article  Google Scholar 

  14. Hong, D., Yokoya, N., Chanussot, J., Zhu, X.X.: An augmented linear mixing model to address spectral variability for hyperspectral unmixing. IEEE Trans. Image Process. 28(4), 1923–1938 (2019)

    Article  MathSciNet  Google Scholar 

  15. Hong, D., Yokoya, N., Chanussot, J., Zhu, X.X.: Cospace: common subspace learning from hyperspectral-multispectral correspondences. IEEE Trans. Geosci. Remote Sens. 57(7), 4349–4359 (2019)

    Article  Google Scholar 

  16. Hong, D., Yokoya, N., Ge, N., Chanussot, J., Zhu, X.X.: Learnable manifold alignment (lema): a semi-supervised cross-modality learning framework for land cover and land use classification. ISPRS J. Photogramm. Remote Sens. 147, 193–205 (2019)

    Article  Google Scholar 

  17. Kanatsoulis, C.I., Fu, X., Sidiropoulos, N.D., Ma, W.K.: Hyperspectral super-resolution: a coupled tensor factorization approach. IEEE Trans. Signal Process. 66(24), 6503–6517 (2018)

    Article  MathSciNet  Google Scholar 

  18. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)

    Google Scholar 

  19. Kruse, F.A.: The spectral image processing system (sips)-interactive visualization and analysis of imaging spectrometer data. Remote Sens. Environ. 44(2–3), 145–163 (1993)

    Article  Google Scholar 

  20. Lanaras, C., Baltsavias, E., Schindler, K.: Hyperspectral super-resolution by coupled spectral unmixing. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 3586–3594 (2015)

    Google Scholar 

  21. Li, S., Dian, R., Fang, L., Bioucas-Dias, J.M.: Fusing hyperspectral and multispectral images via coupled sparse tensor factorization. IEEE Trans. Image Process. 27(8), 4118–4130 (2018)

    Article  MathSciNet  Google Scholar 

  22. Loncan, L., et al.: Hyperspectral pansharpening: a review. IEEE Geosci. Remote Sens. Mag. 3(3), 27–46 (2015)

    Article  Google Scholar 

  23. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  24. Ng, A., et al.: Sparse autoencoder. CS294A Lect. Notes 72(2011), 1–19 (2011)

    Google Scholar 

  25. Qu, Y., Qi, H., Kwan, C.: Unsupervised sparse dirichlet-net for hyperspectral image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2511–2520 (2018)

    Google Scholar 

  26. Rasti, B., et al.: Feature extraction for hyperspectral imagery: the evolution from shallow to deep (overview and toolbox). IEEE Geosci. Remote Sens. Mag. (2020). https://doi.org/10.1109/MGRS.2020.2979764

    Article  Google Scholar 

  27. Simoes, M., Bioucas-Dias, J., Almeida, L.B., Chanussot, J.: A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Trans. Geosci. Remote Sens. 53(6), 3373–3388 (2014)

    Article  Google Scholar 

  28. Vivone, G., et al.: A critical comparison among pansharpening algorithms. IEEE Trans. Geosci. Remote Sens. 53(5), 2565–2586 (2014)

    Article  Google Scholar 

  29. Wald, L.: Quality of high resolution synthesised images: is there a simple criterion? In: 3rd Conference Fusion Earth Data: Merging Point Measurements, Raster Maps, and Remotely Sensed Images (2000)

    Google Scholar 

  30. Wang, Q., Atkinson, P.M.: The effect of the point spread function on sub-pixel mapping. Remote Sens. Environ. 193, 127–137 (2017)

    Article  Google Scholar 

  31. Wang, Z., Bovik, A.C.: A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002)

    Article  Google Scholar 

  32. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  33. Wei, Q., Dobigeon, N., Tourneret, J.Y.: Fast fusion of multi-band images based on solving a sylvester equation. IEEE Trans. Image Process. 24(11), 4109–4121 (2015)

    Article  MathSciNet  Google Scholar 

  34. Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., Xu, Z.: Multispectral and hyperspectral image fusion by MS/HS fusion net. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1585–1594 (2019)

    Google Scholar 

  35. Yao, J., Meng, D., Zhao, Q., Cao, W., Xu, Z.: Nonconvex-sparsity and nonlocal-smoothness-based blind hyperspectral unmixing. IEEE Trans. Image Process. 28(6), 2991–3006 (2019)

    Article  MathSciNet  Google Scholar 

  36. Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process. 19(9), 2241–2253 (2010)

    Article  MathSciNet  Google Scholar 

  37. Yokoya, N., Grohnfeldt, C., Chanussot, J.: Hyperspectral and multispectral data fusion: a comparative review of the recent literature. IEEE Geosci. Remote Sens. Mag. 5(2), 29–56 (2017)

    Article  Google Scholar 

  38. Yokoya, N., Yairi, T., Iwasaki, A.: Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans. Geosci. Remote Sens. 50(2), 528–537 (2011)

    Article  Google Scholar 

  39. Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2016)

    Article  Google Scholar 

  40. Zheng, K., Zheng, K., et al.: Coupled convolutional neural network with adaptive response function learning for unsupervised hyperspectral super-resolution. IEEE Trans. Geosci. Remote Sens. (2020). https://doi.org/10.1109/TGRS.2020.3006534

    Article  Google Scholar 

Download references

Acknowledgements

This work has been supported in part by projects of the National Natural Science Foundation of China (No. 61721002, No. U1811461, and No. 11690011) and the China Scholarship Council.

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, China

    Jing Yao, Deyu Meng & Zongben Xu

  2. Remote Sensing Technology Institute, German Aerospace Center, Weßling, Germany

    Jing Yao, Danfeng Hong & Xiaoxiang Zhu

  3. University Grenoble Alpes, INRIA, CNRS, Grenoble INP, LJK, Grenoble, France

    Jocelyn Chanussot

  4. Technical University of Munich, Munich, Germany

    Jing Yao & Xiaoxiang Zhu

  5. Macau University of Science and Technology, Macao, China

    Deyu Meng

Authors
  1. Jing Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Danfeng Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jocelyn Chanussot
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Deyu Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaoxiang Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zongben Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Danfeng Hong .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yao, J., Hong, D., Chanussot, J., Meng, D., Zhu, X., Xu, Z. (2020). Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12374. Springer, Cham. https://doi.org/10.1007/978-3-030-58526-6_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58526-6_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58525-9

  • Online ISBN: 978-3-030-58526-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation