Skip to main content
SpringerLink
Log in

Query-by-example HDR image retrieval based on CNN

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Due to the expension of High Dynamic Range (HDR) imaging applications into various aspects of daily life, an efficient retrieval system, tailored to this type of data, has become a pressing challenge. In this paper, the reliability of Convolutional Neural Networks (CNN) descriptor and its investigation for HDR image retrieval are studied. The main idea consists in exploring the use of CNN to determine HDR image descriptor. Specifically, a Perceptually Uniform (PU) encoding is initially applied to the HDR content to map the luminance values in a perceptually uniform scale. Afterward, the CNN features, using Fully Connected (FC) layer activation, are extracted and classified by applying the Support Vector Machines (SVM) algorithm. Experimental evaluation demonstrates that the CNN descriptor, using the VGG19 network, achieves satisfactory results for describing HDR images on public available datasets such as PascalVoc2007, Cifar-10 and Wang. The experimental results also show that the features, after a PU processing, are more descriptive than those directly extracted from HDR contents. Finally, we show the superior performance of the proposed method against a recent state-of-the-art technique.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Aydin TO, Mantiuk R, Seidel HP (2008) Extending quality metrics to full luminance range images human. Human vision and electronic imaging XIII (Proceedings of SPIE), pp 6806–6810

  2. Babenko A, Lempitsky V (2015) Aggregating deep convolutional features for image retrieval. International conference on computer vision, pp 1269–1277

  3. Babenko A, Slesarev A, Chigorin A, Lempitsky V (2014) Neural codes for image retrieval. In: European conference on computer vision. Springer, Cham, pp 584–599

  4. Banterle F, Ledda P, Debattista K, Chalmers A (2006) Inverse tone mapping. International conference on Computer graphics and interactive techniques. pp 349–356

  5. Bronislav P, Chalmers A, Zemcík P., Hooberman L, Zadík M. (2016) Evaluation of feature point detection in high dynamic range imagery. J Vis Commun Image Represent 28(C):141–160

    Google Scholar 

  6. Chalmers A (2017) Debattista,K.: HDR video past, present and future: a perspective. Sig Process Image Commun 54:49–55

    Article  Google Scholar 

  7. Debevec PE, Malik J (1997) Recovering high dynamicrange radiance maps from photographs. Proceedings SIGGRAPH, pp 369–378

  8. Dufaux F, Callet PL, Mantiuk R, Mrak M (2016) High dynamic range video: from acquisition, to display and applications. Academic Press

  9. Eilertsen G, Kronander J, Denes G, Mantiuk RK, Unger J (2017) HDR image reconstruction from a single exposure using deep CNNs. ACM Trans Graph 36(6):178:1–178:15

    Article  Google Scholar 

  10. Endo Y, Kanamori Y, Mitani J (2017) Deep reverse tone mapping. ACM Trans Graph 36(6):177

    Article  Google Scholar 

  11. Gao L, Li X, Song Shen HTJ (2020) Hierarchical LSTMs with adaptive attention for visual captioning. IEEE Trans Pattern Anal Mach Intell 42 (5):1112–1131

    Google Scholar 

  12. Gong Y, Wang L, Guo R, Lazebnik S (2014) Multi-scale orderless pooling of deep convolutional activation features. In: European conference on computer vision. Springer, Cham, pp 392–407

  13. Gong Y, Wang L, Guo R, Lazebnik S (2014) Multi-scale orderless pooling of deep convolutional activation features. In: European conference on computer vision. Springer, Cham, pp 392–407

  14. Gordo A, Almazan J, Revaud J, Larlus D (2017) End-to-end learning of deep visual representations for image retrieval. Int J Comput Vis 124 (2):237–254

    Article  MathSciNet  Google Scholar 

  15. He K, Zhang X, Ren S, Sun J (2014) Spatial pyramid pooling in deep convolutional networks for visual recognition. European Conference on Computer Vision, pp 346–361

  16. Husain SS, Bober M (2019) Multi-layer entropy-guided pooling of dense CNN features for image retrieval. IEEE Trans Image Process 28(10):5201–5213

    Article  MathSciNet  Google Scholar 

  17. Kalantari NK, Ramamoorthi R (2017) Deep high dynamic range imaging of dynamic scenes. ACM Trans Graph 36(4):144:1–144:12

    Article  Google Scholar 

  18. Kalantidis Y, Mellina C, Osindero S (2016) Cross-dimensional weighting for aggregated deep convolutional features. In: European conference on computer vision. Springer, Cham, pp 685–701

  19. Khwildi R, Hachani M, Ouled Zaid A (2016) New indexing method of HDR images using color histograms. International conference on machine vision

  20. Khwildi R, Ouled Zaid A (2018) Color Based HDR image retrieval using HSV histogram and color moments. In: International conference on computer systems and applications. IEEE, pp 1–5

  21. Khwildi R, Ouled Zaid A (2018) New retrieval system based on low dynamic range expansion and SIFT descriptor. In: International workshop on multimedia signal processing. IEEE pp 1–6

  22. Khwildi R, Ouled Zaid A (2020) HDR image retrieval by using color-based descriptor and tone mapping operator. Vis Comput 36:1111–1126

    Article  Google Scholar 

  23. Kim BK, Park RH, Chang S (2016) Tone mapping with contrast preservation and lightness correction in high dynamic range imaging. SIViP 10(8):1425–1432

    Article  Google Scholar 

  24. Kovaleski RP, Oliveira MM (2009) High-quality brightness enhancement functions for real-time reverse tone mapping. Vis Comput 25(5):539–547

    Article  Google Scholar 

  25. Kovaleski RP, Oliveira MM (2014) High-quality reverse tone mapping for a wide range of exposures. In: Conference on graphics patterns and images. IEEE, pp 49–56

  26. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  27. Larson GW (1998) Logluv encoding for full-gamut, high-dynamic range images. J Graph Tools 3(1):15–31

    Article  Google Scholar 

  28. Lin K, Lu J, Chen C, Zhou J (2016) Learning compact binary descriptors with unsupervised deep neural networks. In: Conference on computer vision and pattern recognition, pp 1183–1192

  29. Mantiuk RK, Myszkowski KH, Seidel P (2015) High dynamic range imaging. Wiley encyclopedia of electrical and electronics engineering, pp 1–4

  30. Masia B, Serrano A, Gutierrez D (2017) Dynamic range expansion based on image statistics. Multimed Tools Appl 76(1):631–648

    Article  Google Scholar 

  31. Mitsunaga T, Nayar SK (1999) Radiometric self calibration. In: Conference on computer vision and pattern recognition. IEEE, pp 374–380

  32. Mohedano E, McGuinness K, et al. (2016) Bags of local convolution. International conference on multimedia retrieval, pp 327–331

  33. Ng J, Yang F, Davis L (2015) Exploiting local features from deep networks for image retrieval. Conference on computer vision and pattern recognition workshops, pp 53–61

  34. Pan Y, He F, Yu H (2020) Learning social representations with deep autoencoder for recommender system. World Wide Web 23:2259–2279

    Article  Google Scholar 

  35. Quan Q, He F, Li H (2020) A multi-phase blending method with incremental intensity for training detection networks. Vis Comput, pp 1–15

  36. Radenovic F, Tolias G, Chum O (2018) Fine-tuning CNN image retrieval with no human annotation. IEEE Trans Pattern Anal Mach Intell 41 (7):1655–1668

    Article  Google Scholar 

  37. Rana A, Valenzise G, Dufaux F (2015) Evaluation of feature detection in HDR based imaging under changes in illumination conditions. In: IEEE international symposium on multimedia. IEEE, pp 289– 294

  38. Rana A, Valenzise G, Dufaux F (2016) An Evaluation of HDR image matching under extreme illumination changes. In: Visual communications and image processing. IEEE, pp 1–4

  39. Razavian AS, Azizpour H, Sullivan J, Carlsson S (2014) CNN features off-the-shelf: an astounding baseline for recognition. Computer vision and pattern recognition workshops, pp 512–519

  40. Razavian AS, Sullivan J, Carlsson S, Maki A (2016) Visual instance retrieval with deep convolutional networks. ITE Trans Media Technol Appl 4(3):251–258

    Article  Google Scholar 

  41. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. International conference on learning representations

  42. Song J, He T, Gao L, et al. (2020) Unified binary generative adversarial network for image retrieval and compression. Int J Comput Vis 128:2243–2264

    Article  MathSciNet  Google Scholar 

  43. Tang Y (2013) Deep learning using linear support vector machines. International conference on neural information processing, pp 458–465

  44. Tolias G, Sicre R, Jégou H. (2016) Particular object retrieval with integral max-pooling of CNN activations. International conference on learning representations, pp 1–12

  45. Uricchio T, Bertini M, Seidenari L, Del Bimbo A (2015) Fisher encoded convolutional Bag-of-Windows for efficient image retrieval and social image tagging. In: International conference on computer vision workshop, pp 1020–1026

  46. Vaccaro F, Bertini M, Uricchio T, Del BimboImage A (2020) Retrieval using multi-scale CNN features pooling. In: International conference on multimedia retrieval, pp 311–315

  47. Vinyals O, Jia Y, Deng L, Darrell T (2012) Learning with recursive perceptual representations. Annu Conf Neural Inf Process Syst, pp 2834–2842

  48. Ward G (1991) Real pixels. Graphics Gems, New York

    Book  Google Scholar 

  49. Zhang N, Donahue J, Girshick R, Darrell T (2014) Part-based R-CNNs for fine-grained category detection. In: European conference on computer vision. Springer, Cham, pp 834–849

  50. Zhang S, He F (2020) RCDN: Learning deep residual convolutional dehazing networks. Vis Comput 36(9):1797–1808

    Article  Google Scholar 

  51. Zhang S, He F, Ren W (2020) NLDN: Non-local dehazing network for dense haze removal. Neurocomputing 410:363–373

    Article  Google Scholar 

  52. Zhang J, Lalonde JF (2017) Learning high dynamic range from outdoor panoramas. In: International conference on computer vision. pp 4529–4538

  53. Zheng L, Zhao Y, Wang S, Wang J, Tian Q (2016) Good practice in CNN feature transfer. arXiv preprint arXiv:1604.00133

  54. Zhu H, Chen X, Dai W, Fu K, Ye Q, Jiao J (2015) Orientation robust object detection in aerial images using deep convolutional neural network. In: International conference on image processing. IEEE, pp 3735–3739

  55. (2003) OpenEXR. http://www.openexr.org

Download references

Author information

Authors and Affiliations

  1. Laboratoire Systèmes de Communications, École Nationale d’Ingénieurs de Tunis, Université de Tunis El Manar, B.P. 37 le Belvédère, 1002, Tunis, Tunisie

    Raoua Khwildi & Azza Ouled Zaid

  2. Laboratoire des Signaux et Systèmes, Université Paris-Saclay, CNRS, CentraleSupelec, 91190, Gif-sur-Yvette, France

    Frédéric Dufaux

Authors
  1. Raoua Khwildi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Azza Ouled Zaid
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frédéric Dufaux
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Raoua Khwildi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khwildi, R., Ouled Zaid, A. & Dufaux, F. Query-by-example HDR image retrieval based on CNN. Multimed Tools Appl 80, 15413–15428 (2021). https://doi.org/10.1007/s11042-020-10416-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10416-4

Keywords

Search

Navigation