Abstract
The preservation, accessibility, and dissemination of historical artifacts to a wider audience have become increasingly important, and cultural institutions can achieve these goals through the digitization of cultural heritage. In recent years, artificial intelligence (AI) and machine learning (ML) techniques improve the virtualization of cultural artifacts for interactive experiences. In this work, we present a virtualization pipeline for the cultural heritage domain, focusing specifically on paintings, using AI techniques. We outline the basic workflow, including a thorough description of the comparison of various neural network models and their performance metrics. The proposed method creates an immersive experience for viewers to interact with paintings beyond observation. The approach utilizes 2.5D technology by applying depth maps of paintings using deep learning (DL) algorithms. The proof of concept was demonstrated on two real-life paintings of varying complexities, and this innovative approach holds potential for enhancing the appreciation and understanding of cultural heritage in museums and other cultural institutions.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The interactive visualization: https://bit.ly/41oVsWO.
- 2.
- 3.
PTGui official website: https://ptgui.com/.
- 4.
Depth Player official website: https://bit.ly/3nMJsk7),.
References
Depth predictions in art. https://storage.googleapis.com/art/history/depth/data/demo/index.html
Altınbay, R., Gümüş, N.: Social studies teachers’ views on the virtual tour applications. J. Innov. Res. Teacher Educ. 1(1), 60–71 (2020). https://doi.org/10.29329/jirte.2020.321.5
Chen, P.Y., Liu, A.H., Liu, Y.C., Wang, Y.C.F.: Towards scene understanding: unsupervised monocular depth estimation with semantic-aware representation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2619–2627 (2019). https://doi.org/10.1109/CVPR.2019.00273
Chen, T., An, S., Zhang, Y., Ma, C., Wang, H., Guo, X., Zheng, W.: Improving monocular depth estimation by leveraging structural awareness and complementary datasets. CoRR abs/2007.11256 (2020). https://arxiv.org/abs/2007.11256
Ehret, T.: Monocular depth estimation: a review of the 2022 state of the art. Image Processing On Line 13, 38–56 (2023). https://doi.org/10.5201/ipol.2023.459
Elkhuizen, W.S., et al.: Comparison of three 3D scanning techniques for paintings, as applied to Vermeer’s ‘Girl with a Pearl Earring’. Heritage Sci. 7(1), 1–22 (2019). https://doi.org/10.1186/s40494-019-0331-5
Fang, Z., Chen, X., Chen, Y., Van Gool, L.: Towards good practice for CNN-based monocular depth estimation. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1080–1089 (2020). https://doi.org/10.1109/WACV45572.2020.9093334
Furferi, R., Governi, L., Volpe, Y., Puggelli, L., Vanni, N., Carfagni, M.: From 2D to 2.5D i.e. from painting to tactile model. Graph Models 76(6), 706–723 (2014). https://doi.org/10.1016/j.gmod.2014.10.001
Galeazzi, F., Franco, P.D.G.D., Matthews, J.L.: Comparing 2D pictures with 3D replicas for the digital preservation and analysis of tangible heritage. Museum Manage. Curatorship 30(5), 462–483 (2015). https://doi.org/10.1080/09647775.2015.1042515
Giuliani, F., De Paoli, R., Di Miceli, E.: A risk-reduction framework for urban cultural heritage: a comparative study on Italian historic centres. J. Cultural Heritage Manage. Sustain. Dev. 11(4), 499–515 (2021). https://doi.org/10.1108/JCHMSD-07-2020-0099
Guarneri, M., De Collibus, M.F., Francucci, M., Ciaffi, M.: The importance of artworks 3D digitalization at the time of Covid epidemy: case studies by the use of a multi-wavelengths technique. In: 2020 IEEE 5th International Conference on Image, Vision and Computing (ICIVC), pp. 113–117 (2020). https://doi.org/10.1109/ICIVC50857.2020.9177443
Huynh, L., Nguyen-Ha, P., Matas, J., Rahtu, E., Heikkilä, J.: Guiding monocular depth estimation using depth-attention volume. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12371, pp. 581–597. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58574-7_35
Jin, S., Fan, M., Wang, Y., Liu, Q.: Reconstructing traditional Chinese paintings with immersive virtual reality. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–8. CHI EA ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3334480.3382934
Johnston, A., Carneiro, G.: Self-supervised monocular trained depth estimation using self-attention and discrete disparity volume, pp. 4755–4764 (2020). https://doi.org/10.1109/CVPR42600.2020.00481
Khan, F., Salahuddin, S., Javidnia, H.: Deep learning-based monocular depth estimation methods-a state-of-the-art review. Sensors 20(8) (2020). https://doi.org/10.3390/s20082272, https://www.mdpi.com/1424-8220/20/8/2272
Kim, J.H., Ko, K.L., Le Ha, T., Jung, S.W.: Monocular depth estimation of old photos via collaboration of monocular and stereo networks. IEEE Access 11, 11675–11684 (2023). https://doi.org/10.1109/ACCESS.2023.3241348
Lee, J.H., Kim, C.S.: Single-image depth estimation using relative depths. J. Vis. Commun. Image Representation 84, 103459 (2022). https://doi.org/10.1016/j.jvcir.2022.103459
Lee, J., Kim, C.S.: Multi-loss rebalancing algorithm for monocular depth estimation, pp. 785–801 (2020). https://doi.org/10.1007/978-3-030-58520-4_46
Liu, Z.S., Wang, L.W., Siu, W.C., Kalogeiton, V.: Name your style: an arbitrary artist-aware image style transfer (2022)
Pan, J., Li, L., Yamaguchi, H., Hasegawa, K., Thufail, F.I., Brahmantara, Tanaka, S.: 3D reconstruction of Borobudur reliefs from 2D monocular photographs based on soft-edge enhanced deep learning. ISPRS J. Photogram. Remote Sens. 183, 439–450 (2022). https://doi.org/10.1016/j.isprsjprs.2021.11.007, https://www.sciencedirect.com/science/article/pii/S0924271621003051
Park, S., Chon, S., Lee, T., Kim, J.: Toward the experiential VR Gallery using 2.5-D. EasyChair Preprint no. 1091 (EasyChair, 2019)
Poornapushpakala, S., Barani, S., Subramoniam, M., Vijayashree, T.: Restoration of Tanjore paintings using segmentation and in-painting techniques. Heritage Sci. 10(1), 1–6 (2022). https://doi.org/10.1186/s40494-022-00661-1
Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., Koltun, V.: Towards robust monocular depth estimation: mixing datasets for zero-shot cross-dataset transfer. 44, 1623–1637 (2022). https://doi.org/10.1109/TPAMI.2020.3019967
Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction (2021). https://doi.org/10.48550/ARXIV.2103.13413
Romão, X., Paupério, E.: An indicator for post-disaster economic loss valuation of impacts on cultural heritage. Int. J. Architect. Heritage, 1–20 (2019). https://doi.org/10.1080/15583058.2019.1643948
Skamantzari, M., Georgopoulos, A.: 3D visualization for virtual museum development. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLI-B5, 961–968 (2016). https://doi.org/10.5194/isprs-archives-XLI-B5-961-2016
Tatlı, Z., Çelenk, G., Altınışık, D.: Analysis of virtual museums in terms of design and perception of presence. Educ. Inf. Technol. (2023). https://doi.org/10.1007/s10639-022-11561-z
Torre, S.D.: Italian perspective on the planned preventive conservation of architectural heritage. Front. Architect. Res. 10(1), 108–116 (2021). https://doi.org/10.1016/j.foar.2020.07.008
Wang, C., Lucey, S., Perazzi, F., Wang, O.: Web stereo video supervision for depth prediction from dynamic scenes (2019). https://doi.org/10.48550/arxiv.1904.11112
Wang, L., Zhang, J., Wang, O., Lin, Z., Lu, H.: SDC-Depth: semantic divide-and-conquer network for monocular depth estimation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 538–547 (2020). https://doi.org/10.1109/CVPR42600.2020.00062
Wang, L., Zhang, J., Wang, Y., Lu, H., Ruan, X.: CLIFFNet for monocular depth estimation with hierarchical embedding loss, pp. 316–331 (10 2020). https://doi.org/10.1007/978-3-030-58558-7_19
Xian, K., Zhang, J., Wang, O., Mai, L., Lin, Z., Cao, Z.: Structure-guided ranking loss for single image depth prediction. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Xue, A.: End-to-end Chinese landscape painting creation using generative adversarial networks (2020)
Yin, W., Liu, Y., Shen, C., Yan, Y.: Enforcing geometric constraints of virtual normal for depth prediction, pp. 5683–5692 (10 2019). https://doi.org/10.1109/ICCV.2019.00578
Yin, W., Liu, Y., Shen, C., Yan, Y.: Enforcing geometric constraints of virtual normal for depth prediction (2019). https://doi.org/10.48550/ARXIV.1907.12209
Yin, W., et al.: Learning to recover 3D scene shape from a single image (2020). https://doi.org/10.48550/arxiv.2012.09365
Zamir, A., Sax, A., Shen, W., Guibas, L., Malik, J., Savarese, S.: Taskonomy: disentangling task transfer learning (2018). https://doi.org/10.48550/arxiv.1804.08328
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pauls, A., Pierdicca, R., Mancini, A., Zingaretti, P. (2023). The Depth Estimation of 2D Content: A New Life for Paintings. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2023. Lecture Notes in Computer Science, vol 14219. Springer, Cham. https://doi.org/10.1007/978-3-031-43404-4_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-43404-4_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43403-7
Online ISBN: 978-3-031-43404-4
eBook Packages: Computer ScienceComputer Science (R0)