Abstract
This paper explores the implementation of a blockchain-powered metadata catalog in a data mesh architecture. The metadata catalog serves as a critical component in managing data at scale, allowing for efficient discovery, access, and governance. By integrating blockchain technology, the metadata catalog can provide federated control, immutability, and transparency in managing metadata across a distributed network of data domains. This paper discusses the benefits of using blockchain technology in the metadata catalog and provides a proof-of-concept implementation of a blockchain-powered metadata catalog in a data mesh architecture using HyperLedger Fabric. The paper also highlights some challenges and potential solutions for adopting this approach, including scalability, interoperability, and governance concerns. Overall, this paper presents a novel approach for implementing a secure and federated metadata catalog in data mesh architecture that can improve the efficiency, reliability, and transparency of data management.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Androulaki, E., et al.: Hyperledger fabric: a distributed operating system for permissioned blockchains. In: Proceedings of the Thirteenth EuroSys Conference, pp. 1–15 (2018)
Cherradi, M., EL Haddadi, A., Routaib, H.: Data lake management based on DLDS approach. In: Ben Ahmed, M., Teodorescu, H.-N.L., Mazri, T., Subashini, P., Boudhir, A.A. (eds.) Networking, Intelligent Systems and Security. SIST, vol. 237, pp. 679–690. Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-3637-0_48
Child, A.W., Hinds, J., Sheneman, L., Buerki, S.: Centralized project-specific metadata platforms: toolkit provides new perspectives on open data management within multi-institution and multidisciplinary research projects. BMC. Res. Notes 15(1), 106 (2022)
Dehghani, Z.: Data Mesh: Delivering Data-Driven Value at Scale. O’Reilly (2022)
Demichev, A., Kryukov, A., Prikhodko, N.: The approach to managing provenance metadata and data access rights in distributed storage using the hyperledger blockchain platform. In: Ivannikov Ispras Open Conference. IEEE (2018)
Dolhopolov, A., Castelltort, A., Laurent, A.: Exploring the benefits of blockchain-powered metadata catalogs in data mesh architecture. In: Proceedings of the 15th International Conference on Management of Digital EcoSystems. Springer (2023). https://conferences.sigappfr.org/medes2023/
Driessen, S., Monsieur, G., van den Heuvel, W.J.: Data product metadata management: an industrial perspective. In: Troya, J., et al. (eds.) ICSOC 2022. LNCS, vol. 13821, pp. 237–248. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-26507-5_19
Halevy, A.Y., et al.: Managing Google’s data lake: an overview of the goods system. IEEE Data Eng. Bull. 39(3), 5–14 (2016)
Hillmann, D.I., Marker, R., Brady, C.: Metadata standards and applications. Ser. Libr. 54(1–2), 7–21 (2008)
Hooshmand, Y., Resch, J., Wischnewski, P., Patil, P.: From a monolithic PLM landscape to a federated domain and data mesh. Proc. Des. Soc. 2, 713–722 (2022)
Inmon, W., Strauss, D., Neushloss, G.: DW 2.0: The Architecture for the Next Generation of Data Warehousing. Elsevier (2010)
Koscina, M., Manset, D., Negri-Ribalta, C., Perez, O.: Enabling trust in healthcare data exchange with a federated blockchain-based architecture. In: International Conference on Web Intelligence-Companion Volume (2019)
Laurent, A., Laurent, D., Madera, C.: Data Lakes. Wiley, Hoboken (2020)
Liu, L., Li, X., Au, M.H., Fan, Z., Meng, X.: Metadata privacy preservation for blockchain-based healthcare systems. In: Bhattacharya, A., et al. (eds.) DASFAA 2022. LNCS, vol. 13245, pp. 404–412. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-00123-9_33
Machado, I.A., Costa, C., Santos, M.Y.: Data mesh: concepts and principles of a paradigm shift in data architectures. Procedia Comput. Sci. 196, 263–271 (2022)
Majchrzak, J., Balnojan, S., Siwiak, M., Sieraczkiewicz, M.: Data Mesh in Action. Manning Publishing (2022)
Meadows, D.H.: Leverage points: places to intervene in a system (1999)
Newman, S.: Building Microservices. O’Reilly Media, Inc. (2015)
Sawadogo, P., Darmont, J.: On data lake architectures and metadata management. J. Intell. Inf. Syst. 56(1), 97–120 (2021)
Sawadogo, P.N., Darmont, J., Noûs, C.: Joint management and analysis of textual documents and tabular data within the AUDAL data lake. In: Bellatreche, L., Dumas, M., Karras, P., Matulevičius, R. (eds.) ADBIS 2021. LNCS, vol. 12843, pp. 88–101. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-82472-3_8
Zhao, Y.: Metadata management for data lake governance. Ph.D. thesis, Univ. Toulouse 1 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
A Metadata Catalog Assets
A Metadata Catalog Assets
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dolhopolov, A., Castelltort, A., Laurent, A. (2023). Implementing a Blockchain-Powered Metadata Catalog in Data Mesh Architecture. In: Machado, J.M., et al. Blockchain and Applications, 5th International Congress. BLOCKCHAIN 2023. Lecture Notes in Networks and Systems, vol 778. Springer, Cham. https://doi.org/10.1007/978-3-031-45155-3_35
Download citation
DOI: https://doi.org/10.1007/978-3-031-45155-3_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45154-6
Online ISBN: 978-3-031-45155-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)