Fares Hamouda
ABSTRACT
Data migration refers to the set of tasks around transferring data over a network between two systems, either homogeneous or heterogeneous, and the potential reformatting of this data. Combined with large volumes of data, resource constraints and variety in data models and formats, data migration can be critical for enterprises, as it can consume a significant amount of time, incur high costs, and pose a significant risk if not executed correctly. The ability to accurately and effectively predict these challenges and plan for proper resource, time and budget allocation is vital for the proper execution of data migration. In this work, we introduce the concept of load testing and benchmarking for data migration to allow decision-makers for higher efficiency and effectiveness when planning for such tasks. Our framework aims for extensibility and customizability to enable the execution of a greater variety of tests. Here, we present a prototype architecture, a roadmap of how the development of such a platform should proceed and a simple case study of how it can be used in practice.
- Julius Volz and Björn Rabenstein and Matt Bostock. 2012. Prometheus : an opensource monitoring and alerting toolkit. SoundCloud. https://prometheus.io/Google Scholar
- Eric Anderson, Joe Hall, Jason Hartline, Michael Hobbs, Anna R. Karlin, Jared Saia, Ram Swaminathan, and John Wilkes. 2001. An Experimental Study of Data Migration Algorithms. In Algorithm Engineering, Gerth Stølting Brodal, Daniele Frigioni, and Alberto Marchetti-Spaccamela (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 145--158.Google Scholar
- Dwight Merriman, Eliot Horowitz, and Kevin Ryan. 2007. MongoDB: an opensource, document-oriented NoSQL database. DoubleClick. https://www.mongodb. com/Google Scholar
- MElamparithi and V Anuratha. 2015. A Review on Database Migration Strategies, Techniques and Tools. World Journal of Computer Application and Technology 3, 3 (2015), 41--48.Google Scholar
- Zhao JF. and Zhou JT. 2014. Strategies and Methods for Cloud Migration. International Journal of Automation and Computing 11 (2014), 143--152.Google ScholarDigital Library
- Kevin Kline, Denis McDowell, Dustin Dorsey, and Matt Gordon. 2022. Moving Your Data to the Cloud. In Pro Database Migration to Azure: Data Modernization for the Enterprise. Springer, Berlin, Germany, 263--283.Google Scholar
- TN Manjunath, Ravindra S Hegadi, and HS Mohan. 2011. Automated data validation for data migration security. International Journal of Computer Applications 30, 6 (2011), 41--46.Google Scholar
- Johny Morris. 2012. Practical data migration. BCS, The Chartered Institute, London, United Kingdom.Google Scholar
- Stephen Orban. 6. Strategies for Migrating Applications to the Cloud. Medium. Library Catalog: medium. com 6 (6).Google Scholar
- PR Devale P Paygude. 2013. Automated Data Validation Testing Tool for Data Migration Quality Assurance. International Journal of Modern Engineering Research (IJMER) 3 (2013), 599--603.Google Scholar
- Priyanka Paygude and PR Devale. 2013. Automation of data validation testing for QA in the project of DB migration. International Journal of Computer Science 3, 2 (2013), 15--22.Google Scholar
- Prometheus community. [n. d.]. Node Exporter: a software component used in conjunction with Prometheus for monitoring Linux and UNIX system. https: //github.com/prometheus/node_exporterGoogle Scholar
- K. Subramani, Bugra Caskurlu, and Alvaro Velasquez. 2019. Minimization of Testing Costs in Capacity-Constrained Database Migration. In Algorithmic Aspects of Cloud Computing, Yann Disser and Vassilios S. Verykios (Eds.). Springer International Publishing, Cham, 1--12.Google Scholar
- Google Core Team. 2014. cAdvisor: an open-source container monitoring and performance analysis tool. Google. https://github.com/google/cadvisorGoogle Scholar
- Jinesh Varia. 2010. Migrating your existing applications to the aws cloud. A Phase-driven Approach to Cloud Migration (2010), 1--23.Google Scholar
- Bin Wei and Tennyson X Chen. 2014. Verifying Data Migration Correctness: The Checksum Principle. RTI Press, United States.Google Scholar
Index Terms
- DMBench: Load Testing and Benchmarking Tool for Data Migration
Recommendations
Issues in big data testing and benchmarking
DBTest '13: Proceedings of the Sixth International Workshop on Testing Database SystemsThe academic community and industry are currently researching and building next generation data management systems. These systems are designed to analyze data sets of high volume with high data ingest rates and short response times executing complex ...
Benchmarking performance for migrating a relational application to a parallel implementation
Many organizations rely on relational database platforms for OLAP-style querying (aggregation and filtering) for small to medium size applications. We investigate the impact of scaling up the data sizes for such queries. We intend to illustrate what ...
BigBench: towards an industry standard benchmark for big data analytics
SIGMOD '13: Proceedings of the 2013 ACM SIGMOD International Conference on Management of DataThere is a tremendous interest in big data by academia, industry and a large user base. Several commercial and open source providers unleashed a variety of products to support big data storage and processing. As these products mature, there is a need to ...
Comments