Skip to main content
SpringerLink
Log in

Benchmarking NewSQL Cloud-Native or Cloud-Hosted Databases

  • Conference paper
  • First Online:
Software Engineering Application in Informatics (CoMeSySo 2021)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 232))

Included in the following conference series:

  • 874 Accesses

Abstract

Evaluating the performance of high-impact components when building a solution is the process that provides an informed and fitting choice. Choosing a database system and the proper infrastructural configuration underneath it is however a process that can prove to be extremely complex. In the context of the highly volatile NewSQL database systems and Cloud hosting, choosing the fitting combination is not only difficult but can be daunting. This paper provides an overview of the implications of creating reliable benchmarking for NewSQL Cloud hosted databases. It includes the steps involved in accurate performance evaluations, existing tools, proposes a systematic approach for narrowing the number of combinations to test for and lists the standard benchmarks as well as alternatives. With this work we aim to introduce clarity and structure for the database benchmarking process, based on existing research and the current gaps therein.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abadi, D., et al.: The Beckman report on database research. Commun. ACM 59(2), 92–99 (2016)

    Article  Google Scholar 

  2. Altibase: Altibase (2020). https://altibase.com/

  3. Aslett, M.: What we talk about when we talk about newSQL, November 2020. https://blogs.451research.com/information_management/2011/04/06/what-we-talk-about-when-we-talk-about-newsql/

  4. Azqueta-Alzúaz, A., Martinez, M.P., Vianello, V., Péris, R.J.: Fault-tolerance evaluation of a new SQL database. In: 2018 14th European Dependable Computing Conference (EDCC), pp. 81–86. IEEE (2018)

    Google Scholar 

  5. Binani, S., Gutti, A., Upadhyay, S.: SQL vs. NoSQL vs. NewSQL-a comparative study. Database 6(1), 1–4 (2016)

    Google Scholar 

  6. Bitton, D., DeWitt, D.J., Turbyfill, C.: Benchmarking database systems-a systematic approach. Technical report, University of Wisconsin-Madison Department of Computer Sciences (1983)

    Google Scholar 

  7. Buhl, H.U., Röglinger, M., Moser, F., Heidemann, J.: Big data (2013)

    Google Scholar 

  8. Chandra, U.: A comparative study on: NoSQL, NewSQL and Polygot persistence. Int. J. Soft Comput. Eng. (IJSE) 7 (2017)

    Google Scholar 

  9. Chereja, I., Hahn, S.M.L., Matei, O., Avram, A.: Multidimensional analysis of NewSQL database systems. In: Silhavy, R. (ed.) CSOC 2021. LNNS, vol. 230, pp. 221–236. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77442-4_19

    Chapter  Google Scholar 

  10. Chereja, I., Hahn, S.M.L., Matei, O., Avram, A.: Operationalizing analytics with NewSQL. In: Silhavy, R. (ed.) CSOC 2021. LNNS, vol. 230, pp. 249–263. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77442-4_21

    Chapter  Google Scholar 

  11. CitusData: Citus (2020). https://www.citusdata.com/

  12. Cockroachlabs: Cockroachdb (2020). https://www.cockroachlabs.com/

  13. Codd, E.: An evaluation scheme for database management systems that are claimed to be relational. In: 1986 IEEE Second International Conference on Data Engineering, pp. 720–729. IEEE (1986)

    Google Scholar 

  14. Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st ACM Symposium on Cloud Computing, pp. 143–154 (2010)

    Google Scholar 

  15. Copie, A., Fortiş, T.F., Munteanu, V.I.: Benchmarking cloud databases for the requirements of the internet of things. In: Proceedings of the ITI 2013 35th International Conference on Information Technology Interfaces, pp. 77–82. IEEE (2013)

    Google Scholar 

  16. Curino, C.A., Difallah, D.E., Pavlo, A., Cudre-Mauroux, P.: Benchmarking OLTP/Web databases in the cloud: the OLTP-bench framework. In: Proceedings of the Fourth International Workshop on Cloud Data Management, pp. 17–20 (2012)

    Google Scholar 

  17. Fauna: Fauna (2020). https://fauna.com

  18. Ferdman, M., et al.: Clearing the clouds: a study of emerging scale-out workloads on modern hardware. ACM SIGPLAN Not. 47(4), 37–48 (2012)

    Article  Google Scholar 

  19. Ficco, M., Rak, M., Venticinque, S., Tasquier, L., Aversano, G.: Cloud evaluation: Benchmarking and monitoring. In: Quantitative Assessments of Distributed Systems, pp. 175–199 (2015)

    Google Scholar 

  20. Folkerts, E., Alexandrov, A., Sachs, K., Iosup, A., Markl, V., Tosun, C.: Benchmarking in the cloud: what it should, can, and cannot be. In: Nambiar, R., Poess, M. (eds.) TPCTC 2012. LNCS, vol. 7755, pp. 173–188. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-36727-4_12

    Chapter  Google Scholar 

  21. Funke, F., Kemper, A., Neumann, T.: Benchmarking hybrid oltp&olap database systems. Datenbanksysteme für Business, Technologie und Web (BTW) (2011)

    Google Scholar 

  22. Gartner: Gartner, November 2020. https://www.gartner.com

  23. Geelan, J., et al.: Twenty-one experts define cloud computing. Cloud Comput. J. 4, 1–5 (2009)

    Google Scholar 

  24. Gillam, L., Li, B., O’Loughlin, J., Tomar, A.P.S.: Fair benchmarking for cloud computing systems. J. Cloud Comput. Adv. Syst. Appl. 2(1), 1–45 (2013)

    Article  Google Scholar 

  25. solidIT consulting & software development GmbH: Db-engines (2020). https://db-engines.com/

  26. Gray, J.: Database and Transaction Processing Performance Handbook (1993)

    Google Scholar 

  27. Hahn, S.M.L., Chereja, I., Matei, O.: Analysis of transformation tools applicable on NewSQL databases. In: Silhavy, R. (ed.) CSOC 2021. LNNS, vol. 230, pp. 180–195. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77442-4_16

    Chapter  Google Scholar 

  28. Hajoui, O., Dehbi, R., Talea, M., Batouta, Z.I.: An advanced comparative study of the most promising NoSQL and NewSQL databases with a multi-criteria analysis method. J. Theor. Appl. Inf. Technol. 81(3), 579 (2015)

    Google Scholar 

  29. Han, J., Haihong, E., Le, G., Du, J.: Survey on NoSQL database. In: 2011 6th International Conference on Pervasive Computing and Applications, pp. 363–366. IEEE (2011)

    Google Scholar 

  30. Hsu, W.W., Smith, A.J., Young, H.C.: Characteristics of production database workloads and the TPC benchmarks. IBM Syst. J. 40(3), 781–802 (2001)

    Article  Google Scholar 

  31. Google Inc: Spanner (2020). https://cloud.google.com/spanner

  32. Kaur, K., Sachdeva, M.: Performance evaluation of NewSQL databases. In: 2017 International Conference on Inventive Systems and Control (ICISC), pp. 1–5. IEEE (2017)

    Google Scholar 

  33. Knob, R.R., et al.: Análise e benchmarking das soluções newsql CockroachDB, MemSQL, NuoDB e VoltDB (2018)

    Google Scholar 

  34. Knorr, E., Gruman, G.: What cloud computing really means. InfoWorld 7(20–20), 1–17 (2008)

    Google Scholar 

  35. Kuhlenkamp, J., Klems, M., Röss, O.: Benchmarking scalability and elasticity of distributed database systems. Proc. VLDB Endowment 7(12), 1219–1230 (2014)

    Article  Google Scholar 

  36. Lankford, W.M.: Benchmarking: understanding the basics. Coastal Bus. J. 1(1), 57–62 (2000)

    Google Scholar 

  37. Leavitt, N.: Will NoSQL databases live up to their promise? Computer 43(2), 12–14 (2010)

    Article  Google Scholar 

  38. Lehrig, S., Sanders, R., Brataas, G., Cecowski, M., Ivanšek, S., Polutnik, J.: CloudStore-towards scalability, elasticity, and efficiency benchmarking and analysis in cloud computing. Futur. Gener. Comput. Syst. 78, 115–126 (2018)

    Article  Google Scholar 

  39. Mell, P., Grance, T., et al.: The NIST definition of cloud computing. NIST Special Publication, pp. 800–145 (2011)

    Google Scholar 

  40. Mohamed, M.A., Altrafi, O.G., Ismail, M.O.: Relational vs. NoSQL databases: a survey. Int. J. Comput. Inf. Technol. 3(03), 598–601 (2014)

    Google Scholar 

  41. Nambiar, R., Wakou, N., Carman, F., Majdalany, M.: Transaction processing performance council (TPC): state of the council 2010. In: Nambiar, R., Poess, M. (eds.) TPCTC 2010. LNCS, vol. 6417, pp. 1–9. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-18206-8_1

    Chapter  Google Scholar 

  42. Nayak, A., Poriya, A., Poojary, D.: Type of NoSQL databases and its comparison with relational databases. Int. J. Appl. Inf. Syst. 5(4), 16–19 (2013)

    Google Scholar 

  43. NuoDB: Nuodb (2020). https://nuodb.com/

  44. Ohlhorst, F.J.: Big Data Analytics: Turning Big Data into Big Money, vol. 65. Wiley (2012)

    Google Scholar 

  45. OpenStack: Openstack (2021). https://www.openstack.org/

  46. Osborne, J.B.: A quantitative evaluation of the performance impact of Type-I virtualization on a NewSQL relational database management system. Ph.D. thesis, Colorado Technical University (2020)

    Google Scholar 

  47. Palit, T., Shen, Y., Ferdman, M.: Demystifying cloud benchmarking. In: 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 122–132. IEEE (2016)

    Google Scholar 

  48. PingCap: Tidb (2020). https://pingcap.com/

  49. Project, B.: BaaS: benchmarking-as-a-service (2021). https://baas-project.de/

  50. SAP: SAP HANA (2020). https://www.sap.com/products/hana.html

  51. Seybold, D., Domaschka, J., et al.: Is distributed database evaluation cloud-ready? In: Kirikova, M. (ed.) ADBIS 2017. CCIS, vol. 767, pp. 100–108. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67162-8_12

    Chapter  Google Scholar 

  52. Seybold, D., Keppler, M., Gründler, D., Domaschka, J.: Mowgli: finding your way in the DBMS jungle. In: Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering, pp. 321–332 (2019)

    Google Scholar 

  53. Seybold, D., Wesner, S., Domaschka, J.: King Louie: reproducible availability benchmarking of cloud-hosted DBMS. In: Proceedings of the 35th Annual ACM Symposium on Applied Computing, pp. 144–153 (2020)

    Google Scholar 

  54. Stonebraker, M.: SQL databases v. NoSQL databases. Commun. ACM 53(4), 10–11 (2010)

    Google Scholar 

  55. TPC: Transaction processing performance council, June 2021. http://tpc.org/

  56. Van Eyk, E., Scheuner, J., Eismann, S., Abad, C.L., Iosup, A.: Beyond microbenchmarks: the SPEC-RG vision for a comprehensive serverless benchmark. In: Companion of the ACM/SPEC International Conference on Performance Engineering, pp. 26–31 (2020)

    Google Scholar 

  57. Vatika, S., Meenu, D.: SQL and NoSQL databases. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 2(8), 20–27 (2012)

    Google Scholar 

  58. VoltDB: Voltdb (2020). https://www.voltdb.com/

  59. Wang, L., et al.: Cloud computing: a perspective study. N. Gener. Comput. 28(2), 137–146 (2010)

    Article  Google Scholar 

  60. Zairi, M.: Effective Benchmarking. Springer, Dordrecht (1996)

    Google Scholar 

  61. Zhang, Q., Cheng, L., Boutaba, R.: Cloud computing: state-of-the-art and research challenges. J. Internet Serv. Appl. 1(1), 7–18 (2010)

    Article  Google Scholar 

Download references

Acknowledgments

This research was made possible by funding from the ICT-AGRI-FOOD 2020 Joint Call. This work was supported by a grant of the Romanian National Authority for Scientific Research and Innovation, CCCDI - UEFISCDI, project number COFUND-ICT-AGRI-FOOD-MUSHNOMICS 205/2020, within PNCDI III.

Author information

Authors and Affiliations

  1. Computers and Information Technology, Technical University Cluj Napoca, Cluj-Napoca, Romania

    Ionela Chereja, Sarah Myriam Lydia Hahn & Oliviu Matei

Authors
  1. Ionela Chereja
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah Myriam Lydia Hahn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Oliviu Matei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ionela Chereja .

Editor information

Editors and Affiliations

  1. Faculty of Applied Informatics, Tomas Bata University in Zlín, Zlín, Czech Republic

    Radek Silhavy

  2. Faculty of Applied Informatics, Tomas Bata University in Zlín, Zlín, Czech Republic

    Petr Silhavy

  3. Faculty of Applied Informatics, Tomas Bata University in Zlín, Zlín, Czech Republic

    Zdenka Prokopova

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chereja, I., Hahn, S.M.L., Matei, O. (2021). Benchmarking NewSQL Cloud-Native or Cloud-Hosted Databases. In: Silhavy, R., Silhavy, P., Prokopova, Z. (eds) Software Engineering Application in Informatics. CoMeSySo 2021. Lecture Notes in Networks and Systems, vol 232. Springer, Cham. https://doi.org/10.1007/978-3-030-90318-3_26

Download citation

Publish with us

Policies and ethics

Search

Navigation