Abstract
With the rapid development of hardware, a stand-alone computer can employ a memory which has large amounts of volumes. Several industries and research institutions have devoted more resources to develop several in-memory databases, which preload the data into memory for further processing. With the boom of in-memory databases, there emerges requirements to evaluate and compare the performance of these systems impartially and objectively. In this paper, we proposed MemTest, a novel benchmark considering the main characteristics of an in-memory database. This benchmark constructs particular metrics, which cover CPU usage, cache miss, compression ratio, minimal memory space and response time of an in-memory database and are also the core of our benchmark. We design a data model based on inter-bank transaction applications, around which a data generator is devised to support the data distributions of uniform and skew. The MemTest workload includes a set of queries and transactions against the metrics and data model. In the end, we illustrate the efficacy of MemTest through implementations on three different in-memory databases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Fedorova, A., Seltzer, M.I., Small, C., Nussbaum, D.: Performance of multithreaded chip multiprocessors and implications for operating system design. In: USENIX Annual Technical Conference, General Track, pp. 395–398 (2005)
Färber, F., Cha, S.K., Primsch, J., Bornhövd, C., Sigg, S., Lehner, W.: Sap hana database: data management for modern business applications. In: SIGMOD Record, pp. 45–51 (2011)
Lahiri, T., Neimat, M.A., Folkman, S.: Oracle timesten: an in-memory database for enterprise applications. IEEE Data Eng. Bull. 36, 6–13 (2013)
Diaconu, C., Freedman, C., Ismert, E., Larson, P.A., Mittal, P., Stonecipher, R., Verma, N., Zwilling, M.: Hekaton: sql server’s memory-optimized oltp engine. In: SIGMOD Conference, pp. 1243–1254 (2013)
HyperSQL: Hypersql. http://hsqldb.org
SQLite: Sqlite. http://www.sqlite.org/mostdeployed.html
MemSql: Memsql. http://www.memsql.com
Monetdb: Monetdb. https://www.monetdb.org/Home
Carey, M.J., DeWitt, D.J., Naughton, J.F.: The oo7 benchmark. In: Buneman, P., Jajodia, S. (eds.): SIGMOD Conference, pp. 12–21. ACM Press (1993)
Carey, M.J., DeWitt, D.J., Naughton, J.F., Asgarian, M., Brown, P., Gehrke, J., Shah, D.: The bucky object-relational benchmark (experience paper). In: SIGMOD Conference, pp. 135–146 (1997)
Schmidt, A., Waas, F., Kersten, M.L., Carey, M.J., Manolescu, I., Busse, R.: Xmark: a benchmark for xml data management. In: VLDB, pp. 974–985 (2002)
Carey, M.J., Ling, L., Nicola, M., Shao, L.: EXRT: towards a simple benchmark for XML readiness testing. In: Nambiar, R., Poess, M. (eds.) TPCTC 2010. LNCS, vol. 6417, pp. 93–109. Springer, Heidelberg (2011)
Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with ycsb. In: SoCC, pp. 143–154 (2010)
Ghazal, A., Rabl, T., Hu, M., Raab, F., Poess, M., Crolotte, A., Jacobsen, H.A.: Bigbench: towards an industry standard benchmark for big data analytics. In: SIGMOD Conference, 1197–1208 (2013)
Gupta, M.K., Verma, V., Verma, M.S.: In-memory database systems - a paradigm shift, pp. 333–336 (2014)
Gray, J.: Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., San Francisco (1992)
Cole, R., Funke, F., Giakoumakis, L., Guy, W., Kemper, A., Krompass, S., Kuno, H.A., et al.: The mixed workload ch-benchmark. In: DBTest, pp. 8:1–8:6 (2011)
Rabl, T., Poess, M., Jacobsen, H.A., O’Neil, P.E., O’Neil, E.J.: Variations of the star schema benchmark to test the effects of data skew on query performance. In: ICPE, pp. 361–372 (2013)
Tözün, P., Pandis, I., Kaynak, C., Jevdjic, D., Ailamaki, A.: From a to e: analyzing tpc’s oltp benchmarks: the obsolete, the ubiquitous, the unexplored. In: EDBT, pp. 17–28 (2013)
Liu, D., Luan, H., Wang, S., Qin, B.: Main memory database TPC-H workload characterization on modern process. J. Softw. 19, 2573–2584 (2008)
Pirk, H., Funke, F., Grund, M., Neumann, T., Leser, U., Manegold, S., Kemper, A., Kersten, M.L.: Cpu and cache efficient management of memory-resident databases. In: ICDE, pp. 14–25 (2013)
Müller, I., Ratsch, C., Frber, F.: Adaptive string dictionary compression in in-memory column-store database systems. In: EDBT, pp. 283–294 (2014)
Acknowledgement
Our research is supported by the 973 program of China (No. 2012CB316203), NSFC (No. 61370101), Shanghai Knowledge Service Platform Project (No. ZF1213), Innovation Program of Shanghai Municipal Education Commission (14ZZ045) and the Natural Science Foundation of Shanghai (14ZR1 412600).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Kang, Q., Jin, C., Zhang, Z., Zhou, A. (2014). MemTest: A Novel Benchmark for In-memory Database. In: Zhan, J., Han, R., Weng, C. (eds) Big Data Benchmarks, Performance Optimization, and Emerging Hardware. BPOE 2014. Lecture Notes in Computer Science(), vol 8807. Springer, Cham. https://doi.org/10.1007/978-3-319-13021-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-13021-7_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-13020-0
Online ISBN: 978-3-319-13021-7
eBook Packages: Computer ScienceComputer Science (R0)