Abstract
Despite the fast growth and increasing popularity, the broad field of RDF and Graph database systems lacks an independent authority for developing benchmarks, and for neutrally assessing benchmark results through industry-strength auditing which would allow to quantify and compare the performance of existing and emerging systems.
Inspired by the impact of the Transaction Processing Performance Council (TPC) Benchmarks on relational databases, the LDBC consortium formed by University and Industry researchers and practitioners has recently launched a European Commision sponsored project that will offer the first comprehensive set of open and vendor-independent benchmarks for RDF and Graph technologies. The consortium will incorporate the Linked Data Benchmark Council (LDBC) which will survive the project and will supervise the process of obtaining and reporting results as well as fostering the creation and maintenance of new and existing benchmarks. This paper describes the state-of-the-art benchmarks in RDF and Graph databases and overviews the technical challenges that should be addressed in the development of such benchmarks. With this paper we would like to invite the readers to participate in the LDBC effort towards the development of Linked Data Benchmarks, both from the user prospective (by sharing available usage scenarios, datasets, query workloads) and the vendor perspective (by reporting the results of systems and research prototypes).
Notes
Transaction Processing Performance Council: http://www.tpc.org/.
References
The DBLP computer science bibliography. http://www.informatik.uni-trier.de/~ley/db/
Abadi DJ, Marcus A, Data B (2007) Scalable semantic web data management using vertical partitioning. In: VLDB
Alexe B, Tan W-C, Velegrakis Y (2008) STBenchmark: towards a benchmark for mapping systems. In: PVLDB
Bhattacharya I, Getoor L (2006) Entity resolution in graphs. Mining Graph Data. Wiley, New York
Bizer C, Schultz A (2009) The Berlin SPARQL benchmark. Int J Semantic Web Inf Syst 5(2):1–24
Brickley D, Guha R (2004) RDF vocabulary description language 1.0: RDF schema. www.w3.org/TR/2004/REC-rdf-schema-20040210
Carey MJ, DeWitt DJ, Naughton JF (1993) The OO7 benchmark. In: SIGMOD
Cattell RGG, Skeen J (1992) Object operations benchmark. In: ACM TODS
Dominguez-Sal D, Martínez-Bazan N, Muntés-Mulero V, Baleta P, Larriba-Pey J-L (2010) A discussion on the design of graph database benchmarks. In: TPCTC
Dominguez-Sal D, Urbón-Bayes P, Giménez-Vañó A, Gómez-Villamor S, Martínez-Bazán N, Larriba-Pey JL (2010) Survey of graph database performance on the HPC scalable graph analysis benchmark. In: WAIM
Görlitz O, Thimm M, Staab S (2012) SPLODGE: systematic generation of SPARQL benchmark queries for linked open data. In: The semantic web (ISWC 2012). Lecture notes in computer science. Springer, Berlin
Harris S, Seaborne A SPARQL 1.1 query language. http://www.w3.org/TR/sparql11-query/, November 2012. W3C proposed recommendation
Ioannidis YE, Christodoulakis S (1991) On the propagation of errors in the size of join results. In: SIGMOD
Isaac A, Van Der Meij L, Schlobach S, Wang S (2007) An empirical study of instance-based ontology matching. In: ISWC/ASWC
Köpcke H, Thor A, Rahm E (2009) Comparative evaluation of entity resolution approaches with FEVER. Proc VLDB Endow 2(2):1574–1577
Ma L, Yang Y, Qiu Z, Xie G, Pan Y, Liu S (2006) Towards a complete OWL ontology benchmark. In: ESWC
McGuinness DL, van Harmelen F (2004) OWL web ontology language. http://www.w3.org/TR/owl-features/
Mironov V, Seethappan N, Blondé W, Antezana E, Lindi B, Kuiper M (2010) Benchmarking triple stores with biological data. In: SWAT4LS
Morsey M, Lehmann J, Auer S, Ngonga Ngomo A-C (2011) DBpedia SPARQL benchmark—performance assessment with real queries on real data. In: ISWC
Neumann T, Weikum G (2010) The RDF-3X engine for scalable management of RDF data. VLDB J 19(1):91–113
Patni H, Henson C, Sheth A (2010) Linked sensor data. In: CTS
Pham M-D, Boncz P, Erling O (2012) S3G2: a scalable structure-correlated social graph generator. In: TPCTC
Prud’hommeaux E, Seaborne A (2008) SPARQL query language for RDF. www.w3.org/TR/rdf-sparql-query, January
Redaschi N, Consortium U (2009) UniProt in RDF: tackling data integration and distributed annotation with the semantic web. In: Biocuration conference
Schmidt M, Hornung T, Lausen G, Pinkel C (2009) Sp2bench: a SPARQL performance benchmark. In: ICDE
Suchanek FM, Kasneci G, Weikum G (2007) Yago: a core of semantic knowledge. In: WWW
Zeiss K, Vater S, Conrad S (2010) A benchmark for testing instance-based ontology matching methods. In: IEKAW
Acknowledgements
Supported by LDBC EU FP7 project.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Boncz, P., Fundulaki, I., Gubichev, A. et al. The Linked Data Benchmark Council Project. Datenbank Spektrum 13, 121–129 (2013). https://doi.org/10.1007/s13222-013-0125-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13222-013-0125-y