Skip to main content
Log in

The Linked Data Benchmark Council Project

Datenbank-Spektrum Aims and scope Submit manuscript

Abstract

Despite the fast growth and increasing popularity, the broad field of RDF and Graph database systems lacks an independent authority for developing benchmarks, and for neutrally assessing benchmark results through industry-strength auditing which would allow to quantify and compare the performance of existing and emerging systems.

Inspired by the impact of the Transaction Processing Performance Council (TPC) Benchmarks on relational databases, the LDBC consortium formed by University and Industry researchers and practitioners has recently launched a European Commision sponsored project that will offer the first comprehensive set of open and vendor-independent benchmarks for RDF and Graph technologies. The consortium will incorporate the Linked Data Benchmark Council (LDBC) which will survive the project and will supervise the process of obtaining and reporting results as well as fostering the creation and maintenance of new and existing benchmarks. This paper describes the state-of-the-art benchmarks in RDF and Graph databases and overviews the technical challenges that should be addressed in the development of such benchmarks. With this paper we would like to invite the readers to participate in the LDBC effort towards the development of Linked Data Benchmarks, both from the user prospective (by sharing available usage scenarios, datasets, query workloads) and the vendor perspective (by reporting the results of systems and research prototypes).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Notes

  1. Transaction Processing Performance Council: http://www.tpc.org/.

  2. http://simile.mit.edu/wiki/Dataset:_Barton.

  3. http://swat.cse.lehigh.edu/projects/lubm.

  4. http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/BusinessIntelligenceUseCase/.

  5. http://oaei.ontologymatching.org/.

  6. http://oaei.ontologymatching.org/2009/.

  7. http://islab.dico.unimi.it/iimb/.

  8. http://www.tpc.org/reports/status/default.asp.

  9. http://ldbc.eu/events

References

  1. The DBLP computer science bibliography. http://www.informatik.uni-trier.de/~ley/db/

  2. Abadi DJ, Marcus A, Data B (2007) Scalable semantic web data management using vertical partitioning. In: VLDB

    Google Scholar 

  3. Alexe B, Tan W-C, Velegrakis Y (2008) STBenchmark: towards a benchmark for mapping systems. In: PVLDB

    Google Scholar 

  4. Bhattacharya I, Getoor L (2006) Entity resolution in graphs. Mining Graph Data. Wiley, New York

    Google Scholar 

  5. Bizer C, Schultz A (2009) The Berlin SPARQL benchmark. Int J Semantic Web Inf Syst 5(2):1–24

    Article  Google Scholar 

  6. Brickley D, Guha R (2004) RDF vocabulary description language 1.0: RDF schema. www.w3.org/TR/2004/REC-rdf-schema-20040210

  7. Carey MJ, DeWitt DJ, Naughton JF (1993) The OO7 benchmark. In: SIGMOD

    Google Scholar 

  8. Cattell RGG, Skeen J (1992) Object operations benchmark. In: ACM TODS

    Google Scholar 

  9. Dominguez-Sal D, Martínez-Bazan N, Muntés-Mulero V, Baleta P, Larriba-Pey J-L (2010) A discussion on the design of graph database benchmarks. In: TPCTC

    Google Scholar 

  10. Dominguez-Sal D, Urbón-Bayes P, Giménez-Vañó A, Gómez-Villamor S, Martínez-Bazán N, Larriba-Pey JL (2010) Survey of graph database performance on the HPC scalable graph analysis benchmark. In: WAIM

    Google Scholar 

  11. Görlitz O, Thimm M, Staab S (2012) SPLODGE: systematic generation of SPARQL benchmark queries for linked open data. In: The semantic web (ISWC 2012). Lecture notes in computer science. Springer, Berlin

    Google Scholar 

  12. Harris S, Seaborne A SPARQL 1.1 query language. http://www.w3.org/TR/sparql11-query/, November 2012. W3C proposed recommendation

  13. Ioannidis YE, Christodoulakis S (1991) On the propagation of errors in the size of join results. In: SIGMOD

    Google Scholar 

  14. Isaac A, Van Der Meij L, Schlobach S, Wang S (2007) An empirical study of instance-based ontology matching. In: ISWC/ASWC

    Google Scholar 

  15. Köpcke H, Thor A, Rahm E (2009) Comparative evaluation of entity resolution approaches with FEVER. Proc VLDB Endow 2(2):1574–1577

    Google Scholar 

  16. Ma L, Yang Y, Qiu Z, Xie G, Pan Y, Liu S (2006) Towards a complete OWL ontology benchmark. In: ESWC

    Google Scholar 

  17. McGuinness DL, van Harmelen F (2004) OWL web ontology language. http://www.w3.org/TR/owl-features/

  18. Mironov V, Seethappan N, Blondé W, Antezana E, Lindi B, Kuiper M (2010) Benchmarking triple stores with biological data. In: SWAT4LS

    Google Scholar 

  19. Morsey M, Lehmann J, Auer S, Ngonga Ngomo A-C (2011) DBpedia SPARQL benchmark—performance assessment with real queries on real data. In: ISWC

    Google Scholar 

  20. Neumann T, Weikum G (2010) The RDF-3X engine for scalable management of RDF data. VLDB J 19(1):91–113

    Article  Google Scholar 

  21. Patni H, Henson C, Sheth A (2010) Linked sensor data. In: CTS

    Google Scholar 

  22. Pham M-D, Boncz P, Erling O (2012) S3G2: a scalable structure-correlated social graph generator. In: TPCTC

    Google Scholar 

  23. Prud’hommeaux E, Seaborne A (2008) SPARQL query language for RDF. www.w3.org/TR/rdf-sparql-query, January

  24. Redaschi N, Consortium U (2009) UniProt in RDF: tackling data integration and distributed annotation with the semantic web. In: Biocuration conference

    Google Scholar 

  25. Schmidt M, Hornung T, Lausen G, Pinkel C (2009) Sp2bench: a SPARQL performance benchmark. In: ICDE

    Google Scholar 

  26. Suchanek FM, Kasneci G, Weikum G (2007) Yago: a core of semantic knowledge. In: WWW

    Google Scholar 

  27. Zeiss K, Vater S, Conrad S (2010) A benchmark for testing instance-based ontology matching methods. In: IEKAW

    Google Scholar 

Download references

Acknowledgements

Supported by LDBC EU FP7 project.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Thomas Neumann.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Boncz, P., Fundulaki, I., Gubichev, A. et al. The Linked Data Benchmark Council Project. Datenbank Spektrum 13, 121–129 (2013). https://doi.org/10.1007/s13222-013-0125-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13222-013-0125-y

Keywords

Navigation