There is a newer version of the record available.

Published March 13, 2023 | Version 6
Dataset Open

SciQA benchmark: Dataset and RDF dump

Creators

  • 1. Leibniz Information Centre for Science and Technology
  • 2. Institute of Informatics, Federal University of Rio Grande do Sul
  • 3. L3S Research Center, Leibniz University Hannover
  • 4. Department of Informatics and Telecommunications, National and Kapodistrian University of Athens
  • 5. Laboratory of Information Science and Semantic Technologies, ITMO University

Description

SciQA benchmark of questions and queries.

The data dump is in NTriples format (RDF NT) taken from the ORKG system on 14.02.2023 at 02:04PM.
The dump can be imported into a virtuoso endpoint or any RDF engine so it can be queried.

The questions/queries are provided as spread sheets (Excel format & CSV format), also train and test files are provided for each of the sets. Huggingface datasets are also attached in the archive to make it easy to integrate with existing workflows and to enable the automated evaluation of SciQA within challenges.

Types of questions and queries:

  • Handcrafted set of 100 questions
  • Auto-generated set of 2465 questions

More details on certain columns:
"Classification rationale" It may contain the following values:

  • Nested facts in the question
  • Sorting, sum, average, minimum, maximum or count calculation required
  • Filter used
  • Mappings of Asking Point in the question to the ORKG ontology

Explanation of Rationale for Non-factoid:

  • Nested facts in the question. An entity (e.g., a system or a paper) or predicate is requested that is not explicitly stated in the question text and must be inferred while searching for an answer. 
  • Sorting, sum, average, minimum, maximum or count calculation required. To get the answer to the question it is necessary to make an aggregation of the query results. 
  • Filter used. To get the answer to the question it is necessary to use filtering of the query results by some conditions.
     

Files

SciQA.zip

Files (11.7 MB)

Name Size Download all
md5:b6b7adce29c44198b7a51ed265335e02
11.7 MB Preview Download