Skip to main content
SpringerLink
Log in

The 2013 Evaluation of SMT-COMP and SMT-LIB

  • Published:
Journal of Automated Reasoning Aims and scope Submit manuscript

Abstract

After 8 years of SMT Competitions, the SMT Steering Committee decided, for 2013, to sponsor an evaluation of the status of SMT benchmarks and solvers, rather than another competition. This report summarizes the results of the evaluation, conducted by the authors. The key observations are that (1) the competition results are quite sensitive to randomness and (2) the most significant need for the future is assessment and improvement of benchmarks in the light of SMT applications. The evaluation also measured competitiveness of solvers, general coverage of solvers, logics, and benchmarks, and degree of repeatability of measurements and competitions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aziz, M.A.: A novel portfolio solver for satisfiability modulo theory problems (2013)

  2. Aziz, M.A., Wassal, A., Darwish, N.: A machine learning technique for hardness estimation of QFBV SMT problems (work in progress). In: SMT Workshop 2012 10th International Workshop on Satisfiability Modulo Theories SMT-COMP 2012, p. 56 (2012)

  3. Barrett, C., Deters, M., de Moura, L., Oliveras, A.: A stump. 6 years of SMT-COMP (2012)

  4. Barrett, C., Stump, A., Tinelli, C.: The SMT-LIB standard: version 2.0. In: Gupta, A, Kroening, D. (eds.) Proceedings of the 8th International Workshop on Satisfiability Modulo Theories (Edinburgh, England) (2010)

  5. Le Berre, D., Simon, L.: The essentials of the SAT 2003 competition. In: Sixth international conference on theory and applications of satisfiability testing, vol. 2919 of LNCS, pp. 452–467. Springer (2003)

  6. Pelletier, F.J., Sutcliffe, G., Suttner, C.B.: The Development of CASC. AI Communications 15(2–3), 79–90 (2002)

    MATH  Google Scholar 

  7. Stump, A., Sutcliffe, G., Tinelli, C.: StarExec: a cross-community infrastructure for logic solving. In: Demri, S., Kapur, D., Weidenbach, C. (eds.) Proceedings of the 7th international joint conference on automated reasoning, Lecture Notes in Artificial Intelligence. Springer (2014)

  8. Sutcliffe, G.: The CADE ATP System Competition Design and Organization., http://www.cs.miami.edu/tptp/CASC/24/Design.html#Evaluation

  9. Federated logic conference (FLoC) olympic games. http://vsl2014.at/olympics/

  10. The SMT-COMP web site provides results of the SMT competition and links to the system descriptions of the participants., http://smtcomp.org

  11. StarExec web site., http://www.starexec.org

Download references

Author information

Authors and Affiliations

  1. GrammaTech, Inc., Ithaca, NY, USA

    David R. Cok

  2. University of Iowa, Iowa City, IA, USA

    Aaron Stump

  3. Uppsala University, Uppsala, Sweden

    Tjark Weber

Authors
  1. David R. Cok
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aaron Stump
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tjark Weber
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David R. Cok.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cok, D.R., Stump, A. & Weber, T. The 2013 Evaluation of SMT-COMP and SMT-LIB. J Autom Reasoning 55, 61–90 (2015). https://doi.org/10.1007/s10817-015-9328-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10817-015-9328-2

Keywords

Search

Navigation