Abstract
We show that a widely used benchmark set for the comparison of static analysis tools exhibits an impressive number of weaknesses, and that the internationally accepted quantitative evaluation metrics may lead to useless results. The weaknesses in the benchmark set were identified by applying a sound static analysis to the programs in this set and carefully interpreting the results. We propose how to deal with weaknesses of the quantitative metrics and how to improve such benchmarks and the evaluation process, in particular for external evaluations, in which an ideally neutral institution does the evaluation, whose results potential clients can trust.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
AbsInt Angewandte Informatik GmbH: Astrée. http://www.astree.de
AdaCore: CodePeer. http://www.adacore.com/codepeer
DARPA - Defense Advanced Research Projects Agency: Space/Time Analysis for Cybersecurity (STAC). http://www.darpa.mil/program/space-time-analysis-for-cybersecurity
Deutsch, A.: Static verification of dynamic properties. In: ACM SIGAda 2003 Conference (2003)
ETAPS/TAPAS: Competition on Software Verification (SV-COMP). http://sv-comp.sosy-lab.org/2017/
GrammaTech: CodeSonar. http://www.grammatech.com/products/codesonar
Kästner, D., Miné, A., Mauborgne, L., Rival, X., Feret, J., Cousot, P., Schmidt, A., Hille, H., Wilhelm, S., Ferdinand, C.: Finding all potential runtime errors and data races in automotive software. In: SAE World Congress 2017. SAE International (2017)
Lu, S., Li, Z., Qin, F., Tan, L., Zhou, P., Zhou, Y.: BugBench: benchmarks for evaluating bug detection tools. In: Workshop on the Evaluation of Software Defect Detection Tools (2005)
Martin, R., Christey, S., Jarzombek, J.: The case for common flaw enumeration. In: NIST Workshop on Software Security Assurance Tools, Techniques, and Methods, Long Beach, California, USA, November 2015. http://cwe.mitre.org/documents/case_for_cwes.pdf
Mathworks: Polyspace Bug Finder. http://www.mathworks.com/products/polyspace-bug-finder.html
Mathworks: Polyspace Code Prover. http://www.mathworks.com/products/polyspace-code-prover.html
Miné, A., Mauborgne, L., Rival, X., Feret, J., Cousot, P., Kästner, D., Wilhelm, S., Ferdinand, C.: Taking static analysis to the next level: proving the absence of run-time errors and data races with Astrée. In: Embedded Real Time Software and Systems Congress ERTS2 (2016)
MISRA-C:2004 - Guidelines for the use of the C language in critical systems (2004)
MISRA-C:2012 - Guidelines for the use of the C language in critical systems (2013)
NIST - National Institute of Standards and Technology: Juliet Suite for C/C++. http://samate.nist.gov/SRD/view.php?tsID=86
NIST - National Institute of Standards and Technology: SAMATE - Software Assurance Metrics And Tool Evaluation. http://samate.nist.gov/Main_Page.html
Holzmann, G.J.: The power of 10: rules for developing safety-critical code. Computer 39(6), 95–97 (2006). NASA/JPL Laboratory for Reliable Software. http://dx.doi.org/10.1109/MC.2006.212
Rogue Wave Software: Klocwork™. http://www.klocwork.com
Shiraishi, S., Mohan, V., Marimuthu, H.: Test suites for benchmarks of static analysis tools. In: 2015 IEEE International Symposium on Software Reliability Engineering Workshops, ISSRE Workshops, Gaithersburg, MD, USA, pp. 12–15, 2–5 November 2015. http://dx.doi.org/10.1109/ISSREW.2015.7392027
Software Engineering Institute, Carnegie Mellon University: CERT Secure Coding Validation Suite. http://www.cert.org/secure-coding/tools/validation-suite.cfm
Synopsys: Coverity. http://www.synopsys.com/software-integrity/products/static-code-analysis.html
Acknowledgment
The work presented in this paper is supported by the European ITEA3 project ASSUME.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Herter, J., Kästner, D., Mallon, C., Wilhelm, R. (2017). Benchmarking Static Code Analyzers. In: Tonetta, S., Schoitsch, E., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2017. Lecture Notes in Computer Science(), vol 10488. Springer, Cham. https://doi.org/10.1007/978-3-319-66266-4_13
Download citation
DOI: https://doi.org/10.1007/978-3-319-66266-4_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-66265-7
Online ISBN: 978-3-319-66266-4
eBook Packages: Computer ScienceComputer Science (R0)