Skip to main content
SpringerLink
Log in

Robustness Testing of Software Verifiers

  • Conference paper
  • First Online:
Software Engineering and Formal Methods (SEFM 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14323))

Included in the following conference series:

  • 267 Accesses

Abstract

Software verification tools fully automatically prove the correctness of verification tasks (i.e., programs with correctness specifications). With their increasing application on safety-critical software, the quality of such tools becomes of prime importance. This quality is typically assessed via experimental evaluation. In this paper, we present a novel approach for robustness testing of software verifiers. We consider tools to be robust if their output (for a given input task) does not change under small perturbations of the input. The core idea of our technique is to start with tasks of publicly available benchmarks and systematically apply small program transformations on them which preserve program semantics. As a consequence, the ground truth known from the benchmark (i.e., the correct outcome used as an oracle during testing) carries over to all of its perturbed versions. We experimentally evaluate robustness testing on three state-of-the-art software verifiers. To this end, we perturbate 778 tasks from the annual Competition on Software Verification via 8 transformations. Our evaluation shows that all three verifiers are non-robust, however, to different extents.

Partially funded by German Research Council DFG under grant number 418257054.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/FlorianDyck/semtransforms.

  2. 2.

    For our formalization, we employ an artifical programming language; our implementation transforms C programs.

  3. 3.

    In Fig. 1 we used an error-function to make it proper C syntax.

  4. 4.

    To simplify the notation, the mapping \(\phi \) stores both assignments to variables and procedure definitions.

  5. 5.

    We chose CPAchecker for this purpose as it is the only verifier configurable to one particular algorithm.

References

  1. Beyer, D.: Competition on software verification and witness validation: SV-COMP 2023. In: TACAS. LNCS, vol. 13994, pp. 495–522. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-30820-8_29

  2. Beyer, D., Dangl, M.: Strategy selection for software verification based on Boolean features. In: Margaria, T., Steffen, B. (eds.) ISoLA 2018. LNCS, vol. 11245, pp. 144–159. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03421-4_11

    Chapter  Google Scholar 

  3. Beyer, D., Dangl, M., Wendler, P.: Boosting k-induction with continuously-refined invariants. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 622–640. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21690-4_42

    Chapter  Google Scholar 

  4. Beyer, D., Keremoglu, M.E.: CPAchecker: a tool for configurable software verification. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 184–190. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22110-1_16

    Chapter  Google Scholar 

  5. Beyer, D., Keremoglu, M.E., Wendler, P.: Predicate abstraction with adjustable-block encoding. In: Proceedings of 10th International Conference on Formal Methods in Computer-Aided Design, FMCAD 2010, Lugano, Switzerland, October 20–23, pp. 189–197. IEEE (2010). https://ieeexplore.ieee.org/document/5770949/

  6. Beyer, D., Lemberger, T.: CPA-SymExec: efficient symbolic execution in CPAchecker. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3–7, 2018, pp. 900–903. ACM (2018). https://doi.org/10.1145/3238147.3240478

  7. Beyer, D., Löwe, S.: Explicit-state software model checking based on CEGAR and interpolation. In: Cortellessa, V., Varró, D. (eds.) FASE 2013. LNCS, vol. 7793, pp. 146–162. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37057-1_11

    Chapter  Google Scholar 

  8. Beyer, D., Löwe, S., Wendler, P.: Reliable benchmarking: requirements and solutions. Int. J. Softw. Tools Technol. Transf. 21(1), 1–29 (2019). https://doi.org/10.1007/s10009-017-0469-y

    Article  Google Scholar 

  9. Cadar, C., Donaldson, A.F.: Analysing the program analyser. In: ICSE, pp. 765–768. ACM (2016). https://doi.org/10.1145/2889160.2889206

  10. Chalupa, M., Strejcek, J., Vitovská, M.: Joint forces for memory safety checking revisited. Int. J. Softw. Tools Technol. Transf. 22(2), 115–133 (2020). https://doi.org/10.1007/s10009-019-00526-2

  11. Chen, T.Y., Kuo, F., Liu, H., Poon, P., Towey, D., Tse, T.H., Zhou, Z.Q.: Metamorphic testing: a review of challenges and opportunities. ACM Comput. Surv. 51(1), 4:1–4:27 (2018). https://doi.org/10.1145/3143561

  12. Chen, Y.T., Furia, C.A.: Robustness testing of intermediate verifiers. In: Lahiri, S.K., Wang, C. (eds.) ATVA 2018. LNCS, vol. 11138, pp. 91–108. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01090-4_6

    Chapter  Google Scholar 

  13. Clarke, E., Grumberg, O., Jha, S., Lu, Y., Veith, H.: Counterexample-guided abstraction refinement. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 154–169. Springer, Heidelberg (2000). https://doi.org/10.1007/10722167_15

    Chapter  Google Scholar 

  14. Clarke, E., Kroening, D., Lerda, F.: A tool for checking ANSI-C programs. In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 168–176. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24730-2_15

    Chapter  MATH  Google Scholar 

  15. De Nicola, R.: Extensional equivalences for transition systems. Acta Informatica 24(2), 211–237 (1987). https://doi.org/10.1007/BF00264365

    Article  MathSciNet  MATH  Google Scholar 

  16. Dolan-Gavitt, B., Hulin, P., Kirda, E., Leek, T., Mambretti, A., Robertson, W.K., Ulrich, F., Whelan, R.: LAVA: large-scale automated vulnerability addition. In: IEEE Symposium on Security and Privacy, SP 2016, pp. 110–121. IEEE Computer Society (2016). https://doi.org/10.1109/SP.2016.15

  17. Dyck, F., Richter, C., Wehrheim, H.: Robustness testing of software verifiers (2023). https://doi.org/10.5281/zenodo.8186536

  18. Fink, X., Berger, P., Katoen, J.: Configurable benchmarks for C model checkers. In: NFM. LNCS, vol. 13260, pp. 338–354. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-06773-0_18

  19. Gadelha, M.R., Monteiro, F., Cordeiro, L., Nicole, D.: ESBMC v6.0: verifying C programs using k-induction and invariant inference. In: Beyer, D., Huisman, M., Kordon, F., Steffen, B. (eds.) TACAS 2019. LNCS, vol. 11429, pp. 209–213. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-17502-3_15

    Chapter  Google Scholar 

  20. Howar, F., Jasper, M., Mues, M., Schmidt, D., Steffen, B.: The RERS challenge: towards controllable and scalable benchmark synthesis. Int. J. Softw. Tools Technol. Transf. 23(6), 917–930 (2021). https://doi.org/10.1007/s10009-021-00617-z

    Article  Google Scholar 

  21. Kapus, T., Cadar, C.: Automatic testing of symbolic execution engines via program generation and differential testing. In: ASE, pp. 590–600. IEEE Computer Society (2017). https://doi.org/10.1109/ASE.2017.8115669

  22. Le, V., Afshari, M., Su, Z.: Compiler validation via equivalence modulo inputs. In: O’Boyle, M.F.P., Pingali, K. (eds.) PLDI ’14, pp. 216–226. ACM (2014). https://doi.org/10.1145/2594291.2594334

  23. Le, V., Sun, C., Su, Z.: Finding deep compiler bugs via guided stochastic program mutation. In: Aldrich, J., Eugster, P. (eds.) OOPSLA 2015, pp. 386–399. ACM (2015). https://doi.org/10.1145/2814270.2814319

  24. Milner, R.: Communication and Concurrency. PHI Series in Computer Science, Prentice Hall (1989)

    Google Scholar 

  25. Schott, S., Pauck, F.: Benchmark fuzzing for android taint analyses. In: SCAM, pp. 12–23. IEEE (2022). https://doi.org/10.1109/SCAM55253.2022.00007

  26. Steffen, B., Isberner, M., Naujokat, S., Margaria, T., Geske, M.: Property-driven benchmark generation: synthesizing programs of realistic structure. Int. J. Softw. Tools Technol. Transf. 16(5), 465–479 (2014). https://doi.org/10.1007/s10009-014-0336-z

    Article  Google Scholar 

  27. Sun, C., Le, V., Su, Z.: Finding compiler bugs via live code mutation. In: Visser, E., Smaragdakis, Y. (eds.) OOPSLA 2016, pp. 849–863. ACM (2016). https://doi.org/10.1145/2983990.2984038

  28. Zhang, C., Su, T., Yan, Y., Zhang, F., Pu, G., Su, Z.: Finding and understanding bugs in software model checkers. In: ESEC/SIGSOFT FSE, pp. 763–773. ACM (2019). https://doi.org/10.1145/3338906.3338932

Download references

Author information

Authors and Affiliations

  1. University of Oldenburg, Department of Computing Science, Oldenburg, Germany

    Florian Dyck, Cedric Richter & Heike Wehrheim

Authors
  1. Florian Dyck
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cedric Richter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Heike Wehrheim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cedric Richter .

Editor information

Editors and Affiliations

  1. NOVA University Lisbon, Caparica, Portugal

    Carla Ferreira

  2. Eindhoven University of Technology, Eindhoven, The Netherlands

    Tim A. C. Willemse

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dyck, F., Richter, C., Wehrheim, H. (2023). Robustness Testing of Software Verifiers. In: Ferreira, C., Willemse, T.A.C. (eds) Software Engineering and Formal Methods. SEFM 2023. Lecture Notes in Computer Science, vol 14323. Springer, Cham. https://doi.org/10.1007/978-3-031-47115-5_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47115-5_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47114-8

  • Online ISBN: 978-3-031-47115-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation