Skip to main content
SpringerLink
Log in
Book cover

NASA Formal Methods Symposium

NFM 2017: NASA Formal Methods pp 19–34Cite as

Learning from Faults: Mutation Testing in Active Automata Learning

Mutation Testing in Active Automata Learning

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 10227))

Abstract

System verification is often hindered by the absence of formal models. Peled et al. proposed black-box checking as a solution to this problem. This technique applies active automata learning to infer models of systems with unknown internal structure.

This kind of learning relies on conformance testing to determine whether a learned model actually represents the considered system. Since conformance testing may require the execution of a large number of tests, it is considered the main bottleneck in automata learning.

In this paper, we describe a randomised conformance testing approach which we extend with fault-based test selection. To show its effectiveness we apply the approach in learning experiments and compare its performance to a well-established testing technique, the partial W-method. This evaluation demonstrates that our approach significantly reduces the cost of learning – in one experiment by a factor of more than twenty.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    Note that more sophisticated test suite reduction/prioritisation strategies could be used. However, this is beyond the scope of this paper.

References

  1. Aichernig, B.K., Brandl, H., Jöbstl, E., Krenn, W., Schlick, R., Tiran, S.: Killing strategies for model-based mutation testing. Softw. Test. Verif. Reliab. 25(8), 716–748 (2015)

    Article  Google Scholar 

  2. Angluin, D.: Learning regular sets from queries and counterexamples. Inf. Comput. 75(2), 87–106 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  3. Banks, A., Gupta, R. (eds.): MQTT Version 3.1.1. OASIS Standard, October 2014. Latest version: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html

  4. Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. In: Cerioli, M. (ed.) FASE 2005. LNCS, vol. 3442, pp. 175–189. Springer, Heidelberg (2005). doi:10.1007/978-3-540-31984-9_14

    Chapter  Google Scholar 

  5. Cassel, S., Howar, F., Jonsson, B., Steffen, B.: Active learning for extended finite state machines. Formal Asp. Comput. 28(2), 233–263 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  6. Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. 4(3), 178–187 (1978)

    Article  MATH  Google Scholar 

  7. Combe, D., de la Higuera, C., Janodet, J.-C.: Zulu: an interactive learning competition. In: Yli-Jyrä, A., Kornai, A., Sakarovitch, J., Watson, B. (eds.) FSMNLP 2009. LNCS (LNAI), vol. 6062, pp. 139–146. Springer, Heidelberg (2010). doi:10.1007/978-3-642-14684-8_15

    Chapter  Google Scholar 

  8. emqtt. http://emqtt.io/. Accessed 29 Nov 2016

  9. Fabbri, S., Delamaro, M.E., Maldonado, J.C., Masiero, P.C.: Mutation analysis testing for finite state machines. In: ISSRE 1994, pp. 220–229. IEEE (1994)

    Google Scholar 

  10. Fiterău-Broştean, P., Janssen, R., Vaandrager, F.: Combining model learning and model checking to analyze TCP implementations. In: Chaudhuri, S., Farzan, A. (eds.) CAV 2016. LNCS, vol. 9780, pp. 454–471. Springer, Cham (2016). doi:10.1007/978-3-319-41540-6_25

    Google Scholar 

  11. Fujiwara, S., von Bochmann, G., Khendek, F., Amalou, M., Ghedamsi, A.: Test selection based on finite state models. IEEE Trans. Softw. Eng. 17(6), 591–603 (1991)

    Article  Google Scholar 

  12. Howar, F., Steffen, B., Merten, M.: From ZULU to RERS - lessons learned in the ZULU challenge. In: Margaria, T., Steffen, B. (eds.) ISoLA 2010. LNCS, vol. 6415, pp. 687–704. Springer, Heidelberg (2010). doi:10.1007/978-3-642-16558-0_55

    Chapter  Google Scholar 

  13. Isberner, M., Howar, F., Steffen, B.: The TTT algorithm: a redundancy-free approach to active automata learning. In: Bonakdarpour, B., Smolka, S.A. (eds.) RV 2014. LNCS, vol. 8734, pp. 307–322. Springer, Cham (2014). doi:10.1007/978-3-319-11164-3_26

    Google Scholar 

  14. Isberner, M., Howar, F., Steffen, B.: The open-source LearnLib. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 487–495. Springer, Cham (2015). doi:10.1007/978-3-319-21690-4_32

    Chapter  Google Scholar 

  15. Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)

    Article  Google Scholar 

  16. Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines - a survey. Proc. IEEE 84(8), 1090–1123 (1996)

    Article  Google Scholar 

  17. Lee, D., Yannakakis, M.: Testing finite-state machines: state identification and verification. IEEE Trans. Comput. 43(3), 306–320 (1994)

    Article  MathSciNet  Google Scholar 

  18. Margaria, T., Niese, O., Raffelt, H., Steffen, B.: Efficient test-based model generation for legacy reactive systems. In: Ninth IEEE International High-Level Design Validation and Test Workshop 2004, pp. 95–100. IEEE Computer Society (2004)

    Google Scholar 

  19. Moerman, J.: Yannakakis - test-case generator. https://gitlab.science.ru.nl/moerman/Yannakakis. Accessed 30 Nov 2016

  20. Niese, O.: An integrated approach to testing complex systems. Ph.D. thesis, Dortmund University of Technology (2003)

    Google Scholar 

  21. Peled, D., Vardi, M.Y., Yannakakis, M.: Black box checking. In: Wu, J., Chanson, S.T., Gao, Q. (eds.) FORTE XII/PSTV XIX 1999. IFIP AICT, vol. 28, pp. 225–240. Springer, Boston (1999). doi:10.1007/978-0-387-35578-8_13

    Chapter  Google Scholar 

  22. Pretschner, A.: Defect-based testing. In: Dependable Software Systems Engineering, NATO Science for Peace and Security Series, D: Information and Communication Security, vol. 40, pp. 224–245. IOS Press (2015)

    Google Scholar 

  23. Rivest, R.L., Schapire, R.E.: Inference of finite automata using homing sequences. Inf. Comput. 103(2), 299–347 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  24. de Ruiter, J., Poll, E.: Protocol state fuzzing of TLS implementations. In: USENIX Security 15, pp. 193–206. USENIX Association (2015)

    Google Scholar 

  25. Shahbaz, M., Groz, R.: Inferring Mealy machines. In: Cavalcanti, A., Dams, D.R. (eds.) FM 2009. LNCS, vol. 5850, pp. 207–222. Springer, Heidelberg (2009). doi:10.1007/978-3-642-05089-3_14

    Chapter  Google Scholar 

  26. Smeenk, W., Moerman, J., Vaandrager, F., Jansen, D.N.: Applying automata learning to embedded control software. In: Butler, M., Conchon, S., Zaïdi, F. (eds.) ICFEM 2015. LNCS, vol. 9407, pp. 67–83. Springer, Cham (2015). doi:10.1007/978-3-319-25423-4_5

    Chapter  Google Scholar 

  27. Tappler, M.: mut-learn - randomised mutation-based equivalence testing. https://github.com/mtappler/mut-learn. Accessed 07 Dec 2016

  28. Tappler, M., Aichernig, B.K., Bloem, R.: Model-based testing IoT communication via active automata learning. In: ICST 2017. IEEE Computer Society (2017)

    Google Scholar 

  29. TCP models. https://gitlab.science.ru.nl/pfiteraubrostean/tcp-learner/tree/cav-aec/models. Accessed 14 Nov 2016

  30. Vasilevskii, M.P.: Failure diagnosis of automata. Cybernetics 9(4), 653–665 (1973)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgment

This work was supported by the TU Graz LEAD project “Dependable Internet of Things in Adverse Environments”. We would also like to thank the developers of LearnLib and of the test-case generator available at [19].

Author information

Authors and Affiliations

  1. Institute of Software Technology, Graz University of Technology, Graz, Austria

    Bernhard K. Aichernig & Martin Tappler

Authors
  1. Bernhard K. Aichernig
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martin Tappler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martin Tappler .

Editor information

Editors and Affiliations

  1. Stanford University, Palo Alto, California, USA

    Clark Barrett

  2. NASA Ames Research Center, Moffett Field, California, USA

    Misty Davies

  3. NASA Ames Research Center, Moffett Field, California, USA

    Temesghen Kahsai

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Aichernig, B.K., Tappler, M. (2017). Learning from Faults: Mutation Testing in Active Automata Learning. In: Barrett, C., Davies, M., Kahsai, T. (eds) NASA Formal Methods. NFM 2017. Lecture Notes in Computer Science(), vol 10227. Springer, Cham. https://doi.org/10.1007/978-3-319-57288-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-57288-8_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-57287-1

  • Online ISBN: 978-3-319-57288-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation