Skip to main content
SpringerLink
Log in

Model-Based Testing of Probabilistic Systems with Stochastic Time

  • Conference paper
  • First Online:
Tests and Proofs (TAP 2017)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 10375))

Included in the following conference series:

Abstract

This paper presents a model-based testing framework for black-box probabilistic systems with stochastic continuous time. Markov automata are used as an underlying model. We show how to generate, execute and evaluate test cases automatically from a probabilistically timed requirements model. In doing so, we connect classical ioco-theory with statistical hypothesis testing; our ioco-style algorithms test for functional behaviour, while \(\chi ^2\) hypothesis tests and confidence interval estimations assess the statistical correctness of the system.

A crucial development are the classical soundness and completeness properties of our framework. Soundness states that test cases assign the correct verdict, while completeness states that our methods are powerful enough to discover each discrepancy in functional or statistical misbehaviour, up to arbitrary precision.

We illustrate our framework via the Bluetooth device discovery protocol.

This research has been partially funded by STW and ProRail under the project ArRangeer (12238), STW under the project SEQUOIA (15474), NWO under the project BEAT (612.001.303), NWO under the project SamSam (628.005.015), and the EU under the project SUCCESS (509-18240).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The topic of statistical testing, e.g. [1, 52], is concerned with choosing test inputs probabilistically; it does not check the correctness of the random choices made by a system itself.

  2. 2.

    Note that \( freq \left( O\right) \) is not a bijection, but used here for ease of notation.

References

  1. Beyer, M., Dulz, W.: Scenario-based statistical testing of quality of service requirements. In: Leue, S., Systä, T.J. (eds.) Scenarios: Models, Transformations and Tools. LNCS, vol. 3466, pp. 152–173. Springer, Heidelberg (2005). doi:10.1007/11495628_9

    Chapter  Google Scholar 

  2. Bohnenkamp, H., Belinfante, A.: Timed testing with TorX. In: Fitzgerald, J., Hayes, I.J., Tarlecki, A. (eds.) FM 2005. LNCS, vol. 3582, pp. 173–188. Springer, Heidelberg (2005). doi:10.1007/11526841_13

    Chapter  Google Scholar 

  3. Bohnenkamp, H., Stoelinga, M.: Quantitative testing. In: Proceedings of the 8th International Conference on Embedded Software, (EMSOFT), pp. 227–236. ACM (2008)

    Google Scholar 

  4. Böhr, F.: Model-based statistical testing of embedded systems. In: IEEE 4th International Conference on Software Testing, Verification and Validation, pp. 18–25 (2011)

    Google Scholar 

  5. Bozga, M., David, A. Hartmanns, H., Hermanns, H., Larsen, K.G., Legay, A., Tretmans, J.: State-of-the-art tools and techniques for quantitative modelling and analysis of embedded systems. In: DATE, pp. 370–375 (2012)

    Google Scholar 

  6. Briones, L.B., Brinksma, E.: A test generation framework for quiescent real-time systems. In: Grabowski, J., Nielsen, B. (eds.) FATES 2004. LNCS, vol. 3395, pp. 64–78. Springer, Heidelberg (2005). doi:10.1007/978-3-540-31848-4_5

    Chapter  Google Scholar 

  7. Briones, L.B., Brinksma, E., Stoelinga, M.: A semantic framework for test coverage. In: Graf, S., Zhang, W. (eds.) ATVA 2006. LNCS, vol. 4218, pp. 399–414. Springer, Heidelberg (2006). doi:10.1007/11901914_30

    Chapter  Google Scholar 

  8. Cheung, L., Stoelinga, M., Vaandrager, F.: A testing scenario for probabilistic processes. J. ACM 54(6), 29:1–29:45 (2007). Article 29

    Article  MathSciNet  MATH  Google Scholar 

  9. Choi, S.G., Dachman-Soled, D., Malkin, T., Wee, H.: Improved non-committing encryption with applications to adaptively secure protocols. In: Matsui, M. (ed.) ASIACRYPT 2009. LNCS, vol. 5912, pp. 287–302. Springer, Heidelberg (2009). doi:10.1007/978-3-642-10366-7_17

    Chapter  Google Scholar 

  10. Cleaveland, R., Dayar, Z., Smolka, S.A., Yuen, S.: Testing preorders for probabilistic processes. Inf. Comput. 154(2), 93–148 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  11. Cohn, D.L.: Measure Theory. Birkhäuser, Basel (1980)

    Book  MATH  Google Scholar 

  12. D’Argenio, P.R., Katoen, J.-P.: A theory of stochastic systems part I: stochastic automata. Inf. Comput. 203(1), 1–38 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  13. Deng, Y., Hennessy, M.: On the semantics of Markov automata. Inf. Comput. 222, 139–168 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  14. Deng, Y., Hennessy, M., van Glabbeek, R.J., Morgan, C.: Characterising testing preorders for finite probabilistic processes. CoRR (2008)

    Google Scholar 

  15. Duflot, M., Kwiatkowska, M., Norman, G., Parker, D.: A formal analysis of bluetooth device discovery. Int. J. Softw. Tools Technol. Transf. 8(6), 621–632 (2006)

    Article  Google Scholar 

  16. Eisentraut, C., Hermanns, H., Zhang, L.: On probabilistic automata in continuous time. In: IEEE 25th Annual Symposium on LICS, pp. 342–351 (2010)

    Google Scholar 

  17. Gerhold, M., Stoelinga, M.: Model-based testing of probabilistic systems. In: Stevens, P., Wąsowski, A. (eds.) FASE 2016. LNCS, vol. 9633, pp. 251–268. Springer, Heidelberg (2016). doi:10.1007/978-3-662-49665-7_15

    Chapter  Google Scholar 

  18. Gerhold, M., Stoelinga, M.: Model-based testing of stochastic systems with IOCO theory. In: A-TEST 2016, Proceedings of the 7th International Workshop on Automating Test Case Design, Selection, and Evaluation, pp. 45–51. ACM (2016)

    Google Scholar 

  19. Guiotto, A., Acquaroli, B., Martelli, A.: MaTeLo: automated testing suite for software validation. In: DASIA, vol. 532 (2003)

    Google Scholar 

  20. Hartmanns, A., Hermanns, H.: The Modest Toolset: An Integrated Environment for Quantitative Modelling and Verification. In: Ábrahám, E., Havelund, K. (eds.) TACAS 2014. LNCS, vol. 8413, pp. 593–598. Springer, Heidelberg (2014). doi:10.1007/978-3-642-54862-8_51

    Chapter  Google Scholar 

  21. Hermanns, H., Chains, I.M.: Interactive Markov Chains: and the Quest for Quantified Quality. Springer, Heidelberg (2002)

    Book  MATH  Google Scholar 

  22. Hessel, A., Larsen, K.G., Mikucionis, M., Nielsen, B., Pettersson, P., Skou, A.: Testing real-time systems using UPPAAL. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing. LNCS, vol. 4949, pp. 77–117. Springer, Heidelberg (2008). doi:10.1007/978-3-540-78917-8_3

    Chapter  Google Scholar 

  23. Hierons, R.M., Merayo, M.G.: Mutation testing from probabilistic and stochastic finite state machines. J. Syst. Softw. 82(11), 1804–1818 (2009)

    Article  Google Scholar 

  24. Hierons, R.M., Núñez, M.: Testing probabilistic distributed systems. In: Hatcliff, J., Zucca, E. (eds.) FMOODS/FORTE -2010. LNCS, vol. 6117, pp. 63–77. Springer, Heidelberg (2010). doi:10.1007/978-3-642-13464-7_6

    Chapter  Google Scholar 

  25. Hierons, R.M., Núñez, M.: Implementation relations and probabilistic schedulers in the distributed test architecture. J. Syst. Softw. (2017)

    Google Scholar 

  26. Hwang, I., Cavalli, A.R.: Testing a probabilistic FSM using interval estimation. Comput. Netw. 54(7), 1108–1125 (2010)

    Article  MATH  Google Scholar 

  27. Jegourel, C., Legay, A., Sedwards, S.: A Platform for High Performance Statistical Model Checking – PLASMA. In: Flanagan, C., König, B. (eds.) TACAS 2012. LNCS, vol. 7214, pp. 498–503. Springer, Heidelberg (2012). doi:10.1007/978-3-642-28756-5_37

    Chapter  Google Scholar 

  28. Krichen, M., Tripakis, S.: Conformance testing for real-time systems. Form. Methods Syst. Des. 34(3), 238–304 (2009)

    Article  MATH  Google Scholar 

  29. Kwiatkowska, M., Norman, G., Parker, D.: PRISM: probabilistic symbolic model checker. In: Field, T., Harrison, P.G., Bradley, J., Harder, U. (eds.) TOOLS 2002. LNCS, vol. 2324, pp. 200–204. Springer, Heidelberg (2002). doi:10.1007/3-540-46029-2_13

    Chapter  Google Scholar 

  30. Larsen, K.G., Mikucionis, M., Nielsen, B.: Online testing of real-time systems using Uppaal. In: Grabowski, J., Nielsen, B. (eds.) FATES 2004. LNCS, vol. 3395, pp. 79–94. Springer, Heidelberg (2005). doi:10.1007/978-3-540-31848-4_6

    Chapter  Google Scholar 

  31. Larsen, K.G., Skou, A.: Bisimulation through probabilistic testing, pp. 344–352. ACM Press (1989)

    Google Scholar 

  32. Lohrey, M., D’Argenio, P.R., Hermanns, H.: Axiomatising Divergence. In: Widmayer, P., Eidenbenz, S., Triguero, F., Morales, R., Conejo, R., Hennessy, M. (eds.) ICALP 2002. LNCS, vol. 2380. Springer, Heidelberg (2002)

    Google Scholar 

  33. Marsan, M.A., Balbo, G., Conte, G., Donatelli, S., Franceschinis, G.: Modelling with Generalized Stochastic Petri Nets. Wiley, Hoboken (1994)

    MATH  Google Scholar 

  34. Milner, R.: A Calculus of Communicating Systems. Springer, Heidelberg (1980)

    Book  MATH  Google Scholar 

  35. Mostowski, W., Poll, E., Schmaltz, J., Tretmans, J., Wichers Schreur, R.: Model-based testing of electronic passports. In: Alpuente, M., Cook, B., Joubert, C. (eds.) FMICS 2009. LNCS, vol. 5825, pp. 207–209. Springer, Heidelberg (2009). doi:10.1007/978-3-642-04570-7_19

    Chapter  Google Scholar 

  36. Nie, J., Demmel, J., Gu, M.: Global minimization of rational functions and the nearest GCDs. J. Glob. Optim. 40(4), 697–718 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  37. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Hoboken (2014)

    MATH  Google Scholar 

  38. Kwiatkowska, M., Norman, G., Parker, D.: Stochastic model checking. In: Bernardo, M., Hillston, J. (eds.) SFM 2007. LNCS, vol. 4486, pp. 220–270. Springer, Heidelberg (2007). doi:10.1007/978-3-540-72522-0_6

    Chapter  Google Scholar 

  39. Russell, N., Moore, R.: Explicit modelling of state occupancy in hidden markov models for automatic speech recognition. In: IEEE International Conference on Acoustics, Speech, and Signal Processing ICASSP, vol. 10, pp. 5–8 (1985)

    Google Scholar 

  40. Segala, R.: Modeling and verification of randomized distributed real-time systems. Ph.D. thesis, Cambridge, MA, USA (1995)

    Google Scholar 

  41. B. SIG. Bluetooth Specification, version 1.2 (2003). www.bluetooth.com

  42. Song, L., Zhang, L., Godskesen, J.C., Hermanns, H., Eisentraut, C.: Late weak bisimulation for Markov automata. CoRR, abs/1202.4116 (2012)

    Google Scholar 

  43. Stoelinga, M.: Alea jacta est: verification of probabilistic, real-time and parametric systems. Ph.D. thesis, Radboud University of Nijmegen (2002)

    Google Scholar 

  44. Stoelinga, M., Vaandrager, F.: Root contention in IEEE 1394. In: Katoen, J.-P. (ed.) ARTS 1999. LNCS, vol. 1601, pp. 53–74. Springer, Heidelberg (1999). doi:10.1007/3-540-48778-6_4

    Chapter  Google Scholar 

  45. Stokkink, W.G.J., Timmer, M., Stoelinga, M.I.A.: Divergent quiescent transition systems. In: Veanes, M., Viganò, L. (eds.) TAP 2013. LNCS, vol. 7942, pp. 214–231. Springer, Heidelberg (2013). doi:10.1007/978-3-642-38916-0_13

    Chapter  Google Scholar 

  46. Thrun, S.: Probabilistic robotics. Commun. ACM 45(3), 52–57 (2002)

    Article  Google Scholar 

  47. Timmer, M., Brinksma, H., Stoelinga, M., Testing, M.-B., Software, I., Safety, S.: Specification and verification, Volume 30 of NATO Science for Peace and Security, pp. 1–32. IOS Press (2011)

    Google Scholar 

  48. Timmer, M., Katoen, J.-P., Pol, J., Stoelinga, M.I.A.: Efficient modelling and generation of Markov automata. In: Koutny, M., Ulidowski, I. (eds.) CONCUR 2012. LNCS, vol. 7454, pp. 364–379. Springer, Heidelberg (2012). doi:10.1007/978-3-642-32940-1_26

    Chapter  Google Scholar 

  49. Timmer, M., van de Pol, J., Stoelinga, M.I.A.: Confluence reduction for Markov automata. In: Braberman, V., Fribourg, L. (eds.) FORMATS 2013. LNCS, vol. 8053, pp. 243–257. Springer, Heidelberg (2013). doi:10.1007/978-3-642-40229-6_17

    Chapter  Google Scholar 

  50. Tretmans, J.: Test generation with inputs, outputs and repetitive quiescence. Softw. - Concepts Tools 17(3), 103–120 (1996)

    MATH  Google Scholar 

  51. Veanes, M., Campbell, C., Grieskamp, W., Schulte, W., Tillmann, N., Nachmanson, L.: Model-based testing of object-oriented reactive systems with spec explorer. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing. LNCS, vol. 4949, pp. 39–76. Springer, Heidelberg (2008). doi:10.1007/978-3-540-78917-8_2

    Chapter  Google Scholar 

  52. Whittaker, J.A., Rekab, K., Thomason, M.G.: A Markov chain model for predicting the reliability of multi-build software. Inf. Softw. Technol. 42(12), 889–894 (2000)

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank David Huistra for his aid on the case study.

Author information

Authors and Affiliations

  1. University of Twente, Enschede, The Netherlands

    Marcus Gerhold & Mariëlle Stoelinga

Authors
  1. Marcus Gerhold
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mariëlle Stoelinga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mariëlle Stoelinga .

Editor information

Editors and Affiliations

  1. Technical University of Darmstadt, Darmstadt, Germany

    Sebastian Gabmeyer

  2. University of Oslo, Oslo, Norway

    Einar Broch Johnsen

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Gerhold, M., Stoelinga, M. (2017). Model-Based Testing of Probabilistic Systems with Stochastic Time. In: Gabmeyer, S., Johnsen, E. (eds) Tests and Proofs. TAP 2017. Lecture Notes in Computer Science(), vol 10375. Springer, Cham. https://doi.org/10.1007/978-3-319-61467-0_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-61467-0_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-61466-3

  • Online ISBN: 978-3-319-61467-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation