Abstract
We present a formalism, algorithms and tools to synthesise reactive systems that behave efficiently, i.e., which achieve an optimal trade-off between a given cost and reward model. Synthesis aims to automatically generate a program from a specification. Most research in this area focuses on qualitative specifications, i.e., those that define a system as either correct or incorrect. The result can be a system that is correct, but still shows undesired behaviour, e.g., because it is too slow, inefficient or resource-intensive. Quantitative synthesis aims to use additional information to guide the synthesizer towards a desired implementation. Trade-offs between costs and rewards provide a natural source of information in order to guarantee efficiency. The systems we want to synthesize are open, i.e., they react to input signals from their environment. So, we have to specify how to combine the trade-offs the system decides to make for each input. There are several possible ways, e.g., worst or best case, or average case. In this paper we focus on the average case, i.e., we focus on the expected trade-off achieved by a system. We define the problem of finding the system with the best expected behaviour according to a quantitative specification. This specification associates costs and rewards with each decision the system makes and defines a probabilistic environment that the system operates in. We analyze the feasibility of this task (i.e., prove that such systems exist and are computable) and present three algorithms to compute an optimal system for a given specification. We compare a prototypical implementation of these algorithms against each other and, based on the best-performing algorithm, develop a novel symbolic implementation and integrate it into the probabilistic model checker PRISM. We report on experiments showing that our algorithm can analyze models with several million states.
Similar content being viewed by others
Notes
We can model a DDOS attack, e.g., by assuming that it happens with low probability: on attack, the environment suddenly changes its behaviour drastically.
Our approach can also handle liveness specifications resulting in a Ratio-MDP with parity objective, which is then reduced to solving a sequence of MDPs with mean-payoff parity objectives [11].
Pun intended.
A feasible solution is an assignment that fulfills the linear equations.
Note that, due to the way that we model the assembly line here in PRISM, these are different sized MDPs to the ones for the same example used in Table 1.
References
Bahar, I., Frohm, E., Gaona, C., Hachtel, G., Macii, E., Pardo, A., Somenzi, F.: Algebraic decision diagrams and their applications. Form. Methods Syst. Des. 10(2/3), 171–206 (1997)
Baier, C., Clarke, E., Hartonas-Garmhausen, V., Kwiatkowska, M., Ryan, M.: Symbolic model checking for probabilistic processes. In: Degano, P., Gorrieri, R., Marchetti-Spaccamela, A. (eds.) Proceedings of 24th International Colloquium on Automata, Languages and Programming (ICALP’97), volume 1256 of LNCS, pp. 430–440. Springer, New York (1997)
Baier, C., Katoen, J.-P.: Principles of Model Checking. MIT Press, Berlin (2008)
Bloem, R., Chatterjee, K., Henzinger, T.A., Jobstmann, B.: Better quality in synthesis through quantitative objectives. In: Bouajjani, A., Maler, O. (eds.) CAV, volume 5643 of LNCS, pp. 140–156. Springer, Berlin (2009)
Bloem, R., Gabow, H., Somenzi, F.: An algorithm for strongly connected component analysis in n log n symbolic steps. In: Proceedings of 3rd International Conference on Formal Methods in Computer-Aided Design (FMCAD’00), pp. 37–54 (2000)
Bloem, R., Greimel, K., Henzinger, T. A., Jobstmann, B.: Synthesizing robust systems. In: FMCAD, pp. 85–92. IEEE (2009)
Brázdil, T., Brožek, V., Chatterjee, K., Forejt, V.Kučera, A.: Two views on multiple mean-payoff objectives inMarkov decision processes. In: LICS, pp. 33–42. IEEE ComputerSociety (2011)
Bryant, R.E.: Graph-based algorithms for boolean function manipulation. IEEE Trans. Comput. 35(8), 677–691 (1986)
Burch, J.R., Clarke, E.M., McMillan, K.L., Dill, D.L., Hwang, L.J.: Symbolic model checking: \(10^{20}\) states and beyond. Inf. Comput. 98(2), 142–170 (1992)
Chatterjee, K., Henzinger, M.: Faster and dynamic algorithms for maximal end-component decomposition and related graph problems in probabilistic verification. In: Proceedings of 22nd Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’11), pp. 1318–1336 (2011)
Chatterjee, K., Henzinger, T.A., Jobstmann, B., Singh, R.: Measuring and synthesizing systems in probabilistic environments. In: Touili, T., Cook, B., Jackson, P. (eds.) CAV, volume 6174 of LNCS, pp. 380–395. Springer, Berlin (2010)
Chatterjee, K., Majumdar, R., Henzinger, T.: Markov decision processes with multiple objectives. In: Proceedings of 23rd International Symposium on Theoretical Aspects of Computer Science (STACS’06), volume 3884 of LNCS, pp. 325–336. Springer (2006)
de Alfaro, L.: Formal Verification of Probabilistic Systems. PhD thesis, Stanford University (1997)
Derman, C.: On sequential decisions and Markov chains. Manage. Sci. 9(1), 16–24 (1962)
Etessami, K., Kwiatkowska, M., Vardi, M., Yannakakis, M.: Multi-objective model checking of Markov decision processes. In: Proceedings of 13th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’07), volume 4424 of LNCS, pp. 50–65. Springer (2007)
Forejt, V., Kwiatkowska, M., Norman, G., Parker, D., Qu, H.: Quantitative multi-objective verification for probabilistic systems. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS, volume 6605 of LNCS, pp. 112–127. Springer, Berlin (2011)
Fujita, M., Mcgeer, P.C., Yang, J.C.Y.: Multi-terminal binary decision diagrams: an efficient data structure for matrix representation. Form. Methods Syst. Des. V 10(2/3), 149–169 (1997)
Gimbert, H.: Pure stationary optimal strategies in Markov decision processes. In: STACS’07, pp. 200–211. Springer (2007)
Hachtel, G., Macii, E., Pardo, A., Somenzi, F.: Markovian analysis of large finite state machines. Comput. Aided Des. Integr. Circuits Syst. IEEE Trans. 15(12), 1479–1493 (1996)
Haverkort, B.R.: Performance of Computer Communication Systems—A Model-Based Approach. Wiley, New York (1998)
Isbell, J.R., Marlow, W.H.: Attrition games. Nav. Res. Logist. Q. 3, 71–94 (1956)
Kemeny, J., Snell, J., Knapp, A.: Denumerable Markov Chains, 2nd edn. Springer, New York (1976)
Kwiatkowska, M., Norman, G., Parker, D.: Probabilistic symbolic model checking with PRISM: a hybrid approach. In: Katoen, J.-P., Stevens, P. (eds.) TACAS, volume 2280 of LNCS, pp. 52–66. Springer, Berlin (2002)
Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: Verification of probabilistic real-time systems. In: Proceedings of 23rd International Conference on Computer Aided Verification (CAV’11), volume 6806 of LNCS, pp. 585–591. Springer (2011)
Kwiatkowska, M., Norman, G., Parker. D.: The PRISM benchmark suite. In: Proceedings of 9th International Conference on Quantitative Evaluation of Systems (QEST’12), pp. 203–204. IEEE CS Press (2012)
Kwiatkowska, M., Norman, G., Parker, D., Sproston, J.: Performance analysis of probabilistic timed automata using digital clocks. Form. Methods Syst. Des. 29(1), 33–78 (2006)
Lehmann, D.J., Rabin, M. O.: On the advantages of free choice: a symmetric and fully distributed solution to the dining philosophers problem. In: POPL (1981)
Manna, Z., Pnueli, A.: Temporal Verification of Reactive Systems—Safety. Springer, Berlin (1995)
Norman, G., Parker, D., Kwiatkowska, M., Shukla, S.K., Gupta, R.: Using probabilistic model checking for dynamic power management. Formal Asp. Comput. 17(2), 160–176 (2005)
Norris, J.: Markov Chains. Cambridge University Press, Cambridge (2003)
Parker, D.: Implementation of Symbolic Model Checking for Probabilistic Systems. PhD thesis, University of Birmingham (2002)
Parr, R., Russell, S.J.: Reinforcement learning with hierarchies of machines. In: Jordan, M.I., Kearns, M.J., Solla, S.A. (eds.) NIPS. The MIT Press, Cambridge (1997)
Pnueli, A.: The temporal logic of programs. In: FOCS, pp. 46–57. IEEE Computer Society (1977)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley-Interscience, Chichester (1994)
Rabin, M.O.: N-process mutual exclusion with bounded waiting by 4 \(log_2 n\)-valued shared variable. J. Comput. Syst. Sci. 25(1), 66–75 (1982)
Tijms, H.C.: A First Course in Stochastic Models. Wiley, Chichester (2003)
von Essen, C., Jobstmann, B.: Synthesizing systems with optimal average-case behavior for ratio objectives. In: Reich, J., Finkbeiner, B. (eds.) Proceedings International Workshop on Interactions, Games and Protocols, iWIGP 2011, Saarbrücke, Germany, 27th March 2011. EPTCS, vol. 50, pp. 17–32 (2011)
von Essen, C., Jobstmann, B.: Synthesizing efficient controllers. In: International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI), pp. 428–444 (2012)
Wimmer, R., Braitling, B., Becker, B., Hahn, E.M., Crouzen, P., Hermanns, H., Dhama, A., Theel, O.: Symblicit calculation of long-run averages for concurrent probabilistic systems. In: QEST, pp. 27–36. IEEE Computer Society (2010)
Wimmer, R., Derisavi, S., Hermanns, H.: Symbolic partition refinement with dynamic balancing of time and space. In: QEST, pp. 65–74. IEEE Computer Society (2008)
Yue, H., Bohnenkamp, H.C., Katoen, J.-P.: Analyzing energy consumption in a gossiping MAC protocol. In: Müller-Clostermann, B., Echtle, K., Rathgeb, E.P. (eds.) MMB/DFT, volume 5987 of LNCS, pp. 107–119. Springer (2010)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
von Essen, C., Jobstmann, B., Parker, D. et al. Synthesizing efficient systems in probabilistic environments. Acta Informatica 53, 425–457 (2016). https://doi.org/10.1007/s00236-015-0237-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00236-015-0237-y