Abstract
We present an actor-critic scheme for reinforcement learning in complex domains. The main contribution is to show that planning and I/O dynamics can be separated such that an intractable planning problem reduces to a simple multi-armed bandit problem, where each lever stands for a potentially arbitrarily complex policy. Furthermore, we use the Bayesian control rule to construct an adaptive bandit player that is universal with respect to a given class of optimal bandit players, thus indirectly constructing an adaptive agent that is universal with respect to a given class of policies.
This research was supported by the European Commission FP7-ICT, “GUIDE—Gentle User Interfaces for Disabled and Elderly citizens”.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Barto, A., Sutton, R., Anderson, C.: Neuron like elements that can solve difficult learning control problems. IEEE Trans. on Systems, Man and Cybernetics 13 (1983)
Berry, D.A., Fristedt, B.: Bandit problems: Sequential allocation of experiments. Monographs on Statistics and Applied Probability. Chapman & Hall, London (1985)
Bertsekas, D.P., Tsitsiklis, J.N.: Neuro-Dynamic Programming. Athena Scientific, Belmont (1996)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)
Ghavamzadeh, M., Engel, Y.: Bayesian actor-critic algorithms. In: Proc. of the 24th International Conference on Machine Learning (2007)
Gittins, J.C.: Multi-armed bandit allocation indices. Wiley-Interscience Series in Systems and Optimization. John Wiley & Sons, Ltd., Chichester (1989)
Ortega, P.A., Braun, D.A.: A bayesian rule for adaptive control based on causal interventions. In: The Third Conference on Artificial General Intelligence, pp. 121–126. Atlantis Press, Paris (2010)
Ortega, P.A., Braun, D.A.: A minimum relative entropy principle for learning and acting. Journal of Artificial Intelligence Research 38, 475–511 (2010)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ortega, P.A., Braun, D.A., Godsill, S. (2011). Reinforcement Learning and the Bayesian Control Rule. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds) Artificial General Intelligence. AGI 2011. Lecture Notes in Computer Science(), vol 6830. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22887-2_30
Download citation
DOI: https://doi.org/10.1007/978-3-642-22887-2_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-22886-5
Online ISBN: 978-3-642-22887-2
eBook Packages: Computer ScienceComputer Science (R0)