Meta-learning in Reinforcement Learning

Abstract

Meta-parameters in reinforcement learning should be tuned to the environmental dynamics and the animal performance. Here, we propose a biologically plausible meta-reinforcement learning algorithm for tuning these meta-parameters in a dynamic, adaptive manner. We tested our algorithm in both a simulation of a Markov decision task and in a non-linear control task. Our results show that the algorithm robustly finds appropriate meta-parameter values, and controls the meta-parameter time course, in both static and dynamic environments. We suggest that the phasic and tonic components of dopamine neuron firing can encode the signal required for meta-learning of reinforcement learning.

Introduction

Reinforcement learning is a particularly attractive model of animal learning, as dopamine neurons have the formal characteristics of the teaching signal known as the temporal difference (TD) error (Schultz, 1998). However, crucial to successful reinforcement learning is the careful setting of several meta-parameters. We earlier proposed that these meta-parameters are set and adjusted by neuromodulator neurons (Doya, 2002). We introduce here a simple, yet robust, algorithm that not only finds appropriate meta-parameters, but also controls the time course of these meta-parameters in a dynamic, adaptive manner.

The main issue in the theory of reinforcement learning (RL) is to maximize the long-term cumulative reward. Thus, central to reinforcement learning is the estimation of the value function

$$\text{V(x(t))=E}\text{∑}\text{k=0}\text{∞}\text{γ}{}^{\mathrm{k}}\text{r(t+k+1)}\text{,}$$

where r(t),r(t+1),r(t+2),… are the rewards acquired by following a certain action policy x→a starting from x(t), and γ is a discount factor such that 0<γ<1. The value function for the states before and after the transition should satisfy the consistency equation

$$\text{V(x(t−1))=E[r(t)+γV(x(t))].}$$

Therefore, any deviation from the consistency equation, expressed as

$$\text{δ(t)=r(t)+γV(x(t))−V(x(t−1)),}$$

should be zero on average. This signal is the TD error and is used as the teaching signal to learn the value function:

$$\text{Δ}\text{V(x(t−1))=αδ(t),}$$

where α is a learning rate.

The policy is usually defined via the action value function Q(x(t),a), which represents how much future rewards the agent would get by taking the action a at state x(t) and following the current policy in subsequent steps. One common way for stochastic action selection that encourages exploitation is to compute the probability to take an action by the soft-max function:

$$\text{P(a|x(t))=}\text{e}{}^{\mathrm{\beta Q(x(t),a)}}\text{∑}\text{a′=1}\text{M}\text{e}{}^{\mathrm{\beta Q(x(t),a\prime )}}\text{,}$$

where the meta-parameter β is called the inverse temperature.

Crucial to successful reinforcement learning is the careful setting of the three meta-parameters α,β and γ.

• The learning rate α is key to maximize the speed of learning, as small learning rates induce slow learning, and large learning rates induce oscillations.

• The inverse temperature β controls the exploitation–exploration trade-off. Ideally, β should initially be low to allow large exploration when the agent does not have a good mapping of which actions will be rewarding, and gradually increase as the agent reaps higher and higher rewards.

• The discount factor γ specifies how far in the future rewards should be taken into account. If γ is small, the agent learns to behave only for short-term rewards. Although setting γ close to 1 promotes the agent to learn to act for long-term rewards, there are several reasons γ should not be set too large. First any real learning agent, either artificial or biological, has a limited lifetime. A discounted value function is equivalent to a non-discounted value function for an agent with a constant death rate of 1−γ. Second, an agent has to acquire some rewards in time; for instance, an animal must find food before it starves; a robot must recharge its battery before it is exhausted. Third, if the environmental dynamics is highly stochastic or the dynamics is non-stationary, long-term prediction is doomed to be unreliable. Finally, the complexity of learning a value function increases with the increase of 1/(1−γ) (Littman et al., 1995).

In many applications, the meta-parameters are hand-tuned by the experimenter, and heuristics are often devised to schedule the value of these meta-parameters as learning progresses. Typically, α is set to decay as the inverse of time, and β to increase linearly with time. However, these algorithms cannot deal with non-stationary environments.

Several robust methods to dynamically adjust meta-parameters have been proposed. We (Schweighofer & Arbib, 1998) proposed a biological implementation of the IDBD (incremental delta bar delta) algorithm (Sutton, 1992) to tune α. The model improves learning performance by automatically setting near optimal synaptic learning rates for each synapse. To direct exploration, Thrun (1992) used a competence network that is trained to estimate the utility of exploration. Ishii et al. (2002) proposed to directly tune β as a function of the inverse of the variance of the action value function. However, although these algorithms are effective in balancing exploration and exploitation, they are specific to each meta-parameters and often make severe computation and memory requirements.