Policy Generation from Latent Embeddings for Reinforcement Learning

Abstract

The human brain endows us with extraordinary capabilities that enable us to create, imagine, and generate anything we desire. Specifically, we have fascinating imaginative skills allowing us to generate fundamental knowledge from abstract concepts. Motivated by these traits, numerous areas of machine learning, notably unsupervised learning and reinforcement learning, have started using such ideas at their core. Nevertheless, these methods do not come without fault. A fundamental issue with reinforcement learning especially now when used with neural networks as function approximators is their limited achievable optimality compared to its uses from tabula rasa. Due to the nature of learning with neural networks, the behaviours achievable for each task are inconsistent and providing a unified approach that enables such optimal policies to exist within a parameter space would facilitate both the learning procedure and the behaviour outcomes. Consequently, we are interested in discovering whether reinforcement learning can be facilitated with unsupervised learning methods in a manner to alleviate this downfall. This work aims to provide an analysis of the feasibility of using generative models to extract learnt reinforcement learning policies (i.e. model parameters) with the intention of conditionally sampling the learnt policy-latent space to generate new policies. We demonstrate that under the current proposed architecture, these models are able to recreate policies on simple tasks whereas fail on more complex ones. We therefore provide a critical analysis of these failures and discuss further improvements which would aid the proliferation of this work.

Appendices

A Environments

Fig. 4.

Representations of the environments used in these experiments. From left to right: CartPole-v0, MountainCar-v0, LunarLander-v0

CartPole-v0: This environment corresponds to the cart-pole problem introduced by [21]. An agent (i.e. the cart) has a pole attached by an un-actuated joint and moves along a frictionless track (see Fig. 4a). The agent starts at position (0, 0) with the pole upright and can apply a force of $+1$ or $-1$ to reduce the agent’s velocity to prevent the pole from falling over. The agent receives a reward of $+1$ for every timestep that the pole remains upright, including when it reaches the terminal state. However, the agent velocity is affected by the pole angle and centre of gravity which dictates the force needed to be applied to keep the pole upright. An episode terminates either if the pole surpasses 15 degrees vertically or the agent moves more than 2.4 units from the centre (i.e. the agent reaches the edge of the display). The episode terminates if the agent reaches 200 timesteps and is solved when the agent receives a reward of 195 over 100 consecutive episodes. The observation states of this environment are fully observable. The agent receives a four-dimensional observation space containing its position, velocity, pole angle, and pole angular velocity.

MountainCar-v0: This second environment is a bit more complex and was first introduced by [13]. It describes an agent (i.e. a car) on a one-dimensional track positioned between two hills (see Fig. 4b). The agent’s engine cannot climb the mountain in a single pass. The only way for the agent to succeed is to learn to build momentum by driving back and forth. Its goal is to reach the flag located at the top of the hill on its right. The agent can either accelerate left, accelerate right, or do nothing. The agent starts at position (0, 0) and receives a reward of 0 if it reaches the flag (i.e. position (x, 0.5)) and $-1$ if the agent’s position is less than (x, 0.5). The episode terminates if the agent reaches its goal or the episode length exceeds 200 timesteps. The environment is solved when the agent receives a reward of $-110$ over 100 consecutive episodes. In addition, the velocity affected by the gravitational pull acting on the agent is not affected by the action taken. For this environment, the observation states are also fully observable. The agent receives a two-dimensional observation space containing both its position and velocity.

LunarLander-v2: this last environment describes a rocket trajectory optimisation problem and is the hardest out of all. An agent (i.e. the rocket) starts at the top of the screen, and its goal is to reach the landing pad positioned at coordinates (0, 0) (see Fig. 4c). In this environment, the landing pad is fixed for all episodes, whereas the landscape and starting position of the agent are sampled randomly from a finite list of available options. The agent can either fire its right orientation engine, left orientation engine, main engine, or do nothing. The environment terminates either when the agent crashes or comes to rest, receiving a reward of $-100$ and $+100$, respectively. In addition, when the agent comes to rest, each leg ground contact gets a reward of $+10$. There are no fuel restrictions on the agent, although if the agent fires the main engine, it receives a reward of $-0.3$ for each timestep. The environment is solved when the agent receives a reward of 200 over 100 consecutive episodes. As with CartPole and MountainCar, LunarLander has fully observable observation states. The agent receives an eight-dimensional observation space containing horizontal and vertical coordinates, horizontal and vertical speed, angle and angular speed, and whether the first and second legs have ground contact.

B Experiment Hyperparameters

Table 2. List of hyperparameters and their values for DQN

Table 3. List of hyperparameters and their values for CVAE