VisuoSpatial Foresight for physical sequential fabric manipulation

Abstract

Robotic fabric manipulation has applications in home robotics, textiles, senior care and surgery. Existing fabric manipulation techniques, however, are designed for specific tasks, making it difficult to generalize across different but related tasks. We build upon the Visual Foresight framework to learn fabric dynamics that can be efficiently reused to accomplish different sequential fabric manipulation tasks with a single goal-conditioned policy. We extend our earlier work on VisuoSpatial Foresight (VSF), which learns visual dynamics on domain randomized RGB images and depth maps simultaneously and completely in simulation. In this earlier work, we evaluated VSF on multi-step fabric smoothing and folding tasks against 5 baseline methods in simulation and on the da Vinci Research Kit surgical robot without any demonstrations at train or test time. A key finding was that depth sensing significantly improves performance: RGBD data yields an $\mathbf{80 \%}$ improvement in fabric folding success rate in simulation over pure RGB data. In this work, we vary 4 components of VSF, including data generation, visual dynamics model, cost function, and optimization procedure. Results suggest that training visual dynamics models using longer, corner-based actions can improve the efficiency of fabric folding by 76% and enable a physical sequential fabric folding task that VSF could not previously perform with 90% reliability. Code, data, videos, and supplementary material are available at https://sites.google.com/view/fabric-vsf/.

Appendices

Appendix

We structure this “Appendix” as follows:

• “Appendix 9” compares and contrasts various fabric simulators.

• “Appendix 10” lists hyperparameters and provides details for training policies.

• “Appendix 11” provides more details on the smoothing experiments.

Fabric simulators

As in the prior paper (Hoque et al. 2020), we use the fabric simulator originally developed in Seita et al. (2020). This simulator possesses an ideal balance between ease of code implementation, speed, and accuracy, and was able to lead to reasonable smoothing policies in prior work. We considered using simulators from ARCSim (Narain et al. 2012), MuJoCo (Todorov et al. 2012), PyBullet (Coumans and Bai 2016–2019), Blender (Community 2018), or NVIDIA FLeX (Lin et al. 2020), but did not use them for several reasons outlined below.

High-fidelity simulators, such as ARCSim, take too long to simulate to get sufficient data for training visual dynamics models. Furthermore, it is difficult to simulate rudimentary grasping behavior in ARCSim because it does not represent fabric as a fixed grid of vertices, which means grasping cannot be simulated by pinning vertices.

Blender includes a new fabric simulator, with substantial improvements after 2017 for more realistic shearing and tensioning. These changes, however, are only supported in Blender 2.8, not Blender 2.79, and we used 2.79 because Blender 2.8 does not allow background processes to run on headless servers, which prevented us from running mass data collection. Additionally, Blender does not allow the dynamic re-grasping of mesh vertices during simulation which makes long horizon cloth manipulation and data collection difficult.

MuJoCo is a widely utilized physics simulator for deep reinforcement learning benchmarks (Todorov et al. 2012). The first MuJoCo version providing full support for fabric manipulation was released in October 2018. Currently, the only work that integrates the fabric simulator with simulated robot grasps is from Wu et al. (2020), which was developed concurrently with the prior work (Hoque et al. 2020). Upon investigating the open-source code, we found that MuJoCo’s fabric simulator did not handle fabric self-collisions better than the simulator from Seita et al. (2020), and hence did not pursue it further.

The PyBullet simulator code from Matas et al. (2018) showed relatively successful fabric simulation, but it was difficult for us to adapt the author’s code to the proposed work, which made significant changes to the off-the-shelf PyBullet code. PyBullet’s fabric simulator was upgraded and tested for more fabric-related tasks in Seita et al. (2021), but still suffers from self-collisions and fabric which tends to get crumpled.

In concurrent work, SoftGym (Lin et al. 2020) benchmarks deep reinforcement learning algorithms on deformable object manipulation tasks, including those with fabrics. SoftGym provides fabric simulation environments utilizing NVIDIA FLeX, which models deformable objects in a particle and position based dynamical system similar to the mass-spring system used in the fabric simulator from Hoque et al. (2020), Seita et al. (2020) and also incorporates self-collision handling. SoftGym is concurrent work, and future work will investigate the feasibility of utilizing Flex. Additionally, we will compare the performance of the model-based policies presented in this work to the model-free policies evaluated in Lin et al. (2020) on similar smoothing and folding tasks.

Details of learning-based methods

We describe implementation and training details of the three learning-based methods tested: imitation learning, model-free reinforcement learning, and model-based VisuoSpatial Foresight. The other baselines tested—random, highest point, and wrinkles—are borrowed unmodified from prior open-source code (Seita et al. 2020).

1.1 Imitation learning baseline: DAgger

This section contains details and results from our prior work (Hoque et al. 2020). DAgger (Ross et al. 2011) is implemented directly from the open source DAgger code in Seita et al. (2020). This was originally based on the open-source OpenAI baselines (Dhariwal et al. 2017) library for parallel environment support to overcome the time bottleneck of fabric simulation.

We ran the corner pulling demonstrator for 2000 trajectories, resulting in 6697 image-action pairs $({\mathbf {o}}_t, {\mathbf {a}}_t')$, where the notation ${\mathbf {a}}_t'$ indicates the action is labeled and comes from the demonstrator. Each trajectory was randomly drawn from one of the three tiers in the simulator with equal probability. We then perform a behavior cloning (Pomerleau 1991) “pre-training” period for 200 epochs over this offline data, which does not require environment interaction.

After behavior cloning, each DAgger iteration rolls out 20 parallel environments for 10 steps each (hence, 200 total new samples) which are labeled by the corner pulling policy, the same policy that created the offline data and uses underlying state information. These are inserted into a replay buffer of image-action samples, where all samples have actions labeled by the demonstrator. The replay buffer size is 50,000, but the original demonstrator data of size 6697 is never removed from it. After environment stepping, we draw 240 minibatches of size 128 each for training and use Kingma and Ba (2015) for optimization. The process repeats with the agent rolling out its updated policy. We run DAgger for 110,000 steps across all environments (hence, 5500 steps per parallel environment) to make the number of actions consumed to be roughly the same as the number of actions used to train the video prediction model. This is significantly more than the 50,000 DAgger training steps in prior work (Seita et al. 2020). Table 7 contains additional hyperparameters.

The actor (i.e., policy) neural network for DAgger uses a design based on Seita et al. (2020) and Matas et al. (2018). The input to the policy are RGBD images of size $(56 \times 56 \times 4)$, where the four channels are formed from stacking an RGB and a single-channel depth image. The policy processes the input through four convolutional layers that have 32 filters with size $3\times 3$, and then uses four fully connected layers with 256 nodes each. The actor network has 0.8 million parameters.

Table 7 DAgger hyperparameters

Fig. 13

Average coverage over 50 simulated test-time episodes at checkpoints (marked “X”) during the behavior cloning and DAgger phases. For each setting of no action truncation and action truncation, we deploy a single DAgger policy deployed on all tiers. Using dashed lines, we annotate the average starting coverage and the corner pulling demonstrator’s average final coverage

The result from the actor policy is a 4D vector representing the action choice ${\mathbf {a}}_t \in {\mathbb {R}}^4$ at each time step t. The last layer is a hyperbolic tangent which makes each of the four components of ${\mathbf {a}}_t$ within $[-1,1]$. During action truncation, we further limit the two components of ${\mathbf {a}}_t$ corresponding to the deltas into $[-0.4, 0.4]$.

A set of graphs representing learning progress for DAgger is shown in Fig. 13, where for each marked snapshot, we roll it out in the environment for 50 episodes and measure final coverage. Results suggest the single DAgger policy, when trained with 110,000 total steps on RGBD images, performs well on all three tiers with performance nearly matching the 95–96% coverage of the demonstrator.

We trained two variants of DAgger, one with and one without the action truncation to $[-0.4, 0.4]$ for the two deltas $\varDelta x$ and $\varDelta y$. The model trained on truncated actions outperforms the alternative setting, and it is also the setting used in VSF-1.0, hence we use it for physical robot experiments. We choose the final snapshot as it has the highest test-time performance, and we use it as the policy for simulated and real benchmarks in the main part of the paper.

1.2 Model-free reinforcement learning baseline: DDPG

This section contains details and results from our prior work (Hoque et al. 2020). To provide a second competitive baseline, we apply model-free reinforcement learning. Specifically, we use a variant of Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al. 2016) with several improvements as proposed in the research literature. Briefly, DDPG is a deep reinforcement learning algorithm which trains parameterized actor and critic models, each of which are normally neural networks. The actor is the policy, and the critic is a value function (Table 8).

Table 8 DDPG hyperparameters

First, as with DAgger, we use demonstrations (Vecerik et al. 2017) to improve the performance of the learned policy. We use the same demonstrator data of 6697 samples from DAgger, except this time each sample is a tuple of $({\mathbf {o}}_t, {\mathbf {a}}_t', r_t, {\mathbf {o}}_{t+1})$, including a scalar reward $r_t$ (to be described) and a successor state ${\mathbf {o}}_{t+1}$. This data is added to the replay buffer and never removed. We use a pre-training phase (of 200 epochs) to initialize the actor and critic. We also apply $L_2$ regularization for both the actor and critic networks. In addition, we use the Q-filter from Nair et al. (2018) which may help the actor learn better actions than the demonstrator provides, perhaps for cases when naive corner pulling might not be ideal. For a fairer comparison, the actor network for DDPG uses the same architecture as the actor for DAgger. The critic has a similar architecture as the actor, with the only change that the action input ${\mathbf {a}}_t$ is inserted and concatenated with the features of the image ${\mathbf {o}}_t$ after the four convolutional layers, and before the fully connected portion. As with the imitation learning baseline, the inputs are RGBD images of size $(56\times 56 \times 4)$. Further hyperparameters are provided in Table 8.

Fig. 14

Average coverage over 50 simulated test-time episodes at checkpoints (marked “X”) during the pre-training DDPG phase, and the DDPG phase with agent exploration. This is presented in a similar manner as in Fig. 13 for DAgger. Results suggest that DDPG has difficulty in training a policy that can achieve high coverage

We design a dense reward to encourage the agent to achieve high coverage. At each time, the agent gets reward based on:

• A small negative living reward of − 0.05

• A small negative reward of − 0.05 for failing to grasp any point on the fabric (i.e., a wasted grasp attempt).

• A delta in coverage based on the change in coverage from the current state and the prior state.

• A $+$ 5 bonus for triggering 92% coverage.

• A − 5 penalty for triggering an out-of-bounds condition, where the fabric significantly exceeds the boundaries of the underlying fabric plane.

We designed the reward function by informal tuning and borrowing ideas from the reward in Andrychowicz (2020), which used a delta in joint angles and a similar bonus for moving a block towards a target, or a penalty for dropping it. Intuitively, an agent may learn to take a slightly counter-productive action which would decrease coverage (and thus the delta reward component is negative), but which may enable an easier subsequent action to trigger a high bonus. This reward design is only suited for smoothing. As with the imitation learning baseline, the model-free DDPG baseline is not designed for non-smoothing tasks.

Figure 14 suggests that the pre-training phase, where the actor and critic are trained on the demonstrator data, helps increase coverage. The DDPG portion of training, however, results in performance collapse to achieving no net coverage. Upon further inspection, this is because the actions collapsed to having no “deltas,” so the robot reduces to picking up but then immediately releasing the fabric. Due to the weak performance of DDPG, we do not benchmark the policy on the physical robot.

1.3 VisuoSpatial Foresight

Table 9 Visual MPC hyperparameters for CEM and CMA-ES

The main technique considered in this paper and our prior work (Hoque et al. 2020) is VisuoSpatial Foresight (VSF), an extension of Visual Foresight (Ebert et al. 2018). It consists of a training phase followed by a planning phase. An overview of VisuoSpatial Foresight is provided in Sect. 4, and practical implementation details are in Sect. 5. For the planning phase described in Sect. 4.2, we tuned the hyperparameters in Table 9. The CEM variance reported is the diagonal covariance used for folding and double folding. We found that for smoothing, a lower CEM variance (0.25, 0.25, 0.04, 0.04) results in better performance, though it may encourage the policy towards taking shorter actions. For CMA-ES, we use the open source Python implementation PyCMA (https://pypi.org/project/cma/), changing only the number of iterations, initial mean, and initial variance from default parameters. CMA-ES and CEM take a similar amount of computation time.

As described in Sect. 4.3, we evaluate with a Pixel L2 and learned Vertex L2 cost function. For the Pixel L2 cost function (Eq. 3), we remove the 7 pixels on each side of the image to get rid of the impact of the dark border, using only the inner $42\times 42$ region of the $56\times 56$ image. For the Vertex L2 cost function, as described in Sect. 4.3.2, we generate a second dataset from the primary dataset collected. Each of the 9,932 episodes in Fabric-CornerBias can contribute up to $10 \atopwithdelims ()2$ image pairs to use in the second dataset, but we sample only 10 of these possible pairs from each episode to keep the dataset size modest (and the same size as the primary dataset). Specifically, we use the following 10 pairs, chosen for their variable gaps in temporal distance: {$({\mathbf {o}}_1, {\mathbf {o}}_2)$, $({\mathbf {o}}_1, {\mathbf {o}}_3)$, $({\mathbf {o}}_1, {\mathbf {o}}_5)$, $({\mathbf {o}}_1, {\mathbf {o}}_9)$, $({\mathbf {o}}_6, {\mathbf {o}}_8)$, $({\mathbf {o}}_6, {\mathbf {o}}_{10})$, $({\mathbf {o}}_6, {\mathbf {o}}_7)$, $({\mathbf {o}}_3, {\mathbf {o}}_4)$, $({\mathbf {o}}_3, {\mathbf {o}}_7)$, $({\mathbf {o}}_3, {\mathbf {o}}_9)$}. During training, we flip the order of half of the data points to encourage the network to ignore the direction of time in its estimation of mesh distance. As mentioned in Sect. 4.3.2, we annotate all data points with the sum of the distances between corresponding points in the ground truth mesh states, i.e.

$$\begin{aligned} \sum _{i=0}^{625} ||p_1^{(i)} - p_2^{(i)} ||_2^2 \end{aligned}$$

where $p_1^{(i)}$ is the (x, y, z) coordinates of the ith point of the mesh shown in the first image and $p_2^{(i)}$ is the (x, y, z) coordinates of the ith point of the mesh in the second image. We divide all labels by the maximum value for more stable training. Finally, for the network architecture, we use the same CNN as the actor in the DAgger baseline as described in Sect. 10.1. However, to accommodate the second image input, we pass both images through the same convolutional layers and concatenate the outputs to a 5184-dimensional vector before applying the fully connected layers. The resulting network has about 1.5 million parameters.

Supplementary smoothing results

1.1 Statistical significance tests

We run the Mann–Whitney U test (Mann and Whitney 1947) on the coverage and number of action results reported in Table 3 for VSF-1.0 against all baselines other than Imitation Learning, to which we wish to perform similarly. See Table 10 for computed p values. We conclude that we can confidently reject the null hypothesis that the values are drawn from the same distribution for all metrics except Tier 2 coverage for Wrinkle and the Tier 1 and Tier 3 number of actions for DDPG ($p < 0.02$). Note that Tier 3 results are most informative, as it is the most difficult tier.

Table 10 Mann–Whitney test p values for coverage and number of actions of VSF-1.0 compared with Random, Highest, Wrinkle and DDPG baselines across all tiers of difficulty for smoothing

1.2 Domain randomization ablation

For Fabric-Random, we run 50 simulated smoothing episodes per tier with a policy trained without domain randomization and compare with the 200 episodes from Table 3. In the episodes without domain randomization, we keep fabric color, camera angle, background plane shading, and brightness constant at training and testing time. In the episodes with domain randomization, we randomize these parameters in the training data and test in the same setting as the experiments without domain randomization, which can be interpreted as a random initialization of the domain randomized parameters. In particular, we vary the following:

• Fabric color RGB values uniformly between (0, 0, 128) and (115, 179, 255), centered around blue.

• Background plane color RGB values uniformly between (102, 102, 102) and (153, 153, 153).

• RGB gamma correction with gamma uniformly between 0.7 and 1.3.

• A fixed amount to subtract from the depth image between 40 and 50 to simulate changing the height of the depth camera.

• Camera position (x, y, z) as $(0.5+\delta _1, 0.5+\delta _2, 1.45+\delta _3)$ meters, where each $\delta _i$ is sampled from ${\mathcal {N}}(0, 0.04)$.

• Camera rotation with Euler angles sampled from ${\mathcal {N}}(0, 90^{\circ })$.

• Random noise at each pixel uniformly between − 15 and 15.

From the results in Table 11, we find that final coverage values are similar whether or not we use domain randomization on training data, suggesting our domain randomization techniques do not have an adverse effect on performance in simulation.

To analyze robustness of the policy to variation in the randomized parameters, we also evaluate the former two policies (trained with and without domain randomization) with randomization in the test environment on Tier 3 starting states. Specifically, we change the color of the fabric in fixed increments from its non-randomized setting (RGB (25, 89, 217)) until performance starts to deteriorate. In Table 12, we observe that the domain randomized policy maintains high coverage within the training range (RGB (0, 0, 128) to (115, 179, 255)) while the policy without domain randomization suffers as soon as the fabric color is slightly altered.

Table 11 Coverage and number of actions for simulated smoothing episodes from Fabric-Random, with and without domain randomization on training data, where the domain randomized results are from Table 3

Table 12 Coverage and number of actions for Tier 3 simulated smoothing episodes with and without domain randomization on Fabric-Random training data, where we vary fabric color in fixed increments. (26, 89, 217) is the default blue color and (128, 191, 115) is slightly outside the domain randomization range