Recognition, prediction, and planning for assisted teleoperation of freeform tasks

Abstract

The approach of inferring user’s intended task and optimizing low-level robot motions has promise for making robot teleoperation interfaces more intuitive and responsive. But most existing methods assume a finite set of candidate tasks, which limits a robot’s functionality. This paper proposes the notion of freeform tasks that encode an infinite number of possible goals (e.g., desired target positions) within a finite set of types (e.g., reach, orient, pick up). It also presents two technical contributions to help make freeform UIs possible. First, an intent predictor estimates the user’s desired task, and accepts freeform tasks that include both discrete types and continuous parameters. Second, a cooperative motion planner continuously updates the robot’s trajectories to achieve the inferred tasks by repeatedly solving optimal control problems. The planner is designed to respond interactively to changes in the indicated task, avoid collisions in cluttered environments, handle time-varying objective functions, and achieve high-quality motions using a hybrid of numerical and sampling-based techniques. The system is applied to the problem of controlling a 6D robot manipulator using 2D mouse input in the context of two tasks: static target reaching and dynamic trajectory tracking. Simulations suggest that it enables the robot to reach intended targets faster and to track intended trajectories more closely than comparable techniques.

Appendix:

1.1 Gaussian Conditioning

If $X$ and $Y$ are jointly normally distributed as follows:

$$\begin{aligned} \left[ \begin{array}{l} X \\ Y \end{array}\right] \sim \mathcal{N }\left( \left[ \begin{array}{l}\mu _x \\ \mu _y\end{array}\right] , \left[ \begin{array}{ll}\Sigma _{x} &{} \Sigma _{xy} \\ \Sigma _{yx} &{} \Sigma _{y} \end{array}\right] \right) , \end{aligned}$$

(21)

then the conditional distribution over $Y$ given the value of $X$ is another Gaussian distribution $\mathcal{N }(\mu _{y|x},\Sigma _{y|x})$ with

$$\begin{aligned} \begin{aligned} \mu _{y|x}&= \mu _y+\Sigma _{yx}\Sigma _{x}^{-1}(x-\mu _x) \\ \Sigma _{y|x}&= \Sigma _{y}-\Sigma _{yx}\Sigma _{x}^{-1}\Sigma _{xy}. \end{aligned} \end{aligned}$$

(22)

This form is in fact equivalent to the ordinary least-squares fit $y = A x + b + \epsilon$ with $A = \Sigma _{xy}\Sigma _{x}^{-1},\, b = \mu _y-\Sigma _{yx}\Sigma _{x}^{-1}\mu _x$, and where $\epsilon \sim \mathcal{N }(0,\Sigma _{y|x})$ is an error term.

1.2 Kalman update

Given a linear observation model $o = A x + b + \epsilon$ with $\epsilon \sim \mathcal{N }(0,Q)$, and prior $x \sim \mathcal{N }(\mu ,\Sigma )$, the posterior $P(x|o)$ is a Gaussian with mean and covariance

$$\begin{aligned} \begin{aligned} \mu _{x|o}&= \mu -\Sigma A^T C^{-1}(o-A\mu ) \\ \Sigma _{x|o}&= \Sigma -\Sigma A^T C^{-1} A \Sigma \end{aligned} \end{aligned}$$

(23)

where $C = A \Sigma A^T + Q$.