Connecting the CoppeliaSim robotics simulator to virtual reality

Abstract The CoppeliaSim VR Toolbox provides a set of tools to experience CoppeliaSim robot simulation software in Virtual Reality and to return user interactions. Its primary focus is to create a platform that enables the fast prototyping and verification of robotic systems. Moreover, the generality of the toolbox ensures that it can be valuable in other contexts like robotics education, human–robot interaction or reinforcement learning. The software is designed to have a low entry threshold for moderately complex use cases, but can be extended to perform very complex visualizations for more experienced users.


Motivation and significance
CoppeliaSim is a robot simulation environment used for the prototyping, development and verification of robot systems and algorithms.A robotic system can be a complex mechatronic system that is nontrivial to construct and use.The offline programming of the motion of a robot can furthermore be cumbersome, time-consuming and requires a high level of expertise [1].In this work we developed a virtual reality interface for CoppeliaSim.While the importance of virtual reality in virtual prototyping and validation has long been understood [2][3][4], there are only very few software tools that meet the requirements of the robotics community [5].
The developed CoppeliaSim VR Toolbox aims to be more accessible to members of the robotics community than other software solutions.During the development we have focused on the following priorities: lowering the entry threshold for new  The software's efficacy has already been proven in the human guided design of camera networks [6] and robot path planning for optical inspections [7].With this software, it was easy and quick to create new design scenarios that could be used to test human performance in a wider range of problems.The VR Toolbox proved its extensibility by allowing the implementation of advanced volume rendering, which was used to augment user performance. 1 The VR Toolbox uses a commonly used robot simulator, namely CoppeliaSim.This simulator is used as usual to model the experimental scene, and to define the scene control logic.The VR Toolbox contains an application that visualizes this scene in openVR compatible devices, and returns user interactions.These can than be used in the robot simulator to modify the scene logic.The toolbox also contains a standard set of interactors in which a common set of interactions are preprogrammed.These interactors are used directly, or as building blocks for more advanced interactors or can serve as an example in the development of specific custom interactions (see Fig. 1).
Our VR Toolbox links CoppeliaSim [8], to the Visualization Toolkit (VTK) [9,10].VTK provides an extensive library of stateof-the-art scientific visualizations and also links to openVR compatible virtual reality devices.A different approach that combines the Robot Operating System (ROS) with Unity is also available [5].This solution, however, is less accessible to the robotics community since Unity is more challenging to set up, and is less intuitive to use for roboticists than CoppeliaSim [11].Furthermore, the separation between scene logic in ROS and scene modeling in Unity increases the time to develop specific cases.This separation between scene logic and modeling environment is deliberately avoided in our software solution.A disadvantage that is not to be underestimated is that this extra complexity makes user studies more challenging to set up.

Software description and functionalities
The CoppeliaSim VR toolbox builds around a core module that converts CoppeliaSim scene data to VTK actors that can be 1 A video that shows volume rendering with the VR Interface is available online: https://youtu.be/Dsh8oyN4sD0.visualized in VTK.This module is used in two separate included applications.The first application renders any CoppeliaSim scene in openVR compatible hardware, and returns user action to the simulator.We will refer to this application as the VR-Interface.The second application renders omnidirectional stereo images from CoppeliaSim scenes.These can be used to create compelling visualizations of research that was carried out with the VR-Interface. 2We explicitly created a separate application for the VR rendering, instead of integrating these into the simulator, to guarantee a consistent and fast framerate of the VR visualization, even when the simulation speed of the simulator is inconsistent.This is important in user-studies for the comfort of subjects, and it is necessary in order to not bias results.This set of tools (i.e.module and applications) is completed with CoppeliaSim models which program standard interactions (we call these interactors).These interactors are entirely programmed in CoppeliaSim, in its native format (i.e.LUA) and also act as examples.

Software architecture
The most important component of the toolbox is the translation module (schematic version of the software architecture in Fig. 2).This module transforms CoppeliaSim data in VTK actors.To do this, the module interfaces with CoppeliaSim by calling script functions.These script files are located in a CoppeliaSim model that must be added to the scene by the user.After object loading, between renders, all information (e.g.positions) of the actors are updated, based on the scene logic in CoppeliaSim.This translation module is used to build a VTK application (in c++) which can be augmented with custom visualizations.By default, the VR Toolbox features two such applications, one which performs visualizations in VR, and one that renders omnidirectional stereo images.However, users can freely add custom visualizations based on these examples.
The easiest, and most common way of working with the VR Toolbox is by using or programming interactors.These interactors are CoppeliaSim child scripts that modify the scene logic based on interactions from the VR device.The programming of interactors is completely analogous to programming traditional logic in CoppeliaSim, or even in robot simulators in general.The graphical user interface of CoppeliaSim is designed to quickly and efficiently develop and debug such scripts.

Sample code snippets analysis
in Fig. 3 an interactor is programmed in a child script in CoppeliaSim (LUA).This script allows a user to control the robot end-effector with a virtual reality controller.The state of the trigger button of the right controller is stored in the integer signal R_trigger_press.If this button is pressed, the robot target (target of the tip-target combination of an IK object) becomes a child of the controller.This parent-child link will ensure that the relative transformation between the right controller and the robot target remains fixed.The IK calculation module in CoppeliaSim will ensure that the robot tip will match the robot target.As a result, the target will follow the movements of the controller.The parent-child link is broken if the user releases the trigger button of the right controller.This code snippet shows how easy it is to integrate virtual reality interactions in CoppeliaSim scene logic.(see Fig. 4).

Robotic inspection planning
We used the VR-Interface to investigate if inexperienced users are capable of generating high quality robotic inspection paths [7].Users could move the end-effector of the robot by moving the VR controller.To help them in the process of recording inspection paths, we visualized the inspection quality on the geometry that needed to be inspected.This was achieved by programming custom vertex colors in VTK.A 360VR video was generated with the omnidirectional render tool that is also available in the VR toolbox to explain the experimental procedure. 3This path can naturally be used for inspections on a real robot. 4

Camera network design
Recently, we used the developed VR Toolbox to investigate how well users could design camera networks compared to automated algorithms5 [6].To do this we implemented an interactive volume visualization in VTK.This interactive volume showed the users which parts of the scene was invisible to a camera system, as an interactive red cloud.The users could move the cameras of the camera system by a dragging motion of the VR controllers.This is shown in Fig. 5. Upon the movement of the cameras, the interactive volume changed based on the new camera positions.With this visualization, camera network design, corresponds to placing cameras, such that there are no red areas in the environment.With this software we demonstrated that the gap between user performance, and the performance of automated algorithms is limited.The flexibility of the software allowed us to investigate this in a wide range of different (dynamic) environments.

Impact and conclusion
The CoppeliaSim VR Toolbox provides a set of tools that add VR support to a powerful robot system prototyping environment.CoppeliaSim itself is known for its flexibility and ability to prototype every robot system [8].Interactors can easily be programmed in the same way and with the same flexibility.The complexity of extending the CoppeliaSim VR Toolbox is layered.The most common extension, making new interactors, can be performed with standard tools, which have a low entry threshold.However, more experienced users can build their own applications, in which they can unleash all the visualizations that VTK offers, to aid users in performing their tasks.This extensibility has already been shown with advanced volume rendering [6].
This software solution is significantly easier and faster to set up than other software solutions.This is important since it is common to perform pilot studies in human-computer interaction to guide the design process, and to eliminate experimental errors [12].The complexity of setting up and installing all the required software is minimized which reduces the effort to set up user studies.Furthermore, all the required data that makes up a test scene, is stored in a single file.This further reduces the effort to set up user studies.

Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Fig. 1 .
Fig. 1.The user sees the V-REP scene (right) in VR (top-left).The user can interact with a robot to teach paths.users, simplifying the software setup, and minimizing the time to develop and test user interactions.The software's efficacy has already been proven in the human guided design of camera networks[6] and robot path planning for optical inspections[7].With this software, it was easy and quick to create new design scenarios that could be used to test human performance in a wider range of problems.The VR Toolbox proved its extensibility by allowing the implementation of advanced volume rendering, which was used to augment user performance.1The VR Toolbox uses a commonly used robot simulator, namely CoppeliaSim.This simulator is used as usual to model the experimental scene, and to define the scene control logic.The VR Toolbox contains an application that visualizes this scene in openVR compatible devices, and returns user interactions.These can than be used in the robot simulator to modify the scene logic.The toolbox also contains a standard set of interactors in which a common set of interactions are preprogrammed.These interactors are used directly, or as building blocks for more advanced interactors or can serve as an example in the development of specific custom interactions (see Fig.1).Our VR Toolbox links CoppeliaSim[8], to the Visualization Toolkit (VTK)[9,10].VTK provides an extensive library of stateof-the-art scientific visualizations and also links to openVR compatible virtual reality devices.A different approach that combines the Robot Operating System (ROS) with Unity is also available[5].This solution, however, is less accessible to the robotics community since Unity is more challenging to set up, and is less intuitive to use for roboticists than CoppeliaSim[11].Furthermore, the separation between scene logic in ROS and scene modeling in Unity increases the time to develop specific cases.This separation between scene logic and modeling environment is deliberately avoided in our software solution.A disadvantage that is not to be underestimated is that this extra complexity makes user studies more challenging to set up.

Fig. 2 .
Fig. 2. A high level schematic overview of the software architecture.

Fig. 3 .
Fig. 3. Sample code that lets the user control the end-effector of a robot with a virtual reality controller.

Fig. 4 .
Fig.4.The user can control the end-effector of a robot by moving a VR controller, and gets feedback on the inspection quality of an attached measurement system.This visualization allows the user to generate high quality inspection paths[7].

Fig. 5 .
Fig. 5.The user can move virtual cameras, and see which areas of the scene are visible, here shown as a green cloud.This visualization allows users to effectively design camera networks.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)