TOWARDS IMMERSIVE DESIGNING OF PRODUCTION PROCESSES USING VIRTUAL REALITY TECHNIQUES

The article provides a novel approach to the implementation of virtual reality within planning and design of manual processes and systems. The use of hardware and software required to perform different production  especially assembly  tasks in a virtual environment, using CAD parts as interactive elements, is presented. Considering the CAD parts, the format conversion problem is comprehensively described and solved using format conversion software to overcome the present poor data connectivity between the CAD system and virtual reality hardware and software. Two examples of work processes have been made in a virtual environment: peg-in-hole and wall socket assembly. In the latter case, the traditional planning approach of manual assembly tasks using predetermined motion time system MTM-2 has been compared with a modern approach in which the assembly task is fully performed within a virtual environment. The comparison comprises a discussion on the assembly task execution times. In addition, general and specific advantages and disadvantages that arise in the immersive designing of production processes using virtual reality are presented, as well as reflections on teamwork and collaborative man-machine work. Finally, novel technologies are proposed to overcome the main problems that occur when implementing virtual reality, such as time-consuming scene defining or tedious CAD software data conversion.


INTRODUCTION
Virtual reality (VR) is a computer-generated interface that imitates reality and allows user interaction in that imaginative world by specific interaction devices such as head mounted displays, haptic gloves, motion trackers and sensors. A key element of a created interface is interaction. A special impact of VR can be foreseen regarding further exploitation of human mental and motor sensing in work processes [1]. Thanks to the interaction devices, it is possible to interact with different objects within VR which help in carrying out different types of work or production processes. In the past, VR devices were rare, and researchers were forced to make special type of device that could deal with VR content since commercial VR devices were not available as they are today (e.g. Oculus and HTC Vive). Regardless of these limiting circumstances, a lot of quality research is done in the field of virtual assembly, virtual prototyping and virtual manufacturing. De Sa and Zachmann [2] investigated the steps needed to apply VR for virtual prototyping to verify assembly and maintenance processes in automotive and aerospace industries. Such processes are also applicable for shipbuilding or any other industry and even services. Further, manufacturing can also be observed and improved using modern VR techniques: Iwata et al. [3] stated that "virtual manufacturing is defined as a computer system which is capable of generating information about the structure, status, and behaviour of a manufacturing system as can be observed in a real manufacturing environment." So, "VR holds great potential in manufacturing applications to solve problems before being employed in practical manufacturing, thereby preventing costly mistakes." [4]. Boothroyd and Dewhurst [5] revealed a long ago that "the assembly process often drives the majority of the cost of a product", what makes research in this area constantly important. Thus, VR is applicable for design and analysis of human work and activities in manufacturing and production, but generally and moreover, for all processes, industries and services which include some kind of human labour. In the case of very complex products such as ships, there is a big database of 3D models with a huge amount of data. VR can help to manage and visualise model lot easier using supporting VR prototyping and is therefore widely used in early design phases (concept design), simulations and training methods [6] that can enhance the overall experience to overcome the negativity of physical models. And finally, immersification of a process by VR, eventually and gradually leads to a fully automatic process.
When it comes to process design and human resource planning, various empirical methods, as well as the concurrent engineering (CE), yielded results that showed the importance of training in achieving higher performance [7]. Virtual training can be very easily accomplished in a virtual environment that is by itself safe and without a possibility of hurting workers. As an example, in the field of the shipbuilding industry, VR is applied in the study of the escape routes or army crew training in the vessels like submarines where problems with lack of evacuation space exist: "Simulation is the other important activity where jobs like painting and welding are known to be difficult and one of the most important processes in ship construction. These processes are cost-effective and also directly related to the quality of the ship under construction. So, it is particularly important to train workers so that they can paint evenly on large surfaces with a uniform thickness and [that] they can weld properly too." [8]. Further, Smith [9] believes that "concurrent engineering can be seen as a summary of best practice in product development." Helander et al. [10] concluded that "the trend toward global manufacturing and geographically distributed work appear to be shaped by the consumer demand for more individualized products and a more direct connection between the customer, designer and manufacturer." When constructing the large-scale products such as a ship or any other vessel, where construction offices are often not located at the same place, the connection of all types of people involved in developing process can be very cost effective. Using VR techniques, the connection between the customer, designer and manufacturer is easier to achieve, even without physical presence. The advantage of 3D immersive visualisation of the ship in an immersive environment is the "ability to have geographically distributed participants sharing space with each other and the objects under discussion. This allows the different participants to point at specific objects in the scene or set the parameters of the simulation to specific values to clarify the information. It gives the users a common context for their discussions, especially in international collaborations." [8].
Except in the field of virtual training and virtual design, VR techniques can further significantly enhance the existing processes through their virtual process planning. In addition to manufacturing and production, the potential of virtual process planning is recognised in medical [11] and dental [12] services.
The unavoidable term implied by VR is immersion. The concept of immersion comprises generating of a 3D virtual reality image that appears to surround the user, where the image is generated through a computer screen or system. The sense of immersion can vary from nonimmersive VR to fully immersive VR. With regard to the levels, VR has recently been divided into a non-immersive, semi-immersive and fully immersive virtual reality [13]. However, due to the newly presence of commercial versions of VR systems, the situation changes and the level of semi-immersive virtual reality is practically lost. Therefore, the new division of VR could only be done in two categories: non-immersive and fully immersive virtual reality. Table 1 shows the differences between these two levels. VR has an increasing importance in product design and process and system planning. It is about the use of an immersive  powerful interactive tool, which includes visualisation and haptics for simulation  consideration and evaluation of previously faithfully crafted constructs in an equally faithful environment. VR thus enables better design results; so, on the one hand, it is a natural technological development of design activities, while on the other hand it involves the evaluation and verification of traditional design approaches and methods. Furthermore, even the entire plant can be simulated in a virtual environment. Factory simulation for immersive investigation has been presented by [14], and although no direct manipulation of objects is possible, VR produced quality results and earlier identification of problems than when using conventional simulations. In industry, the competition is challenging and new methods and enhancements need to be developed fast. To keep the track with the competition, novel technologies must be implemented as soon as possible, immediately when their efficiency is approved. From all the aforementioned, it is evident that the VR provides a unique approach to problem solving, especially in the area of designing of product and work processes. Moreover, software packages have been developed for virtual applications in manufacturing (e.g. CATIA/DELMIA, Siemens NX). For example, according to [4]: "DELMIA package provides authoring applications that can be used to develop and create virtual manufacturing environment to address process planning, cost estimation, factory layout, ergonomics, robotics, machining, inspection, factory simulation, and production management." Likewise, Siemens NX software is an integrated product design (CAD), engineering (CAE) and manufacturing (CAM) solution that helps deliver better products faster and more efficiently. In the field of shipbuilding, FVIEWER VR software developed by SENER and fully integrated into FORAN environment gives the user a possibility to review the ship 3D model, detect collisions, walk through and interrogate with the model, measure clearances, make annotations and many other features [8].
However, apart from the mentioned above, VR is generally still not satisfactory represented in various CAx software packages: despite the advances regarding the nearer past, when the user of VR systems "in spite of using some immersive visualisation tools, e.g. head-mounted display (HMD) with tracking technology and VR gloves, does not perceive an artificial computer world as real, because while interacting with any VE object it is not possible to feel the interaction with gravity (and other) forces and inertia of mass-objects" [15].
To change that, bimanual haptic interaction backed by powerful hardware and software tools should be used, as proposed: "Considering the importance of bimanual interaction in real life, not using both of our hands could lead to a loss of efficiency or immersion for a certain number of tasks. ... Haptics can greatly enhance the immersion of a user into a simulated or remote environment by stimulating the tactile and proprioceptive senses" [16]. Not including haptic devices will result in limiting environment due to the lack of force feedback which is extremely important in the most assembly operations.
This paper will focus on creating a virtual work environment as realistic as possible, without haptic devices, and it is based on the research conducted in [17]: Manual assembly task analysis may be performed using Oculus Rift DK2 HMD and Leap Motion Controller, with bimanual interaction with objects in the environment. Designing of manual production or work processes, especially in large-scale production like shipbuilding where thousands of parts should be assembled, significantly contribute to the overall costs of a product and therefore should be carefully developed, measured and evaluated. For manual assembly task analysis, tools like predetermined motion time system MTM (Methods-Time Measurement), are nowadays traditionally established in the industry, enabling description, design, evaluation and improvement of human manual work. MTM is based on subdividing of a worker's manual operations into basic motions such as GET, PUT, REGRASP etc. that have corresponding  predetermined and standard execution times. Thus, while many supporting tools are available for production planning (e.g. Tecnomatix Plant Simulation or visTABLE), there is no such support for planning manual work using VR. The results of VR assembly planning and classic assembly planning using the system MTM-2 (one of the versions of the MTM system) will be compared on the basis of wall socket assembly task [18]. Recently, new methods to improve the traditional approach to MTM system with support for real walking was implemented [19].

CONNECTING CAD AND VR
VR techniques can lead to enhancement and improvement of existing processes and services, especially in environments where interaction between human and system elements is important. As the techniques of VR have an increasing significance, it is most likely that the new principles and advantages that the virtual environment can provide will result with even broader use of VR techniques. Apart from the application in the areas of assembly of components, quality control, virtual plant simulations and virtual training tasks, many other studies are conducted on this subject. The interaction elements in created virtual environment will be parts generated in CAD software. The realisation of connecting CAD and VR system is relatively new because commercial VR systems came to the market only a few months ago. On the other hand, CAD systems like CATIA were developed decades ago and even such newer systems are still relatively old in comparison with novice VR technology. Due to all of the above, it can be expected that interaction between CAD and VR will result in numerous problems such as incompatibility and lack of mutual support. However, format conversion issue existed in the near past, when it was more complex to show CAD file on VR platform using VRML. Although there are certain improvements, the problem remains the same. As de Sa and Zachmann [2] concluded, "to avoid that designers have to become familiar with different software tools, the number of interfaces must be kept low. To achieve that the two worlds need to be integrated, at least to a higher degree than present today." Also, they emphasised that "VR will not become a widespread tool in manufacturing industries before it is seamlessly and completely integrated into the existing CAx and IT infrastructure." In order to connect CAD geometry and VR platforms such as low-budget Google Cardboard or Oculus Rift, it is necessary to adapt the output geometry of the CAD part to a particular type of software that allows creating a virtual environment for selected platforms.
Considering intuitive and user-friendly interface, the development software that will be used in this work is Unity3D. Apart from the Unity3D, other 3D game engine software such as CryEngine or Unreal could have been used to create a virtual environment in which the following tasks would be performed. However, using other engine would result in different procedures and setup parameters to define the environment, so the most convenient software to use in this project was shown to be Unity3D.
Unity3D is the game engine and has support for more than 20 different platforms. The software has direct support for recognising formats .FBX, .dae, .3DS, .dxf and .obj. Since conventional CAD software does not provide the ability to save output geometry in any of the before mentioned formats, a format converter will be used to achieve the desired compatibility between CAD parts and VR systems. The most convenient software for format conversion is Autodesk 3DS Max, with the ability to import various file formats, such as .CATPart & .CATProduct (CATIA), SLDPRT & SLDASM (SolidWorks), PRT & ASM (ProE) but also for universal formats like STEP or IGES. The ability to import nearly all types of formats from the most widely used CAD software allows to save and export desired geometry in a format supported by Unity3D (Fig. 1). After format conversion, to display the crankshaft on the Google Cardboard platform (Fig. 2), there are several additional things that should be done, in terms of software installation, settings and programming.
Incorporating CAD geometry into virtual environment provides 3D immersive experience, easier visualisation and creation of an impression of reality, but also the ability to scale objects, where objects of bulky dimensions can be reduced, resulting in easier imagery and manipulation of parts. However, without including advanced interaction devices into a virtual environment, the user has no ability to move or interact with objects because they are static. For the objects to be dynamic, intensive programming work should be done for each individual object (if there are different movement rules between objects), resulting in time consumption and limiting environment. Also, it is important to use scripts to define behaviour, the functionality of objects and more complicated interactions and relationships between them [15]. Due to the disadvantages of displaying content on the Google Cardboard platform, the Oculus DK2 platform will be used. Oculus provides a number of advantages primarily in terms of the quality and reality of virtual visualisation compared to the mobile platform. Also, Oculus uses computer resources to display content directly, unlike Cardboard, where smartphone resources are used, which significantly deviate from the computer component's performance.

PROCESS DESIGN USING VR
In a virtual environment, peg-in-hole and assembly of wall socket tasks will be solved using Oculus Rift and Leap Motion Controller (Fig. 3). In the example of the assembly of the wall socket, the task execution time in the virtual environment will be measured and then compared with the results of a traditional planning approach which uses predetermined motion time system MTM-2.

PEG-IN-HOLE
Peg-in-hole still remains as one of the major problems in part assembly processes, regardless of whether the parts are assembled manually by human hand or automatically by a robot. Numerous studies have been carried out in this area (e.g. [20]), but solving peg-in-hole task in virtual environment actualises this problem yet again.
In order to successfully complete the assembly task, several actions are required: 1. Locate the peg and hole in virtual environment (Fig. 4); 2. Grab the peg; 3. Place the peg above the hole; 4. Align the peg and the hole axes; 5. Insert the peg in the hole. Carrying out tasks in a virtual environment cannot entirely replicate the real process of assembly task (where due to tolerancing and even surface damage, large extraction force or the complete inability to pull out the peg out of the hole may occur), but it may sufficiently well articulate the action of the difficult insertion of the peg into the hole, primarily due to the physics that is an integral part of Unity3D software. In the first experiment, dimensions of the peg were deliberately greater than the dimensions of the hole, so it was impossible to insert the peg in the hole even after multiple attempts of insertion  Fig. 5a). In the second case, dimensions of peg were reduced so that a peg could be placed into the hole  Fig. 5b). There was a slight clearance between components, that made it possible to solve assembly task. a) b) Figure 5. a) Impossible insertion, b) clearance between the peg and the hole.

WALL SOCKET ASSEMBLY
Since the use of the wall socket (Fig. 6) is inevitable in every facility where single-phase electric power is used, the wall socket is, therefore, one of the basic electrical installation products with a demand for large production volumes. The virtual task of wall socket assembly will allow a comparison of the time taken to perform assembly actions in the virtual environment, with the time obtained through traditional methods of designing manual assembly task procedures using predetermined motion time system, MTM-2. A wall socket assembly task will be performed for selected components and by appropriate work elements. The work element is the smallest rational part of the work that can be independently run and defines such a state of a partially mounted assembly that it can be moved to another location without unwanted disassembly. The work element is divided into movements which can be described and measured (in VR or by MTM-2). Components of wall socket assembly [18].
The movements of the left and right hand will be dissected on the basic movements, whereby each basic movement will be assigned the standard time towards the type of motion and the condition under which it is performed using normative values of the MTM-2 system. After work analysis and assembly plan have been made, the time taken using traditional process planning using MTM-2 system amounts to 400 TMUs, corresponding to a duration of 14,40 seconds. Subsequently, it is necessary to define the virtual work environment (Fig. 7) in which the assembly task will be executed. It is important that the components for assembly should be properly positioned and oriented, as specified in the assembly plan developed by MTM-2, so it is possible to expect a faithful replication of movements (defined also by MTM-2). Assembly of individual parts will be performed using simple functions which turn on or turn off the graphical representation of individual components at the moment when a collision is detected between parts that need to be assembled.
When the environment is fully defined with so-called colliders, scripts and other required software components, wall socket assembly task can be performed (Fig. 8). The execution time of wall socket assembly will be measured with a timer inside the virtual environment. The execution time of wall socket assembly is shown in Table 2. The total duration of the wall socket assembly process in the virtual environment takes 11,70 seconds. For comparison, assembly time of identical process with traditional planning method using MTM-2 technique was estimated at 400 TMUs, which equals a period of 14,40 seconds. Therefore, assembly in a virtual environment has been done for 2,70 seconds less, which means that the process is done roughly 19 % faster than the time scheduled to perform assembly operations through predetermined motion time system.
A shorter time to complete the assembly task in virtual environment may in some way be expected due to the fact that the assembly process, following work analysis, could not be fully replicated due to the lack of virtual environment in terms of the difficult execution of certain actions such as screwing and tightening the screws due to the lack of force feedback. If haptic devices were used, it is expected that the assembly process could be fully replicated, so it may be concluded that virtual assembly and assembly in reality are coincident. Such simple conclusion is of extreme importance for manufacturing and production.

ADVANTAGES AND DISADVANTAGES OF USING VR
Performing the assembly tasks in an immersive environment does not require too long learning, and creates an intuitive environment that can serve to gain experience and simulate work to make it easier to perform in reality. Given the fact that the wall socket is not an assembly that consists of complex geometry, special materials or particularly difficult assembly task for assemblers, the advantage of this approach may not be fully apparent. Virtual training would result in considerably higher savings in complex assembly products, where more advanced assembly operations are required and where potential damage may be caused to highly valuable components if the assembler does not possess certain foreknowledge about the assembly components. Further, in the field of large products (plane and ships), there are great advantages in terms of VR presentation. In a communication between potential owner/investor and manufacturer, VR presentation makes realistic model and is much cheaper than making a physical prototype (1:1), which even may not be possible. VR presents a great advantage in comparison with conventional presentations used in past. Using VR techniques owners/investors can monitor the design and production progress of their order and be important collaborate in early and critical phases where any changes are least expensive.
The advantages and disadvantages of immersive designing of production processes using VR are listed as follows.
Advantages: -the virtual environment creates an intuitive space where users are easily trained, -the user in a very short time, within a matter of minutes, can learn how to use the virtual reality device, such as Oculus Rift headset or the Leap Motion Controller, -there are great savings in design due to easy definition of environment and work tasks, -excellent visualisation of geometry from any point of view, -in addition to realistic visualisation, the great advantage of this system is the ability to interact with all types of parts regardless of their size, -getting acquainted with the product before making a physical prototype, -once the parts are saved in the appropriate format, one can quickly define the scene for a different geometry, -it is possible to perform a variety of tasks due to the flexibility of the system, -assembly can be done at different locations (e.g. at home), -easy virtual training for different activities, -the system is suitable for the implementation of ergonomic research (biometrics), -a virtual environment is a safe environment without the risk of injury to human operators and damage to work parts and tools.
Disadvantages: -the user may feel slight nausea and discomfort when using the headset device, -for each particular task, scene should be newly set, -required conversion of CAD geometry formats into a format suitable for VR development platform, -limitation of user movements due to geometrical features of the equipment, -misconfiguring of the user's hands when there is a physical barrier between hand and the Leap Motion Controller sensors, or when device is used in a too bright environment.

TEAMWORK AND COLLABORATIVE MAN-MACHINE WORK USING VR
The VR also presents new views on team work for multiple users involved in a process and collaborative man-machine work, typically simultaneous work of human and robot. Teamwork involving multiple Leap Motion Controllers is a good idea to overcome the limitations that arise when using only the single one. Some restrictions that occur when using Leap Motion Controller are: erroneous detection of user's hands if multiple hands in scene or when obstacles, such as wire (e.g. from a VR headset) exist between the camera, overlapping of user's fingers (e.g. if one hand is below the other, controller will not be able to perceive the lower hand and it will not be detected). The problem that persists when using Leap Motion Controller is well known and is related to the most motion capture systems that use a camera device to capture the user's movements. To overcome mentioned problems, the idea of using multiple Leap Motion Controllers is conceived. That would allow a larger Field-Of-View (FOV) of the device, but also the teamwork of multiple users and their common task execution in the virtual environment. However, using multiple Leap Motion Controllers on a single computer within Unity3D software is not possible at this time without modifications that require the creation of a virtual machine.
Due to built-in physics, Unity3D can also be used in the field of robotics. Initially, the CAD model of robot needs to be imported into Unity3D. Then, it is necessary to define the constraints and relationships between the individual joints of the robot via the embedded joint options. Once the joints are brought into the mutual relationship, it is necessary to approach solving the kinematic problem (e.g. using C# or similar language). When everything is set, the robotic model in the Unity3D can communicate with the real robot via the TCP/IP protocol. The aforementioned principle would be used for direct motion-oriented programming of the robot, where the robot could perform human-induced motion replication from Leap Motion Controller. A particular advantage of human-robot interaction would be visible in some actions of certain production tasks where it is necessary to combine human work (which is sometimes more cost-effective than robots) and robot work. In such situations, the robot would perform tasks where it achieves better results than humans, while humans would execute the tasks that are unprofitable to use robots (e.g. due to complex and changing work environment and high costs). Such work would result in greater flexibility and productivity.

CONCLUSIONS AND FURTHER WORK
Virtual reality has an increasing importance and application in the diverse fields of human activity. The benefits of VR are mostly visible currently in product visualisation and marketing activities, but even larger potential lays in a treatment of VR as a specific tool in integrated and concurrent product, process and system design and planning, including ergonomics and teamwork man-machine research. Also, great potential lies in large and highly competitive industries such as automotive, aerospace and shipbuilding industry where the competition is globally distributed and extremely advanced. The immersive environment provides the ability to join geographically distributed participants in the same discussion within the same environment which is helpful when clarifying essential pieces of information about product characteristics.
The widening availability of equipment and recent commercialisation of VR devices resulted in the easier simulation of processes in production plants such as virtual assembly, without too large investment, at least in the first stage of the research and implementation. Solving tasks in a virtual environment can serve to compare (and re-evaluate) traditional approaches to designing production processes with a modern approach, using VR techniques. Here in the paper, the results of VR assembly planning and classic assembly planning using MTM were compared on wall socket assembly task. Assembly task in a virtual environment was performed faster than the execution time predicted by the MTM-2 system. The difference in time is the consequence of the limitations of the available VR equipment (inability to accomplish all the assignments from work analysis and assembly plan) where the use of the haptic devices would most likely result in overlap of the assembly time in the virtual environment and projected time using the MTM-2 system. This would mean that performing tasks in the virtual environment does not cause additional delays. Reported approach and work done by the authors contribute by the improvements in better connection between CAx and VR software, as well as in immersification of planning of manual work (assembly). Based on the studied literature, the authors did not find that such or similar approach has been implemented so far.
With technology advancement and better quality headsets, the adverse consequences will most likely be reduced to a minimum. A further enhancement is possible using technologies such as 3D scanning that would allow faster incorporation of complex real-world models into CAD software or even directly into VR software, without using formatting tools.
Also, incorporating VR module in CAx software will result with no need to use tedious work of format conversion in the future.
Future work would involve the use of haptic devices for a more faithful replication of the assembly processes and exploration of human-robot interaction (collaborative man-machine work). Obviously, with time span, the area of computer-mediated reality will experience further expansion: this could result not only in a thorough change of the current work and lifestyle of people, but also in unprecedented addressing of problems of the material world and human sensing and reasoning about it.