Real-virtual components interaction for assembly simulation and planning

Highlights

• Enable real-virtual components interaction during augmented reality assembly.

• Facilitate intuitive manipulation of virtual components with bare-hand interface.

• An industrial case is conducted to validate the effectiveness of the system.

Abstract

Assembly motion simulation is crucial during conceptual product design to identify potential assembly issues and enhance assembly efficiency. This paper presents a novel assembly simulation system incorporating real-virtual components interaction in an augmented-reality based environment. With this system, users can manipulate and assemble virtual components to real components during assembly simulation using natural gestures through an enhanced bare-hand interface. Accurate and realistic movements of virtual components can be obtained based on constraints analysis, determination of resultant forces from contacts with real components and manipulation from the user's hands (forces and torques applied, etc.) during assembly simulation. A prototype system has been developed, and a case study of an automobile clutch was conducted to validate the effectiveness and intuitiveness of the system.

Introduction

Assembly simulation, which allows users to evaluate an assembly process and address assembly issues, is crucial during conceptual product design [17]. Nowadays, assembly operations are becoming more complex due to the growing numbers of product variants, smaller batch sizes, and shorter life cycles of products; there has been a growing demand for flexible and reconfigurable assembly simulation and planning systems. The main task of assembly process simulation and planning is to formalize the details of assembly constraints and relationships [38], so as to simulate the assembly operations and processes accurately. Typically, expert assembly planners use traditional interaction tools (e.g., mouse, keyboard) to manipulate 3D CAD models of the components of an assembly, so as to determine the geometrical characteristics of the components as well as the assembly features of the assembly [21], [27]. However, expert assembly planners are usually required to identify constraints manually. Pure virtual reality (VR) based assembly motion simulation and planning does not allow users to relate their assembly experience to the physical context, and thus leading to a lack of real spatial feeling. Augmented reality (AR) technology, a seamless interface that bridges the gap between the real and virtual world, can be used to enhance the assembly simulation process [22]. An AR-based assembly simulation system can be implemented to augment useful virtual contents (e.g., virtual prototypes, information, tools, etc.) onto a user's view to assist the user to verify assembly operations and sequences and assess ease of assembly, thus closing the gap between product design and assembly operation.

AR has been applied to simulate assembly operations and address various assembly-related issues throughout a product's lifecycle, e.g., assembly planning and design [23], ergonomics assessment [24], operation guidance and training [20], etc. However, previous work on AR assembly simulation and planning focused primarily on manipulating virtual objects (assembly components or tools) and carrying out virtual assemblies in a real environment, and has neglected real-virtual components interaction and contact phenomena. Information from real-virtual components interaction, e.g., the spatial relation of a virtual new part design with respect to an existing real component, physical constraints and manipulation obstructions from the real component, etc., are important for decisions making in assembly design and planning [33], [34]. Accurate and realistic motions of the virtual components should be determined based on their interaction with real components, and manipulation forces from the user's hands. Essential factors contributing to a realistic and reliable AR assembly simulation system should be considered, e.g., intrinsic properties of virtual components (e.g., material, shape, dimensions, etc.), human factors (e.g., ergonomics such as assembly position and orientation, accessibility to component), etc.

This paper presents a novel AR-assisted assembly planning and simulation (ARAS) system. ARAS provides (1) an enhanced bare-hand interface (EBHI) enabling users to manipulate virtual components realistically using natural hand gestures, (2) a method to calculate the resultant forces exerted on virtual components from contacts with real components and manipulation from the user's hands (forces and torques applied, etc.) during an assembly process, and (3) constraint analysis and virtual snapping force generation. Thus, ARAS can position virtual components accurately, simulate the assembly motions realistically, allow users to assess product assembly design and identify potential assembly issues. The calculated force/torque (e.g., position, orientation, magnitude, etc.) can be rendered with 3D symbols when virtual components are manipulated by the users. In order to validate the benefits of ARAS for assembly operators, a case study is conducted.

The rest of the paper is organized as follows. Section 2 presents a review of the state-of-the-art assembly motion simulation systems and AR based interaction methods for assembly applications. An overview of the proposed system is presented in Section 3, and the details are described in 4 Enhanced bare-hand interface, 5 Assembly information management, 6 Assembly motion simulation. Section 7 presents the system implementation and a case study. Section 8 summarizes the study and future work.