A Virtual Environment for Collaborative Inspection in Additive Manufacturing

1 INTRODUCTION

Additive manufacturing (AM) or 3D printing techniques have been used to enhance manufacturing processes that can fabricate optimal part designs irrespective of part complexity. AM has impacted a wide range of applications in many industrial sectors. Compared to traditional techniques, e.g., subtractive manufacturing, in which manufactured parts are created by cutting solid blocks of material, AM provides unique benefits that could offer innovative designs. For example, allowing internal density gradients or stiff, but lightweight infills using complex lattice structures [19].

AM reduces the cost of the prototyping stage and material waste generation compared to conventional approaches. Furthermore, intricate truss structures and similarly complicated geometries produced by AM techniques are not possible via traditional manufacturing [17]. The workflow for AM typically converts computer-aided design (CAD) models to print instructions that are used to fabricate parts, followed by intensive inspection methods of the parts to ensure they meet user requirements. Despite the advantages of AM (and consistent with all novel manufacturing methods), AM systems are nascent and could produce unexpected defects in fabricated parts. Defects of AM parts are often related to excess or insufficient material, but can also include unwanted material impurities. In the context of defects of lattice structures, trusses can be missing/broken, have an overly rough surface finish, or have unwanted pores within. Therefore, identifying and classifying those defects is of utmost importance.

The inspection of manufactured parts can be categorized into four elements, such as internal defects, external defects, dimensional accuracy, and surface roughness [10]. Most inspection analyses rely on desktop-based systems with high-resolution computed tomography (CT) scans or other related modalities [9, 12]. Volume rendering and model representations could serve as an approach to explore and analyze the parts. However, desktop-based systems offer limited visualization and interaction opportunities, in particular depth perception, compared to a virtual reality (VR) system [5]. VR offers great potential and several benefits for AM. Apart from the immersive experience and intuitive interactions, it can be used to enhance the overall workflow. This includes visualization of 3D modeling to identify flaws in the early process, improving the manufacturing process and performance with digital twins by combining virtual-physical representations and data-driven models, and virtual inspection of manufactured parts [2, 12]. In addition, collaboration throughout the design and production process, which often involves separate teams, is crucial. The current process is often realized with in-person meetings and video conferencing, e.g., Webex and Zoom meetings with screen sharing and PowerPoint slide presentations. Collaborative VR can be used to improve several aspects of these processes, including team communication, intuitive interactions, and real-time synchronization for collaborative tasks over distance. However, current integrations for AM parts exploration and inspection are still underutilized and are not flexible concerning the aim of the process, e.g., collaborative scenarios and team setup [23, 24].

In this work, we present a VR environment that allows multiple users to collaborate, explore, and inspect AM parts (see Figure 1). The environment allows users to join either in a co-located or remote space. It can be used to improve AM processes ranging from data exploration and visualization to in-depth team discussion. Various features are developed and enhanced in a real-time synchronization manner. We followed Nielsen’s heuristics approach [21, 22] for usability and interface verification. Furthermore, we conducted exploratory and semi-structured interviews with six domain experts. Qualitative feedback regarding the potential benefits, applicability, and current limitations are collected and described. The proposed VR environment opens new directions for team communication and collaboration in AM. Our contributions are the following:

Figure 2:

View Figure

2 RELATED WORK

In recent years, there has been a growing interest in creating virtual analogous physical components, e.g., digital twins, to aid the overall AM process [2, 13]. Compared to desktop-based systems, VR offers immersive visualization, intuitive interactions, realistic rendering, navigation, and other functionalities. Mathur et al. [18] proposed a virtual training environment of AM for design students in VR. Their results show that VR can be used in training programs and curricula to improve design and problem-solving skills.

Klacansky et al. [12] presented a virtual inspection tool for comparing the CAD model of designed parts with the CT scan of the corresponding AM parts. They concluded that using immersive 3D rendering is more effective and allows measurements for 3D metrology in AM straightforwardly compared to existing tools. Pirker et al. [24] reviewed application scenarios that benefit from a combination of VR and digital twins. Similarly, Del Vecchio et al. [8] provided a systematic literature review to investigate the intersection of VR and digital twins in industrial contexts. Their results demonstrate the potential and opportunities of VR. The potential benefits include a low-cost approach, direct and natural interactions, advanced data collection, advanced visualizations, and remote collaboration.

While VR has the potential to provide better visualization and interactions, most of the systems are designed as a single-user approach. On the other hand, collaborative VR allows multiple users to join and collaborate simultaneously in the same shared virtual environment [6]. It can be used to support communication, team training, group discussion, and particularly collaborative work between multidisciplinary teams [4]. Stacchio et al. [26] presented a framework by integrating user annotations to support the design in the manufacturing context for both augmented and virtual environments. Oppermann et al. [23] introduced a mixed reality tool to support remote maintenance in the collaborative environment. They also discussed some of the practical challenges and complexities during the development of collaborative interactions, such as remote rendering, mixed reality hardware limitation, and data synchronization.

Kuts et al. [14, 15] proposed a VR environment for industrial digital-twin robot synchronization. They concluded that the concept of a digital twin in a collaborative VR environment is practically viable, and can be used for industrial applications in the near future. Havard et al. [11] developed an environment for industrial design and assessment between the digital twin and VR on a human-robot collaborative use case. They proposed a solution using client-server architecture and real-time machine-to-machine communication for data exchange. Collaborative data management could play an essential part in digital twin for AM [27]. Liu et al. [16] presented a framework to support the development of digital-twin data management in different product lifecycle stages. It includes the stages for product design, process planning, manufacturing, post-processing, and quality measurement.

Compared to previous work, the proposed collaborative VR environment offers unique advantages for team-based inspection and collaboration ranging from immersive data visualization and interactions to team discussion in a real-time synchronization manner.

3 COLLABORATIVE VR FOR AM INSPECTION

In the following sections, we describe the design and implementation of the proposed collaborative VR environment, including requirement analysis, system architecture, system features, and interface verification.

3.2 Collaborative Exploration and Inspection

Once the users join the virtual environment, they can start exploring the environment as well as interaction techniques, e.g., navigation and UI interactions. We used the dataset of AM octet lattice structures (1200 × 1200 × 1200) [20] as an example for collaborative inspection. The defect in AM can potentially result in a bent strut, broken strut, missing strut, thin structs caused by insufficient material, or dross defect caused by excessive material.

Figure 2 shows inspection features and interaction possibilities in the virtual environment. The users can use the virtual tablet to facilitate exploration and inspection features, such as loading datasets, a cross-section plane, box and sphere cutouts with inclusive and exclusive features, and slicing views with different axes (axial, coronal, and sagittal plane). The users can use the cross-section plane to slice the volume data from different angles. It can be moved and rotated freely based on the interaction with the VR hand (see Figure 2a). To get the view from the exact axis angle, the users can use the axis slicing view to change the image slices (see Figure 2b). Furthermore, they can use the box or sphere cutout to explore the region of interest of the dataset (see Figure 2c). There are options to change the volume intensity, i.e., window width and level, on the virtual tablet as well. All interactions are synchronized in real time. Besides the exploration features, the users can draw lines to point to the region of interest and initiate the discussion (see Figure 2d). A virtual whiteboard was also implemented allowing the users to draw and illustrate their ideas, which could be essential for discussion with their collaborators.

Figure 3:

View Figure

4 EXPERT FEEDBACK

We evaluated the proposed environment with six domain experts (E1–E6). Three of them were computer scientists working with AM and the other three were engineers/material scientists. One had nine years of working experience, three rated themselves between 3–5, and the other two had between 1–2 years of working experience. Regarding VR experience, one had none, four had little experience (using VR a few times), and one had much experience (using VR several times). We conducted exploratory and qualitative interviews to obtain their feedback about benefits, applicability, and current limitations as well as research directions. After explanation and introduction, the participants were asked to join in the virtual environment with one of the researchers. The participants were guided through all the features and interaction possibilities. Moreover, a think-aloud protocol was used during the session. After that, we conducted a semi-structured interview to collect qualitative feedback. During the interview, one researcher (interviewer) took notes of all relevant comments. Those comments were then collated in a database, and redundancies were removed.

Team-based Inspection and Collaboration. All participants stated that current approaches for their team communication and collaboration are realized with in-person meetings, which require back-and-forth travels, and video conferencing by using screen sharing, screenshots, and even using Microsoft paint to visually explain ideas. E5 said “We use WebEx video conferencing with screen sharing and PowerPoint slides to give presentations to our collaborators. Being able to visualize the machine, hardware, and parts and handle it in the virtual context is a game changer.” While in-person meetings are crucial in reducing misunderstandings, this is not always the case, especially during the COVID-19 pandemic. The experts confirmed that using collaborative VR is extremely helpful in visualizing and guiding their collaborators through the data, machine, and installation setups, particularly with new experimental results. It offers real-time synchronization, intuitive interactions, and team engagement, which are beneficial for communication and discussion cycles, in particular between design and production agencies. E2, E3, E5, and E6 clearly stated that it is invaluable during the AM design phase by allowing designers, engineers, and stakeholders to collaboratively explore various design aspects, fit, form, and functions. This leads to better-informed decisions, optimized designs, and reducing the need for physical prototypes and travels.

Exploration and Inspection Features. Regarding features for data exploration, E4, E5, and E6 commented that scaling and slicing the data in VR are the most useful features for them because they could easily explore and inspect the printing artifacts, especially with their collaborators. E3 expressed that drawing annotation was very helpful in highlighting the point of interest for discussion. However, it would be beneficial to allow users to draw with different colors or a unique color for each user. Similarly, E6 suggested adding the drawing of annotations on the slicing plane, this could avoid drawing through to the other side in 3D space. VR hand representations with animations were assessed as supportive while grabbing virtual objects. Nonetheless, adding VR tooltips or a help button could help users easily understand the button-mapped functionalities. E1 mentioned that visualizing multimodal data, including printing toolpath, sensing data, and captured images during the printing process would be very helpful. By visualizing each step, the team can identify bottlenecks, optimize workflows, and enhance efficiency accordingly. Moreover, E3 and E4 commented that providing additional metadata information, e.g., strut diameter, and a measurement tool would be advantageous for measuring and understanding the data.

Applicability. For applicability, E2 mentioned that the adaptation of technology could be easily adapted by their team. E3 also added that there seem no issues with senior technicians since it is just a learning curve, and obviously, the system can be used to enhance their process. E1 and E2 suggested investigating the approach to handle data privacy and security for collaboration over geographical locations. E3 and E6 expressed that incorporating it into their workflow might face challenges due to logistics and accessibility. Safety while using VR in a limited space in their workplace should be considered as well. The experts also confirmed the advantages of employing collaborative VR for team training and skill development in AM processes, such as machine setup, material handling, and post-processing without the potential risk of damaging expensive equipment.

5 DISCUSSION

Collaborative VR can enable users to design, test, and optimize AM components in a realistic and immersive way, as well as to collaborate with other users across different locations and disciplinary. We demonstrated the prototype to the experts following guided exploration and interviews, and the sessions were informal. One researcher joined in the virtual environment with them in each session. They confirmed its benefits and usefulness, in particular the collaborative capabilities (R1). The environment opens a new approach to exploring and qualifying AM parts. In the beginning, they were struggling with the use of controllers and interaction techniques. However, it was a learning curve; after they familiarized themselves with the technology, the performance was faster. They also stated that they learned it quickly because another user was showing the interaction possibilities via real-time synchronization. VR is still a nascent technology; thus, allowing users to join as a spectator mode with a 2D desktop was considered as helpful.

All participants were positive about the proposed VR environment. When it comes to the question of integrating into their workflow, they highlighted the potential viability. The proposed environment can be used for collaborative data exploration (R2). They confirmed that using collaborative VR is of utmost useful to enhance their process, e.g., visualizing and guiding their collaborators through data and machine setup. There were no issues regarding discomfort or any motion sickness in the VR environment, which could indicate that the system rendering performance is acceptable (R3). However, quantitative assessments are needed in future work. Apart from developed features, integrating more scenarios that go beyond a research prototype would be interesting, e.g., data streaming from a corresponding server, measurement, and alignment tools. Data streaming could play an essential part in their workflow because it could greatly enhance the process and data security. One potential approach is to develop a custom WebAPI and utilize OpenViSUS for data management. It also could be beneficial for data storage and sharing in the collaborative VR environment as well.

Besides the inspection features, investigating natural and intuitive VR interaction techniques, including high-precision drawing and annotations using input devices, e.g., VR pen, could be essential in increasing precision, making the annotation and drawing smoother and engaging, and reducing mental load [1, 3]. For measurement, it would be interesting to integrate a tool measuring the length between two points in a correct unit. Furthermore, alignment techniques, e.g., AMP-IT and WISDOM [25] could be essential to provide a precise manipulation and alignment between objects.

VR is primarily used with visual and auditory senses. However, AM involves physical properties, such as material strength, thermal conductivity, elasticity, sensor data, and archived datasets. Representing this information accurately in the virtual environment could remain a challenge. AM also comprises various techniques, e.g., fused deposition modeling (FDM), selective laser sintering (SLS), direct ink write, laser powder bed fusion, and others, with distinct parameters and processes. Replicating and simulating all these processes in VR could be highly complex and may require simplifications. There are also other challenges to utilizing collaborative VR for the AM process. These include high-performance hardware and network requirements. Collaborative VR requires high bandwidth and low latency to ensure smooth and synchronous communication among users. Additionally, it could pose some human factors issues, such as discomfort, which may affect the user experience and performance while wearing it for a longer time. Future work should investigate the effects of collaborative VR on the user’s cognitive, affective, and behavioral outcomes in AM with an extensive user evaluation.

Integrating artificial intelligence (AI), e.g., generative AI assistant [7], and cloud-based sensing data center, including Omiverse and AWS Digital Twins in the virtual environment would be interesting. Moreover, exploring the integration of multimodal data and progressive refinement interfaces to enhance the realism and fidelity of AM could be crucial as well.

Compared to 2D interfaces with a variety of systems and features, including video conferencing and web-based collaborative tools, VR is still nascent and evolving. However, it is an emerging technology and is increasingly developed and used in various applications. VR offers several unique advantages, such as immersive experience, improved depth perception, intuitive interactions, and enhanced collaboration. Apart from immersive VR, mixed reality (MR) has also been used and applied to various fields. It provides an alternative approach to visualizing and interacting with virtual objects in the physical world. Investigating in MR, e.g., situated analytics and collaboration, would be interesting for future work to provide and enhance understanding and situational awareness.

The proposed collaborative VR offers a new approach and pushes the boundaries of current inspection methods in AM. It also provides new opportunities to integrate with digital twins and has the potential for other related fields, such as medical aerospace, and consumer products industries.

6 CONCLUSION

We have presented a collaborative VR environment for AM parts exploration and inspection. We developed and enhanced collaborative features ranging from data exploration to team discussion. The proposed environment allows users, particularly design and production agencies, to communicate and enhance their workflow accordingly. We followed the guidelines of usability and interface verification using Nielsen’s usability heuristics. Moreover, qualitative feedback from experts was collected and summarized. To sum up, collaborative VR is a promising tool for advancing AM and opening new research directions in related domains.

ACKNOWLEDGMENTS

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. The project has been supported by LLNL LDRD (23-SI-003). The work is reviewed and released under LLNL-PROC-859114. We are thankful to study participants for their support and valuable feedback.