Overview of Computer Vision Techniques in Robotized Wire Harness Assembly: Current State and Future Opportunities

Wire harnesses are essential hardware for electronic systems in modern automotive vehicles. With a shift in the automotive industry towards electrification and autonomous driving, more and more automotive electronics are responsible for energy transmission and safety-critical functions such as maneuvering, driver assistance, and safety system. This paradigm shift places more demand on automotive wire harnesses from the safety perspective and stresses the greater importance of high-quality wire harness assembly in vehicles. However, most of the current operations of wire harness assembly are still performed manually by skilled workers, and some of the manual processes are problematic in terms of quality control and ergonomics. There is also a persistent demand in the industry to increase competitiveness and gain market share. Hence, assuring assembly quality while improving ergonomics and optimizing labor costs is desired. Robotized assembly, accomplished by robots or in human-robot collaboration, is a key enabler for fulfilling the increasingly demanding quality and safety as it enables more replicable, transparent, and comprehensible processes than completely manual operations. However, robotized assembly of wire harnesses is challenging in practical environments due to the flexibility of the deformable objects, though many preliminary automation solutions have been proposed under simplified industrial configurations. Previous research efforts have proposed the use of computer vision technology to facilitate robotized automation of wire harness assembly, enabling the robots to better perceive and manipulate the flexible wire harness. This article presents an overview of computer vision technology proposed for robotized wire harness assembly and derives research gaps that require further study to facilitate a more practical robotized assembly of wire harnesses.


Introduction
The shift towards electrification and autonomous driving in the automotive industry places increasing importance on the automotive electronic system, where more and more wire harnesses are installed as they are essential for connecting automotive electronics and supporting signal transmission in the electronic system.This shift, in turn, makes the efficient, safe, and high-quality assembly of wire harnesses in vehicles critical for manufacturers to increase their competitiveness and gain more market share.Fig. 1 illustrates an example of an automotive wire harness.
However, a large proportion of current automotive wire harness assembly operations remains manual and skill-demanding, leading to potential quality problems.Some of the manual assembly processes also lead to severe ergonomic issues.Meanwhile, the automotive industry persists in a consistent requirement for productivity.Hence, it is desired to ensure assembly quality and improve ergonomics while keeping the utilization to optimal levels.
Robotized assembly has been implemented to facilitate automation in various industries.Although different automation solutions have been proposed under simplified industrial configurations [26,7], robotized assembly of wire harnesses re-  mains challenging in production due to the deformability of wire harnesses and short takt time.In theory, as a special Deformable Linear Object (DLO) manipulation, robotized assembly of wire harnesses is challenging in modeling, state estimation, and operation [32,17].The high flexibility of DLOs also limits transplanting methods for manipulating rigid objects [23,32] and makes it complex to design robot programs in production applications.Other practical issues on general automation, such as risk management between robots and human operators, production management, and generalization of automation for multiple product variants on the same production line, also limit the robotization of wire harness assembly.
Computer vision presents a potential to the automotive industry and computer vision combined with robots can be the game changer in solving ergonomic issues whilst increasing quality and productivity.Increasing research efforts in robotics have explored the complex DLO manipulation, addressing both theoretical research [22,24,28] and engineering practices [3,7,8,9,23,4].Although various proposals under laboratory configurations have been discussed in previous studies [3,26,7,8,2,27,25,32], vision-based robotized wire harness assembly remains challenging in actual production, considering extracting image features from an intricate background [13] and fast recognition of wire harnesses for following robot operations [4].Besides, vision-based automation solutions have not reached the robustness levels of human capabil-ities, especially under dynamic configurations in practical production.
This article aims at providing an overview of what type of vision systems have been proposed for robotized assembly of wire harnesses and exploring the future research directions toward a more practical vision system for robotized wire harness assembly.Based on the previous literature, this study identifies three major trends for future research, including 1) the adaptation of more advanced object detection and recognition algorithms to facilitate the robotized wire harness assembly; 2) new product design on the components of wire harnesses to facilitate the visual machine perception; and 3) the evaluation of proposed vision systems in actual production to examine the practicality and reliability.
In the rest of this article, Section 2 introduces the basics of automotive wire harnesses and the task for automation of the final assembly of wire harnesses according to current assembly workflow in production, followed by the methodology of this study in Section 3.Then, Section 4 and Section 5 describe vision systems proposed in previous research for manipulating components of wire harnesses and perceiving the structure of wire harnesses, respectively, followed by discussions in Section 6. Section 7 concludes the article with summarized findings and outlooks on future research direction.The follow-up research of this study is briefly introduced in Section 8.

Basics of automotive wire harnesses
A wire harness has a tree-like structure, consisting of a bundle of routed cables with various components, such as clamps and connectors, which are used to transmit current or signals within electrical equipment [1,29].The usage of wire harnesses keeps enlarging in modern vehicles due to the increasing number of electronic devices installed for various functions, which can be reflected in the heavily-increasing length of automotive wire harnesses installed in automobiles over time, as illustrated in Fig. 2.
Wire harnesses can be categorized according to the location of installation, for example, engine harness, instrument panel harness, floor harness, and bumper harness.Based on the observation in a car manufacturing plant, the current wire harness assembly in the passenger cabin of a passenger vehicle can be divided into (1) automated preprocessing on the packed wire harness; (2) transfer of the preprocessed wire harness into the car body using lifting equipment; (3) disentangle and route the wire harness manually; and (4) fix and connect the wire harness manually.The various heavyweights of wire harnesses and several far-reaching assembly positions pose challenges to human operators to manipulate and assemble them manually in the final assembly in terms of quality and ergonomics.In addition, several high-voltage wire harnesses, especially in an electric car, need to be handled more carefully regarding assembly quality, safety, and reliability, making automation desired and in urgent need.
Robots are widely implemented as automated solutions in manufacturing and have demonstrated tremendous potential for 2 the automation of wire harness assembly.Besides mechanical control, a robot needs to know the position of picking and placing the target objects before operation.Considering the complexity of wire harness assembly, computer vision may facilitate a more flexible positioning and prove considerable aid in pick-n-place and assembly operations.

Methodology
To acquire an overview of what vision-based technology has been discussed for robotized wire harness assembly, this study first implemented an inquiry on the Scopus database with the search string, TITLE-ABS-KEY((wir* OR cabl*) AND (harness* OR bundl*) AND assembl*).According to the main purpose of this study, the searching results were examined thoroughly by the co-authors to select the articles focusing on the final assembly of wire harnesses onto other products and proposing vision systems for the robotized assembly.The selected articles were then analyzed to identify the current state and future research needs.Besides, only articles in English were included in the analysis, and secondary studies, i.e., review and conference review, were excluded from the analysis.

Clamp insertion
Clamps are used for fixing wire harnesses on target locations.Four articles discussed clamp insertion onto an automobile instrument panel frame with different vision systems [13,7,8,9], where CCD cameras were implemented to recognize clamps so that the end-effector on a robot arm could reach, grasp, and manipulate the detected clamps.All four studies implemented hand-eye vision systems by mounting different numbers of cameras on the end-effectors of robot arms [13,7,8,9].Two of them also adopted global vision systems with multiple cameras fixed around the operation area to support recognition, for example, avoiding occlusion [7,8].
Koo et al. [13] first proposed to facilitate the clamp recognition and manipulation by installing cubic clamp covers with markers, whose poses were recognized by identifying the markers with SIFT [15,16] using stereo vision systems consisting of two CCD cameras with different focal lengths mounted on the end-effectors of two robot arms.Later, Jiang et al. [7] and Jiang et al. [8] improved the clamp covers to a cylinder-like shape with more markers from ARToolKit [10,11] and implemented a global vision system comprising ten fixed cameras with different angles surrounding the work-frame alongside hand-eye cameras to relief the occlusion problem.Furthermore, Jiang et al. [9] proposed to replace visual clamp detection with a tracing operation.Nonetheless, one wrist CMOS camera on the right robot arm remained to estimate the pose of the cover later following the similar design of pattern recognition in Jiang et al. [7] and Jiang et al. [8].

Connector mating
Mating of connectors is critical in wire harness assembly to ensure quality and functionality, which is complicated with demanding manipulation accuracy and intricate structures and non-rigid materials of connectors [26].There are seven articles proposing various vision systems for different tasks of connector mating, including state estimation [27,30,32], visionguided mating [2,25], and fault detection [3,26].
State estimation of the 3D geometric information is essential for robot control in vision-based robotized assembly.Tamada et al. [27] implemented a global-fixed high-speed camera to recognize the types and poses of connectors and monitor the mating process by detecting the corners of connectors with image processing at high speed.Zhou et al. [32] used three cameras, one fixed global camera and one hand-eye camera per arm of a dual-arm robot, to achieve a rough locating-then-fine positioning detection based on MobileNet-SSD [6] and CAD model registration.Instead of using 2D cameras, Yumbla et al. [30] adopted a RealSense D435 depth camera, where RGB images were first processed for connector detection with image processing methods and then the depth image was integrated to obtain the 3D geometry.
Dedicated to the vision-guided mating process, Di et al.
[2] designed a scheme for monitoring relative motions between two connectors with two mutually perpendicular cameras based on basic pattern matching.Song et al. [25] proposed a pattern matching-based visual servoing for locating connector headers using a hand-eye camera and markers on connector headers.
Fault detection has also been discussed to ensure the quality of grasp and insertion so that the control system could react to problematic manipulations earlier.Di et al. [3] adopted an In-Sight 5100 camera to detect faults in grasp and insertion by checking the relative translational and rotational displacements between the gripper and the connector, which was further improved by Sun et al. [26] by adding one camera perpendicular to the first one to supplement the displacement detection.

Structure perception
Recently, three studies have discussed different tasks in perceiving the structure of wire harnesses, including interpretable classification of branches [12], 3D profile extraction [19], and wire recognition [4], as listed in Table 2.
Kicki et al. [12] focused on the interpretable classification of wire harness branches and proposed an RGB-D image dataset of four branches of an automotive wire harness captured by a global-fixed RealSense D435 depth camera.This research compared different networks sharing the same Downsample layer from ERFNet [21] for different data modalities and evaluated the impact of elastic transformation and pre-training with inpainting task.Saliency maps based on class activation mapping CCD cameras Global-fixed + Hand-eye 10 fixed + 6 on end-effectors [9] Point Grey Firefly MV Hand-eye 1 Connector [27] MC1362, Mikrotron Global-fixed 1 [30] RealSense D435, Intel Hand-eye 1 [32] Industrial cameras Global-fixed + Hand-eye 1 fixed + 2 on robot arms [3] In-Sight 5100 Global-fixed 1 [26] CCD cameras Global-fixed 2 [2] CCD cameras Global-fixed 2 [25] FL2G-13S2C-C, PGR Hand-eye 1 3D profile extraction Helios Time-of-Flight camera Hand-eye 1 [4] Visual recognition RGB-D --(CAM) [31] were adopted to visualize the interpretability of the classification results.Nguyen and Yoon [19] proposed to obtain the 3D geometry of a wire harness through clamp detection and profile tracking and correction using a hand-eye depth camera (Helios Time-of-Flight Camera) mounted on the right UR3 robot to guide the selection of next robot picking point.
Guo et al. [4] proposed an RGB-D-based segmentation and estimation for aircraft wire harness recognition, where the complete segmented wires were obtained through supervoxel oversegmentation, segmentation based on Cartesian distance, color similarity, and bending continuity, and estimation with Gaussian Mixture Model [20] on the raw point cloud data acquired by an RGB-D camera.

Discussion
In general, previous research efforts designed different vision systems to recognize different components of wire harnesses [13,3,7,26,8,2,27,9,25,30,32], estimate the state of sub-processes [3,2,25,12,4], and detect errors in assembly [3,26] so that the control system of robotized wire harness assembly could conduct manipulations on wire harnesses and monitor the operation process according to the real-time configurations.
Previously, the most focused components of wire harnesses were clamps [13,7,8,9] and connectors [3,26,2,27,30,32,25]. Earlier studies on clamp insertion took advantage of designed clamp covers with unique markers to facilitate detection and manipulation [13,7,8,9].However, these studies [13,7,8,9] focused on the assembly of wire harnesses to an automobile instrument panel, which could be significantly different from other assembly locations that demand a different number of wire harnesses, for example, in the engine room or cabin.Clamp covers would occupy space in other installation areas with more compact installation, and post-assembly cover removal would lead to further challenges.Thus, new product designs to facilitate the visual machine perception and robotic manipulation are desired.Other studies on connectors discussed how to detect connectors [27,30,32] and monitor the assembly process [3,26,2,25].2D image recognition was primarily adopted in previous research except for one that introduced depth information alongside RGB color information captured by an RGB-D camera [30].However, the 2D RGB image was processed initially to detect connectors, and the depth information of detected connectors was simply extracted from the depth image [30].
Considering more recent studies on perceiving the structure of wire harnesses with RGB-D and ToF cameras [12,19,4] and the increasingly advanced and affordable imaging technology, it is promising to implement depth or 3D cameras to acquire spatial information and conduct detection by processing 3D information directly.The recent renaissance of convolutional neural networks (CNN) and the successful development of deep learning in computer vision research [14] also facilitate numerous research on learning-based 2D and 3D computer vision problems [5,18,33], which inspires the adaptation of latest object detection and recognition algorithms to the future computer vision-driven robotized assembly of wire harnesses to facilitate the detection, manipulation, and tracking of various components of wire harnesses.
In addition, although previous studies have demonstrated the potential of vision-based solutions for robotized wire harness assembly tasks, the practicality and reliability of the proposals in practical applications remain unknown.The actual production has a strict requirement on the production rate, which makes the operating speed and quality of the vision system critical and validation of vision systems in actual circumstances necessary.Thus, future studies on the evaluation of computer vision techniques for robotized wire harness assembly in prac-tical manufacturing configurations are desired and necessary to assure the practicality and reliability as well as the successful integration of the vision systems in actual manufacturing.

Conclusion
In conclusion, previous studies have noticed and explored various computer vision-based solutions for different tasks in robotized assembly of wire harnesses, including vision-guided manipulation of different components and visual machine perception of the structure of wire harnesses.Yet, further development on robotized assembly and the enabling computer visionbased techniques are needed to promote a sound work environment, less ergonomic stress, and sustainable production, especially considering the increasing number of cables in high-tech products, e.g., cars, as depicted above.The state of the art of computer vision technology in robotized wire harness assembly was reviewed in this paper and some future research opportunities were discussed, including: • Developing new learning-based computer vision algorithms to exploit 3D information captured by depth or 3D cameras to facilitate the detection, manipulation, and tracking in robotized wire harness assembly; • Evaluating the practicality and reliability of vision systems in actual production to promote the integration of vision systems into practice; • Exploring new product designs of wire harnesses to enable a more efficient component detection and manipulation without affixing additional parts.

Future work
In the future, vision-guided robotized manipulations on various components of wire harnesses will be proposed and evaluated in both laboratory and practical scenarios.A systematic literature review will also be conducted to better understand the state of the art of vision-guided robotized wire harness assembly and distinguish the challenges in the task.

Fig. 1 .
Fig. 1.An example of an automotive wire harness.

Fig. 2 .
Fig. 2. The heavily-increasing length of wire harnesses in passenger cars over time.(Courtesy of Volvo Car Corporation).

Table 1 .
Vision systems in articles for manipulation on components of wire harnesses.

Table 2 .
Vision systems in articles for perceiving the structure of a wire harness.