Next Article in Journal
Taking a Load Off: User Perceptions of Smart Offloading Walkers for Diabetic Foot Ulcers Using the Technology Acceptance Model
Next Article in Special Issue
A Cost–Benefit Analysis Simulation for the Digitalisation of Cold Supply Chains
Previous Article in Journal
Novel Robotic Arm Working-Area AI Protection System
Previous Article in Special Issue
A Novel Simulated Annealing-Based Hyper-Heuristic Algorithm for Stochastic Parallel Disassembly Line Balancing in Smart Remanufacturing
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 23
Issue 5
10.3390/s23052766
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Analysis of Training Deep Learning Models for PCB Defect Detection

by
Joon-Hyung Park
1,†,
Yeong-Seok Kim
1,†,
Hwi Seo
1 and
Yeong-Jun Cho
2,*
1
Data Science Team, Hyundai Mobis, 203 Teheran-ro, Gangnam-gu, Seoul 06141, Republic of Korea
2
Department of Artificial Intelligence Convergence, Chonnam National University, 77 Yongbong-ro, Buk-gu, Gwang-ju 61186, Republic of Korea
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Sensors 2023, 23(5), 2766; https://doi.org/10.3390/s23052766
Submission received: 2 February 2023 / Revised: 20 February 2023 / Accepted: 24 February 2023 / Published: 2 March 2023
(This article belongs to the Special Issue Advanced Sensing Technology and Data Analytics in Smart Manufacturing)
Browse Figures
Review Reports Versions Notes

Abstract

:
Recently, many companies have introduced automated defect detection methods for defect-free PCB manufacturing. In particular, deep learning-based image understanding methods are very widely used. In this study, we present an analysis of training deep learning models to perform PCB defect detection stably. To this end, we first summarize the characteristics of industrial images, such as PCB images. Then, the factors that can cause changes (contamination and quality degradation) to the image data in the industrial field are analyzed. Subsequently, we organize defect detection methods that can be applied according to the situation and purpose of PCB defect detection. In addition, we review the characteristics of each method in detail. Our experimental results demonstrated the impact of various degradation factors, such as defect detection methods, data quality, and image contamination. Based on our overview of PCB defect detection and experiment results, we present knowledge and guidelines for correct PCB defect detection.

1. Introduction

A printed circuit board (PCB) consists of various electronic parts, such as capacitors, resistors, semiconductors, and conductive paths. The PCB performs an electrical connection between the parts through conductive paths on the substrate. Additionally, PCBs simplify the development of prototypes and enable the mass production of electronic products. They are generally used in home electronic products or industrial electronic facilities. As the industry changes, PCBs have also been installed in automobiles and mechanical facilities, where PCBs were previously not essential [1]. In recent years, most electrical, electronic, and mechanical products in our daily lives have contained a variety of PCBs. Thus, accurate PCB production is becoming critical as the demand for these electronic products increases [2].
Accordingly, many companies have employed automated inspection systems for PCB manufacturing [3]. Automatic optical inspection (AOI) systems automatically detect PCB components in terms of their surfaces, arrangements, and circuits. The system takes images for defect detection. (Usually, AOI systems take two-dimensional color images (RGB) but can take other images, such as three-dimensional, ultrasonic, X-ray, and infrared images, depending on the purpose of the system [4,5]. In this study, we focus on two-dimensional RGB images.) For highly reliable PCB manufacturing, defect detection methods embedded in AOI systems should be accurate. However, traditional defect detection methods operate only on rule-based algorithms [6,7], which are vulnerable to contaminated PCB images and have difficulty coping with new types of PCB defects. Recent attempts using deep learning have been studied to overcome the limitations of the rule-base methods, and they display improved defect detection performance compared with conventional methods [8,9,10,11].
However, even if deep learning achieves positive results in PCB defect detection, deep learning models are not perfect for all cases. Extensive training data and various training data distributions are required to train deep neural networks with numerous learning parameters, such as weights and biases. Particularly, the balance of the training data is essential for performing deep learning [12]. A deep learning model is preferred in industrial fields with varied and balanced training data. Industrial images, such as PCB images, are challenging to collect compared with other typical image data [13,14] because PCB images can only be collected during the manufacturing process in the factory. Furthermore, the filming environment of PCB images is different from other standard images, making model training difficult, as noted in Section 3. In this case, introducing deep learning techniques should be more carefully performed for PCB manufacturing.
Recently, many attempts have been studied to improve existing defect detection models or proposed new types of defect detection methodologies for specific industrial sites [15,16]. Despite these attempts, the models can still be degraded by several factors such as a lack of training data, unbalanced training data, image data contamination during system maintenance, and inappropriate model selection (Figure 1b). The application of technology in industrial fields requires considerable cost and time. Therefore, it is essential to review the various performance degradation factors in advance and apply an appropriate model training method.
In this study, we analyze training methodologies for deep learning models that can stably perform PCB defect detection in various industrial situations. The main contributions of this study are as follows: We first summarize the properties of PCB images and review possible degradation factors such as image contamination in industrial sites. Second, we organize applicable defect detection methods according to system purpose and we review the properties of each method in detail. Finally, we present methodologies and guidelines of proper deep learning for defect detection by analyzing various performance degradation factors in real industrial sites. To sum up, this study considers various performance degradation factors while presenting a possible methodology for PCB defect detection. We hope that this study provides meaningful guidelines to readers who desire to apply defect detection methods using deep learning in many industrial fields.

2. Related Work

In traditional image processing and computer vision fields, research has focused on extracting and learning discriminating features in images. Many studies have extracted critical features, such as the shape, pattern, and color of images [17,18], to learn high-performance classifiers [19,20]. However, the emergence of deep learning in recent years, which can be optimized with one objective function from feature extraction to the classification stage, has led to a paradigm shift in many research fields [21]. In particular, recent deep neural network structures aimed at image understanding, such as convolutional neural networks, have achieved significant performance improvements that far exceed existing image understanding techniques, regardless of application [22]. Computer vision methods have steadily improved, e.g., image classification, understanding, and object detection methods. These methods have been applied to various fields, including defect detection at industrial sites.
It has long been attempted to detect defects in industrial data using AOI systems. There are two main approaches for automated PCB defect detection: (1) rule-based defect detection, (2) deep learning-based defect detection. In the rule-based approach, many studies have determined defects based on designed rules after performing simple binary image processing on PCB data [3,6,23]. However, these simple analyses have difficulty coping with various situations that may occur in industrial sites. For example, they have difficulty handling subtle image changes due to position and lighting differences [24]. The studies have not adequately reviewed the factors or situations that can potentially degrade the performance of designed algorithms.
Meanwhile, many methods have been studied to perform defect detection via convolutional neural networks to overcome the limitations of these rule-based methods [8,25]. For more efficient applications, Ren et al. proposed a method to use a pre-trained deep learning network on surface datasets that have very few data points and require accurate identification of defect regions [26]. Kim et al. also performed defect detection using an auto-encoder structure [15]. They first generated a nondefective image from a defective image, then compared the two images (defective and nondefective) to perform defect detection rather than learning a detector that directly detects PCB defects. Liao et al. proposed the YOLOv4-MN3 model, a modified version of YOLOv4 that can detect PCB defects quickly and efficiently [16]. In addition, Adibhatla et al. optimized the YOLO structure for improving defect detection accuracy [27]. Lian et al. utilized a mask R-CNN for defect detection and added a geometric attention-guided mask branch into the fully connected CNN for increasing efficiency of the model [28]. As mentioned, studies have attempted various methods for defect detection in specific industrial applications. However, factors that can interfere with learning in the industrial field are very diverse, so these factors should be identified and addressed in advance. While previous studies have focused on network design or learning methods for PCB defect detection, our study instead focuses on various degradation factors that can affect the model’s performance. We aim to review and analyze potential model degradation factors on industrial data, which, to our knowledge, has not been explored in prior research.

3. Properties of PCB Images

3.1. Public PCB Datasets

Available PCB image data for research purposes are summarized in Table 1. The PCB [29] has synthetic six-class defect images from 10 independent PCB images. Five classes (e.g., mousebite, open, short, spurs, and spurious) are defects of PCB circuits, and the other class (missing hole) is a defect of a PCB component. Each defective class randomly generated 115 PCB images, providing 690 total PCB images, including defects provided in the dataset. The TDD-PCB [30] extends the PCB [29] dataset to over 10,000 images through geometric and photometric image transformations. The augmented image has 600 × 600 pixels and includes two defects, but both datasets (PCB [29] and TDD-PCB [30]) have several limitations. The source PCB images comprise only 10 images, and the defective images are not real images but are generated synthetic data in the datasets [29,30].
In contrast, the FICS-PCB [31], PCB DSLR [32], and PCB-METAL [33] datasets provide a relatively large number of source PCB images compared with PCB and TDD-PCB. However, they do not provide defective PCB parts because defective data are scarce in actual industrial sites, and even if defects occur, companies do not share their defective parts with researchers. In this work, we used TDD-PCB [30] data, which provide various PCB defect images.

3.2. Attributes of Industrial Data

Attributes of industrial data in manufacturing sites are summarized in Table 2. According to the work in [34], industrial data are generally not rich in volume and are imbalanced. The PCB image data (TDD-PCB [30]) generated at an industrial site also have the attributes in Table 2. For example, TDD-PCB has only 10 original source images (small amount), and the defective parts account for a small proportion of the PCB (small object). The dataset has 21,336 defective images, but all other image areas except the defective area can be extracted as nondefective images. In other words, defective and nondefective images are imbalanced, as illustrated in Figure 2a (imbalanced). In addition, six types of defective classes exist in TDD-PCB, which can be divided into two main categories: (1) part-related defects (missing hole) and (2) circuit-related defects (mousebite, open, short, spur, and spurious). As demonstrated in Figure 2b, it is difficult to distinguish the circuit-related defects due to the high similarity of appearance (fine-grained). The attributes mentioned above, such as small amount, imbalance, and fine-grained, lead to difficulties in training and applying deep learning models. Training numerous parameters in deep neural networks without sufficient training data can cause over-fitting problems. Similarly, imbalanced data between each class can cause model bias.
Furthermore, the industrial data are likely to be contaminated due to the environment of the manufacturing space (strong interference) and the long-term maintenance of facilities (temporality). The product manufacturing process involves chemical and physical reactions on boards and parts; thus, illumination changes and blurring can occur in the obtained images. Moreover, although products are not contaminated yet, they can become contaminated when the manufacturing facilities are not adequately maintained. We discuss the possible image contamination of the industrial data in Section 3.3. When analyzing industrial data, such as PCB images, all attributes should be considered before training the models.

3.3. Image Contamination

This section examines possible image contamination due to the attributes of industrial data discussed in Section 3.2. Hendrycks et al. [35] propose various image contamination models to evaluate the robustness and stability of deep neural networks. They considered common and industrial image acquisition conditions. Table 3 summarizes the possible contamination according to the image acquisition environment of PCB data. Examples of possible contamination are depicted in Figure 3. We categorized industrial environments for image acquisition according to four factors as follows:
  • Close-up imaging: Close-up imaging is required to inspect small electronic parts in detail. However, a small vibration of the camera or product causes significant blurring and image degradation. For example, subtle changes in intrinsic camera parameters occur, such as defocus and zoom blurring. In addition, subtle camera pose changes or PCB movements occur, such as elastic transforms and motion blur.
  • Illumination changes: Generally, manufacturing equipment blocks external light and uses internal lighting to avoid the influence of external light sources. The quality of the image changes sensitively depending on the lighting condition. According to the lighting variation, images are affected by noise (e.g., Gaussian and impulse) and color variations (e.g., brightness change, contrast change, and saturation).
  • Long-term maintenance: During long-term manufacturing, various image degradation factors, such as dirt on the lens and PCB, dust and steam particles in the air, and others, occur, which can cause glass blur and spatter contamination.
  • Systematic issues: Factory facilities require enough storage, but some older factories may not be equipped with enough storage capacity to handle the generation of vast amounts of image data. For example, multiple high-resolution images (e.g., 4 K image) are taken for each product and thousands of products are produced per day, a single production line can easily generate about 1 TB of data each week. Managing large-scale images while the factory is in operation is a challenging task. Therefore, image compression is necessary within the industrial field. A lossy image compression can be an effective solution to reduce image capacity, but the image quality degrades. In this case, JPEG compression and pixelation can occur.
These factors prevent the training of high-performance deep neural networks. Generally, deep neural networks are stable and robust as a result of learning numerous weights on large-scale training data. They can make high-performance predictions for new testing data through their complex hierarchies and rich representations of networks. However, several studies have proved that deep neural networks can be significantly degraded with minimal differences or data distribution contamination [35,36].
Manufacturing applications require defect detection performance close to 100% because only one minor defective part can cause a severe product defect. Therefore, we should very carefully employ deep learning models and review possible degradation factors in advance for industrial image data with contamination. In this study, the TDD-PCB [30] dataset is augmented with possible contamination in industrial sites based on [35], and we analyzed model performance under these challenges.

4. Methods of PCB Defect Detection

This section summarizes three methods using deep learning for PCB defect detection: (1) part image classification, (2) whole image understanding, and (3) direct defect detection. Selecting the appropriate method according to the manufacturing process situation (e.g., the volume of collected data, the purposes of the defect inspection systems, and possible image contamination during manufacturing) is critical. Detailed descriptions of each method are summarized in Table 4. (Anomaly detection [37] is a possible solution for PCB defect detection. However, we exclude the method from this study because anomaly detection is very different from the three methods (i.e., image classification, understanding, and defect detection) and has limited applications in industrial sites.)
  • Part image classification is a method that distinguishes classes of input images. It takes cropped image patches as model input and predicts the class of input cropped data. The prediction results include the class of electronic component images and whether they are defective. We can apply image classification when the location and size of each electronic part are known. This method focuses on classifying the electronic parts at a specified location.
  • Whole image understanding examines whole PCB images and determines whether parts or circuits in the image are defective. Training for image understanding does not require the specified location of each part but only the images and their labels, so the training data collection is simple. This method is suitable to employ where the locations of defects at the PCB are not specified. It does not directly infer the location and size of the defective part but only determines whether the whole PCB image contains defective parts. Therefore, we must implement an additional algorithm, such as the class activation map (CAM) [38], to visualize predictions of the image understanding model.
  • Direct defect detection receives the whole PCB image as input and predicts the location, size, and class of the defects. The PCB images and annotations on the location, size, and type of the defects are necessary to learn the defect detection model. If the defect can occur at an unspecified location rather than a specified location in the image, applying the defect detection method is advantageous.
The size of the defect is small compared with that of the PCB, and it is inefficient to perform deep learning using the whole image. Traditional AOI [39] systems have used two approaches to perform defect detection effectively. The first is to determine a defective candidate using a standard sample. Due to the nature of PCB manufacturing, which produces the same type of product in large quantities, it is often possible to obtain the same type of standard sample in advance as the test product. In this approach, a differential image between the test product and the standard sample can be used to determine a candidate location with a possible defect and then cut around the area for deep learning prediction [40]. The location of the defect is known; thus, the most suitable method is part image classification.
Second, if it is difficult to obtain a standard sample, the whole PCB image is divided equally into a specific size and inspected. In this approach, if it is necessary to identify the location of the defect accurately, direct defect detection can be applied. If only the presence or absence of defective parts in the PCB image is needed, image understanding can be applied. This study analyzes the defect detection performance for the three methods and reviews the possible performance degradation factors according to industrial data attributes mentioned in Section 3.

5. Experiments

5.1. Settings

This section analyzes how the quality and quantity of training data affect the performance of each PCB defect detection method. We used the TDD-PCB dataset [30], which was augmented through image flipping, resulting in a total of 21,336 images. The dataset is composed of six classes of defects, with each class containing approximately 3550 images. The proportion of training images among the total images was divided into the following five categories to check the effect of the number of training data: 80%, 50%, 20%, 10%, and 5%. For fairness in the performance evaluation, the image ratio and the number of defective classes were set to be the same for each type. Thus, 20% of the total images were separated in advance as testing images and were used equally for all experiments. Next, we generated contaminated PCB images that frequently occur in industrial sites to analyze how each contamination affects defect detection. To this end, we referred to a study [35] that proposed several image transformation methods to generate contaminated and distorted images that could occur in real-world environments. The 16 contamination types discussed in Section 3.3 were taken, except for weather-related contamination proposed in [35], which rarely occurs at PCB manufacturing sites. We set the parameter settings in the mentioned study [35] as the default for image contamination. Contamination is applied to test images, rather than training images, since it is unpredictable during manufacturing.
As introduced in Section 4, we validated three methods for PCB defect detection, and the experimental settings for them are as follows. For standard settings, we set batch sizes and epochs to 16 and 50 for model training, respectively. For data augmentation, we performed image flipping, which is simple but effective for improving model performance. We followed default settings for other hyper-parameters such as the learning rate, loss function, and dropout rate. For part image classification, we trained ResNet50 [41], a successful deep neural network structure for many computer vision applications. Unlike with other methods, we randomly generated nondefective electronic part images. As the part image classification test specified and cropped part images, nondefective part images are essential for a fair comparison with other methods. We measured classification accuracy for each part image to evaluate model performance. For whole image understanding, the same ResNet50 [41] was employed because it allows a clear comparison of methods for the same goal (i.e., defect detection). To evaluate model performance, we also measured the classification accuracy. To visualize the positions of estimated defects, we employed Grad-CAM [38]. For direct defect detection, we applied an object detector called YOLOv7 [42], which currently displays superior performance compared with other detectors. The mean average precision (mAP) was measured for performance evaluations. We reflected the pretrained model by ImageNet [13] as the initial weight parameters for ResNet50 [41]. For initial weights of YOLOv7 [42], we used the pretrained model by MSCOCO [14].

5.2. Defect Detection Performance According to Training Data Volume

We validated the defect detection performance for three methods according to the training data volume, as summarized in Table 5. Part image classification maintains classification performance better than other methods under a lack of training data. When 80% of learning data were used, the performance of the entire class was 98.8%, and even if only 5% of the learning data were used, the average performance was 96.8%. This method does not need to determine the location of the part but only to predict the class of cropped images. Therefore, maintaining classification performance with a small training data volume (5%) is possible. In particular, missing hole and nondefective classes, which are quite discriminating, prevent performance degradation compared with other classes because the shape and appearance of these classes are clearly distinguished from other circuit-related defects, such as mousebite, open, short, spurs, and spurious. Moreover, the classification accuracy for spur was the lowest at 93.2%, using only 5% of the training data.
Second, the degradation in classification accuracy can be clearly observed for whole image understanding, and the degradation step is the steepest among the three methods. Therefore, the image understanding method requires more data, and if the number of data is limited, we do not recommend this method for defect detection. The performance drop for the missing hole (part-related) defect is relatively gradual compared with other circuit-related defects. In this case, more circuit-related part images are required to mitigate the degradation of classification accuracy. In direct defect detection, despite the sufficient epochs for model training, the detection rate decreases rapidly as the number of training data decreases. Similar to other methods, the detection rate of the missing hole class is maintained while circuit-related defects decrease rapidly. To sum up, as you can see in Figure 4, whole image understanding was most sensitive to the lack of training data. On the other hand, the part image classification maintained its performance despite extreme learning data reduction. Model performance must be verified during the learning process according to the amount of training data and methods.

5.3. Defect Detection Performance According to Possible Contamination

The performance of the part image classification for the contaminated images is summarized in Figure 5a. Compared to other methods (i.e., whole image understanding and direct defect detection), it still displays stable performance for most contamination, but performance significantly declined for several contamination types, such as impulse noise, zoom blur, and saturation. Even in simple noise, the classification performance significantly decreased from 6% to 36%. Thus, deep neural networks are vulnerable to even slight perturbations. In addition, the model could not completely cope with blur and digital distortions, which significantly affect the shape or color of the PCB. Missing hole and positive classes still exhibited stable performance compared to other circuit-related classes for most distortion types but displayed unstable results for zoom blur or saturation changes. For part image classification, the average performance decreased by about 0.252% after contamination.
The results of the whole image understanding model are summarized in Figure 5b. Among various contamination types, zoom blur most significantly degrades image quality, and the resulting performance drop was also the most rapid in this method. Based on the experiment results, the most robust class for contamination was missing hole, and the most vulnerable class was mousebite. However, mousebite was the best-performing class with missing hole, achieving 99.4% in Table 5. Although the classification accuracy of the class is high, the method is not always robust to contamination. For whole image understanding, the average performance decreased by 0.264% after contamination. We visualized the difference in robustness through Grad-CAM [38], as presented in Figure 6. If the prediction is successful, the area of defective parts in Grad-CAM is also specifically determined (Figure 6a). Otherwise, the peak of the Grad-CAM is unclear, and there is no clear distinction (Figure 6b). Thus, whether the understanding of a specific part has been correctly performed can be effectively checked through the Grad-CAM.
Finally, the results of direct defect detection for contamination are summarized in Figure 5c. For zoom blur types whose images are severely damaged, the performance was very low for all classes, and the missing hole class performance was strong, as in the previous experiments. In the case of other circuit-related classes, except missing hole, the performance was significantly degraded when noise and geometric transformation were applied. The performance graph clearly confirms this tendency. As the defect detection method focuses on the appearance information of defects distinct from the background, performance degradation due to contamination is significant compared to other methods. For direct defect detection, mean average precision decreased by 0.165% after contamination. In Figure 7, we present the detection results for several challenging types of contamination. The detector exhibited a 100% detection rate for the images without contamination. However, after applying the contamination, it struggled to detect PCB defects, resulting in frequent false positives and negatives. Compared to the other two methodologies, it was robust to most contamination except for zoom blur and elastic transform. Data augmentation for each contamination type can be attempted in advance to prevent performance degradation. However, the data augmentation rate for each contamination type should be appropriately adjusted. If too much data augmentation occurs for contaminated images, the defect detection performance for noncontaminated images can be degraded.

6. Discussion on PCB Defect Detection

To apply deep learning models to industrial sites, data collection and processing, model learning, model application and verification, and system maintenance should be carefully considered. In addition, compatibility with existing facilities, such as the programmable logic controller [43] and manufacturing execution system [44], is also essential. While many problems can arise during the training of deep learning models, it is highly recommended to thoroughly review and address any potential issues before deploying the model for use in a manufacturing line. This study conducted experiments on various methods (Section 4) that can be employed to detect PCB defects. Various image contamination types (Section 3) that can occur in industrial sites during the process were also considered. This section discusses PCB defect detection by considering the challenges in manufacturing conditions.
First, the class and quantity of training data that significantly influence model performance should be reviewed in detail. The TDD-PCB [30] dataset used in this study provides six types of defective classes. Among them, the missing hole class occurs in electronic components (part-related class), and the remaining five classes occur in the circuit area (circuit-related class). All five defects can be regarded as one class if we determine whether a circuit region is defective. However, if it is necessary to determine the detailed type of each defect during the manufacturing process, it is relatively difficult to distinguish all circuit defects into five different defects due to their similar appearance. As proved in the experimental results, all three methods commonly perform stably for the missing hole class, even with insufficient training data or severe image contamination due to the difference in characteristics of the class from other defective classes. As much as possible, classes with highly similar appearances must be identified, and real data must be collected. Thus, we can investigate and review the training history of the initial model. Collecting defective data at industrial sites cannot be performed in a short period. A sufficient development period must be secured until the model is applied, and data must be continuously collected even after the model is applied. It is possible to augment data during preprocessing, before the model application. However, data augmentation is not significantly different from the collection data distribution; thus, too much data augmentation can drop the model.
Second, image contamination that can occur during system maintenance should be considered, and the model performance should be checked at all times. As summarized in Section 3.3, the PCB is manufactured through the chemical action of complex mechanical facilities in industrial sites, so the images are more likely to be contaminated than those in a general environment. Based on the experimental results in Section 5.3, model performance is poor, especially for blur-related contamination (e.g., defocus and zoom) caused by the movement of the camera or PCB. In particular, if the classes are similar in appearance, they are more vulnerable to such contamination. In addition, when few data are available, it is more difficult to cope with such contamination. Therefore, contamination factors that may occur in the manufacturing process must be avoided, and the facility must be managed steadily by monitoring model performance.
Third, appropriate models should be used according to the manufacturing environment and purpose. In this study, we categorized three applicable methods for PCB defect detection in Section 4. In summary, part image classification is suitable for inspecting parts and circuits at known locations. Part image classification cannot detect defects outside the specified inspection area. Compared to other methods, part image classification displayed stable performance, even with various disturbances, but it requires the most annotation (e.g., location and class of each part). Moreover, whole video understanding has the advantage of being able to train the model with less data annotation. Generally, this method is not highly accurate and does not specify the defect location. Thus, additional methods, such as Grad-CAM [38], should be used to specify the defect location. Whole video understanding can be used meaningfully by quickly applying it to assist workers during manufacturing. Direct defect detection directly detects the locations of defects. It requires additional annotation on the location and size of the defect. This method has the advantage of being able to detect a defect even when a defect occurs in an unspecified location in a PCB circuit. However, it is difficult to train a model that performs well in all situations because the method must predict the class, location, and size of defects at once. They are vulnerable to contamination, as indicated in Figure 5; therefore, we must carefully apply the model to industrial sites and continuously maintain the model.
The article describes the application of deep learning models to industrial sites, highlighting the importance of data collection, processing, model learning, application, and maintenance. The study focuses on PCB defect detection and discusses the challenges faced in manufacturing conditions. The authors suggest that appropriate models should be used based on the manufacturing environment and purpose and that factors such as data quality, image contamination, and model performance must be carefully considered. The study categorizes three applicable methods for PCB defect detection: part image classification, whole video understanding, and direct defect detection.

7. Conclusions and Future Works

In this study, we described the significance of data collection, processing, model learning, application, and maintenance in the application of deep learning models to industrial sites. By categorizing PCB defect detection approaches into three applicable methods—part image classification, whole video understanding, and direct defect detection—we provide a comprehensive overview of the challenges faced in manufacturing conditions. Our experimental results demonstrate the impact of various degradation factors, such as defect detection methods, data quality, and image contamination. Ultimately, we hope that this study provides readers with valuable guidelines for implementing defect detection methods using deep learning in a wide range of industrial fields.
Based on our study, there are a few areas that could be explored for the future work: (1) Developing new methods for PCB defect detection. While the study identified three applicable methods, there may be other methods that could be developed for detecting PCB defects that are more effective in certain manufacturing environments or for specific types of defects. (2) Optimizing the use of deep learning models in industrial sites. We emphasized the importance of carefully considering data collection, processing, model learning, application, and maintenance when using deep learning models. We can further focus on optimizing each of these areas to improve the effectiveness and efficiency of deep learning models in industrial applications. (3) Application of deep learning models in other industrial settings. While the study focused on PCB defect detection, the principles and methods discussed could be applied to other industrial settings where the detection of defects or anomalies is important.

Author Contributions

Methodology, J.-H.P., Y.-S.K. and Y.-J.C.; Validation, Y.-J.C.; Formal analysis, Y.-J.C.; Investigation, J.-H.P. and H.S.; Data curation, Y.-S.K.; Writing—original draft, J.-H.P. and Y.-S.K.; Writing—review and editing, Y.-J.C.; Supervision, H.S. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported in part by the research project entitled “Development of Technology to Evaluate and Improve the Robustness of AI Models” by Data Science Team in Hyundai Mobis (2022) and in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2022-00165919).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

Not applicable.

References

  1. Cucchiella, F.; D’Adamo, I.; Rosa, P.; Terzi, S. Automotive printed circuit boards recycling: An economic analysis. J. Clean. Prod. 2016, 121, 130–141. [Google Scholar] [CrossRef]
  2. Malge, P.; Nadaf, R. PCB defect detection, classification and localization using mathematical morphology and image processing tools. Int. J. Comput. Appl. 2014, 87, 40–45. [Google Scholar]
  3. Moganti, M.; Ercal, F.; Dagli, C.H.; Tsunekawa, S. Automatic PCB inspection algorithms: A survey. Comput. Vis. Image Underst. 1996, 63, 287–313. [Google Scholar] [CrossRef] [Green Version]
  4. Lee, J.H.; Oh, H.M.; Kim, M.Y. Deep learning based 3D defect detection system using photometric stereo illumination. In Proceedings of the 2019 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), Okinawa, Japan, 11–13 February 2019; pp. 484–487. [Google Scholar]
  5. Malge, P.; Nadaf, R. A survey: Automated visual pcb inspection algorithm. Int. J. Eng. Res. Technol. (IJERT) 2014, 3, 223–229. [Google Scholar]
  6. Putera, S.I.; Ibrahim, Z. Printed circuit board defect detection using mathematical morphology and MATLAB image processing tools. In Proceedings of the 2010 2nd International Conference on Education Technology and Computer, Shanghai, China, 22–24 June 2010; Volume 5, p. V5-359. [Google Scholar]
  7. Chang, P.C.; Chen, L.Y.; Fan, C.Y. A case-based evolutionary model for defect classification of printed circuit board images. J. Intell. Manuf. 2008, 19, 203–214. [Google Scholar] [CrossRef]
  8. Wei, P.; Liu, C.; Liu, M.; Gao, Y.; Liu, H. CNN-based reference comparison method for classifying bare PCB defects. J. Eng. 2018, 2018, 1528–1533. [Google Scholar] [CrossRef]
  9. Abd Al Rahman, M.; Mousavi, A. A review and analysis of automatic optical inspection and quality monitoring methods in electronics industry. IEEE Access 2020, 8, 183192–183271. [Google Scholar]
  10. Chen, J.; Ran, X. Deep learning with edge computing: A review. Proc. IEEE 2019, 107, 1655–1674. [Google Scholar] [CrossRef]
  11. Huang, R.; Gu, J.; Sun, X.; Hou, Y.; Uddin, S. A rapid recognition method for electronic components based on the improved YOLO-V3 network. Electronics 2019, 8, 825. [Google Scholar] [CrossRef] [Green Version]
  12. Johnson, J.M.; Khoshgoftaar, T.M. Survey on deep learning with class imbalance. J. Big Data 2019, 6, 1–54. [Google Scholar] [CrossRef] [Green Version]
  13. Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar]
  14. Lin, T.Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft coco: Common objects in context. In Proceedings of the Computer Vision—ECCV 2014: 13th European Conference, Zurich, Switzerland, 6–12 September 2014; Springer: Berlin/Heidelberg, Germany, 2014; pp. 740–755. [Google Scholar]
  15. Kim, J.; Ko, J.; Choi, H.; Kim, H. Printed circuit board defect detection using deep learning via a skip-connected convolutional autoencoder. Sensors 2021, 21, 4968. [Google Scholar] [CrossRef]
  16. Liao, X.; Lv, S.; Li, D.; Luo, Y.; Zhu, Z.; Jiang, C. YOLOv4-MN3 for PCB Surface Defect Detection. Appl. Sci. 2021, 11, 11701. [Google Scholar] [CrossRef]
  17. Ojala, T.; Pietikäinen, M.; Harwood, D. A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 1996, 29, 51–59. [Google Scholar] [CrossRef]
  18. Dalal, N.; Triggs, B. Histograms of oriented gradients for human detection. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA, 20–26 June 2005; Volume 1, pp. 886–893. [Google Scholar]
  19. Ho, T.K. Random decision forests. In Proceedings of the 3rd International Conference on Document Analysis and Recognition, Montreal, QC, Canada, 14–16 August 1995; Volume 1, pp. 278–282. [Google Scholar]
  20. Hearst, M.A.; Dumais, S.T.; Osuna, E.; Platt, J.; Scholkopf, B. Support vector machines. IEEE Intell. Syst. Their Appl. 1998, 13, 18–28. [Google Scholar] [CrossRef] [Green Version]
  21. LeCun, Y.; Jackel, L.; Bottou, L.; Brunot, A.; Cortes, C.; Denker, J.; Drucker, H.; Guyon, I.; Muller, U.; Sackinger, E.; et al. Comparison of learning algorithms for handwritten digit recognition. In Proceedings of the International Conference on Artificial Neural Networks, Perth, Australia, 27 November–1 December 1995; Volume 60, pp. 53–60. [Google Scholar]
  22. Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1–9. [Google Scholar]
  23. Dave, N.; Tambade, V.; Pandhare, B.; Saurav, S. PCB defect detection using image processing and embedded system. Int. Res. J. Eng. Technol. (IRJET) 2016, 3, 1897–1901. [Google Scholar]
  24. Malaca, P.; Rocha, L.F.; Gomes, D.; Silva, J.; Veiga, G. Online inspection system based on machine learning techniques: Real case study of fabric textures classification for the automotive industry. J. Intell. Manuf. 2019, 30, 351–361. [Google Scholar] [CrossRef]
  25. Takada, Y.; Shiina, T.; Usami, H.; Iwahori, Y.; Bhuyan, M. Defect detection and classification of electronic circuit boards using keypoint extraction and CNN features. In Proceedings of the Ninth International Conferences on Pervasive Patterns and Applications Defect, Athens, Greece, 19–23 February 2017; Volume 100, pp. 113–116. [Google Scholar]
  26. Ren, R.; Hung, T.; Tan, K.C. A generic deep-learning-based approach for automated surface inspection. IEEE Trans. Cybern. 2017, 48, 929–940. [Google Scholar] [CrossRef]
  27. Adibhatla, V.A.; Chih, H.C.; Hsu, C.C.; Cheng, J.; Abbod, M.F.; Shieh, J.S. Applying deep learning to defect detection in printed circuit boards via a newest model of you-only-look-once. Math. Biosci. Eng. 2021, 18, 4411–4428. [Google Scholar] [CrossRef]
  28. Lian, J.; Wang, L.; Liu, T.; Ding, X.; Yu, Z. Automatic visual inspection for printed circuit board via novel Mask R-CNN in smart city applications. Sustain. Energy Technol. Assess. 2021, 44, 101032. [Google Scholar] [CrossRef]
  29. Huang, W.; Wei, P. A PCB dataset for defects detection and classification. arXiv 2019, arXiv:1901.08204. [Google Scholar]
  30. Ding, R.; Dai, L.; Li, G.; Liu, H. TDD-net: A tiny defect detection network for printed circuit boards. CAAI Trans. Intell. Technol. 2019, 4, 110–116. [Google Scholar] [CrossRef]
  31. Lu, H.; Mehta, D.; Paradis, O.; Asadizanjani, N.; Tehranipoor, M.; Woodard, D.L. Fics-pcb: A multi-modal image dataset for automated printed circuit board visual inspection. Cryptol. ePrint Arch. 2020. Available online: https://eprint.iacr.org/2020/366 (accessed on 1 February 2023).
  32. Pramerdorfer, C.; Kampel, M. A dataset for computer-vision-based PCB analysis. In Proceedings of the 2015 14th IAPR International Conference on Machine Vision Applications (MVA), Tokyo, Japan, 18–22 May 2015; pp. 378–381. [Google Scholar]
  33. Mahalingam, G.; Gay, K.M.; Ricanek, K. Pcb-metal: A pcb image dataset for advanced computer vision machine learning component analysis. In Proceedings of the 2019 16th International Conference on Machine Vision Applications (MVA), Tokyo, Japan, 27–31 May 2019; pp. 1–5. [Google Scholar]
  34. Liu, T.; Bao, J.; Wang, J.; Wang, J. Deep learning for industrial image: Challenges, methods for enriching the sample space and restricting the hypothesis space, and possible issue. Int. J. Comput. Integr. Manuf. 2021, 35, 1077–1106. [Google Scholar] [CrossRef]
  35. Hendrycks, D.; Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. arXiv 2019, arXiv:1903.12261. [Google Scholar]
  36. Huang, S.; Papernot, N.; Goodfellow, I.; Duan, Y.; Abbeel, P. Adversarial attacks on neural network policies. arXiv 2017, arXiv:1702.02284. [Google Scholar]
  37. Bergmann, P.; Fauser, M.; Sattlegger, D.; Steger, C. MVTec AD—A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection. In Proceedings of the the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
  38. Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 618–626. [Google Scholar]
  39. Wu, W.Y.; Wang, M.J.J.; Liu, C.M. Automated inspection of printed circuit boards through machine vision. Comput. Ind. 1996, 28, 103–111. [Google Scholar] [CrossRef]
  40. Kaur, B.; Kaur, G.; Kaur, A. Detection and classification of printed circuit board defects using image subtraction method. In Proceedings of the 2014 Recent Advances in Engineering and Computational Sciences (RAECS), Chandigarh, India, 6–8 March 2014; pp. 1–5. [Google Scholar]
  41. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778. [Google Scholar]
  42. Wang, C.Y.; Bochkovskiy, A.; Liao, H.Y.M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv 2022, arXiv:2207.02696. [Google Scholar]
  43. Jones, C.T. Programmable Logic Controllers: The Complete Guide to the Technology; Brilliant-Training: Kepulauan Riau, Indonesia, 1998. [Google Scholar]
  44. Saenz de Ugarte, B.; Artiba, A.; Pellerin, R. Manufacturing execution system–a literature review. Prod. Plan. Control 2009, 20, 525–539. [Google Scholar] [CrossRef]
Sensors 23 02766 g001 550
Figure 1. Results and issues of applying deep learning model for PCB defect detection. (a) Example of PCB defect detection. (b) Issues of applying a deep learning model for PCB manufacturing.
Figure 1. Results and issues of applying deep learning model for PCB defect detection. (a) Example of PCB defect detection. (b) Issues of applying a deep learning model for PCB manufacturing.
Sensors 23 02766 g001
Sensors 23 02766 g002 550
Figure 2. Distributions of TDD-PCB dataset [30]. (a) Imbalanced number of non-defective and defective samples. (b) Overlapped distributions of circuit-related defects (mousebite, open, short, spur, and spurious), whereas the distribution of a part-related defect (missing hole) is distinguishable.
Figure 2. Distributions of TDD-PCB dataset [30]. (a) Imbalanced number of non-defective and defective samples. (b) Overlapped distributions of circuit-related defects (mousebite, open, short, spur, and spurious), whereas the distribution of a part-related defect (missing hole) is distinguishable.
Sensors 23 02766 g002
Sensors 23 02766 g003 550
Figure 3. Example of deformation due to PCB image distortion.
Figure 3. Example of deformation due to PCB image distortion.
Sensors 23 02766 g003
Sensors 23 02766 g004 550
Figure 4. Performance graph of each methodology according to training data volume.
Figure 4. Performance graph of each methodology according to training data volume.
Sensors 23 02766 g004
Sensors 23 02766 g005 550
Figure 5. Defect detection model performance graphs due to various image contamination types. Best viewed in color.
Figure 5. Defect detection model performance graphs due to various image contamination types. Best viewed in color.
Sensors 23 02766 g005
Sensors 23 02766 g006 550
Figure 6. Examples of Grad-CAM [38] according to PCB circuit image understanding results.
Figure 6. Examples of Grad-CAM [38] according to PCB circuit image understanding results.
Sensors 23 02766 g006
Sensors 23 02766 g007 550
Figure 7. Examples of direct defect detection results under several types of contamination. Bounding boxes with red denote false negatives, blue denotes true positives, and yellow denotes false positives.
Figure 7. Examples of direct defect detection results under several types of contamination. Bounding boxes with red denote false negatives, blue denotes true positives, and yellow denotes false positives.
Sensors 23 02766 g007
Table
Table 1. Summary of public PCB datasets.
Table 1. Summary of public PCB datasets.
DatasetImagesCropped ImagesTypes of Defects# of DefectsTypes of Positives
PCB [29]10690Sensors 23 02766 i0012070Any circuits or parts
except for defects
TDD-PCB [30]1010,66821,336
FICS-PCB [31]31400NoneNoneIC Chip (3243)
Capacitors (36,639)
Resistors (33,182)
Inductors (1292)
Transistors (1398)
Diodes (1593)
PCB DSLR [32]165849NoneNoneIC Chip (9313)
PCB-METAL [33]123984NoneNoneIC Chip (5844)
Capacitors (3175)
Resistors (2670)
Inductors (542)
Table
Table 2. Attributes of industrial data. We referred to [34] and summarized the primary attributes and their details.
Table 2. Attributes of industrial data. We referred to [34] and summarized the primary attributes and their details.
AttributeDescriptions
Small amount
–
The amount of data collected is small.
–
Collecting industrial data usually takes a long time and many steps.
Small object
–
PCB includes small and sophisticated electronic parts and circuits.
–
Close-up shots are required for detailed inspection of the individual small parts.
Imbalance
–
The volume difference between non-defective and defective data can be huge.
In general, it is more difficult to obtain data on defective parts.
–
Volume differences can also be large between defective parts.
Fine-grained
–
PCB includes a variety of similar parts.
–
In addition, each part is further subdivided into various parts.
Strong interference
–
Industrial process involves physicochemical reactions.
–
Considerable dust, steam particles, and vibrations can occur during manufacturing.
–
All the factors interfere with the clear imaging.
Temporality
–
Industrial process consists of numerous sequential processes.
–
Long maintenance varies from the initial setup of the manufacturing facility.
Table
Table 3. Possible image contamination due to four factors of industrial image acquisition environment.
Table 3. Possible image contamination due to four factors of industrial image acquisition environment.
FactorPossible Contamination
Close-up imaging• Defocus blur, Motion blur, Zoom blur, Gaussian blur
• Elastic transform
Illumination changes• Gaussian noise, Shot noise, Impulse noise, Speckle noise
• Brightness • Contrast change • Saturate
Long-term maintenance• Glass blur • Spatter
Systematic issues• JPEG compression • Pixelation
Table
Table 4. Summary of PCB defect detection methods.
Table 4. Summary of PCB defect detection methods.
Part Image ClassificationWhole Image UnderstandingDirect Defect Detection
Training data
–
Cropped images
–
Annotations: images
–
Whole PCB images
–
Annotations: images
–
Whole PCB images
–
Annotations: images, defect positions, and sizes
Test dataCropped part imagePCB imagePCB image
Model predictionClass of the part imageClass of PCB imageDefect location and class
Result examplesSensors 23 02766 i002Sensors 23 02766 i003Sensors 23 02766 i004
Table
Table 5. Accuracy of three methodologies for PCB defect detection according to the amount of training data. Performance between 90% and 80% is in blue, 80–50% is in magenta, 50–20% is in red, and below 20% is in bold red. Best viewed in color.
Table 5. Accuracy of three methodologies for PCB defect detection according to the amount of training data. Performance between 90% and 80% is in blue, 80–50% is in magenta, 50–20% is in red, and below 20% is in bold red. Best viewed in color.
Accuracy of Part Image Classification (Model: ResNet50 [41])
Data RatioMissingholeMousebiteOpenShortSpurSpuriousNon-DefectiveAvg.
80/200.9960.9720.9930.9820.9620.9970.9980.988
50/500.9620.9800.9940.9880.9690.9990.9610.975
20/800.9970.9830.9800.9720.9760.9630.9640.974
10/900.9880.9720.9790.9280.9620.9620.9950.975
5/950.9790.9800.9660.9400.9320.9670.9830.968
Accuracy of Whole Image Understanding (Model: ResNet50 [41])
Data RatioMissingholeMousebiteOpenShortSpurSpuriousNon-DefectiveAvg.
80/200.9940.9940.9940.9880.9650.960-0.983
50/500.9910.4930.3790.7890.2850.241-0.532
20/800.9890.2580.1920.2220.2050.233-0.354
10/900.9700.1590.2290.2370.2620.190-0.344
5/950.4110.1910.1350.1670.2420.088-0.207
Mean Average Precision (mAP) of Direct Defect Detection (Model: YOLOv7 [42])
Data RatioMissingholeMousebiteOpenShortSpurSpuriousNon-DefectiveAvg.
80/200.9810.9920.9840.9330.9500.981-0.970
50/500.9900.9770.9790.8900.9420.973-0.958
20/800.9720.9160.9270.8670.8340.917-0.906
10/900.9800.9350.9300.8710.8590.935-0.918
5/950.9760.9120.8740.8300.7770.852-0.870
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Park, J.-H.; Kim, Y.-S.; Seo, H.; Cho, Y.-J. Analysis of Training Deep Learning Models for PCB Defect Detection. Sensors 2023, 23, 2766. https://doi.org/10.3390/s23052766

AMA Style

Park J-H, Kim Y-S, Seo H, Cho Y-J. Analysis of Training Deep Learning Models for PCB Defect Detection. Sensors. 2023; 23(5):2766. https://doi.org/10.3390/s23052766

Chicago/Turabian Style

Park, Joon-Hyung, Yeong-Seok Kim, Hwi Seo, and Yeong-Jun Cho. 2023. "Analysis of Training Deep Learning Models for PCB Defect Detection" Sensors 23, no. 5: 2766. https://doi.org/10.3390/s23052766

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop