Automatic visual inspection: An approach with multi-instance learning

Highlights

• An approach for quality inspection with multi-instance learning is proposed.

• Using weakly labeled images reduces the labeling effort in quality inspection.

• Experiments show that the approach can be effectively used in real-world applications.

Abstract

One of the industrial applications of computer vision is automatic visual inspection. In the last decade, standard supervised learning methods have been used to detect defects in different kind of products. These methods are trained with a set of images where every image has to be manually segmented and labeled by experts in the application domain. These manual segmentations require a large amount of high quality delineations (on pixels), which can be time consuming and often a difficult task. Multi-instance learning (MIL), in contrast to standard supervised classifiers, avoids this task and can, therefore, be trained with weakly labeled images. In this paper, we propose an approach for the automatic visual inspection that uses MIL for defect detection. The approach has been tested with data from three artificial benchmark datasets and three real-world industrial scenarios: inspection of artificial teeth, weld defect detection and fishbone detection. Results show that the proposed approach can be used with weakly labeled images for defect detection on automatic visual inspection systems. This approach is able to increase the area under the receiver-operating characteristic curve (AUC) up to 6.3% compared with the naïve MIL approach of propagating the bag labels.

Introduction

Inspection is the process of determining whether a product deviates from a given set of specifications [1]. Traditionally, the inspection has been done by human operators; however, machine vision is being used to automate this process [1], [2], [3], [4]. An automatic inspection offers many advantages, such as an increased productivity, high standards of product quality in mass production, and the elimination of human errors [2], [5]. An automatic visual inspection system could be developed by one of the following three approaches:

• Template-matching, where an image of the object under inspection is compared with one or more reference images [6].

• The design-rule verification approach, that checks for the violation of a set of generic rules [6].

• Inspection based on machine learning techniques, where the inspection system learns and generalizes the relations between the object features and the defects found in it [7].

The first two approaches have the disadvantage of requiring an explicit definition of rules and thresholds [8]. It leads to trial and error process that could take a long time before the system is ready to perform a specific inspection task. To deal with these issues, the use of machine learning techniques attempts to assimilate the reasoning skills of human inspectors [9].

Many systems have been proposed in the literature for automatic visual inspection at different domains, for example, for the quality control assurance in semiconductor [10], food [9], paper [11], fabric [12], metal [13] and ceramic tiles [14] industries, among others. Most of these systems use different standard supervised learning algorithms, such as decision trees [15], statistical classifiers [16], artificial neural networks [17], and support vector machines [18]. In order to learn to recognize defects in the products to be inspected, these algorithms should be trained with a dataset of images labeled by human operators. The decisions that the operators make about the defects will inevitably be different for some of the products to be inspected due to the subjectivity of the process (inter and intra operator contradictions) [19]. In this way, it is difficult to accurately and consistently assign labels to input images. Even in the industry, it is common having only weakly labeled images where the absence or presence of defects in an image is known, but the location of the defect and its precise delimitation are not available. In fact, this happens because manual annotation of the defects is time consuming, laborious or even impracticable in industrial applications.

We consider a relatively new learning paradigm called multi-instance learning (MIL) [20] applied to the automatic visual inspection process for defect detection. MIL allows to train a classifier with data that is assumed to have some ambiguity in how the labels are assigned, as occurs in visual inspection applications. The basic idea of this learning paradigm is that an object is represented by a bag, which is a set of feature vectors called instances, and the objective is to classify the bag as either positive or negative, in a two-class problem. MIL has been used in several computer vision applications, such as image categorization [21], image retrieval [22], object recognition [23] and target tracking [24], among others.

In contrast to other existing approaches, we propose to model the defect detection problem as a MIL problem, where the images of the objects to be inspected are taken as bags and their potential defects as instances. We tested the proposed approach with three artificial benchmark datasets from the inspection of textured materials and three real-world datasets from the artificial teeth industry, the welding industry and the fish fillet industry. The structure of the remaining parts of the paper is as follows: Section 2 presents the MIL paradigm. Section 3 introduces the proposed approach. Experimental results and their discussion are presented in Section 4. Section 5 depicts our conclusions.