Real-Time System for Automatic Cold Strip Surface Defect Detection

Detection and classification of surface defects of the rolled metal are one of the main tasks for correctly assessing product quality. Historically, these tasks were performed by a human. However, due to a multitude of production factors, such as high rolling rate and temperature of the metal, the results of such human work are rather low. Replacing a human controller with an artificial intelligence system has been relevant for a long time; it is merely necessary within the concept of Industry 4.0. This paper is devoted to the development of the prototype system automatic detection and classification of defects for one of the Iron-and-Steel Works of the Chelyabinsk region in the Russian Federation. The prototype consists of the Preprocessor, Classifier, Server, Database, and User interface. The main focus is on achieving high classification accuracy, which is planned to be obtained through the use of convolutional neural networks.


INTRODUCTION
Product quality control is an essential step in metallurgical plants because quality assurance means the compliance of mechanical and other properties with specific values.However, when it comes to metalrolling, the condition of surface or coating of rolling strips creates a separate important quality factor, which also needs to be controlled.
Surface control is usually carried out visually (by a human) at the final stages of production.However, even here, the rolling rate is very high, and there is minimal time to carry out control.At the end of the process, the metal strip is rolled up into a roll.After that, it becomes even harder to detect the defect.Often this leads to shipment of low-quality metal to the consumer, which further entails significant financial losses for the manufacturer.
Thus, the need for metallurgical enterprises for automation systems for detecting and classifying surface defects is very high.However, despite this high need, there are very few developers of such systems (Codnex, EES, Matra, Parsytec, Siemens-VAI, and Sipar).At the same time, existing systems perform defect detection well, but still, have difficulty with their classification.The challenge is since steel surfaces rolled on different rolling mills even of the same plant often differ in appearance.It leads to the complication of the algorithms for detecting and classifying surface defects [1].Also, the complexity is due to a large number of varieties of surface defects.Characteristics and classification of defects vary for each specific production and rolling mill and are not regulated by the generally accepted standard.Even small changes in the production process can cause the emergence of new types of defects [4].
The architecture and detailed specifications of commercial vision-based steel surface inspection systems are not publicly available.However, there are many research projects on this topic.In [5], an example was described using the algorithm for feature extraction, created without using machine learning in combination with a neural network for classification.In [6], a similar approach is used to extract features, but the neural network is used to detect defects.The paper [7] presents the architecture of the entire system, including the hardware part.In [8], a convolutional neural network was used, showing a high percentage of efficiency, similar to the last work.The process of detecting and classifying defects of the surface of metal strips usually occurs in three stages [9]: • finding regions of interest (ROIs) by segmentation; • extracting features from the regions of interest; • defects classification based on the detected features.In Industry 4.0, the quality assurance of the products produced at the metallurgical plants will be paramount, because otherwise, it will be necessary to introduce systems for detecting defects in input materials at each plant using metal products.According to the concept of Industry 4.0, metallurgical production will be represented by a cyber-physical system [10].Also, for this, it is necessary that the cyber-physical system can independently control the quality of the metal and recognise founded defect.It will help to assess the need to replace the failing mechanisms involved in the production or to decide whether to slow down or completely stop the production line.In a modern smart enterprise, the introduction of basic automation will not be enough, because new defects may appear in production, for which new decisions will have to be made.This work is devoted to the development of a prototype system for automatic detection and classification of defects, which can be applied as part of the cyber-physical system within the framework of The Fourth Industrial Revolution [11].The main goal of the research is to achieve high accuracy of defect classification and the possibility of automatic further training of the artificial neural network using cloud technologies.
In Figure 1, there is a diagram of the Automatic Defect Recognition System (ADRS).The rectangles denote modules.The arrows indicate the interaction of the modules.
The preprocessor is responsible for obtaining images from video cameras, binding images to physical coordinates, splitting images into parts, finding ROIs and sending them to the classification module.The classifier is responsible for image classification, preparation of training data, and neural network training.The user interface is responsible for managing the system.The database stores system data.The server is responsible for dataflow managing, API providing and business logic.

HARDWARE
ADRS will automatically detect and classify visible defects on a coated metal tape in real time before rolling.ADRS will promptly notify the operator about the appearance of the defect, its class and location, as well as record the defects in the roll passport.The system will reduce the number of unnoticed defects to almost zero.
The installation of the ADRS will include the following main components: • Hardware system for obtaining images: a system of cameras and related equipment that captures an image of moving metal tape and transfers it to the next ADRS subsystems; • Hardware-software and hardware system of image preprocessing: a system that performs process and reduction images obtained from cameras to the necessary form (determination of tape edges, normalisation); • Hardware-software system for recognition and classification of metal-roll defects: a system that searches for defects on images received from the preprocessing system and, classifies them; • Dispatcher's automated workplace: operator's workplace with developed software for managing ADRS and monitoring the current state of the tape and roll as a whole.

Camera connection interface selection
Gigabit Ethernet, USB 3.0 Vision, and Camera Link interfaces are usually used for connection of computer vision cameras.The PCI bus is used to connect controllers of corresponding interfaces.The main problem of camera connection is the necessity to have dozens of high-speed ports on a server.Extension cards are used for this purpose.One PCI Express 3.0 4x extension card can have either up to 4 USB 3.0 ports, that work in parallel and does not share its bandwidth or only one Camera Link port.In case of using Gigabit Ethernet cameras, one dual port 10 Gigabit Ethernet card with a standard network switch in enough.Camera connection interfaces comparison is presented in Table 1.

Camera Requirements Calculation
The developed prototype is focused on the use of area scan cameras, as, according to the authors, commercial steel surface inspection systems of high resolution, working with small defects (0.5 mm square) and high speed of the strip (7 m/s), are built on their basis.Line scan cameras are usually used only as an auxiliary part of such systems to search for specific types of defects.Input parameters for calculating the parameters of cameras and the maximum speed of the tape is presented in Table 2: • the minimum size of a visible defect by width and height (mm) -min_defect_size; • the minimum number of pixels by width and height required for confident detection of a minimum defect -min_defect_pixels; • parameters of a metal strip -width (mm); • speed (m/s); • percentage of an image intersection from neighbours' cameras -overlap; • video camera parameters -width x and height y of a matrix in pixels.The maximum physical size of a rectangle (x×y mm) that can be captured by a single camera on a metal strip with sufficient resolution to recognise minimal defect are calculated by formulas ( _ _ _ _ x min defect size X min defect pixels (1) and ( _ _ _ _ y min defect size Y min defect pixels x min defect size X min defect pixels Minimum required frame rate of the camera at a known speed of the metal line is calculated by the formula (3): Minimum required frame rate of the camera at a known speed of the metal line is calculated by the formula (4): Basler® monochrome infrared cameras listed in Table 3 were considered as an example.Monochrome cameras have been used to reduce the amount of data transmitted from cameras and to speed up preprocessing algorithms, as these algorithms typically use monochrome images.The average cost of a lens that completes a camera set is $600.

ADRS installation diagram
Main hardware components of ADRS for one stand will be: • framework for placing cameras at the top and bottom of the metal tape -1 pc; • sealed protective covers with an integrated cooling system to protect the camera complex -2 pcs; • industrial video cameras -16 pcs.; • IR flash illumination system -1 set; • switching system -1 set; • video camera synchronisation system -1 set; • speed sensor of the metal tape -1 pc.; • roll end sensor -1 pc.; • server with GPU for preprocessing images -1 pc.; • server with GPU for the recognition and classification of the defects on metal -1 pc.; • data storage system -1 pc.; • uninterruptible power supply -1 pc.; • server rack -1 pc.; • cooling system; • operator's AWP based on PC with monitor -1 set.The diagram of ADRS in presented in Figure 2. The diagram of ADRS.The base of the communication network will be network switches with 12 Gigabit Ethernet (GE) ports with Power over Ethernet (PoE) support and 10 Gigabit Ethernet port (10 GE) to connect to the image pre-processing server.The cameras will be connected via the GE interface directly to the network switch.The mechanical features of the installation cause the use of two switches instead of one with 24 ports.The image preprocessing server will be connected to the switches by two 10 GE interfaces.Switching between servers and storage will be performed using a conventional 16-port Gigabit Ethernet switch with port aggregation because the image preprocessing server will reduce the number of images coming from cameras by removing frames that do not contain defects and normalising frames sent to the server for defect classification to determine a class of the defect.
The look of the rolls may differ from one another.Therefore, ADRS has a subsystem for adapting to such deviations, and can also control lighting automatically.The backlight modules use up their resource with time, and their brightness decreases.The system calculates the optimal operation mode of the backlight modules based on the constant analysis of images received from cameras.The ADRS system uses the LED backlight that works in the infrared range.Due to the installation of several separate LED modules, the system has high maintainability.

IMAGE PREPROCESSING UNIT
Preprocessor unit processes image dataflow from cameras and split it into pieces.Then every piece is checked with computer vision algorithms.Pieces that have a defect are then sent to the defect classification module.

Image stitching and blending
The most-known computer vision library nowadays is OpenCVError!Reference source not found..
Because of the width of a metal sheet, one camera cannot get a picture of a surface with sufficient resolution for spotting defects.Because of this, several parallel cameras are used in stand prototype.Images from all cameras need to be united in one panoramic to ease monitor faults detection process for the operator and to get one complete image of a defect, captured from two and more cameras.
An algorithm for creating a panorama usually includes a stage of finding special points and matching pictures with it.This stage can be skipped because critical points that are not defects are challenging to find on a metal sheet.Besides, the cameras' offset is fixed and known.
Next stage for creating a panoramic image is image stitching and brightness equalisation.A gradient-mixing [12] method was chosen for stitching because of its simplicity in implementation and more hidden stitch than in alpha-channel approach.

Defect Detection
It is necessary to locate regions on an image that stands out from the background since it might be defects.
The input is a grayscale image.Sobel filter is then implemented [13] for finding gradients along both axes.After that, a binary image is created [14].At this stage, regions that alter along both axes are highlighted with white colour.Next, morphological opening [15][15] is used to enhance the results, and both of the images overlap.Then morphological closure takes place, and contours of white zones are located and filtered by area, relative to the minimum possible area of the defect.

Design
After the analysis, the next functional requirements were distinguished.The system must: • locate visible defects on the surface of a metal sheet; • provide API for getting images of located defects; • stitch several images to one; • get images from several cameras with a particular frequency; • be able to output image to debugging interface; • provide UI for setting parameters of image stitching; • split input images into equal-sized parts; • be capable of highlighting visible defects on images visually.

The computer vision stand prototype
The computer vision stands prototype was built in «Supercomputer modelling» laboratory of South Ural State University.Stand includes two cameras, fixed on a static frame, pointed down and connected to a computer.All elements are immobile.The test-bed prototype diagram is presented in Figure 3.
The operator is a user who interacts with the system.Defect classification system represents a program module that gets data from the image preprocessor module.The operator can watch a selected type of video stream -combined or from one camera.In the first case, images from all the cameras are united in one piece.In the second, image if shown form a single camera.
The operator can also calibrate cameras, i.e., change the parameters of image stitching.The defect classification system can also get a region of interest, i.e., get an image with a defect that was obtained from the image preprocessing system.

Module implementation
The preprocessing module is a part of the system of automatic visible defects recognition on technological lines.This module gets the video stream from cameras with a specific frequency and separates it into equalsized parts.Then, every part is checked on defects with computer vision algorithm.Parts of images where the defect is located are then sent to the defect classification module.Also, for debugging purposes, the module supports the stitching of images and highlighting defects on the image.

Description of system components
Next classes were created to store data: • ImagePart -an object that keeps metadata and part of the input image; • Config -an object that holds values from the configuration file.Classes that were created to control data: • Preprocessor -the main class that initialises the rest of the module's classes; • Splitter -splits an image into equal-sized parts; • Detector -detects defects on an image; • Blender -class that combines several images into one.For system interaction with external systems, next interfaces were implemented: • Streamer -a class that connects the system to cameras and gets images; • Sender -a class that sends image parts with metadata into defect classification module.

System architecture implementation
Diagram of a main preprocessing cycle activity is presented in Figure 4.
Input image splits into equal-sized parts.The size is determined by the input data format of a neural network in the defect classification module.The meaning of the defect detection algorithm is visible defects has bounds, on which pixel brightness of a grayscale image changes dramatically.A defect then might be distinguished completely after finding its borders by Sobel filer along both axes.

Figure 5. Activity diagram of the defect detection algorithm
If the array of received bounds is not empty, a part of an image, on which defects were located, marked as «has_defects» and sends to defect classification module.This array might also be used for defect detection in debugging purposes (see Figure 6).

Computational experiments
Computational experiments were conducted for defect detection algorithm on images of defects on hot rolled metal surfaces from Northeastern University surface defect database, USA.
After conduction of experiments, there were found coefficients of the algorithm that allows locating the highest number of defects.

DEFECTS CLASSIFICATION BASED ON A NEURAL NETWORK
The main tasks of the classifier are defect classification, preparation of training dataset for the neural network, and its training.

Classification approaches
Support vector machine and neural networks are among the most known classification approaches.Also, knearest-neighbours and fuzzy-logic are used in a number of tasks [9], [16].Support Vector Machine is a binary classification approach.It is used to distinguish two similar classes [17], [18].The approach was improved to identify more classes, also known as multiclass classification [19], [20].In this method, a few binary classifications used to identify every class.More generally it requires N classification (where N is the number of classes) to distinguish one class from the [21]The method is described in [21], but there are some cons, the most significant of which is resource consumption in terms of hardware requirements.
Neural Networks is a universal approach in machine learning.It is widely used in image processing tasks, like classification, detection, segmentation (both semantic and instance).Its properties are flexible and suitable for the solution of a large class of problems [22], which includes defects classification [5], [23].It is enough to use one-hidden-layer networks to classify defects, with input-layer size, hidden-layer size, and output-layer size parameters.Application of neural network in terms of defect on a metal surface is popular because of its capability to distinguish a lot of features [5], [23].This approach is used for its performance and efficiency.
Northeastern University provides a dataset for defects classification and detection problems [24].There are six different classes of most typical defects in metal manufacturing: rolled-in scale, patches, crazing, pitted surface, inclusion, and scratches.Each image is represented in black and white, has the size of 200x200, and every class has 300 images.

System requirements
The system should have the following requirements: • accuracy of classification at least 95%; • variable size of existing classes; • retraining on the updated dataset.

Neural network topology
Yann LeCun's LeNet-5 topology has been taken for a basis [25].Improvements were applied.Activation function.ReLU for its properties replaces sigmoid: ReLU significantly increases the rate of con-vergence of stochastic gradient descent in comparison with the sigmoid [26].
Neural Networks topology is presented in Figure 7Error!Reference source not found.;there are convolutional and max-pooling layers responsible for feature extraction.The origin 200x200x3 tensor image transforms into 6x6x128 (128 maps of 6x6 size).Then classification part follows of 295366 learning parameters.It consists of a flatten layer, which allows representing maps as a vector of features, fully-connected layers, dropout layer, and one more fully-connected layer.
Topology and accuracy were achieved after some iterative process.Many topologies with different parameters were compared (according to the accuracy), and the best are selected.This process stopped when one topology out of two had been chosen.

Dataset processing
There are 1800 marked images in dataset divided into several classes of defects.This quantity is insufficient to obtain high accuracy of predictions.It means that data should be augmented to prevent overfitting.Image Data Generator module based on Keras framework was provided to achieve this goal.Abstract class DataPreparer is the base for data processing before learning.We provide NeuDataPreparer based on DataPreparer to prepare dataset from Northeastern University.Normalisation (see Figure 8) brings brightness and brightness spread in the input images into one range by Z-Score function [27].

Neural network programming representation
The Sequential model represented in Keras is a linear stack of layers.Sequential was used as a container for the neural network implementation.Details of implementation are described below.
Dropout layer was used with a 0.5 coefficient to prevent overfitting.Usually, it also provides better learning efficiency and final score [28].Result layer is represented as a vector of probabilities of some class and computed by Softmax function [29].
During computational experiments, Adam optimiser [30] showed the best score.
Categorical Cross-Entropy was used as loss function because it applies to multiclass classification.
Keras provides different tools to manipulate the parameters of a neural network.The most convenient way to make a sequential model more accurate is callback functions.It gives an ability to manage the state of the neural network on each epoch of learning.The following functions were provided: • Check Point is a function which allows saving the best parameters during the learning process (according to the specific metric).The metric used in neural networks is accuracy on the validation set (val acc).Callback function would save the parameters of a neural network if it scored better accuracy than a score on the previous step.

Programming implementation
TensorFlow as backend and Keras as the main framework were used for the implementation of the neural network.
We aggregated all requirements in one module named SteelDefectClassifier.The module provides the following functions: • __get_callbacks returns a list of callback functions; • __get_model returns the Sequential model of the neural network described above; • classify returns vector of probabilities of provided images; • load_model loads pre-trained network which provided as an argument; • fit_model allows training neural network on provided DataPreparer.Comparison of models is presented in Table 4. NVIDIA GeForce GTX 1080 Ti was used to train and test the neural network.

GRAPHICAL USER INTERFACE OF OPERA-TOR'S AUTOMATED WORKPLACE
One of the parts of ADRS is the graphical interface.It is designed to ensure the interaction of the operator with other parts of the system.The graphical interface consists of various windows, each of which serves for a specific purpose.

Description of the main windows of the graphical interface
The main window of the graphical interface of the operator's automated workplace is shown in Figure 9.
The window contains an image of the last detected defect, its description, a coil map, a list of produced coils and production information about the current coil.
The coil map demonstrates where the defects are located on the top and bottom sides of the metal sheet.Each defect class has a unique colour.Defects on a coil map are indicated by coloured dots, depending on their class.Information on the main window is updated automatically, but the operator can disable update on the image with the last detected defect and its description.
He can also disable an audible warning signal when a new defect appears.In the top part of the main window, there are 20 indicators of high-speed cameras and four indicators of other components of ADRS.Each indicator displays the status of its component using a different colour.It allows the operator to quickly respond to any malfunctions that may occur in the system during its operation.
The coil list window displays a list of produced coils and brief information about it.In this window, the operator can search a coil by various criteria and go to the window of the control protocol for this coil.
The coil control protocol window displays detailed production information about a coil, a list of defects, a coil map of a top and bottom sides.Production information contains an identifier, physical parameters, coil production time, number of defects, and their classes.The defect list contains all defects that were found on this coil and allows the operator to go to the defect report window.This window also contains buttons for printing and editing the coil control protocol.The defect report window contains a detailed description of the defect and its image.The description of the defect contains the physical parameters of the defect, class, and its coordinates on the coil.This window contains buttons for printing and editing a defect report.
The editor window allows the operator to edit information about a coil or detected defects and add a new defect.This is necessary if ADRS incorrectly classified or missed a defect.
The graphical interface allows the operator to get visual information about detected defects, statistics of their occurrence, and generate coil control protocols.The operator can decide on the expediency of stopping rolling mill and carrying out maintenance work based on this information.

The Functional Requirements for the GUI
The operator's automated workplace interface must provide the following main features: • promptly notify the operator about a new defect on the rolled steel coil; • display the image and information of detected defect; • display and print out the defect's statistics; • display the list of produced steel coils; • form and print out steel coil control protocols and defects reports; • display the current state of the steel coil; • display diagnostic information about the health of the hardware imaging system; • display diagnostic information about the health of other components of ADRS; • provide users management in the system; • edit information about steel coils and defects.

Implementation of GUI
The graphical interface was implemented in C# using Microsoft.NET Framework software platform.
To interact with other components of ADRS, the graphical interface of an operator's automated workplace has an API.The interaction is based on REST architecture.The data is transmitted in JSON format.Web sockets are used to display the status of unit and ADRS in real time.

CONCLUSIONS AND FUTURE WORK
ADRS system prototype was developed.It consists of the Preprocessor, Classifier, Server, Database, and User interface.Preliminary experiments that were carried out on the prototype showed the stability of the detection of defects and rather high accuracy of their classification -98.1% on test data with six classes of defects.High classification accuracy was achieved through the use of convolutional neural networks.Despite this, work is currently underway to collect and augment a training set for an artificial neural network to further improve the classification results.
Using the results of the paper, we will be able to automate the process of faulty components detection of the production line by having high accuracy of defect classification and information about the causes of certain types of defects.The automatic detection of faulty production line components is one more step towards Industry 4.0 and full automation of production.Therefore, further research will be related to this task.

Figure 1 .
Figure 1.Scheme of System of Automatic Defect Detection (ADRS)

Figure 2 .
Figure 2. The diagram of ADRS

Figure 4 .
Figure 4. Main preprocessing cycle activity diagram Activity diagram of the defect detection algorithm is presented in Figure 5.The meaning of the defect detection algorithm is visible defects has bounds, on which pixel brightness of a grayscale image changes dramatically.A defect then might be distinguished completely after finding its borders by Sobel filer along both axes.

Figure 6 .
Figure 6.Example of distinguishing and highlighting of defects

Figure 8 .
Figure 8. Original and normalized defect

Figure 9 .
Figure 9.The main window GUI of the operator's workstation

Table 3 . Cameras specification
• Early Stopping is a function which allows stopping learning if it did not achieve better score during the specified amount of epoch.• Reduce Learning Rate is a function which allows as to decay learning rate during the learning process.The global parameters of a neural network are: • steps per epoch are the number of steps before the end of one iteration during learning (epoch).