Digital Image Processing and Computer Graphics Automated Region of Interest Retrieval of Metallographic Images for Quality Classification in Industry

The aim of the research is the development and testing of new methods to classify the quality of metallographic samples of steels with high added value, for example, grades X70 according API. In this paper, we address the development of methods to classify the quality of slab samples images with the main emphasis on the quality of the image center called as a segregation area. For this reason, we introduce an alternative method for automated retrieval of the region of interest. In the first step, the metallographic image is segmented using both spectral method and thresholding. Then, the extracted macrostructure of the metallographic image is automatically analyzed by statistical methods. Finally, the automatically extracted region of interest is compared with the results of human experts. A practical experience with retrieval of non-homogeneous noised digital images in an industrial environment is discussed as well.


Introduction
The center area of continuously cast products as billets, blooms and slabs can be affected by several types of defects.This area can be called as a segregation area.During the steel solidification process, the liquid phase becomes more concentrated with segregated elements.This more and more concentrated liquid metal finally solidified in the last place of the solidification of the cast product.As a result of this process, the last place of the solidification is located at the centre area.For this reason, the majority of these steel defects are located at the centre area of the casting product.
There can be found a lot of parameters, which can influence the segregation area, for example, the casting speed, temperature and steel chemical composition.Segregation modeling is a complex procedure.For example, Beckermann provided a review with more than 150 references, including basic physical models, coupled with microstructure-macro segmentation models and direct numerical simulations [1].
Our research is focused on an automated analysis of the metallographic macrostructures, especially on the centre line segregation in continuously cast steel slabs [2] [3].Similarly to [4], our approach is based on image analysis.In the past, the research of methods for information retrieval in digital photographs of an industrial environment was carried out.The aim was to model the visual similarity of digital images in a metallographic laboratory.For this reason, digital images of metallographic samples were automatically retrieved and classified using wavelets and the Latent Semantic Indexing methods [5].A detailed description of the industrial technologies used, is given in [6].
Currently, we are dealing with methods of machine analysis of digital photographs in industrial metallography.The aim is to retrieve the region of interest, which contains hardly visible objects located at the central part of the analysed image sample.These small but important dots, which are only hardly seen in Fig. 2-5, represent the sample quality of the segregation area.
More precisely, for the center scoring quality estimation in metallography, it is interesting to retrieve a single sub-region of the image, leaving other regions unchanged.This can be referred to as the region of interest (ROI) retrieval [7], [8].For this reason, the problem of automated retrieval of the region of interest in real metallographic images from the steel plant of ArcelorMittal Ostrava a.s.(Ostrava, the Czech Republic) is addressed in this paper.The objective is to monitor the process quality in a steel plant.Thus, two different image segmentation algorithms are used and compared in this contribution.The first algorithm computes the histogram of grey levels of the image and estimates an ideal threshold for object recognition, while the second algorithm is based on spectral segmentation.Then, the extracted objects of the metallographic image sample are automatically analyzed by statistical methods.
T h e p a p e r i s s t r u c t u r e d a s f o l l o w s .S e c t i o n 2 describes the used image segmentation techniques.In Section 3, an automated method for the region of interest retrieval is presented.An experimental comparison of these two approaches for image segmentation follows in Section 4. In section 5, modifications of algorithms for real industrial use are presented.Finally, Section 6 closes the paper with final remarks.

Image Segmentation
In this paper we use a well known threshold based image segmentation technique in comparison to a spectral segmentation method [9].We present industrial images for which the threshold segmentation works well, whereas the full automatic alternative spectral segmentation fails.Moreover, we will see that, in our case, the threshold method is significantly faster than the spectral method.
The typical distribution of gray intensities represented by a histogram of an input metallographic digital image is seen on Fig. 1.It is easy to see that the choice of an optimal threshold T, i s c r u c i a l f o r a successful image segmentation.
An obvious choice for the threshold T would be one of the values marked by the red arrows in Fig. 1.Indeed, only the left arrow, which represents a color division between the "dark" and "gray" shade of black, is our preferred threshold level.This is due to the center scoring quality estimation being more sensitive to "darker" colors, as the important central defects are usually "darker" than the rest of the metallographic sample.In other words, this threshold selection is based on an a-priori known structure of the input image and on a histogram matching procedure as described in [10].
In contrast to the left arrow, the arrow on the right symbolizes a color division between the "gray" and "white" colors, which mainly correspond to the sample marking, which is not currently important in our central defects recognition task.In order to segment the image correctly, the morphological post-processing has to be done.Thus, the morphological post-processing operations called "open and fill" are necessary in order to eliminate the effects of the heterogeneity of the image background and in order to fill the image holes and close all extracted objects of the image sample.The segmented image sample, with and without post processing, is shown in Fig. 2. It is easy to see the post-processing operations successfully filter artificial objects located in the upper left image corner.On the other hand, Fig. 4 contains trouble free results of the same samples segmented via the threshold method.It is obvious that Sample A is segmented without any problems while Sample B has some minor defects, which are not important for the future analysis of the center area.
In our case, the threshold segmentation method seems to be superior in reference to the quality and the speed efficiency of the algorithm.The comparison of results, presented in Tab. 1, also indicates a possibility of the real-time image segmentation for the threshold based method.
All metallographic images here presented, are acquired by the Canon EOS 550D digital camera mounted on a tripod with a fixed distance to the metal sample.The average digital image size is roughly 5 MB.The picture resolution is around 4504Í3093, depending on an automatic crop software procedure.

Automated Region of Interest Retrieval
In this section we present a simple, but an effective method for the automated ROI detection.The method is based on an average value obtained from a moving window that moves down along the rows of an analyzed image.In this way, we find a one-dimensional vector that characterizes the rows of the colors of the image.An example of this "sample mean vector" is plotted in Fig. 6.
In order to achieve a successful localization of the ROI, there has to be done an intensity normalization described in Eq. ( 1), which enables a better dark-white contrast highlight as illustrated in Fig. 5: The window's height has been experimentally estimated to be 100 pixels.This window dimension enables an acceptably smooth graph of average values.
At the final step, we used an explicit knowledge of the problem structure: All false positive matches located too far from the image centre are automatically deleted.
As the region with central defects (dots) is already known, detected defects are compared with their probability of occurrence and are processed for a future evaluation.The dots are filtered: dots of insignificant size, that is, where the area of such dots is less than 4 pixels, are identified as a noise.The remaining significant dots are counted and measured.Especially, their size, position and area are estimated and stored in a database.Here, blue lines represent the detected regions of interest.

Results of the Automated Region of Interest Retrieval
As a result of the intermediate state of the ROI detection algorithm, there are false positive matches (see blue lines located close to peripheral areas of Fig. 7 and Fig. 9).
It is obvious that our needed ROI is located in the center of the image.As a result of the final step of the ROI algorithm, the false-positive hits are identified and deleted.Finally, the final ROI of both samples have been successfully detected (see red lines of Fig. 10).Because of space limitations, the detected region of interest of Sample A is not presented.
Although the metal sample B has some minor residue from the segmentation, this residue does not have a significant impact on the successful ROI localization.As a result of ROI, this final automatically extracted region of interest is consistent with an expert opinion.

Industrial Experience
The current version of the detection algorithm relies on the supplied image samples.Unfortunately, in the industrial environment, digital images of these metal samples were made in the past by two different digital cameras, as well as illuminated by two different light sources.For this reason, the region of interest detection algorithm did not seem to be too successful in real industrial conditions, because there was too much noise for a reliable estimation of the right position of the threshold in the detected region.The main complication was caused by a non-homogeneous brightness of the sample, which complicated the process of correctly setting the "threshold" values for all parts of the central region of interest.Consequently, different levels of brightness on the metal-sample increased the frequency of dark areas, which interfered with the visibility of the central segregation.Fortunately, this problem has been recently resolved by a new lighting system, which has been successfully developed and installed at the metallographic laboratory of the ArcelorMittal Ostrava plc.
Another modification of the industrial application is based on the removing of redundancy data from the detected central region of interest (see Fig. 8).The whole procedure is based on automated sub-region decomposition, as illustrated in Fig. 11.In order to attain a better signal-to-noise ratio, the proposed algorithm for removing redundancy data from the detected central region of interest can be summarized as follows: • An area of the same dimensions as a localized central region is selected in the lower and upper half of the sample.These areas are denoted in Fig. 9 as Lower and Upper reference windows.
• The histogram that represents the grayscale distribution of pixels contained in the window is computed for both reference windows.These two histograms are then averaged.
• The area with the highest frequency is identified in the average histogram (modus).
• Gray shades corresponding to the modus are removed from the histogram of the central window.In this way, the redundant information located across the whole metal sample is removed and the remaining central dots are exposed for extraction.
• The dots location algorithm based on thresholding is applied to the reduced histogram of the central window (see the thresholding algorithm).Our experiments indicate that these modifications seem to be robust enough for the analyzed industrial environment.

Conclusion
In this paper, we presented a comparison of two algorithms for the automatic segmentation of real metallographic images.We introduced a modification of the histogram-based segmentation technique for metallographic images and we described the method for automated estimation of the region of interest.The obtained results prove the high efficiency of the histogram segmentation technique.
The central region with small defects that influence the quality of the metal samples has been successfully automatically detected.The classification results obtained by the proposed methods are close to the classification of human experts.
The obtained results are promising and indicate that the proposed method for ROI can be used for a future evaluation of a crosscut macro structure of the slab samples.In order to achieve more precise evaluation results, individual areas of a sample crosscut image should be analyzed in detail.In our future research, we plan to use techniques for the automated extraction of dots [2] from the segmented region of interest.The detailed analysis of extracted central defects will also be important for the future connecting of metallurgical relations in images [5] and process variables, which are hidden in industrial databases.

Fig. 2 :
Fig. 2: Effects of the morphological post processing: The segmented image without any morphological post-processing is located up, whereas the original image with the morphological post processing is located down.

Fig. 3 :
Fig. 3: Insufficient results of the spectral segmentation: Sample A (up) and Sample B (down).

Figure 3
Figure 3 highlights, in red, an example of the spectral segmentation failure of Sample B. The problems with spectral segmentation occur probably due to the presence of image noise in the industrial environment.For calculations we used the default values of the spectral segmentation algorithm [9].

Fig. 4 :
Fig. 4: Results of the histogram segmentation: Sample A (up) and Sample B (down).

Tab. 1 :
Comparison of computation time for histogram and spectral segmentation.

Fig. 5 :
Fig. 5: Effects of the histogram normalization: The segmented image with enhanced contrast by the histogram normalization is located up, whereas the same image without any histogram normalization is located down.

Figure 6 and
Figure 6 and Fig. 8 present mean values of each row of Samples A and B. The information from these figures is displayed in Fig. 7 (Sample A) and Fig. 9 (Sample B).Here, blue lines represent the detected regions of interest.As a result of the intermediate state of the ROI detection algorithm, there are false positive matches (see blue lines located close to peripheral areas of Fig.7and Fig.9).

Fig. 6 :
Fig. 6: Metal-Sample A: Mean values of the each row in the analyzed image.The detected local maxima are expressed by the red circles.

Fig. 8 :
Fig. 8: Metal-Sample B: Mean values of each row in analyzed image.The detected local maxima are expressed by the red circles.

Fig. 9 :
Fig. 9: Metal-Sample B: Identification of region of interest with several false-positives.

Fig. 10 :
Fig. 10: Metal-Sample B: the final detected region of interested is highlighted in red.
The experimental metallography images have been provided by the company ArcelorMittal Ostrava a.s.(Ostrava, the Czech Republic).gained more theoretical experience in the field of Metallurgy of Iron making and Steelmaking during his doctorate study.The doctoral thesis was successfully completed in 1997.Since leaving the university, he has attained practical experience at the Research Dept. of ArcelorMittal Ostrava a.s.for the last fourteen years.He has held several positions, from research/process engineer to the head of various organization units.Currently, he is managing one group in research concerning liquid steel, billets and slab production.He has contributed to, and successfully completed, several grant projects with the financial support of grant agencies.The results of his scientific works are published continuously.Vit VONDRAK is Associate Professor in the Department of Applied Mathematics and Head of the Research Programme of IT4Innovations at VSB-Technical University of Ostrava, Czech Republic completed his Ph.D. in Numerical Mathematics at Palacky University in Olomouc, Czech Republic in 1993.During his scientific career, he has focused on numerical solution of large engineering problems like contact mechanics, design optimization problems which include modeling in biomechanics and hydrology.His work mainly concentrates on the adaptation and development of solving algorithms to parallel computing.