Abstract
Deep learning techniques, especially Convolutional Neural Networks (CNN), dominate the benchmarks for most computer vision tasks. These state-of-the-art results are typically obtained through supervised learning, for which large annotated datasets are required. However, acquiring such datasets for manufacturing applications remains a challenging proposition due to the time and costs involved in their collection. To overcome this disadvantage, a novel framework is proposed for data augmentation by creating synthetic images using Generative Adversarial Networks (GANs). The generator synthesizes new surface defect images from random noise which is trained over time to get realistic fakes. These synthetic images can be used further for training of classification algorithms. Three GAN architectures are trained, and the entire data augmentation pipeline is implemented for the Northeastern University (China) Classification (NEU-CLS) dataset for hot-rolled steel strips from NEU Surface Defect Database. The classification accuracy of a simple CNN architecture is measured on synthetic augmented data and further it is compared with similar state-of-the-arts. It is observed that the proposed GANs-based augmentation scheme significantly improves the performance of CNN for classification of surface defects. The classically augmented CNN yields sensitivity and specificity of 90.28% and 98.06% respectively. In contrast, the synthetically augmented CNN yields better results, with sensitivity and specificity of 95.33% and 99.16% respectively. Also, the use of GANs is demonstrated to disentangle the representation space and to add additional domain knowledge through synthetic augmentation that can be difficult to replicate through classic augmentation. The proposed framework demonstrates high generalization capability. It may be applied to other supervised surface inspection tasks, and thus facilitate the development of advanced vision-based inspection instruments for manufacturing applications.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Antoniou, A., Storkey, A., & Edwards, H. (2018). Augmenting image classifiers using data augmentation generative adversarial networks. In Artificial neural networks and machine learning—ICANN 2018.
Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN.
Badmos, O., Kopp, A., Bernthaler, T., & Schneider, G. (2020). Image-based defect detection in lithium-ion battery electrode using convolutional neural networks. Journal of Intelligent Manufacturing, 31, 885–897.
Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary equilibrium generative adversarial networks.
Bousmalis, K., Silberman, N., Dohan, D., Erhan, D., & Krishnan, D. (2017). Unsupervised pixel-level domain adaptation with generative adversarial networks. In IEEE conference on computer vision and pattern recognition (CVPR).
Carreira-Perpiñán, M. Á., & Hinton, G. E. (2005). On contrastive divergence learning. In AISTATS.
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016). Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS.
Davtalab, O., Kazemian, A., Yuan, X., & Khoshnevis, B. (2020). Automated inspection in robotic additive manufacturing using deep learning for layer deformation detection. Journal of Intelligent Manufacturing.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, Miami, FL, USA.
Devadas, C., Samarasekera, I. V., & Hawbolt, E. B. (1991). The thermal and metallurgical state of steel strip during hot rolling: Part III. Microstructural evolution. Metallurgical Transactions A, 22(2), 335–349.
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., et al. (2015). Flownet: Learning optical flow with convolutional networks. In ICCV.
Feng, S., Zhou, H., & Dong, H. (2019). Using deep neural network with small dataset to predict material defects. Materials and Design, 162, 300–310.
Frid-Adar, M., Klang, E., Amitai, M., Goldberger, J., & Greenspan, H. (2018). Synthetic data augmentation using GAN for improved liver lesion classification. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, DC.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., et al. (2014). Generative Adversarial Nets. In Advances in neural information processing systems 27.
Grzenda, M., & Bustillo, A. (2019). Semi-supervised roughness prediction with partly unlabeled vibration data streams. Journal of Intelligent Manufacturing, 30, 933–945.
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017). Improved training of Wasserstein GANs.
Hao, R., Lu, B., Cheng, Y., Li, X., & Huang, B. (2020). A steel surface defect inspection approach towards smart industrial monitoring. Journal of Intelligent Manufacturing.
Hinton, G. E., Dayan, P., & Frey, B. J. (1995). The “Wake-Sleep” Algorithm for Unsupervised Neural Networks. Science, 268, 1158–1161.
Hjelm, R. D., Jacob, A. P., Che, T., Trischler, A., Cho, K., & Bengio, Y. (2018). Boundary-seeking generative adversarial networks. In ICLR.
Huang, Y., Qiu, C., Guo, Y., Wang, X., & Yuan, K. (2018). Surface defect saliency of magnetic tile. In IEEE international conference on automation and engineering, At Munich, Germany.
Izadi, S., Mirikharaji, Z., Kawahara, J., & Hamarneh, G. (2018). Generative adversarial networks to segment skin lesions. In IEEE 15th international symposium on biomedical imaging, Washington, DC, USA.
Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes. In ICLR.
Lai, Y. T. K., Hu, J. S., Tsai, Y. H., & Chiu, W. Y. (2018). Industrial anomaly detection and one-class classification using generative adversarial networks. In IEEE/ASME international conference on advanced intelligent mechatronics (AIM).
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In IEEE conference on computer vision and pattern recognition (CVPR).
Li, J., Liang, X., Wei, Y., Xu, T., Feng, J., & Yan, S. (2017). Perceptual generative adversarial networks for small object detection. In IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, USA.
Luo, J., Huang, J., & Li, H. (2020). A case study of conditional deep convolutional generative adversarial networks in machine fault diagnosis. Journal of Intelligent Manufacturing.
Madani, A., Moradi, M., Karargyris, A., & Syeda-Mahmood, T. (2018). Chest x-ray generation and data augmentation for cardiovascular abnormality classification. In Medical imaging 2018: Image processing, vol. 10574.
Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., & Smolley, S. P. (2017). Least squares generative adversarial networks. In IEEE international conference on computer vision, Venice, Italy.
Mirza, M., & Osindero, S. (2014). Conditional generative adversarial nets.
Moeskops, P., Veta, M., Lafarge, M. W., Eppenhof, K. A., & Pluim, J. P. (2017). Adversarial training and dilated convolutions for brain MRI segmentation. In Workshop on deep learning in medical image analysis.
, V., Mohammed Safwan, K. P., Chennamsetty, S. S., & Krishnamurthi, G. (2017). Generative adversarial networks for brain lesion detection. In SPIE medical imaging, Orlando, Florida, United States.
Odena, A., Olah, C., & Shlens, J. (2017). Conditional image synthesis with auxiliary classifier GANs.
Pan, J., Canton, C., McGuinness, K., O’Connor, N., Torres, J., Sayrol, E., et al. (2017). SalGAN: Visual saliency prediction with adversarial networks. Computer Vision and Image Understanding.
Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.
Richter, S.R., Vineet, V., Roth, S., & Koltun, V. (2016). Playing for data: Ground truth from computer games. In ECCV.
Scime, L., & Beuth, J. (2018). Anomaly detection and classification in a laser powder bed additive manufacturing process using a trained computer vision algorithm. Additive Manufacturing, 19, 114–126.
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2014). Overfeat: Integrated recognition, localization and detection using convolutional networks. In International conference on learning representations, Banff.
Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., & Webb, R. (2017). Learning from simulated and unsupervised images through adversarial training. In CVPR.
Song, K., Hu, S., & Yan, Y. (2014). Automatic recognition of surface defects on hot-rolled. Journal of Computational Information Systems, 10(7), 3049–3055.
Song, K., & Yan, Y. (2013). A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects. Applied Surface Science, 285, 858–864.
Song, K., & Yan, Y. (2019). NEU surface defect database. Northeastern University. http://faculty.neu.edu.cn/yunhyan/NEU_surface_defect_database.html. [Accessed 5 4 2019].
Sun, T., Tien, F., Tien, F., Tien, F. C., & Kuo, R. J. (2016). Automated thermal fuse inspection using machine vision and artificial neural networks. Journal of Intelligent Manufacturing, 27, 639–651.
Tabernik, D., Šela, S., Skvarč, J., & Skočaj, D. (2020). Segmentation-based deep-learning approach for surface-defect detection. Journal of Intelligent Manufacturing, 31, 759–776.
Tian, Y. (2017, April 16). Master Chinese calligraphy with conditional adversarial networks. https://kaonashi-tyc.github.io/2017/04/06/zi2zi.html. [Accessed 10 5 2019].
Wolterink, J. M., Leiner, T., Viergever, M. A., & Išgum, I. (2017). Generative Adversarial Networks for Noise Reduction in Low-Dose CT. IEEE Transactions on Medical Imaging, 36(12), 2536–2545.
Yang, D., Xu, D., Zhou, S. K., Georgescu, B., Chen, M., Grbic, S., et al. (2017) Automatic liver segmentation using an adversarial image-to-image network. In International conference on medical image computing and computer-assisted intervention, Quebec City, QC, Canada.
Yu, H., Tieu, K., Lu, C., Deng, G., & Liu, X. (2013). Occurrence of surface defects on strips during hot rolling. International Journal of Advanced Manufacturing Technology, 67, 1161–1170.
Zhai, W., Zhu, J., Cao, Y., & Wang, Z. (2018). A generative adversarial network based framework for unsupervised visual surface inspection. In IEEE international conference on acoustics, speech and signal processing (ICASSP), Calgary, AB, Canada.
Zhang, Z., Song, Y., & Qi, H. (2017). Age progression/regression by conditional adversarial autoencoder. In IEEE conference on computer vision and pattern recognition (CVPR).
Zhao, J., Mathieu, M., & LeCun, Y. (2017). Energy-based generative adversarial networks. In ICLR.
Zhu, X., Liu, Y., Li, J., Wan, T., & Qin, Z. (2018). Emotion classification with data augmentation using generative adversarial networks. In Advances in knowledge discovery and data mining. PAKDD 2018, 2017.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Jain, S., Seth, G., Paruthi, A. et al. Synthetic data augmentation for surface defect detection and classification using deep learning. J Intell Manuf 33, 1007–1020 (2022). https://doi.org/10.1007/s10845-020-01710-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-020-01710-x