Automated segmentation of computed tomography images of fiber-reinforced composites by deep learning

Abstract

A deep learning procedure has been examined for automatic segmentation of 3D tomography images from fiber-reinforced ceramic composites consisting of fibers and matrix of the same material (SiC), and thus identical image intensities. The analysis uses a neural network to distinguish phases from shape and edge information rather than intensity differences. It was used successfully to segment phases in a unidirectional composite that also had a coating with similar image intensity. It was also used to segment matrix cracks generated during in situ tensile loading of the composite and thereby demonstrate the influence of nonuniform fiber distribution on the nature of matrix cracking. By avoiding the need for manual segmentation of thousands of image slices, the procedure overcomes a major impediment to the extraction of quantitative information from such images. The analysis was performed using recently developed software that provides a general framework for executing both training and inference.

Graphic abstract

Appendices

Appendix A: Deep-learning image segmentation by CNNs

Semantic image segmentation—the labeling of pixels in an image according to the object they constitute—is a deep-learning method that was first applied to scientific imaging with the description of the U-Net architecture in 2015 [37], although non-scientific applications predate that work [40]. These network models are built as CNNs, in which image data are subdivided into patches and fed through a network of neurons, which are arranged in sequential layers. Each neuron in a given layer receives input from neurons in the previous layer, transforms the input signal, and then passes the result to a set of neurons in the next layer. The signals are integrated in successive layers of the network, where ultimately higher order neurons may have remarkable discriminative value by selectively amplifying various signals from previous layers. CNNs employ convolution operations as their first layer(s) of neurons. The coefficients in the convolution kernels are seeded randomly initially and are refined iteratively in the learning phase, where the output of the CNN is compared with the manually segmented image (training). The learned weights of different neurons confer the extreme selectivity which gives the networks their power. Similar to biological neural networks, these models are able to interpret texture that is observed in the image and use that to help distinguish hallmarks of one visual object from another. The early neurons are able to encode texture, but not because they are programmed to recognize specific patterns, edges, gradients, or other primitive image descriptors. Rather, the coefficients of the kernels in those early-stage convolutional filters are learned through the reinforcement process of network training. Consequently, if a convolutional kernel conveys a meaningful signal that can provide discriminating value in the network, it will be preserved and up-weighted. CNNs are also used in object detection and other deep learning enabled computer vision techniques. Further discussion of CNNs can be found in Refs. [41, 42].

Appendix B: Implementation of deep learning

To make the deep-learning method easy to apply to a broad class of image segmentation problems, a general framework for executing both the training and inference stages of the deep-learning cycle was developed. Both the training and inference tools rely on software libraries TensorFlow and Keras available from Google (Mountain View, California USA) [27]. These libraries have been integrated into a desktop software platform for image manipulation and analysis named Dragonfly (developed and licensed by Object Research Systems, Montreal, Canada), available at no cost under non-commercial licensing terms. Further details on the software integration and instructions on how to download Dragonfly, the software described in this paper, can be found on the main Object Research Systems website [23]. To download the trained deep-learning models, training images and raw data used for this work along with the CMC image data set refer to the Materials Data Facility repository [43]. Other deep learning models can be found at the Object Research Systems online tool-sharing community repository (Infinite Toolbox).

It is an important goal to make the deep learning solution accessible by non-experts, but provide high-value flexibility for experts that want to use the same system. There are no major runtime parameters associated with the inference stage of deep learning, so this goal of serving both classes of users is accomplished in the interface of how users configure and train their models.

When setting up model training, the basic panel exposes standard parameters: patch size, stride ratio, and batch size, optimization function, and loss function. The training for the microstructural segmentation in “CMC microstructure segmentation” section was done using the neural network architecture FCDenseNet [29, 30], with a patch size of 64 × 64 pixels, stride ratio of 0.5 and batch size 16. The loss function “categorical-crossentropy” and optimization algorithm “Adam” were used as the default functions of the Keras Library. For the segmentation of matrix cracks, the U-net [36] CNN was used with patch size of 128 pixels, stride to input ratio 1, batch size 32, loss function “categorical crossentropy,” and optimization function “Adadelta.” Initially, a U-Net model was trained for the microstructural segmentation. However, the inference results had many errors, so a model with deeper neural net architecture, FCDenseNet, was deployed (with the cost of a much longer time taken for training). The patch size was set to 64 × 64 since the features of the microstructure, such as individual fibers, spanned small areas in the image. For the crack segmentation, a U-net model proved adequate. The cracks spanned larger horizontal pixel areas in the longitudinal images, thus allowing use of larger patch size (128 × 128 pixels). The default optimization function in the Keras library was used for both models.

To support greater control, an optional advanced parameters panel that exposes options for additional logging (including support for TensorBoard), fine tuning parameters for the optimization function, conditions for early termination of training, and conditions for learning rate reduction. These parameters are understood to be beyond the scope of non-experts, but match many of the controls experts would have if they were directly programming their own solution with lower level tools. This platform also integrates methods for data augmentation to reduce the raw training data that must be manually prepared, and data set aside for validation during the course of training. Models that were previously trained can be accessed for further iterations of training.

In the inference stage, the deep-learning model behaves as a simple image transform engine, which can take a single gray-scale image and return a single segmented image. From the user perspective, the trained deep-learning model behaves as a simple image filter. To simplify user interaction, the interface allows the selection from a library of trained models that can be applied like any standard image filter. A preview mode allows viewing of the output from any of the trained models applied to any single image or any sub-area of an image. This permits rapid assessment of whether any of those models is suitable for application at hand.

Appendix C: Validation of deep learning

The general application of the deep-learning solution described here was validated by applying it to a set of serial-section transmission electron micrographs of Drosophila melanogaster neurons that has been used as an image segmentation challenge [44, 45]. These images have many structural features (plasma membranes) that are difficult to discriminate with standard algorithms. The challenge micrographs include a stack of 30 raw images that have not been processed and a stack of 20 manually segmented images to be used as training data for machine learning technique development. A recent report documented the successful segmentation of these micrographs with a CNN called FusionNet [46]. The FusionNet architecture was reproduced here and implemented as a model in the new Dragonfly toolkit. Following training, a set of 10 validation slices were used to assess the inference quality. Visual inspection confirmed that proper segmentation of the plasma membrane was achieved.