Microplastic pollution monitoring with holographic classification and deep learning

The observation and detection of the microplastic pollutants generated by industrial manufacturing require the use of precise optical systems. Digital holography is well suited for this task because of its non-contact and non-invasive detection features and the ability to generate information-rich holograms. However, traditional digital holography usually requires post-processing steps, which is time-consuming and may not achieve the final object detection performance. In this work, we develop a deep learning-based holographic classification method, which computes directly on the raw holographic data to extract quantitative information of the microplastic pollutants so as to classify them according to the extent of the pollution. We further show that our method can generalize to the classification task of other micro-objects through cross-dataset validation. Without bulky optical devices, our system can be further developed into a portable microplastics detection system, with wide applicability in the monitoring of microplastic particle pollution in the ecological environment.


Introduction
Microplastic particulate pollution generated by industrial manufacturing has become a serious environmental problem [1][2][3]. These plastic particles with a diameter of less than 5 mm have been detected in different places of the world in recent years, including both terrestrial and marine environment. In some industrial processes such as industrial pharmaceuticals, medical wastes such as disposable syringes and catheters will be generated, which contain polyethylene plastics [4]. Inadequate degradation of such medical waste will also lead to the accumulation of microplastic pollutants. Since the outbreak of COVID-19, a large amount of plastic waste has been generated, which further aggravates the pollution due to microplastic particles. The textile industry is also considered to be a major source of fiber microplastics pollutants, with synthetic fibers in the waste water of textile washing [5]. At the same time, microplastic particles have been reported to be much higher in areas where vehicle manufacturing, large packaging companies and office furniture companies are concentrated, compared with non-industrial areas [6]. Microplastic particles can also come from the degradation of large plastics, waste from landfills or incinerators, and emissions from transportation and industry [7]. The continuous accumulation of microplastic pollutants results in serious environmental problems, such as concentration of microplastics in the ocean [3] and soil [4], the death of marine animals [8], etc.
Normally, microplastics are mixed with other particles in the ecological environment, such as gravel and soil particles, microalgae, bubbles, leaf fragments and large plastic waste [8], obfuscating some of their features. Consequently, it is difficult to accurately detect and show the quantitative information of microplastics without expensive optical instrumentation or complicated image processing technologies. Conventional approaches to detect and count microplastic particles normally consist of the following steps: (1) micro-object sample collection from the ecological environment; (2) sample preparation and handling by biochemical or physical methods [8,9]; (3) image capturing with optical systems; (4) image pre-processing, manual image examination and sample classification. The latter two steps in particular require expert knowledge, and are time-consuming in general.
Several research groups have explored the optical detection and identification of microplastic particles. Bianco et al used the support vector machine to extract the characteristics of microplastics and classify them [10]. Takahashi et al combined integrated holography with Raman spectroscopy to identify the microplastics [11]. Compared with conventional 2D images, holograms recorded by the digital holography system contain abundant phase information of the objects [12,13] which provide the possibility for further image classification. Moreover, with the advent of deep learning, digital holography can automatically improve [14,15], analyze [16] and classify images by using data training models [17]. However, most of the current classification and identification methods based on digital holography require relatively complex sample pretreatment, image reconstruction, denoising and other steps [18], which increase the complexity of related work and reduce the efficiency of microplastic particle classification.
In this study, we propose a holographic image classification system, which combines deep learning to automatically detect the microplastic particles and classify the images according to their quantitative information, so as to monitor microplastic particle pollution. We first introduce the detailed structure of holographic-classifier convolutional neural network (HC-CNN) layer by layer. Then, we comprehensively evaluate the performance of the network in the classification of microplastic particles, including its ability to extract the features from holographic images, the accuracy, precision and the efficiency of the classification. We also provide details of the HC-CNN training to avoid network overfitting and keep it stable. The classification performance is compared with other leading methods, and the generalization of HC-CNN is evaluated on other cross-dataset tests.
Our main contributions can be summarized as follows: • We propose a deep learning-based digital in-line holography classification system for automatic detection and classification of the microplastic pollutants. • The tasks of automatic classification for microplastic particles are accomplished directly based on the raw holograms without extra image pre-processing steps, such as denoising and reconstruction. The system yields good performance in classification accuracy, time consumption, network robustness and acrossmodel generalization. • We construct a well-labelled hologram dataset and make it publicly available, which fills the gap of open source and well-labeled hologram data in this field. • The automatic classification of microplastic particles based on our system provides a powerful tool for environmental protection organizations and institutions to control microplastic particles pollution.

Related work
There are two main approaches for the classification of microplastic particles. One is manual sample screening combined with biological and chemical detection [8,9]. The other is automatic detection based on imaging and spectroscopy [10,11,19,20]. Because the former often involves several hours or even days of sample staining, precipitation, filtration and other processes, optical imaging methods, such as Raman spectroscopy [11], Fourier infrared spectroscopy [21], environmental scanning electron microscopy [9], digital holography [12], etc, have become the focus of researchers in recent years because of their ability to achieve non-contact rapid imaging and object classification. Among them, holographic images have abundant phase information and image features that can be extracted, which is beneficial to improve the accuracy of microplastics classification. Based on the advantages of holographic technology, our laboratory has made a lot of research and exploration in image reconstruction [22], autofocusing [23] and resolution improvement [24] in the field of digital holography. One way to achieve digital hologram classification of microplastic particles is by manual feature extraction and then classification based on machine learning such as support vector machines [10,25]. On the other hand, our method leverages a deep learning network to provide better feature extraction and higher classification efficiency, paving the way for real-time classification of microplastics particle and microplastic pollution monitoring. Compared with other deep learning detection method such as that proposed by Wu et al [26], our method does not require image autofocusing and reconstruction, which simplifies the processing steps and can achieve quick detection.
The work reported here builds upon our previous publications on the detection and classification of microplastic particles [20,27,28]. In [20], we focus on the classification task with a small and imbalanced dataset. We develop deep transfer learning to enhance the feature extraction capability of the CompNet, which is a specially designed network for small dataset classification. Then, in [27], a generative adversarial network is used for an effective holographic image augmentation and dataset expansion. Both methods tackle the situation with a small dataset and insufficient image labeling, which is different from the training and classification of the well-labeled larger dataset in the present work. The data augmentation by transfer learning [20] and generative adversarial network [27] not only cannot guarantee that augmented images contain valid features, but also consume additional computing resources and slow down the classification speed when dealing with well-labeled datasets. The detailed classification performance comparisons will be shown in detail in section 4.
On the other hand, this work can be considered as a further extension of that presented in [28]. The detection and classification of microplastic particles are handled by using a specially designed HC-CNN based on a single raw hologram without image pre-processing steps, such as denoising and reconstruction. In addition, here we introduce the network structure and training process of HC-CNN, which is specially designed for the characteristics of microplastic particle holographic images, and comprehensively evaluate the classification performance of this network on various aspects such as accuracy, precision, and recall rate. This paper provides a detailed explanation for the classification method of microplastic particles based on digital holography.

Holographic classification method with deep learning
A high-level overview of the holographic classification method is shown in figure 1. Firstly, the raw images are captured by the digital in-line holography system, which is shown in figure 1(a). Secondly, these data are well labelled and built into a dataset. Thirdly, this dataset is feed into the specially designed HC-CNN for the model training. And finally, this model is utilized for the microplastic particles classification and monitoring.

Lensless digital in-line holography system with a low coherence light source
As shown in figure 1(a), the optical system is a lensless digital in-line holography system with an LED (440 nm wavelength and 24 nm bandwidth), a complementary metal oxide semiconductor (CMOS) camera (Mako-507b, 2464(H) × 2056(V), 3.45 µm pixel size) and a sample slide. LEDs consume less energy and are smaller and lighter than laser light sources. We use LED as the illumination, because it will be more suitable for installation in portable devices. The detailed experimental configurations are as follows: the distance between the LED light source and the imaging surface of the CMOS camera is 18.5 cm, and the sample slide is placed 2.5 cm away from the imaging surface. This experimental setup places the camera on the image plane of the system, thus building a unit amplification optical system. The detectable range of the system is about 100 µm to 5 mm.
Mathematically, the optical wave with the object information has a complex amplitude, denoted by: where A o is the amplitude of the object wave and the ϕ o is the phase. The reference wave has a similar format as the object wave, which can be written as: where A r and ϕ r are the amplitude and phase of the reference wave, respectively. During imaging, the reference wave and the object wave travel along the same optical path after being emitted by the LED and form an interference pattern, or a hologram, in the imaging plane. The CMOS camera records the intensity of the hologram, given by [12]: where * denotes complex conjugation. It is worth noting that |E o (x, y)| 2 and |E r (x, y)| 2 are the zero-order terms, which contribute to the background of the hologram. The interference terms E o (x, y)E * r (x, y) and E * o (x, y)E r (x, y) are complex. Therefore, holograms contain more information, which require more complex processing for full feature extraction.
Compared with other traditional optical holography systems [12], such as off-axis digital holography and optical scanning holography, our system does not need bulky optical devices. In addition, the presence of optical devices, such as the microscope objective, beam splitter cube and lenses, may introduce the aberrations [29] including wavefront curvature, spherical aberration, chromatic aberration and astigmatism in mobile use, which will dramatically reduce the image resolution and affect the image quality. In order to The front layers with convolution layers and max-pooling layers are used for depth holographic feature extraction, and the back layers with flatten, dense and softmax layers are used to reduce the number parameters to achieve classification. ensure a stable imaging performance, we put the CMOS camera as close to the sample as possible, and remove the microscope objective and external lenses. Furthermore, because the reference light and object light travel along the same optical path, the influence of the external jitter can be counteracted.

Holographic-classifier convolutional neural network
The HC-CNN network is specially designed for the task of microplastic particle classification considering the following characteristics of the dataset: (a) Holograms contain richer information than digital images, so it is necessary to make full use of the features by a deeper feature extraction network structure. (b) The number of category of the hologram dataset is relatively few. A light network structure and appropriate network parameters are needed to avoid network over-fitting. (c) In order to achieve the goal of real-time detection, the convergence time of the network needs to be relatively short.
As shown in figure 1(e), to fully extract the abundant phase information features, we use more convolution layers in the upper part of the network. At the same time, to avoid over-fitting of the network, the Max-pooling layer is appended to the convolution layers. To realize the real-time classification and speed up the training process, batch normalization layer is utilized in the network. In addition, considering the number of classes in the dataset is relatively small, the dense function is used to drop the excess parameters. The final possibility of each class is coded with the expression of one-hot O(c), which is defined as: where integer c ∈ [1, C] for a given number of classes C. The final possibility of each class is calculated by a softmax function.
In addition, we have found that the abundant features contained in the hologram may not be fully extracted. In order to achieve better parameter and feature matching, we consider various loss functions to assist the training of the network on the hologram dataset without increasing the parameters of the network. Inspired by the work of Demirkaya et al [30] on the role of loss functions in multiclass classification problems, we introduce over-parameterization into the network training on hologram dataset and use the correct-class quadratic loss L CCQL as the loss function, given by: where G(c) is the corresponding label and α is the weight parameter used to adjust the influence of the correct class on the guidance of the network training process.

Sample selection and preparation
Polyethyelene (PE), polystyrene (PS), low-density polyethylene (LDPE) and high-density polyethylene (HDPE) are the main types of microplastic particles in the world [8] and they are chosen as the detection samples in our experiment. Some industrial processes, such as air-blasting system, may produce a large quantity of PS [6]. Besides, polyhydroxyalkanoate (PHA), one of the new thermoplastic pollutants, is also selected as another detection sample of our experiment. The sample size ranges from 90 to 450 µm. In order to simulate the in situ microplastic detection environment, the experimental samples are made of standard microplastic particles mixed with gravel, large plastic fragments, soil particles and leaf fragments. The sample making and mixing process is completed by a professional. Detailed information of the standard microplastic particles we used for our experiments are: HDPE (Sigma-Aldrich #427 985); LDPE (Sigma-Aldrich #428 043); PP (Sigma-Aldrich #427 888); PS (Millipore Sigma GF63567030); PHA (Millipore Sigma GF53774545).

Dataset preparation and labelling
We have built a well-labelled hologram dataset for network training and microplastics classification. We also select several sample holograms for each class from the dataset and show them in figure 2 The network structure is implemented by TensorFlow and the training process is accelerated by NVDIA TITAN V GPU with a 1455 MHz core frequency. We use the Adam optimizer [31] to update the gradient with a learning rate of 0.0001. In the L CCQL , we set α = ( √ C − 1 − 1), with the logic that the correct-class quadratic loss will automatically assign a weight of α + 1 on the correct class and give a weight of C − 1 to all other classes. The square-root operation is used to reduce the linearity of the function.

Results and discussion
To demonstrate the power of HC-CNN on hologram classification, we show the feature maps of a randomly selected sample image in figure 3, which are extracted by each convolutional layer of the neural network in sequence. In addition, we use EDA (exploratory data analysis) [32] to reduce the dimension of each feature map and convert them to a two-dimensional space, showing with t-distributed stochastic neighbor embedding (t-SNE) plots [33] in figure 4. This is a nonlinear dimensionality reduction tool to show the feature distribution in a 2D or a 3D space. From figures 3(b)-(k), as the number of convolution layer increases, the feature areas in the original hologram become smaller, and the features are effectively extracted. Moreover, figure 4, feature points are more segregated and are easier to classify in (k) than in (a), which is consistent with the results of the corresponding feature maps. To assess the performance of HC-CNN during the training process more visually, the accuracy and loss of the training set and the validation set with respect to the number of iterations are plotted in figure 5. The validation loss dramatically decreases after about 20 iterations, while the accuracy of the validation set improves quickly. Therefore, the network starts to converge effectively after about 20 iterations. From about 100-130 iterations (region I), both accuracy and loss values of the validation set become more stable. In this region, the network is well trained and suitable for classification. However, in region II, the data cannot be accurately classified due to the over-fitting of the network, and there are some fluctuations with both accuracy and loss curves. The network may behave worse in this region. On the whole, when the number of iterations of the network is properly controlled to be in region I, the HC-CNN network can show a stable classification performance for microplastic holograms.
In order to compare the performance of HC-CNN network in microplastics classification more objectively, we select other leading methods, namely, multilayer perceptron (MLP) [34], visual geometry group network-16 (VGG-16) [35], CNN, and ResNet [36], and conducted classification tests under the same conditions. For a fair comparison, the parameters of the above-mentioned networks are well adjusted to obtain the best performance. They are evaluated in terms of accuracy (A), precision (P) and recall rate (R), which are calculated by: where X TP represents the HC-CNN output results correctly predicting the well-classified samples, X TN represents the output correctly predicting the misclassified samples, X FP represents the output incorrectly predicting the well-classified samples, and X FN represents the output results incorrectly predicting the misclassified samples. H is the size of the test set. To evaluate the performance of the classification in some cases where the accuracy and precision cannot be successfully applied, we also use the harmonic mean of P and R, called F1-score (F 1 ), to measure the accuracy of the classification, which is expressed as: In addition, the decision time (T D ) is measured to assess the computational resources required for classification, which is the sum of the network training time on the training set and the time for the test set classification. The network robustness test is carried out by evaluating 15 repeated independent experiments. In order to quantitatively compare the performance among different algorithms, we tabulate the values of A, P, R, F 1 and T D in table 2. The results of each method are the mean values of fifteen independent experiments. They indicate that HC-CNN network has the highest accuracy, precision, recall rate and F1-score and it also has the smallest variance, demonstrating good robustness of our approach. In spite of the ResNet network showing a close performance with HC-CNN, it takes three times longer than that of the HC-CNN network. HC-CNN network contains more lightweight feature extraction structure, which effectively reduces the time needed for feature extraction and saves computing resources. In addition, though MLP network costs less time than HC-CNN, its accuracy is significantly lower for there is no convolutional layer to accurately extract the features from holograms. We also compare the performance of the methods proposed in our two previous studies [27] and [20] in table 2. Although they have comparable performance as HC-CNN, the decision time (T D ) required is far beyond that of the HC-CNN model due to the additional data augmentation. We therefore can conclude that, compared to other tested networks, HC-CNN is an accurate and fast network for microplastics classification, which increases the possibility of further being implemented on a portable device and used for real-time microplastic pollution monitoring.
In order to further analyze the performance of the multi-class classifier quantitatively, we calculate the normalized confusion matrix of the HC-CNN network. As shown in figure 6, with the unbalanced dataset (shown in table 1), the network has good classification performance on the class of 1, 2, 3, 4 and ≥ 5 MP, while having only a 17% error rate in the class of 0 MP. This shows that our network is less susceptible to the small and unbalance-distributed dataset.
To evaluate the generalization ability of HC-CNN, we perform the cross dataset validations on an open source hologram data (Wave Glider [37]). It includes 9140 raw holographic images of hemiaulus cells Table 2. Classification performances of MLP [34], VGG-16 [35], CNN, CompNet [20], ResNet [36], Zhu et al [27] and HC-CNN. A, P, R and F1 are defined in equations (6)- (9). TD is measured in hour.   For a fair comparison, we resize the images from Wave Glider to 128 × 128 pixels, which is the same as our own dataset, and re-train all the networks separately. All the training parameters, such as the learning rate, are the same as the training of our own dataset before. The classification accuracies of different networks on Wave Glider are showed in table 3. Compared with other leading methods, HC-CNN also shows higher accuracy on this dataset, which verifies the effectiveness and generalization of HC-CNN on micro-object classifications.  [34], VGG-16 [35], CNN, CompNet [20], ResNet [36], method proposed in [27] and HC-CNN on Wave Glider [37].

Conclusion
In this work, we propose a deep learning-enabled lensless digital in-line holography system to automatically detect and classify the microplastic particles directly based on the raw holograms. The performance of this system is experimentally evaluated by training the HC-CNN both on our own hologram dataset and an open source dataset. The experimental results show that our method has good performance in terms of the classification accuracy, network robustness, time efficiency and the generalization ability to other micro-objects. Future work can be carried out to classify and identify the types of microplastics, since the current classification is only based on their number. In order to better simulate the real environment of underwater microplastic particles, the types of particles in the samples can be more diverse, such as including microalgae and bubbles.
Due to the low complexity of the devices used in this experiment, it can be further developed to be a portable real-time microplastic particle detection device in future experiments. Environmental researchers can use it to monitor microplastic pollutants generated during industrial manufacturing. In addition to microplastic pollutants, our proposed method can also be used for the classification of other micro-objects, such as the particulate pollution (PM2.5) monitoring or micro-defects detection on an integrated circuit. Additionally, our work is a good example that combines the typical optical system with artificial intelligence technologies to improve the imaging performance, which offers an attractive direction for the development of other optical systems, such as light field microscopy [38] and hyperspectral imaging [39].

Data availability statement
The data that support the findings of this study are openly available at the following URL/DOI: https:// github.com/ymzhu19eee/dataset_microplastics.