Enhancing particle string detection in electrorheological plasmas using asymmetrical kernel convolutional networks

To achieve this binarization we make use of a fully convolutional network with an encoder-decoder architecture. The neural network is split up into two parts: an encoder and a decoder. The encoder constitutes the first part of the network and is designed to sample down the input to a certain degree. Afterwards, the input image is sampled back up to original size as seen in figure 3. This is meant to reduce the original image to just the necessary information that is needed. Using this information, the neural network is then able to reproduce a mask of the image highlighting all strings. To regain the positional information of the input image after downsampling, several forward concatenations are built into the model after each sampling step. The downsampling steps are realized by two dimensional convolutions, followed by activation functions, and finalized by two dimensional pooling layers. The upsampling is then realized by two dimensional convolutional layers, activation functions and followed by transposed two dimensional convolutional layers. These layers form the basis of a U-Net model [15].

Zoom In Zoom Out Reset image size

The disadvantage of this U-Net is its immense size. With all downsampling steps the number of channels is increased, reaching a value of 1024. This is not just memory intensive but also computationally intensive and increases latency times of the neural network. Especially, when large images are fed through the network, large memory and computing resources are needed. The high amount of channels is not needed in this case and is therefore reduced starting with 4 channels for the first convolution, doubling in size every convolution until 32 channels are reached in the bottleneck of the network and then halving in size each convolution until 4 channels are reached for the last convolutions. This largely decreases the size of the network from 389 521 to 25 581 trainable parameters, while simultaneously decreasing latency times. Furthermore, all particle strings form in the direction of electric field, which in the case of PK-4 means, that all particle strings form horizontally. With these directed phenomena to be detected, asymmetric kernel sizes for convolution, deconvolution and pooling layers are deployed to account for this asymmetry and to get better results. Thus all kernel sizes are also elongated horizontally with decreasing kernel elongation towards the network constriction. This enables a broader activation of neurons to increase the recognition of strings. An illustration of the used neural network is shown in figure 3.

The finalized convolutional network needs an appropriate training before being applicable. Sufficient training data is needed for training. In the case of this binarizing encoder-decoder network suitable training images with corresponding labels, in form of binary masks, are required. While enough real image data of complex plasmas with isotropic and string-like particle distributions is present, no fitting string masks to operate as training labels are available. With several thousands of training images and masks needed for training, labelling images by hand would not just be too inaccurate but also too elaborate. Therefore, artificially created data is used for training.

The process of generating training images in a way that the output string masks reliably detect string formations is a trial and error process. To display a string on the input image, randomized coordinates for a single particle in the field of view are generated. Next to that, six additional particle coordinates are generated to be situated on the left and on the right side of the initial particle in a string formation. These coordinates are generated with a fixed distance in horizontal direction.

A limited random offset in vertical direction is also added for each particle on the side to make the generated string not entirely straight. The x and y distances to neighboring particles are additionally superimposed with a certain degree of random noise in x and y direction to create more realistic string formations. In addition to that, the whole string is rotated by a limited random angle, to make the string not entirely horizontal. This way an artificial string is created. With a single string visible on the input image, the network would take large amounts of images for sufficient training. Therefore, and to make the input images even more realistic, randomly chosen either two or three strings are generated per image in a limited random vertical distance to each other, preventing overlapping. Looking at real string formation data, some strings show a sort of kink, where two string ends come together. To further improve the realness of artificial training images, this kink is also introduced with a small vertical offset in the string. All limits for random values and noises are chosen to resemble real data as much as possible.

Next to training the network on recognizing string formations it also needs to be trained to recognize isotropic particle formations and not to mask these in the output mask. To address this, randomly distributed particles are added into the areas, where no strings are generated. Furthermore, additional images with randomly distributed particles containing no strings at all are also generated. All particles are displayed on images of 128 to 128 pixels in size. This resolution has shown to be high enough for successful training, while being small enough to be handled in large numbers by a conventional GPU.

The input images are generated by displaying particles at the position of generated particle coordinates on the image. When particle sizes, form and brightness are chosen to resemble real image data from preceding PK-4 campaigns, the low input values are not decisive enough for recognising particle strings. Therefore, binarized particle images are used as input for the neural network. The original image data is fed through a convolutional network, as described in [14], outputting a binarized mask of the original image, where all pixels containing a particle are set to 1, while all others are set to 0. This process is used for particle localization on PK-4 data and is therefore already available at no further computational costs. In a similar manner, the artificially created training images consist of a binary mask of all particles at the position of the generated coordinates in varying sizes.

The according labels are then generated, by simply displaying four pixel wide connection lines between the coordinates of all particles, that are situated in a string. The thickness is chosen so that the labels are distinct enough to be reconstructed by the network and to make sure that the marked particles are reliably covered by the output mask. The string masks are also displayed binary with value 1 on a background with value 0. This way, only strings are visible on the labels and all labels for training images with only an isotropic particle distribution are completely empty. An example of the artificial training images is shown in figure 4.

Zoom In Zoom Out Reset image size

Best results are then achieved by training with 3000 training images containing strings and 3000 images containing no strings and only isotropic distributions. These 6000 images are shuffled and used in training for 20 epochs with a batch size of 50 images. The Adam optimizer with a constant learning rate of 0.001 was used. The loss was calculated with a weighted binary cross entropy function as in

$$\begin{equation} \text{Loss} = -\frac{1}{N}\sum\limits_{i = 1}^{N}\frac{w_1\cdot T_i\cdot \log\left(P_i+\epsilon\right)+w_0\cdot \left(1-T_i\right)\cdot \log\left(1-P_i+\epsilon\right)}{w_0+w_1}\end{equation} \tag{ 1 }$$

with T_i and P_i being the truth and predicted value of each pixel, N being the number of pixels and ε being a small value of 10⁻⁴, to prevent taking the logarithm of zero. The weight of the value 1 w₁ was chosen to be 10, while the weight of the value 0 w₀ was chosen to be 1, since the amount of pixels with the value 1 is significantly lower for the used images. With this loss function the cross entropy loss for each pixel is calculated, while giving the fewer but more meaningful pixels with value 1 more weight.

To evaluate the trained model it is first tested on artificial data. For this, additional artificial validation data is generated in the same way as for the training process, to gather data that is unknown to the trained network. Evaluated on 240 images the accuracy of the network is 99.25% considering the prediction of pixels in the output image, which normally tends to be high, since most pixels are background pixels. When it comes to recognizing particle formations, the model detects 96.06% of particles in the correct formation. Of the incorrectly recognized particles roughly 98% are particles, that do not belong to a string, but have been detected in one. The model therefore tends to detect more particles in strings than are present.

To confirm the high accuracy of the model on artificial data, the model needs to be verified on real data to assure the the neural network has generalized to problem and has not just adapted to the training data. Since the artificial data is of the same kind as the training data, these high accuracies were foreseeable. The strings in the training data are randomly generated to only resemble strings in real data. Real images are much more particle filled and tend to form many more and longer strings, than present in training data. Furthermore, the resolution of real images with 2048 to 2048 is much higher than on the training images. It needs to be shown, that the network has learned to recognize particle strings, by recognizing real particle strings. Because no labelled string data exists for this case, manually labelled data is used. Particles in real images of preceding PK-4 experiments are marked considering their string-likeness. This is done on 20 images with 10 images predominantly showing particle strings and 10 images predominantly showing isotropic distributions. In some cases, deciding on a particles configuration is not possible by hand, since no clear rules exist. Therefore, clearly string-like and isotropic particles are marked as such, while undecidable particles were left out of consideration. These particles are used as an estimation, whether the network is more likely to mask particles as string-like or not.

The trained model predicts 95.3% of all marked particles in real data correctly. This proves that the model has generalized the problem and is able to recognize string-like formation also on new and unknown data, that looks entirely different than the training data. Of the incorrectly evaluated particles 53% have been falsely predicted as string-like and 47% have been falsely marked as isotropic. This demonstrates, that the network does not prefer a configuration over the other on real data. In the manually labelled images 4.26% of particles have been marked as undecidable by hand. Of these particles 22% have been marked as string-like and 78% as isotropic. So it seems as the network rather marks particles as isotropic, when the distinction is not possible by hand.

To verify the applicability of the model to real data, it is helpful to determine the limits to which the string detection accuracy holds up. Since the neural network uses camera image data to generate string masks, its sensitivity to input image quality, such as noise, lighting conditions, and image resolution, should be assessed. As mentioned in section 2.2, the original camera images must be fed through a convolutional neural network in a preprocessing step to binarize them, otherwise they would be too indecisive for analysis. This ensures a consistent, high signal intensity and low noise. In the following, these binarized input images are modified in different ways to assess the influence of image quality. Figure 6 shows the accuracy results of the same labelled data used for testing as above, degraded for lighting conditions by scaling the input intensity from 1 to 0, reduced image resolution by scaling the image resolution by factors of 1 (original) to 16, and image noise by adding Gaussian noise with signal to noise ratios of 50 to 1. It can be seen that the accuracy of the model drops significantly when the input intensity is more than halved, the image resolution is more than halved and the signal to noise ratio is less than 20. These results show that the sensitivity of neural networks to image quality degradation is not high. Typically, the signal to noise ratio of real PK-4 image data is between 100 and 150, which is far from the point where accuracy starts to degrade. In addition, the preprocessing step of image binarization prevents such drops in accuracy due to image quality entirely.

Zoom In Zoom Out Reset image size

String detection performance has been tested and verified in a range of complex plasma conditions. Our real experimental data includes pressures from 0.4 to 1 mbar, plasma currents from 0.5 to 1 mA, AC electric field frequencies of 250 and 1000 Hz and particle sizes of 2.05, 3.4 and 6.8 µm. Changing the background plasma conditions does not noticeably change the visual appearance of the plasma particles and therefore does not have a major effect on the string detection performance. Particle sizes and densities, however, do change the visual appearance of the plasma strings. In order to cover all the different configurations, the artificial training data has been matched as closely as possible to the real data. As a result, there is no significant loss of performance during the experiments. Strings with very different particle sizes and densities could potentially lead to losses in the accuracy of string detection. In this case, the artificial training data could be extended with such examples and the model retrained to cover such cases. Other disturbances in complex plasmas, such as dust acoustic waves, can possibly distort the strings to such an extent that the model is no longer able to detect them. Plasma waves should therefore be avoided or kept to a minimum in string formation experiments.

At first the anisotropic weighted scaling index is compared to the machine learning approach. According to [6, 10, 11] the weighted scaling index α is implemented by calculating the logarithmic derivative of $\rho (\textbf{p}_i,r)$ with respect to r, where ρ is the local weighted cumulative point distribution of each point particle p_i and r being the scale parameter. With

$$\begin{align} \rho \left(\textbf{p}_i, r\right) = \sum^N_{j = 1}e^{-\left(d_{ij}/r\right)^q}\end{align} \tag{ 2 }$$

and

$$\begin{align} d_{ij} = \Vert \textbf{p}_i-\textbf{p}_j\Vert\end{align} \tag{ 3 }$$

the scaling index simplifies to

$$\begin{align} \alpha \left(\textbf{p}_i,r\right) = \frac{\sum^N_{j = 1}q\left( \dfrac{d_{ij}}{r}\right)^qe^{-\left(d_{ij}/r\right)^q}}{\sum^N_{j = 1}e^{-\left(d_{ij}/r\right)^q}}.\end{align} \tag{ 4 }$$

The exponent q controls the weight of the points contribution to α based on their distance to p_i . With increasing q points with $d_{ij} \gt r$ are more and more neglected, leading to a more local measure. In this case q = 2 has been used. This way a scaling index for every particle is calculated. The value of α is a measurement of the particles surrounding structure with α ≈ 0 suggesting a cluster-like structure, α ≈ 1 suggesting a filamentary structure, α ≈ 2 suggesting a sheet-like structure, α ≈ 3 suggesting a uniform distribution and α > 3 indicating points in underdense regions but in the vicinity of structures [11].

Furthermore, longitudinal and transverse scaling indices $\alpha_{\parallel}$ and $\alpha_{\perp}$ can be calculated by adding an eigenvector q to d_ij as in

$$\begin{align} d_{ij} = \sqrt{q_x^2\left(x_i-x_j\right)^2+q_y^2\left(y_i-y_j\right)^2}.\end{align} \tag{ 5 }$$

Since in this case all strings lay parallel in x-direction the eigenvectors $\textbf{q}_x = (1,0)$ and $\textbf{q}_y = (0,1)$ are used to calculate $\alpha_{\parallel}$ and $\alpha_{\perp}$.

The anisotropic weighted scaling index could be an indication of string-like structure in particle clouds and the differences of longitudinal and transverse scaling index distribution a classification of if the present image is filled with string-like structures or not.

As shown in [12] the modified bond orientational order parameter

$$\begin{align} p_n\left(i\right) = \dfrac{2\pi}{\left(2n+1\right)N\left(i\right)}\sum^{N\left(i\right)}_{j = 1}P_n\left(\cos{\theta_{ij}}\right)\end{align} \tag{ 6 }$$

can be calculated for each particle i, using a number of nearest neighbors N(i) and Legendre Polynomial of the order n, with θ_ij being the angle between the two particles i and j. This parameter is used to illustrate structural transitions from weakly anisotropic fluids to string fluids and gives out a value of string-likeness for each particle. Since this approach is heavily dependent on the nearest neighbors, its accuracy for two-dimensional string structures is limited. In particular, the two ends of each string cannot be detected. By choosing a threshold value for $p_n(i)$ a distinction between string-like and isotropic particles can be made. In this evaluation six nearest neighbors N(i) and the second order Legendre polynomial n = 2 was used.

Previous studies evaluating string formations in complex plasmas in three [17, 18] and two [19] dimensions make use of pair correlation functions to determine the structural properties of particles. The pair correlation function in two dimensions, expressed in polar coordinates, is given by [19]

$$\begin{align} g_{2D}\left(\rho,\alpha\right) = \dfrac{1}{\sigma_dN}\sum^N_{\substack{i,j = 1 \\ i\neq j}}\dfrac{\delta\left(\rho_{i,j}-\rho\right)\delta\left(\alpha_{i,j}-\alpha\right)}{\rho}\end{align} \tag{ 7 }$$

with N being the number of particles, σ_d being the particle density, $\rho_{i,j}$ and $\alpha_{i,i}$ being the distance and angle of the vector connecting particle i and j and ρ and α being the current particles polar coordinates.

Visualizing the pair correlation function with respect to ρ and α a distinction between an isotropic and string-like ordered system is visible.

For an isotropic distribution the pair correlation function is distance and angle independent and for ordered structures maxima in the pair correlation function are visible at certain distances and angles. After estimating the ordered distance and angle regions, all particles with neighbors that have an according distance and angle can be viewed as string-like. In this case it needs to be decided, whether one string-like neighbor is enough for defining a particle as in string, while enabling possible wrong classifications, or if two string-like neighbors are needed, while then not recognizing string ends as part of the string.

To analyze phase transitions in electrorheological complex plasmas a machine learning approach based on a small convolutional neural network is presented in [9]. Here, the particle positions are extracted out of the original image at first. A 24 to 24 pixel image is then generated for each particle displaying an area of 5 times the median particle distance in total. The image is binary with every pixel containing a particle being 1. This image is fed through two convolutional layers, each followed by a batch normalisation, activation, pooling and dropout layer, and finally flattened by several fully connected layers to a two dimensional output. An illustration of this neural network is shown in figure 7. This way every particle is classified into either string-like or isotropic. The network is trained on partly real and artificial data. After classifying all particles falsely categorized strings are filtered out by reviewing the neighborhood of each particle.

Zoom In Zoom Out Reset image size

After training this convolutional neural network reaches an accuracy of 96.6% [9]. This evaluation is, at least partly, done on artificially created data. While this approach is very promising, it will be applied to the same manually labelled data to verify its performance.

The results of applying each implementation of the different methods on the same set of manually labelled data is shown in table 1. Applying the anisotropic weighted scaling index method to the real data it is quickly visible that no clear distinction between string-like and isotropic particles can be made. Even modifying the parameters of index calculation does not give results that could be decisive by some kind of threshold particle-wise. Therefore, no result is shown in table 1. Calculating the longitudinal and transverse scaling indices for the whole particle cloud image-wise, the delta between both indices is a good indication of if the particle cloud is filled with strings or not. But in this case no information about individual particles is given. Using the modified bond orientational order parameter to recognize strings is equally prone to errors. While in this case a connection between the order parameter and a particles string-likeness is perceptible, the accuracy is just slightly higher than 50% which can be compared to just guessing the particles configuration without seeing it. Even adjusting the decision threshold and the number of neighbors used in the calculations does not improve the results.

The pair correlation function method delivers much more useful results in comparison. Table 1 shows both outcomes for implementations considering one and two nearest neighbors. With over 80% accuracy the method using only one neighbor performs best. This way the last particle of each string is also detected as string-like. On the other hand this way also some number of false classifications are made with particles that do not form a string but coincidentally have a neighbor in the fitting distance and angle. The smaller convolutional neural network approach from [9] performs comparably well with over 77% accuracy, while not reaching the 96.6% accuracy that was evaluated after training.

The machine learning approach using a convolutional encoder-decoder network with asymmetrical kernels presented here performs best in comparison. While the improvement in accuracy is very apparent and distinct, it is crucial to consider, that these yielded lower values could be influenced by differences in implementation details and the adaptation on the new data.

Comparing computational efficiency is a difficult task, since it depends very much on the code implementation and also on the hardware used, since in this case the conventional analysis methods are executed on a CPU and the neural networks on a GPU. Besides these differences, the main advantage of the presented approach is its independence from the number of particles. While the presented neural network has a fixed interference time of only about 0.03s on a NVIDIA RTX3060M graphics card, including the preprocessing convolutional network, for fullsize images (2048 × 2048 pixels), the previous methods analyse the particles one by one, which has scaling disadvantages. On smaller images (512 × 512 pixels) with about 2000 particles, these methods already have latencies of 2 to 6s per image. When scaled up to fullsize images with easily over 20 000 particles, the latency times rise to over a minute on a CPU core of an Intel i7-12 700 H. Of course, these processes could be accelerated using different implementations and multiprocessing techniques, but these approaches would always suffer from scaling effects. Therefore, the new approach shows a significant acceleration compared to previous methods.