Simulated HRTEM images of nanoparticles to train a neural network to classify nanoparticles for crystallinity

Machine learning approaches for image analysis require extensive training datasets for an accurate analysis. This also applies to the automated analysis of electron microscopy data where training data are usually created by manual annotation. Besides nanoparticle shape and size distribution, their internal crystal structure is a major parameter to assess their nature and their physical properties. The automatic classification of ultrasmall gold nanoparticles (1–3 nm) by their crystallinity is possible after training a neural network with simulated HRTEM data. This avoids a human bias and the necessity to manually classify extensive particle sets as training data. The small size of these particles represents a significant challenge with respect to the question of internal crystallinity. The network was able to assign real particles imaged by HRTEM with high accuracy to the classes monocrystalline, polycrystalline, and amorphous after being trained with simulated datasets. The ability to adjust the simulation parameters opens the possibility to extend this procedure to other experimental setups and other types of nanoparticles.


Introduction
High-resolution transmission electron microscopy (HRTEM) is an important analytical tool in nanoparticle research as size, shape, and atomic structure of nanoparticles are directly re-ected in the image contrast.However, analysing such HRTEM images is oen a time-consuming and tedious, mostly manual process.A conventional manual annotation can also lead to a considerable degree of human bias in the data processing.Manual analysis is especially limited when large amounts of image data, for instance from in situ electron microscopy experiments, are processed.Therefore, automated image analysis, including machine learning (ML), is increasingly used for image analysis tasks in electron microscopy (for comprehensive overviews on machine learning in electron microscopy see ref. 1  and 2).These are typically faster and more objective than manual analyses.][12] For bulk analyses of HRTEM images of a given sample it is not only of interest to know their size and shape, but also features of their internal structure, e.g. to distinguish amorphous, single-crystalline, or polycrystalline congurations.The nanoparticle crystallinity inuences their physical properties, e.g.their luminescence, 13,14 their metallic nature, 15,16 and the stability towards dissolution 17 which can also effect their biological properties. 18,19Notably, a given sample may contain a mixture of nanoparticles with different crystallinity. 20In that case, the relative proportions of particles falling into one of these classes are of interest.The principal difference between the three classes of crystallinity is the degree of periodicity of the atomic structure in the particle volume or its projected area, which manifests itself as a corresponding periodicity in the image contrast.The task of classifying samples according to qualitative differences in periodicity in conned areas of an image is a typical task of pattern recognition, which can be performed in real space or in reciprocal space.
Usually, crystallographic analysis is performed by Fouriertransformed HRTEM images and on the electron diffraction patterns on individual particles, for example diffraction using a parallel coherent electron beam. 21,22While electron diffraction in cutting-edge microscopes offers the sensitivity to fully characterize a single nanostructure, its success is usually limited to larger features exceeding 3 nm. 23With other techniques like Xray powder diffraction, it is generally difficult to obtain Nanoscale Advances PAPER quantitative information on the ratio of amorphous to crystalline particles. 24Furthermore, X-ray diffraction averages information over a large number of particles (unlike electron microscopy which probes individual particles), making it blind to variations within smaller clusters or nanoparticles.It also does not give the particle sizes but the averaged size of crystalline domains in a sample.Thus it cannot distinguish between twinned particles and individual particles. 24he assessment of crystallinity is particularly challenging when ultrasmall nanoparticles (1-3 nm) are considered. 257][28] Furthermore, they are sensitive to internal change (like recrystallization) under the high-dose conditions during electron diffraction. 26,27Gold nanoparticles are suitable to address the question of crystallinity because they give a high contrast (unlike the light platinum metals) and because they are not sensitive to oxidation. 26,27Thus, gold represents a good role model for ultrasmall nanoparticles and atom-sharp clusters which has been studied to a considerable extent.
We have presented earlier a program based on machine learning to analyse individual nanoparticles for their shape and size from HRTEM images. 29,30Here we extend this approach to an automated classication of nanoparticles with respect to their crystallinity.As the generation of manually labelled training data for this task is not only time-consuming but also highly error-prone, different image simulation approaches were tested to establish a feasible training pipeline.This follows earlier approaches to train networks with simulated scanning electron microscopy images for particle size analysis, 7,31 created by generative adversarial networks (GANs). 32,33We present a fully automated classication of nanoparticles by machine learning with respect to their crystallinity, fully based on simulated training data.

Results and discussion
Even particles with the same crystal structure are usually found in random orientations in a TEM image and therefore can lead to a large number of different patterns in HRTEM images.Thus, a machine learning-based procedure to classify the particles in an HRTEM image-based on their crystallinity is urgently needed.One of the most important factors for a successful machine learning model is the quality and quantity of adequate training data.For a crystallographic classication of nanoparticles, a large amount of accurately labelled HRTEM images is necessary.Manually classied HRTEM images would be ideal, but it is a tedious process to classify thousands of particles.Furthermore, the human bias with respect to the classication of borderline cases immediately affects the quality of the training data. 34Therefore, we have investigated approaches applying synthetic HRTEM images to train the neural network.Two different approaches to generate synthetic images were explored, rst a simple pattern-based simulation, and second, a more advanced simulation of HRTEM images by the soware package "Dr Probe". 35The quality of the classication aer training was tested on a subset of the simulated images (test dataset) and also on a manually labelled set of experimental HRTEM images.This ensured that the network was applicable to the experimental images and that the simulated images were an adequate representation of experimental images.
As a rst very basic approach, the classication network was trained on simple pattern images as shown in Fig. 1.The training was performed with such patterns without the background signal of the thin amorphous support that is typical for HRTEM images of supported nanoparticles, i.e. with the depicted quadratic images.
The training was rst performed for two classes (amorphous and crystalline) and then extended to three classes (amorphous, monocrystalline, polycrystalline).The network showed a very good performance on the simulated test dataset which was a subset of 20% of the simulated images that were not used in the training process for the classications amorphous/ crystalline (denoted as "two-class" in the following) and amorphous/monocrystalline/polycrystalline (denoted as "threeclass" in the following).However, a test on experimental HRTEM images of ultrasmall gold nanoparticles (1-3 nm) gave disappointing accuracies (Table 1).This indicates that simple patterns are not suitable to train a network to classify experimental HRTEM images.
Simulations of HRTEM images with atomic structure models of gold nanoparticles and thin amorphous support lms were performed with the soware Dr Probe. 35A dataset was created that consisted of simulated images of ultrasmall gold nanoparticles on a support of amorphous carbon as shown by the Fig. 1 Example images of the simple pattern simulation approach to train a neural network to classify particles into the categories amorphous, monocrystalline, and polycrystalline.

Paper
Nanoscale Advances example in Fig. 2. Gold nanoparticles on a carbon sample holder can be considered as a good model system which is also easily experimentally accessible.Note that we did not consider strict crystallographic structures in this approach, i.e. all patterns with a regular pattern indicating a translation symmetry were considered and classied as crystalline.Different models of gold nanoparticles were used for the simulations, taken from the ChemTube3D database (Fig. 3). 37In addition, spherical cut-outs of the gold fcc structure were prepared.Furthermore, amorphous gold nanoparticles were simulated by a custom-made Python script.The presence of amorphous (or disordered) nanoparticles is a peculiarity in the ultrasmall size regime where each particle consists of only a few hundred atoms. 26n addition to variations of the structure models, some imaging parameters (including the most volatile optical parameters like defocus and two-fold astigmatism) were varied with each simulation within reasonable ranges.Examples of the simulated HRTEM images are shown in Fig. 4 together with an experimental HRTEM image for comparison.Extensive data augmentation of the primary dataset by rotation, brightness and contrast augmentation, x-and y-axis rotation, noise addition etc. was carried out to increase the number of available

Nanoscale Advances Paper
training images (see Materials and methods part).Before training on these images, they were processed by the ANTEMA soware to separate the particle from the background (cut-out procedure based on machine learning) as described earlier. 30n inadvertent inclusion of background into the particle area of interest was therefore avoided.Thus, the training process was kept as similar as possible to the processing of experimental HRTEM images.The rst network trained by a more realistic image simulation by the Dr Probe soware was named "SimulationC" and consisted of images based on the ChemTube3D models, spherical fcc cut-outs, and the generated amorphous particles, all on a thin amorphous carbon support.The network was trained to distinguish two classes (amorphous and crystalline) and reached an accuracy of 91.2% on the test dataset and of 75.2% on the dataset of experimental HRTEM images (Table 2).A closer inspection showed that the network was especially error-prone on images with a strong amorphous background signal.The low precision of 60.9% for the class "crystalline" indicates that the network tended to falsely classify crystalline particles as amorphous.
For this reason, further images were simulated with stronger amorphous background signal.Instead of increasing the thickness of the amorphous carbon lm, which would require a serious increase of computation time of the simulation, the background signal was effectively enhanced by preserving the support lm thickness, and with this keeping the number of atoms the same but substituting the carbon atoms by silicon atoms.Now the signal of the amorphous background was stronger, reducing the contrast between the background and an amorphous particle (Fig. 5).
The network was trained on an extended dataset that contained the images of nanoparticles from SimulationC and the new nanoparticles on a silicon support.It was denoted as "SimulationC+Si".For two classes, this network showed a much higher accuracy of 98.7% on the test dataset than the network SimulationC.The accuracy of the network on experimental HRTEM images was also strongly enhanced with 89.3% (Table 3).Obviously, the inclusion of images with stronger amorphous background signals improved the network performance on experimental HRTEM images by generating a more realistic simulation of the level of disturbing background signal.
To extend this approach to three classes, simulations of polycrystalline particles were necessary.The polycrystalline particles were simulated on carbon and silicon supports by stitching together either two or three differently rotated monocrystalline fcc cut-outs (Fig. 6).The crystallographic orientation of the domains was not considered.The network trained on this dataset is denoted as "Poly" in the following.This network reached an accuracy of 96.3% on the simulation test dataset for three classes.As might have been expected, errors were mainly made in the distinction between polycrystalline and monocrystalline particles.This was also found with experimental HRTEM test images where the network achieved an accuracy of 78.0%.The main error occurred for polycrystalline particles that were wrongly labelled as monocrystalline, leading to a low precision score of 48.9% for the class monocrystalline (Table 4).Aer classications with a low certainty of assignment (<80%) were excluded and categorized as undened, the accuracy increased to 85.4% and the precision for the class monocrystalline increased to 63.2%.However, this put many particles into the non-assignable category "unknown".Further errors occurred in the class polycrystalline as shown in the confusion matrix (Fig. 7).The deletion of all classications with a certainty of assignment below 80% le 19.3% of all particles in the manually labelled dataset categorized as unknown, an acceptable small fraction given that much larger datasets can be evaluated with our automated approach.
The classication network "Poly" together with the 80% omission rule was then included into the soware package ANTEMA 30 to fully analyse particles in HRTEM images in terms of size, shape and structure.Fig. 8 shows a visualization of the combination of particle detection with ANTEMA and the clas-sication by the network trained with the Poly dataset for an

Nanoscale Advances Paper
image of gold nanoparticles.The ANTEMA soware was able to detect the particles, and the classication algorithm classied the nanoparticles based on their crystallinity.The particles at the border of the image were removed by post-processing in the ANTEMA soware to avoid incomplete particles.The analysis by the combined programs took only a few seconds, i.e. this approach was much less time intensive than the usual manual analysis.Clearly, the automated analysis gives correct results in most cases.The classication of nanoparticles by size and shape by ANTEMA has been reported earlier. 30t should be emphasized that gold nanoparticles represent a particularly good system for this approach because they have a high electron contrast and do not tend to be oxidized. 38herefore, this analysis was possible even for the challenging case of ultrasmall nanoparticles (1-3 nm).This approach will become easier for larger particles (like plasmonic particles), but more difficult for metal particles of lighter elements like silver or the light platinum metals.][28] In principle, it is also possible to analyse crystalline nanoparticles by 2D-Fourier Transformation (2D-FT).This has been demonstrated by Zhu et al. 39 who have applied this method to 7 nm iron oxide nanoparticles.However, the contrast of ultrasmall nanoparticles analysed here is much lower, therefore the analysis will be much more difficult.Furthermore, this is just another method of image analysis, based on training the neural Table 4 Performance evaluation metrics for the network "Poly", trained on all data that had been simulated by Dr Probe, 35 i.e. from atomic structure models of amorphous, monocrystalline, and polycrystalline gold nanoparticles on amorphous carbon and silicon supports.The performance was tested on the test dataset and a dataset of manually labelled real HRTEM images

Conclusions
Ultrasmall nanoparticles with a diameter of 1-3 nm can have an internal crystallinity which is difficult to assess due to their small size, even in high-resolution transmission electron microscopy.Nevertheless, the automated analysis of ultrasmall gold nanoparticles with respect to their crystallinity is possible by application of a suitable machine learning procedure.The necessity for huge training datasets was solved by realistic simulations of HRTEM images of gold nanoparticles on an amorphous support lm.Thus, the human bias and the extensive work required by manually classifying thousands of particles can be avoided.The simple approach of pattern-based images was not successful as obviously these patterns are not sufficiently similar to HRTEM images for training.In contrast, HRTEM image simulations can be used to train a neural network for particle classication into the categories amorphous, monocrystalline, and polycrystalline.However, it turned out that small experimental details like the disturbing signal due to the amorphous support lm had a strong inuence on the quality of the training.This was demonstrated by the signicant increase in the level of assignment by changing the support material from carbon to silicon in the simulated images.The simulation is based on specic experimental parameters of the electron microscope used but can in principle be generalized to create any kind of dataset representing a variety of HRTEM imaging conditions.Furthermore, it easily permits to change the chemical nature of the nanoparticle, e.g. from gold to other metals or oxides, thus it can be used for different kinds of materials.Of course, the classication of ultrasmall nanoparticles is particularly challenging due to the small number of atoms involved.Consequently, the Fig. 7 Normalized confusion matrix and performance evaluation metrics for the network "Poly" after omission of all classifications with an accuracy below 80%.
Fig. 8 HRTEM image of gold nanoparticles and the combination of the particle detection software ANTEMA 30 with the particle classification based on crystallinity as implemented here.The particles were classified as either amorphous, monocrystalline, or polycrystalline.

Nanoscale Advances Paper
classication of larger particles should be possible along the same way with even higher accuracy as the periodicity in the HRTEM images is stronger.Further adaptations of the simulation les to produce more realistic images with more noise may further increase the accuracy of a network trained on such simulated data.In summary, the combination of a particle detection approach with ANTEMA with the particle classication presented here enables an automated large-scale analysis of particle crystallinity from HRTEM images with the possibility of analysing thousands of particles within a few minutes.This strongly speeds up the analysis of samples that would otherwise remain insufficiently characterized and gives a statistically reliable assessment of the properties of a particle population.

Electron microscopy
High-resolution transmission electron microscopy (HRTEM) was performed with an aberration-corrected FEI Titan transmission electron microscope equipped with a Cs-image corrector (CEOS Company), operating at 300 kV. 40The nanoparticle dispersion was drop-cast on a copper grid that was coated with an ultrathin amorphous carbon lm.Representative TEM images of ultrasmall metallic nanoparticles (1-3 nm) were used for validating the neural networks trained on articial images.

Image simulation
Image patterns for the classes amorphous, monocrystalline, and polycrystalline were generated from synthetic patterns with a custom-made MATLAB script. 41Images of the class amorphous were generated by placing random black and white dots on a grey background and adding salt and pepper noise with a noise density of 0.5.Images of the class monocrystalline were generated by overlaying two sinus functions with random frequency values in range 0.05 to 0.55 at randomly set angles and adding salt and pepper noise with a noise density of 0.5.Images of the class polycrystalline were generated by stitching together two or three images of the class monocrystalline with the same sinus frequency.All training images were quadratic as shown in Fig. 1.
HRTEM images were simulated with the soware Dr Probe, based on a Python interface. 35,42,43All generated images depicted gold nanoparticles.The atom packing models were partially acquired from ChemTube3D which are based on calculations by Barnard et al. 44,45 and also generated by dedicated scripts with the tools implemented in the Dr Probe soware and the emilys Python package. 37,46The data from ChemTube3D provided 16 monocrystalline and 6 twinned models (Fig. 3).Further monocrystalline models were generated by cutting out spheres of random sizes between 1 and 3 nm from the fcc structure of gold (ICSD 52700). 47Further polycrystalline particles were generated by cutting two differently rotated monocrystalline spheres of the same size (1 to 3 nm) along the same axes with a random distance from the particle centre between 0 nm and half of the radius of the particle.The rst part of the rst sphere and the second part of the second sphere were then stitched together to produce a polycrystalline particle.With a 50% chance this procedure was repeated with the resulting polycrystalline particle and another rotated monocrystalline particle of the same size.For this, the polycrystalline particle was randomly rotated before cutting it so that the previous cutting axis and the new cutting axis were not parallel.Amorphous particle models were generated by randomly positioning atoms in a spherical volume and then removing all positions that had a distance to other atom positions below 0.248 nm, following the procedure given by Novaes et al. 48ach particle was then placed into a cubic box with a side length of about 6 nm with the emilys toolbox. 46An amorphous carbon support layer, representing the sample holder, was added below the particle by the same generative approach as used above with the amorphous gold nanoparticles.The lled volume was a cuboid with the length and width of the cubic box and a randomly set thickness between 1 and 3 nm.The minimum distance between the carbon atoms was set to 0.160 nm.The support was generated individually for each simulation, ensuring a variable support structure, a variable support thickness, and a variable background noise in the simulation.To increase the amorphous background signal, images were also generated by replacing the carbon atoms in the support by silicon atoms, leaving all other parameters and atom positions unchanged.

Paper Nanoscale Advances
The simulation of particles as depicted in Fig. 2 was performed for an acceleration voltage of 300 kV.The focus spread was randomly set to values between 4.5 and 5.5 nm.The defocus was set to values in the range of −4 to 5 nm.The two-fold astigmatisms in x-and y-coefficients were independently set to values between −3.0 and 3.0 nm.In total, three different datasets were generated as shown in Table 5.

Manually labelled set of HRTEM images
To test the performance of the trained network with real HRTEM data, a set of metal nanoparticles from HRTEM images was manually labelled.These particles were cut out from the images with the ANTEMA soware. 30This soware is based on a neural network trained to distinguish particles from background in HRTEM images by performing semantic segmentation.Thus, individual particles can be detected and isolated from the background.The particles were manually classied as monocrystalline, polycrystalline, or amorphous.Particles that could not be assigned to a class by the examiner with a high certainty were not used for the dataset and excluded from training.The nal dataset consisted of 110 monocrystalline particles, 380 polycrystalline particles, and 247 amorphous particles.Chemically, it consisted mainly of gold nanoparticles in a size range of 1 to 10 nm as well as some platinum and silver-platinum nanoparticles for comparison.

Machine learning procedure
Multiple trainings were performed with different simulated image datasets for training (Table 5) with the goal to assign the particles cut out from real HRTEM images into either two classes (amorphous and crystalline) or three classes (amorphous, monocrystalline and polycrystalline).The datasets generated with the soware Dr Probe were pre-processed by treating them with the ANTEMA soware to cut out the particle from the image as was done with the real images.This removed the background outside the particle (the support) but not the inherent background noise level caused by the support lm below the particle.The patterned datasets were not further preprocessed.All simulated datasets were split into training, validation, and test datasets in a number ratio of 60 : 20 : 20.
Different neural networks that are available in the MathWorks Deep Learning Toolbox were tested for the two-class classication. 41The best results were achieved with ResNet-101. 49Therefore, this network was used for all further trainings.The weights were initialized with pretrained weights from training with the ImageNet dataset. 50As ResNet-101 has an image input size of 224 × 224 pixels, all images were resized to that size.To enhance the training by presenting the network more variable data, extensive data augmentation was applied.The images were augmented by random scaling, rotation, x-and y-axis reection, as well as brightness and contrast variation.Furthermore, a random Gaussian lter with a square kernel was applied for image blurring with a maximum Gaussian standard deviation of 2.
The training parameters were optimized by a Bayesian optimization.Training was performed for maximum of 80 epochs.Validation was performed once every epoch to prevent overtting.If the validation loss did not decrease for more than ve validation cycles, the training was terminated.The initial learning rate was set to 0.0085 and decreased every 20 epochs by a drop factor of 0.62.
The computations were performed on a Dell Precision 7920 Tower equipped with an NVIDIA Quadro RTX 5000, 32 GB RAM, and an Intel® Xeon® Gold 6226R processor.
The networks performance was evaluated on the test dataset by the parameters accuracy, precision and recall. 51The accuracy is a global metric, dened as the ratio of the correctly classied true positives (TP) and true negatives (TN) to all classied images including the false positives (FP) and false negatives (FN).
The precision and recall values are class-based metrics.The precision is the ratio of correctly classied images of one class to the full number of images belonging to that class.
The recall is the ratio of correctly classied images of one class to the full number of images that were classied into this class.
Furthermore, the performance was evaluated on the manually labelled dataset of particles from HRTEM images to test whether the network was applicable to real data.

Paper
Nanoscale Advances

Fig. 2 36 Fig. 3
Fig.23D model of a gold nanoparticle on a support of amorphous carbon used for the HRTEM image simulation.The edge length of the cubic box is approximately 6 nm.The rendering was performed with the program Mercury.36

Fig. 4
Fig. 4 Left: Representative simulated images: two examples of an Au 147 icosahedron structure from the ChemTube3D database, 37 two spherical fcc cut-outs, and two examples of generated amorphous particles.Right: A cut-out from an experimental HRTEM image showing two crystalline gold nanoparticles is shown for comparison.

Fig. 5
Fig. 5 Change of the amorphous support film in the simulation from carbon to silicon.Note the increased contrast of the support film due to the stronger scattering power of silicon compared to carbon.

Fig. 6
Fig. 6 Examples of simulated HRTEM images of polycrystalline particles generated by stitching together differently rotated monocrystalline particles with either two or three different domains.The support in the simulation was silicon in all cases.

Table 1
Performance evaluation metrics for the network trained on a dataset that consisted of patterned images, tested on the test dataset, and on a dataset of manually labelled experimental HRTEM images of ultrasmall gold nanoparticles

Table 2
Performance evaluation metrics for the SimulationC network trained on simulated images of gold nanoparticles on amorphous carbon support, tested on a separate dataset of simulated images and a dataset of manually labelled experimental HRTEM images © 2024 The Author(s).Published by the Royal Society of Chemistry Nanoscale Adv., 2024, 6, 4196-4206 | 4199

Table 3
Performance evaluation metrics for the SimulationC+Si network trained on of gold nanoparticles on a layer of amorphous carbon and a layer of amorphous silicon support, tested on the simulation test dataset and a dataset of manually labelled experimental HRTEM images

Table 5
Overview of the training datasets generated with image patterns and with Dr Probe after data augmentation, including the number of particles used in each class