Machine learning at the (sub)atomic scale: next generation scanning probe microscopy

We discuss the exciting prospects for a step change in our ability to map and modify matter at the atomic/molecular level by embedding machine learning algorithms in scanning probe microscopy (with a particular focus on scanning tunnelling microscopy, STM). This nano-AI hybrid approach has the far-reaching potential to realise a technology capable of the automated analysis, actuation, and assembly of matter with a precision down to the single chemical bond limit.


Taking the pain out of probes
Scanning probe microscopy provides arguably the ultimate control of condensed matter: single atoms and molecules can not only be imaged with a resolution down to the single chemical bond/electron orbital limit, but they can be pushed, pulled, poked, prodded, picked up and put down with the tip of the microscope [1][2][3]. This is because a probe microscope is like no other imaging instrument; instead of bending light or electron waves via the appropriate lenses and optics, an SPM works in the near-field regime where the resolution depends on the atomistic structure of the apex of a probe which is scanned back and forth across a surface (as depicted schematically in figure 1). By applying a bias voltage between the tip and the sample, movements of the probe across a surface result in a spatially varying electrical current (due to quantum tunnelling). These currents are directly related to the atomistic structure of the sample surface. When the tip is atomically (or molecularly) sharp then not only can atoms and molecules be imaged or probed for spectroscopic information (by measuring tunnel current with fixed/swept probe voltage, respectively), but they can also be controllably manipulated and positioned to build up artificial nanostructures. Particularly impressive recent examples of probe microscope-driven assembly of artificial atomic and molecular lattices are described in an informative review by Khajetoorians et al [2].
This exceptional control does not come without its substantial frustrations, however. A major bottleneck in the imaging and manipulation process is the component that is at the very core of the technique: the probe itself. Currently, probe microscopists spend a substantial fraction of their time coercing the apex of the probe into a state that provides not just atomic resolution but the correct type of atomic or (sub)molecular resolution. To add to the frustration, a tip apex that provides very high resolution may not be best suited to other tasks, such as spectroscopy or single atom/molecule positioning [5]. At worst, it can take days of frustration to produce a probe that does what the microscopist needs it to do.
Image interpretation, the analysis of spectroscopic data, and the evaluation of probe-induced atomic/molecular manipulation events are also each often fraught with difficulty; the microscopist must continually guard against, and correct for, artefacts due to instrumental effects and/or a non-ideal tip apex [6][7][8][9][10]. Correcting for each of these is, however, often a matter of exploiting the tacit knowledge and expertise of seasoned 'SPM-ers' , with each research group often having its own recipes for scanning, tip preparation/recovery, and atomic manipulation that have been transferred from scientist to scientist over a period of time. Although these strategies are occasionally outlined in the literature, with some groups going so far as to provide detailed reports of the efficacy of different tip preparation and functionalisation protocols [11][12][13][14], it is still very much the case that a great deal of probe microscopy relies on rather subjective criteria, Figure 1. Scanning probe microscopy in a nutshell. An atomically/molecularly sharp probe is raster-scanned back and forth across a surface-which is, preferably, but not necessarily, flat at the atomic level-and an interaction between the tip apex and the underlying sample is measured. This interaction can be of a number of different forms but most commonly, at least under ultrahigh vacuum conditions, arises from quantum mechanical electron tunnelling and/or a frequency shift experienced by an oscillating cantilever/tuning fork [3] onto which the tip has been glued. A feedback loop is used to keep the tunnel current or frequency shift constant by adjusting the height of the tip, and this controlling the strength of the tip-sample interaction, although constant height mode is also possible. For a non-specialist introduction to (ultra)high resolution probe microscopy see reference [4]. often accompanied by somewhat ad hoc experimental heuristics, and that there has historically been a lack of image/data sharing across the (inter)national SPM communities.
Machine learning offers the opportunity for a distinctive and disruptive departure from the type of experimental (and, indeed, theoretical) methodologies that have formed the bedrock of the field of scanning probe microscopy ever since its inception in the eighties [15,16]. Almost forty years since the invention of the technique, it is still the case that the patience of the experimentalist remains the key parameter in cajoling the apex of the tip into yielding (sub)molecular or (sub)atomic [17][18][19] resolution. Even then, unless an 'inverse imaging' [20][21][22] strategy is adopted, the atomistic structure of the apex remains unknown. 1 Moreover, en route to attaining atomic resolution (or better), we probe microscopists reject a great deal of data which, if mined in an appropriate manner, can potentially provide a great deal of information on the various kinetically-limited and thermodynamically stable tip structures that are formed as we strive for the highest resolving power and/or stability.
There have been exceptionally encouraging advances in machine-learning-enabled probe microscopy over the last few years that would suggest that we are on the cusp of a radically new approach to the technique: next-generation SPM, where the effective throughput and analytic limits of probe microscopy are increased substantially by artificial intelligence methods. (Parallels could perhaps be drawn with the evolution of the automation of DNA sequencing, as described by Heather and Chain in their brief history of the technique [24].) In this Perspective we briefly review those key advances before making a number of what we hope are fairly grounded predictions as to the evolution of machine learning-enabled SPM over the next decade or so. Our thoughts and suggestions have been very much informed by a recent meeting of a number of UK scanning probe groups-whose research focuses on ultrahigh vacuum-based atomic and (sub)molecular resolution analysis of surfaces, interfaces, adsorbates and assemblies-held at the Institute of Physics in London on Sept 20 2019 [25].

More human than human: beyond the single molecule limit
Despite the central role played by STM/AFM in state-of-the-art condensed matter science [2,3,26,27], and the rapid exploitation of machine learning methods across all areas of the physical and materials sciences [28], the embedding of automated data mining and image classification protocols in probe microscopy has arguably taken rather longer than might have been expected given the core role of visual data in the field 2 . There is now, however, a small, but steadily growing, subset of SPM groups who are adopting machine 1 Inverse imaging nonetheless still only provides limited information on the geometric structure of the apex and is often difficult to interpret [22,23]. 2 As Vasudevan et al [29] point out, however, excitement about AI-driven data analysis and experimental design has ebbed and flowed for many decades, with periods of intense interest followed by 'AI winters' . . Architecture unpinning many important recent advances in machine-learning-enabled SPM. By using supervised learning with simulated data, Zhang et al were able to extract an undiscovered phase/symmetry of quantum electronic matter from a previously acquired STM dataset [31]; (B)Using a somewhat similar methodology to that employed for (A), Aldritt and co-workers [34] have trained a convolutional neural net to recognise specific molecular geometries in ultrahigh resolution AFM images; (C) Ziatdinov et al determined the orientation/rotation of single molecules on a solid surface via an artificial neural network. In this case the architecture of a convolutional neural network (CNN) is shown but a variety of other machine vision tools are described by Ziatdinov, Maksov, and Kalinin [35].
learning frameworks for the analysis and classification of images, spectra, and spectromicroscopy data (catalysed by the remarkable success of graphical-processing-unit (GPU)-powered deep learning methods [30] in a staggeringly diverse range of applications in science and beyond.) Perhaps the most impressive example to date of the power of machine learning to extract 'buried' information and insights from experimental SPM data is Zhang et al's [31] use of an artificial neural net architecture to discover electronic correlations in voltage-dependent STM data of doped copper oxide samples uncovering signatures of electronic quantum matter (EQM) that are practically impossible for humans to detect in the dataset. The authors make the key point that traditional reciprocal space analysis methods-in this case, Fourier treatments of STM and x-ray scattering data-tend to focus on very limited regions (i.e. peaks) in k-space, discarding the information on localised/non-periodic regions (such as defects) that is present, but entirely washed out, in the diffraction/scattering/Fourier background 3 . Although Fourier space representations are, of course, exceptionally useful throughout physics-and, indeed, the convolutions at the heart of convolutional neural networks can be cast, like any convolution, as a product of Fourier transforms [33]-by relying on Fourier space analysis of SPM data we negate a signature advantage of STM and AFM as compared to diffraction and scattering techniques: they are real space, local probes. By basing the analysis on the entirety of the real space data-rather than a highly limited region of a k-space representation of those data-the AI cuts through the experimental noise and instrumental limitations to find features and correlations that are otherwise undetectable (or practically undetectable) for a human.
The key advance of Zhang et al's paper, therefore, is that an artificial neural net approach to the interpretation of SPM measurements can not only provide additional insights and understanding to traditional analyses, but that AI can discover structure in the data that is outside the (practical) limits of human-only detection. We hesitate to use the clichéd 'paradigm shift' description but this type of automated mining of experimental data to find specific features encoded in simulated inputs (that have been used as the basis of a neural network training set) is clearly a very different approach to theory-experiment synergy than has been the norm in condensed matter physics (although the core process remains the same: theoretical models are effectively fitted to experimental data.) Zhang et al rightly highlight this advance in the conclusion to their paper: 'the demonstration that artificial neural networks can process and identify specific broken symmetries of highly complex image arrays from non-synthetic experimental EQM data is a milestone for general scientific technique. ' Shortly before Zhang et al's work was published in July this year, a related approach to the automated extraction of 'buried' information from AFM, rather than STM, data was described in what is very likely to be a highly influential paper, by Benjamin Aldritt and his colleagues (and recently accepted in the journal Science Advances) [34]. Building on previous machine learning protocols developed by Sergei Kalinin and co-workers at Oak Ridge National Laboratories (among others) [29,36,37], Aldritt et al have developed what they describe as automated structure discovery AFM (ASD-AFM), a deep learning framework based on a similar methodology to that of Zhang et al, whereby a large set of simulated images is generated-in this case via a combination of density functional theory (DFT) optimisation of molecular structures and the probe-particle model of tip-sample interactions developed by Hapala et al [38,39] -and a convolutional neural net is used to determine the best match between experimental data and a molecular geometry. Figure 2 illustrates the general CNN methodology adopted by Aldritt et al, alongside Ziatdinov, Maksov, and Kalinin's earlier work on determining the rotational state of adsorbed molecules from STM data [35].

Big data, (ultra)small science: nanoinformatics
The exciting examples of machine discovery in probe microscopy described above are a subset of a much wider, and rapidly expanding, field: nanoinformatics. Barnard et al's very recent mini-review, Nanoinformatics, and the big challenges for the science of small things [40], is an engaging and exceptionally timely overview of the power and pitfalls of bringing the 'big data' strategy, applied so successfully in bioinformatics and cheminformatics, to bear on the nanoscopic (and sub-nanoscopic) world. They highlight a number of key, but surmountable, issues with the emerging nanoinformatics discipline including, in particular, the perennial problem of limited datasets. Simulation of large training datasets, as discussed in the preceding section, is clearly an effective strategy to deal with this problem, and the ASD-AFM strategy developed by Aldritt and co-workers could be seen as a sub-field within the much broader area of cheminformatics and AI-powered quantum chemistry, including, for example, the MoleculeNet initiative of Wu et al [41] -a benchmark collection/database for molecular machine learning.
We are, of course, not restricted to theoretical datasets when it comes to machine-learning-enabled probe microscopy but, until recently, there has been a lack of 'connectivity' between various SPM groups with regard to experimental data sharing and collation. SPMImages [42], a collaboration between Ruben Perez's SPM Theory & Nanomechanics Group at Universidad Autonoma de Madrid and Quasar Science Resources SL, has the potential to address this issue, providing a database and 'curation' service for probe microscope images from groups across the world. Ultimately, they plan to incorporate machine learning tools within the web server itself.
Scanning probes are much more than an imaging tool, however. A variety of spectroscopic and spectromicroscopic techniques have been developed over the years since the inception of the technique, providing complementary data that are often invaluable both in interpreting SPM images and in providing insights into the physicochemical properties of the sample (with a resolution down to the single atom or chemical bond limit.) For one, scanning tunnelling spectroscopy falls into a variety of categories including, but certainly not limited to, I(V), dI/dV vs V (which provides information on the joint tip-sample density of states), d 2 I/dV 2 vs V (which yields vibrational information via inelastic scattering events [43]), and spin-polarised tunnelling [44].
Atomic force microscopy also has its own variants of spectroscopy and spectromicroscopy 4 including single point force-distance curves (where 'distance' refers to the tip-sample separation), force-distance maps, potential energy landscapes, damping/dissipation variations, phase maps, Kelvin probe force microscopy, and higher harmonic signal variations as a function of both lateral and vertical position of the probe. Taken together, these various information channels provide an exceptionally rich multimodal (or hyperspectral) multidimensional dataset, which, either in isolation or combined with image data, can be mined via machine learning strategies to not only provide significant improvements in both post-experiment [37] and real-time [45] effective signal-to-noise ratio but, importantly, to classify and determine material properties at the nanoscale and below. Burzawa et al [46] have taken his strategy one step further and adopted machine learning not to extract materials properties but to determine which particular physical model/dynamics drives pattern formation in a system (in their case, the 2D Ising model). A second essential issue identified by Barnard et al [40] in their review of nanoinformatics is the bias inherent in very many AI frameworks. Bias, of course, is not just an issue for machine learning in nanoscience and probe microscopy; it is a fundamental issue with machine learning per se [47]. Although the xkcd [48] cartoon shown in figure 3 perhaps overstates the case a little, there is clearly more than a kernel of truth in the observation that we sift and filter data until it gives us what we want. Supervised training, whereby a human operator (or operators) 'hand-classify' a large collection of data is clearly rather prone (to put it mildly) to the introduction of bias. In this context, it was sobering to note that when we asked a set of probe microscopists to classify different types of image of a hydrogen-passivated silicon surface (more on this particular surface in the following section), we found agreement for less than 80% of the data [49]. When probe microsopists themselves cannot agree fully on image classifications, then we can hardly expect a supervised neural net to do much better. (Nonetheless, and as also discussed below, careful classification and selection strategies in supervised training clearly enable artificial neural nets to compete convincingly with, if not even outperform, their organic counterparts.) One clear route to reducing (human) bias in machine learning is to move from a supervised to an unsupervised learning protocol, where the neural network discovers correlations in the data without being guided by a human. In a review that we highly recommend, Belianinov et al [50] discussed both supervised and unsupervised learning in the context of electron microscopy and SPM, covering principal component analysis, independent component analysis, Bayesian de-mixing, clustering, and a variety of other methodologies and algorithms for the automated analysis of microscopy data of various forms-images, spectra, and combinations thereof. Another very important overview of both supervised and unsupervised learning in SPM is William Dusch's (Penn State University) PhD thesis, available online [51]. (Dusch also describes a fascinating application of machine learning to vibration cancellation for an STM, an extension of the ANITA system developed by Pabbi et al [52].) A very good recent example of unspervised learning is Wahl et al's application [53] of a variant of Lloyd's algorithm (which is closely related to k-means clustering) to find nanoscale heteroegenities in the tunnelling conductance spectra of an iron chalcogenide superconductor, FeSe 0.4 Te 0.6 .

The trouble with tips
Thus far, we have assumed that probe microscope images and data are wholly representative of the sample surface. Any scientist with even a modicum of experience with scanning probe techniques knows that this is a dangerous assumption to make. The literature contains many examples of misinterpreted SPM images in which the influence of the probe was not taken into consideration. While for STM a perturbative approach to the tip-sample interaction is often sufficient (though not as often as one might hope), state-of-the-art atomic force microscopy (of the type pioneered by Gross et al a decade ago [54]) relies on operating in a mode whereby Pauli exclusion plays a key role in defining image contrast. This is far from a perturbation; the tip and sample are coupled to the extent that their separation can be smaller than the equilibrium distance in the associated interatomic or intermolecular pair potential. Figure 4 shows a number of images of two different types of silicon surface taken with the qPlus variant of AFM [3] (top row) and STM (bottom row). (Schematic ilustrations of the atomic structure of each surface are also shown.). STM and AFM studies of this particular silicon surface, Si(100), have a long and chequered history, where it took quite some time for the SPM community to realise that the tip of the microscope is very far from a non-invasive probe of the structure [58]. The set of images in figure 4 clearly illustrates the dramatic effect that changes in the atomistic structure of the tip apex can have on image quality, contrast, and symmetry. Each image is of the same Si(100)-c(4x2) (top row) or H:Si(100)-(2x1) (bottom row) surface, but the convolution of the tip and surface structure gives rise to a wide variety of different image types; it is the tip, not the surface, that is playing the key role in determining the 'sample' structure seen in the images. Note also that these are just a selection of 'good' data from the atomic resolution subset of the very much larger number of images of the surface. A particularly poor tip will not show any atomic resolution at all. This is the problem with probes to which we allude at the start of this Perspective. Although there are strategies to recover, improve, and even sculpt the apex of the tip, including the use of field ion microscopy [59], annealing and ion bombardment [60], and focussed ion beams [61], it is still the case that a very large percentage of a scanning probe microscopist's time is wasted on tip preparation. Moreover, the traditional approaches to tip preparation and recovery are often ad hoc, with a pulse(-or-crash)-move-scan-repeat protocol adopted until the tip is coerced into providing the type of image required. That, in turn, leads to uncomfortable questions re. the objectivity of the microscopist in tip formation and control: which of the images shown in figure 4 is the 'right' image? Moreover, is the highest spatial resolution image necessarily the most appropriate when it comes to spectroscopic analysis or single atom/molecule manipulation?
Almost a decade ago, one of the authors (PJM) and colleagues explored the use of evolutionary strategies (genetic algorithms, GAs), combined with simple rules-based protocols that attempted to mimic human microscopists' behaviour, to improve the imaging capability of an STM tip [62]. (We also explored the use of a combination of GAs and image morphometry to classify the outcomes of Ising model-type simulations of the formation of nanoparticle assemblies [63], with similar motivations to that of the work by Burzawa et al [46] described above.) Figure 5A shows the evolution of the fitness of an STM image of graphite as a function of the number of generations of a GA. In this case the fitness was defined by comparison of the experimental image (via a robust mutual information (RMI) metric) with an ideal, simulated image of the graphite lattice; the imaging parameters were 'mutated' and cross-bred from generation to generation such that the image shown in inset (a) evolved to that shown in (b). A key goal with this GA strategy was to coerce the tip apex into a particular imaging state (beyond a simple good/bad binary selection); in other words, we wanted to select the atomistic structure of the probe. This is shown in figure 5B-E, where the GA drove the tip apex towards a state where it yielded a trigonal (as in figure 5(B)) or honeycomb (figure 5(C)) lattice.
Although the GA approach to tip optimisation/correction performed reasonably reliably in some cases, it suffered, unsurprisingly, from the typical limitations of evolutionary strategies and was difficult to implement as a routine, reliable tool. We generally found that the best strategy to adopt was to use simple rules-based measures (related to corrugation amplitude or RMS roughness, in particular) to recover a very poor tip and then rely on a human, rather than the GA, to 'tweak' the image quality. (An example of this approach is shown in Video 1 in the Supplementary Information (stacks.iop.org/MLST/1/023001/mmedia)), where a tip that has been 'ground' into a graphite sample is recovered to the extent that it provides atomic resolution on the Si(111)-(7x7) surface.) All of this could be said to be in the 'dark ages' of AI, before GPU-powered deep learning had been introduced and when reliable image classification and demarcation was at a much more rudimentary stage than has become possible in the intervening decade or so. A major advance in autonomous probe microscopy was made by Mohammad Rashidi and Bob Wolkow at the University of Alberta in a ground-breaking paper published in 2018 [64], where they not only showed that a CNN framework could be used to identify image artefacts arising from a double (or multiple) tip but embedded the identification and classification algorithms in their STM control software. Their focus is on the H-passivated variant of the Si(100) surface discussed above, comprising rows of H-terminated dimers (see the sketch in figure 4); the resulting H:Si(100)-(2x1) surface [65] has been studied extensively in the context of atomic scale lithography [66][67][68][69][70], single atom electronics [71,72], and fundamental quantum mechanics (including quantum information [73,74]). Desorption of single H atoms is possible via injection of electrons from the STM tip, leading to vibrational heating, and ultimately dissociation, of the H-Si bond.  [3] of the Si(100)-c(4x2) surface. The basic building block of this surface configuration is the silicon dimer, whose formation enables each silicon atom at the surface to form three bonds with its neighbours (rather than the two bonds per atom that arise for the (100) termination of the silicon crystal in the absence of dimerisation). This pairing considerably lowers the surface free energy but a further energy gain [55][56][57], is achieved by the dimers buckling so that one atom moves out of the surface plane, while the other moves towards the underlying crystal. The 'upper' and 'lower' atom of each dimer is shown as a large or small filled circle in the sketch. With an 'ideal' tip, the zig-zag arrangement of 'up' atoms of the dimers is clearly visible in the images. Changes to the atomistic structure of the tip apex during scanning (both deliberate and inadvertent) give rise to rather less easily interpretable images. (See Sweetman et al [18] for a more in-depth discussion.) Bottom Row: Exposing the Si(100)-c(4x2) surface to hydrogen lifts the buckling of the dimers, producing a much simpler (2x1) reconstruction, as sketched schematically at the left of the figure. In this case, the red open circles represent single H atoms, which terminate the silicon dimers, removing all dangling bonds and producing a passivated surface with a low free energy. Again, the atomistic structure of the tip determines whether the individual atoms are observed, or if dimers, rows, or, indeed, any type of atomic scale structure can be observed. Note in this case that although dimer asymmetries akin to buckling are sometimes observed (as in the second image from the left), this is entirely a tip effect. See Gordon et al [49] for more information. The software developed by Rashidi and Wolkow can autonomously detect when the STM shows 'double tip' artefacts and, crucially, can correct accordingly, as shown in figure 6. In a more recent paper (Feb 2019) [75], the UoA group has extended their approach to enable detection, classification, and avoidance of various defect structures on the H:Si(100) surface, which, they argue, is an important step towards autonomous atomic-scale manufacturing.
Although image quality/resolution is, of course, an essential metric to optimise in automated probe microscopy, it is by far not the only parameter of interest. As noted above, the highest resolving tip apex may not necessarily be best suited to spectroscopy and/or atomic/molecular manipulation. (Indeed, some probe microscopists have deliberately blunted the tip apex so as to trade off spatial resolution for higher spectroscopic energy resolution [77]. On this point, Hofer [78] makes some interesting and important observations regarding the role of the uncertainty principle in probe microscopy.) Our analysis of over two thousand STM-induced hydrogen desorption events [79] for the H:Si(100)-(2x1) surface provided strong evidence that the probability of hydrogen extraction was highest for a tip apex that did not yield optimal spatial resolution (due, we postulate, to the role of density of states in defining the relevant tunnel current channels). Similarly, our experiments with tip-induced pushing and pulling of C 60 molecules [80][81][82][83] also often involved drawing a compromise between highest imaging capability and the efficacy of single molecule manipulation.
As such, the capability to coerce the tip into a particular type of state, rather than focussing exclusively on acquiring the highest possible spatial resolution, is particularly valuable in state-of-the-art probe microscopy. Over the last eighteen months, and in collaboration with Ingmar Swart and colleagues at Utrecht University, we have therefore focussed on the issue of discerning the tip state in SPM data. The use of majority voting ensembles of CNNs has proven to be particularly efficient at classifying a set of 13,789 images-subsequently enlarged substantially via a variety of augmentations including arbitrary rotations, crops, and pans-of H:Si(100), Cu(111), and Au(111) into a variety of types including 'atomic resolution' , 'double tip' , 'step edge' , 'tip change' , and 'impurity' [49].
There is, however, a very significant problem with basing machine learning strategies for probe microscopy tip optimisation on complete images: bandwidth. Probe microscopy is already a tediously slow technique when compared to its electron microscopy counterparts (which boast video rate imaging as standard). Human probe microscopists therefore do not wait until a complete image has been acquired before making a decision as to the state of the tip (and how to correct it); they almost invariably base their assessment on partial images, sometimes deciding that tip treatment is (or is not) necessary on the basis of just a few scan lines. In a paper published in the inaugural issue of Machine Learning Science and Technology [84], we describe an effective approach-based on the type of temporal network analysis used in video, rather than image, classification and interpretation-to embedding the heuristics used by probe microscopists in a ML framework. Remarkably, and very encouragingly, we find that an analysis of a rolling window of scanlines (see figure 7) often out-performs the full-image analysis and classification strategies described above.

Are the nanobots nigh?
Around 2030, we should be able to flood our brains with nanobots that can be turned off and on and which would function as "experience beamers" allowing us to experience the full range of other people's sensory experiences. Nanobots will also expand human intelligence by factors of thousands  [84] for the analysis of the state of an STM image/probe via a small subset of scanlines, rather than the complete image. This mimics the approach almost invariably adopted by human probe microscopists who do not wait for an image to complete before making an assessment as to tip quality. The neural network was trained to recognise four categories: 'Rows' (green line in the confidence vs line number graph on the right above); 'Atoms' (orange); 'Dimers/Asymmetries' (blue); and 'Bad/Blurry' (red). In this case the image quality is deliberately poor for the majority of the image. In the first 25% of the image of the H:Si(100)-(2x1) surface (the tip scanned from the bottom to the top of the image on the left) dimer rows are clearly visible and the neural network returns a high confidence value for the "Rows" category. Soon after, a spontaneous tip change occurs, row resolution is lost, and the network's confidence in the 'Rows' category plummets. Row resolution is then recovered for a few scan lines (with a concommitant increase in the 'Row' category confidence and decrease in the 'Bad/Blurry' prediction.) From~Line 256 (of the 512 line image), the network returns a "Bad/Blurry" classification, entirely in line with the poor quality of the image due to a poor tip, substantial thermal drift, and piezoelectric creep. See Video 2 of the Supplementary Information for the line-by-line classification in real time.
or millions. By 2030, nonbiological thinking will be trillions of times more powerful than biological thinking.
Thus spake the entrepreneur, technologist, and futurist Ray Kurzweil at the Foresight Institute's 8th Molecular Nanotechnology Conference held in Washington D.C. back in the year 2000. (The 2030 prediction has been subsequently revised back to 2045 [85]. For now.) Although it is not entirely unknown for pessimism about the pace of technological development to be somewhat misplaced-Thomas J Watson famously predicted in 1940 that there was a world market for 'about five computers' -we think it is safe to say that very few who work at the sharp end of nanoscience and nanotechnology share Kurzweil's dream of a 'singularity' , fusing artificial intelligence, nanotech, and the human condition, happening on anything like a 25 year horizon.
Kurzweil's dystopian vision aside, we can make some slightly less grandiose and rather more grounded suggestions as to the trajectory that machine-learning enabled nanotech is likely to follow over the next decade or so. In this Perspective we discussed how machine learning is more than capable of classifying not just complete SPM images but small numbers of linescans with an accuracy that is at the very least comparable to that of a human operator. It would therefore be very surprising if machine learning frameworks were not increasingly adopted both by scanning probe microscopists (and, more broadly, nanoscientists and condensed matter physicists) and the manufacturers of commercial microscope systems. The day when an "Optimise Tip" button is an integral part of the GUI for an SPM controller is almost certainly fast approaching, with a "Select Tip State" button hopefully not too far behind-no longer will PhD and postdoc probe microscopists have to spend a very significant fraction of their time coercing and cajoling the tip into submission.
Given the multiple tasks expected of the probe (imaging, spectroscopy, and atomic/molecular manipulation), we are also very likely to see an increasing reliance on 'blended' approaches to classifying, characterising, and controlling the tip apex. In particular, and borrowing from our colleagues in astronomy and astrophysics, we have in mind the type of recurrent neural net strategy that has been adopted to characterise supernovae on the basis of their light curves rather than the images themselves [86]; a similar strategy would be particularly interesting to apply to SPM spectroscopic or force-distance data and to cross-correlate with image characteristics.
The type of international data sharing strategy exemplified by Perez et al's SPMImages initiative [42] will hopefully be increasingly the norm in the SPM community. As recognised by early leaders in the field, including, in particular John Pethica (whose comments on interatomic forces in STM inspired the invention of AFM [87]), we probe microscopists have a wealth of unpublished (and generally rather confusing) data squirrelled away on hard drives, floppy disks, DVDs, CDs, and even filing cabinet drawers, that has accrued over the forty years since the invention of the STM. That is, collectively, very many petabytes of data that could be used to train artificial neural nets, or that could be mined (a la Zhang et al and Wahl et al for example) to uncover subtleties that have previously escaped notice. In addition, there is particular scope for a more open source approach to data acquisition. To this end, we have benefitted immensely from a close collaboration with Ingmar Swart's group at Utrecht University (particularly Stephan Zevenhuizen) on the development of open-source [88] Python libraries to interface with the MATRIX control software that 'drives' a very large number of ultrahigh vacuum scanning probe microscopes worldwide. This has facilitated and greatly simplified interfacing of the popular tensorflow machine learning framework with closed, proprietary control software made for a time when automated control of the probe on a scripting level was less practical then constant manual intervention.
Exploitation of machine learning for SPM-driven atomic manipulation, as pioneered by Bob Wolkow and co-workers [64,75], is an especially exciting new direction in nanotech, extending the ideas and strategies described in Celotta et al's groundbreaking paper on the autonomous assembly of atomically perfect nanostructures [89]. There is immense scope to not only teach SPM systems how to move atoms but to let them determine the best approach to building on the atomic scale via reinforcement/Q-learning. 5 Ultimately, the type of nanoinformatics strategies for materials discovery we have very briefly reviewed in this Perspective could be combined with autonomous SPM to not only probe but construct nanostructures with 'dial-in' properties. As ever, bandwidth is the limiting factor-SPM is traditionally a serial technique and is thus very slow. IBM's Millipede project [90], which incorporated very many AFM cantilevers working in concert, was, however, a very clear demonstration of the power of parallel probes.
Extending the Millipede methodology to the atomic level, combined with autonomous control and correction of the apices of the tip array, would enable a dramatic step change in our control of matter with atomic precision. We might not quite put in place the widely critiqued and castigated 'machine phase' vision of nanotechnology put forward by K. Eric Drexler [91] in the 80s and 90s [92]), but it is nonetheless intriguing to note that the convergence of artificial intelligence, single atom precision engineering, and 'tip tech' would not be entirely orthogonal to the type of nanosystems engineering approach Drexler had in mind many decades ago. Drexler over-optimistically envisaged that "assemblers will be able to make virtually anything from common materials without labour, replacing smoking factories with systems as clean as forests" [93]. Given the far-from-negligible environmental impact both of current atomic manipulation strategies and machine/deep learning, Drexler's 100% green nanotech vision is unfortunately not likely to come to fruition any time soon (if the understatement can be excused.) Nonetheless, the dramatic improvements in the control of matter at the single chemical bond limit enabled by the maturation of the nascent hybrid AI-nanotech strategies we have outlined in this Perspective will have substantial knock-on effects in the development of much more energy-efficient technologies.