Direct detectors and their applications in electron microscopy for materials science

The past decade has seen rapid advances in direct detector technology for electron microscopy. Direct detectors are now having an impact on a number of techniques in transmission electron microscopy (TEM), scanning electron microscopy, and scanning TEM (STEM), including single particle cryogenic electron microscopy, in situ TEM, electron backscatter diffraction, four-dimensional STEM, and electron energy loss spectroscopy. This article is intended to serve as an introduction to direct detector technology and an overview of the range of electron microscopy techniques that direct detectors are now being applied to.


Introduction
Direct detectors are an important technological development that have been applied to transmission electron microscopy (TEM), scanning electron microscopy (SEM), and scanning TEM (STEM). The use of direct detectors is becoming more common in both materials and biological electron microscopy due to their ability to record images with a higher signal to noise ratio (SNR) than conventional scintillator-coupled detectors. This brief review article is intended to provide an introductory guide to the fundamentals of direct detectors, and the ways in which they have been applied in electron microscopy. (c) Illustration of CCD readout architecture. After exposure, accumulated charge from each pixel is shifted in parallel, row-by-row to a serial readout register where pixel charge is sequentially converted to voltage. Very high CTE is required for CCDs, but CTE is degraded by radiation damage, necessitating the use of a scintillator for electron microscopy applications. (d) Illutstation of CMOS readout architecture. Here, charge is converted to voltage in each pixel and entire rows of pixels are connected to column wires for simultaneous readout, conferring a significant readout speed advantage compared to CCDs. Most direct detectors for electron microscopy employ a CMOS readout architecture, as do some scintillator-coupled detectors. (a) and (b): Adapted with permission from www.directelectron.com/de-series/. © 2021 DIRECT ELECTRON. microscopy. In contrast to a scintillator-coupled detector, the sensor of a direct detector is radiation hardened to allow it to be exposed directly to electrons. The basic difference between scintillator coupled and direct detectors is illustrated in figures 1(a) and (b). Direct exposure of the sensor to electrons avoids the inefficiency inherent in electron to photon conversion in a scintillator coupled system and allows direct detectors to achieve a DQE equal to or greater than that of film [8].
Most direct detectors for electron microscopy employ a CMOS readout architecture due to the challenges involved radiation hardening CCDs. As illustrated in figure 1(c), accumulated charge from each pixel of a CCD is shifted in parallel, row-by-row to a serial readout register where pixel charge is converted sequentially to a voltage. Very high charge transfer efficiency (CTE) is required for the signal charge from each pixel to make the long transit without loss, but CTE is degraded by radiation damage, which makes direct exposure to the electron beam problematic [9]. In contrast, in the CMOS architecture (figure 1(d)), charge is converted to voltage within each pixel [10]. This confers an advantage for CMOS sensors in terms of readout speed [11], as well as negating the need for charge shifting and high CTE, which makes radiation hardening a CMOS sensor less challenging. Direct detectors can be divided into two broad categories, based on their different pixel designs (figures 2(a) and (b)), which are described in more detail below.

Types of direct detectors for electron microscopy
Hybrid pixel array detectors (HPADs) (figure 2(a)) are one type of direct detector, consisting of a sensor chip bump bonded to a separate application specific integrated circuit (ASIC) CMOS readout chip [12]. The sensor chips used in HPADs are typically relatively thick, for example 300 µm in the case of some detectors based on the Medipix sensor [13], or 500 µm for the electron microscope pixel array detector (EMPAD) described by Tate et al [14]. The thick sensor is employed in order to fully stop incident electrons and thereby capture all of the energy deposited by each electron in the sensor. Pixel width, also commonly referred to as  120 keV, and (e) 300 keV. Incident electrons can travel tens to hundreds of microns into a material, with a lateral spread of tens to hundreds of microns. For HPADs, this necessitates the use of thick, wide pixels. (f)-(h) show the same trajectories, but focus on the initial 8 µm of depth, similar to the thickness of the sensitive layer of a MAPS based direct detector. Under these conditions, the lateral spread of incident electrons is much smaller, allowing MAPS based detectors to incorporate smaller pixel sizes and therefore more pixels per sensor. Trajectories were simulated using the CASINO software [15]. pixel pitch, is determined both by the area required for the readout circuitry in the ASIC and by the diffusion of signal charge in the sensor chip, which is driven by an electric field impressed across the chip [14]. Pixel pitches for HPADs used in TEM vary from 55 × 55 µm for Medipix based sensors [13] to 150 × 150 µm for an EMPAD [14]. Figures 2(c)-(e) shows plots of electron trajectories in silicon (a typical sensor material) at different incident electron energies, simulated using the Casino software [15]. From these simulated trajectories, it is evident that high energy electrons incident on a sensor typically travel a significant depth into the material, and also travel a significant distance laterally. Large pixels are therefore necessary to localize as much of the energy as possible to a single pixel, in order to maximize DQE. This strategy is typically more effective for lower electron beam energies (such as 60 keV) than at higher beam energies (such as 300 keV) because electrons travel a shorter distance within the sensor at lower beam energies. Heavier materials than silicon, for which electrons travel a shorter distance, such as GaAs:Cr and CdTe are being explored for use in HPAD sensor chips as a way to extend the range of voltages for which hybrid detectors perform well [16]. The large pixel size employed in HPADs allow each pixel to accommodate a large amount of charge, and therefore each pixel can accommodate a relatively large number of primary electrons within a single frame without saturation. However, large pixel size restricts the number of pixels that can be incorporated into the sensor, which limits the resolution HPADs can achieve at a given field of view [14].
In contrast to hybrid pixel detectors, monolithic active pixel sensors (MAPS) employ a single chip containing electronics, and a thin sensitive layer with a thickness as low as 8 µm [17]. Almost all incident electrons pass through this thin sensitive layer, depositing only a fraction of their energy within a pixel. Figures 2(f)-(h) shows plots of electron trajectories in silicon with the same incident electron energies as in figures 2(c)-(e), but focus only on the initial 8 µm depth from the silicon surface. Within this thin layer of silicon, the lateral distance travelled by electrons is relatively small. This relatively narrow lateral spread of electrons, allows for smaller pixel sizes to be used in a MAPS e.g. 6.5 × 6.5 µm for a Direct Electron DE-16 or DE-64 detector [18]. Given their smaller pixel size, each pixel of a MAPS can accommodate less charge than the pixels of HPADs, and therefore MAPS pixels can accommodate fewer primary electrons within each frame without saturation. The advantage of smaller pixel size is that it allows for a large number of pixels to be incorporated into a sensor (e.g. 8192 × 8192 in the case of a DE-64) [19], which allows for a high resolution to be obtained at a given field of view. MAPS based direct detectors for TEM have typically been optimized for best performance at higher electron beam energies e.g. 200 to 300 keV, due to the lower lateral spread of electrons at higher beam energies in a thin sensitive layer. However, differently designed MAPS detectors have been demonstrated that are optimized for lower electron beam energies (5-30 keV) [20,21], and also very high electron beam energies (1 MeV) [22].
A third type of direct detector used in electron microscopy is the pnCCD, which uses a different readout architecture to HPADs and MAPS, and is described in detail by Ryll et al [23].

Noise and modes of operation of direct detectors
With the electron beam directly incident on the sensor, all sources of noise and loss of signal due to electron to photon conversion in a scintillator, and photon transfer through a fibre-optic coupling are eliminated. However, images acquired by direct electron detectors are still subject to different sources of noise. All electron microscope images are subject to shot noise [24], which occurs due to fluctuations in the number of electrons emitted within a given time from the electron source. Other sources of noise affect images acquired with direct detectors differently depending upon the mode of operation of the detector. One mode of operation of direct detectors is a charge-integrating mode where the charge generated in the sensitive layer by primary electrons is integrated to form the signal. This integrating mode is used in the EMPAD detector, and can be used in some of the MAPS based detectors. Images acquired using integrating mode are subject to readout noise, and may also be subject to reset noise, although reset noise can be corrected using correlated double sampling. These sources of noise are typically small relative to the most probable intensity generated in direct detectors by a single electron. For example, the most probable single electron SNR, which varies as a function of electron beam energy, has been measured as ∼20:1 in a Gatan K2 MAPS detector at 300 keV, 50:1 in a DE-20 MAPS detector at 300 keV [8], and 70:1 in an EMPAD at 100 keV [25]. For MAPS detectors, an additional source of noise in integrating mode is Landau noise, which arises because each electron incident on the sensor will deposit a differing amount of energy in the sensitive layer, subject to a Landau distribution [26].
An alternative mode of operation of direct detectors is the electron counting mode, which aims to eliminate readout noise, as well as Landau noise in the case of MAPS detectors [26]. Electron counting images recorded by direct detectors are therefore limited only by shot noise. The implementation of electron counting differs between MAPS detectors and HPADs.
Electron counting mode for conventional MAPS detectors involves operating the detector at a high frame rate, with a low probe current, such that each raw frame generated by the camera contains a sparse signal, (e.g. each frame contains only one electron event per approximately 50 pixels). This sparsity allows a counting algorithm to be applied to to each frame, after applying a threshold. The counting algorithm assigns each electron incident on the detector to an individual pixel. Counting each electron in this way eliminates both readout noise, and the Landau noise [26]. The sparsity of each raw frame is necessary to avoid coincidence loss, i.e. events where two or more electrons hit the same part of the sensor at almost the same time, and are incorrectly counted as only one electron [26]. In practice, this means that a large number of counted frames must be summed to form an electron counting image, which means that the effective frame rate of the detector, i.e. the rate at which usable images are generated, is much slower in elecron counting mode than it is in integrating mode. This may be an acceptable trade-off in applications for which obtaining the highest quality images at given electron beam dose is critical. Commercially available MAPS detectors from Direct Electron, Gatan, and Thermo Fisher Scientific all offer an electron counting mode. Direct Electron offers an additional HDR counting feature, which identifies sparse and non-sparse areas in each frame of data, and performs electron counting in sparse regions, and charge integration in non-sparse regions [18].
Electron counting for HPADs involves assigning a count of 1 electron to each event detected within each pixel above a threshold energy, which may be adjusted. Instead of acquiring sparse frames as in conventional MAPS detectors, elecron counting HPADs typically operate the counting electronics in each pixel at a faster rate than the frame readout speed, allowing each pixel to record multiple events in each frame [27]. This electron counting mode is used in Dectris and Medipix based HPADs. Electron counting HPADs typically work up to electron fluxes of 10 6 -10 7 e − pixel −1 s −1 , at which point multiple electrons begin to be registered as a single event, leading to coincidence loss [27,28]. At high electron beam energies, greatly exceeding the threshold energy, where electron charge spreads over multiple pixels (e.g. figure 2(e)), single electrons can also be counted multiple times [16].
The description of modes of operation above applies to frame-based detectors, i.e. detectors that are configured such that when data is read out from the sensor, data is generated from all pixels, or a region of pixels, to generate an image frame. An alternative configuration is an event-based detector, which was originally devised for more data-efficient tracking of moving objects [29]. Instead of generating data from all pixels, event-based detectors only generate data from pixels that experience a significant increase in intensity (above a threshold). In electron microscopy, such intensity changes are due to electron events, and information from multiple electron events can be compiled to generate images. Event-based direct detectors available in electron microscopy include HPADs based on the Timepix sensor [30], and the recently introduced Apollo MAPS detector, in which event-based detection is being explored as a more efficient way to perform electron counting [31].

DQE, MTF, and dynamic range of direct detectors
DQE and modulation transfer function (MTF) are two parameters commonly used to characterize detectors. The MTF and DQE of direct detectors have been evaluated in detail a number of past publications [19,32,33], and will be summarized briefly here. MTF essentially describes how well spatial details are preserved in an image, and is influenced by the point spread fuction of the detector, which describes how the signal due to an incident electron spreads over multiple pixels [34]. DQE essentially describes the strength of the detected signal relative to noise [35], and is important when an experiment is constrained by electron beam dose, as a detector with higher DQE will return a higher quality image at a given beam dose [33]. It is important to understand that DQE and MTF are not constants, but vary with the spatial frequency of the incident signal, and with electron imaging conditions such as electron beam energy, as well as the mode of operation of the detector. For scintillator-coupled detectors, particularly when operated at higher electron beam energies (200-300 keV), although DQE at 0 spatial frequency can be ∼0.4 or better, DQE rapidly falls to <0.2 at 1 /2 Nyquist frequency [32][33][34]. A number of publications have compared different direct detectors and scintillator coupled detectors, finding that MAPS based direct detectors in both integrating and counting mode exhibit significantly higher MTF and DQE, than scintillator coupled detectors at 200-300 keV, especially at higher spatial frequencies [8,19,[32][33][34], and that HPADs exhibit higher MTF and DQE than scintillator coupled detectors at lower electron beam energies [33]. Newer direct detectors have been developed since the date of these publications that further improve upon the performance of earlier direct detectors. For example, the MTF of a DE-64 MAPS detector in counting mode has been measured as ∼1 at 0 spatial frequency and ∼0.9 at 1 /2 Nyquist frequency at 300 keV electron beam energy [36], and the corresponding values for DQE have been measured as ∼0. 95 and ∼0.7 [19].
Dynamic range is another parameter used to characterize detectors and is important for certain applications such as four-dimensional STEM (4D STEM) and selected area electron diffraction, where there may be multiple order of magnitude differences in intensity between bright and weak features of interest in the data. Dynamic range normally refers to the difference between maximum and minimum signals that a detector is capable of recording in an image. Since direct detectors in electron microscopy are capable of single electron sensitivity, manufacturers typically focus on the maximum signal that can be recorded when discussing dynamic range. This depends on both the maximum number of electrons that can be recorded in each pixel in each frame, and also detector frame rate, since a fast detector can produce images with higher dynamic range by averaging or summing frames. Dynamic range is therefore normally reported by manufacturers as a maximum number of electrons per pixel per unit time (typically per second), or as a maximum probe current per pixel. It should be noted that, like single electron SNR, dynamic range will vary as a function of electron beam energy, as primary electrons with different energy will deposit different amounts of energy in the sensor. It should also be noted that in principle, detectors with a high pixel count, but lower maximum probe current per pixel may still be capable of imaging bright features if they are spread over many pixels. Electron counting HPADs have reported a maximum probe current per pixel on the order of 0.2-1 pA [27,28] at 60 keV, whilst the integrating mode EMPAD has reported a maximum probe current per pixel of ∼4.5 pA at 100 keV [25]. Current MAPS detectors exhibit a smaller dynamic range than HPADs when operated in integrating or HDR counting mode, for example, ∼7 fA per pixel for a DE-16 detector, and dynamic range is approximately a further two orders of magnitude lower for MAPS detectors operated in counting mode, due to the need to acquire sparse raw frames. Forthcoming MAPS detectors under development at Lawrence Berkeley National Laboratories [37], and at Direct Electron (the Celeritas detector) [31] aim to substantially increase the frame rate of MAPS detectors, which will in turn increase the maximum probe current that be used without saturating the detectors.

Low dose imaging
The high DQE and SNR of direct detectors compared to scintillator-coupled detectors allows higher quality images to be obtained at given electron dose. This is particularly important when imaging specimens for which electron beam damage is a critical limiting factor to resolution. A prominent example is of course biological cryogenic electron microscopy, for which the 2017 Nobel Prize in Chemistry was awarded [38], and for which the introduction of direct detectors has resulted in a significant improvement in the spatial resolution attainable [39][40][41]. In materials science, there are also numerous examples of beam sensitive specimens that have benefited from low dose TEM imaging using direct detectors, such as metal organic frameworks [42], halide perovskites [43,44], lithium battery materials [45,46], graphitic carbon nitride [47], paint from art masterpieces [48], and polymers [49,50]. MAPS based direct detectors operated in electron counting mode [51] are most commonly employed for low dose imaging due to their high pixel count. Examples of low dose images of a beam sensitive zeolite material acquired with a CCD, and with direct detector in counting mode, are shown in figure 3. Improved SNR allows clearer resolution of the zeolite lattice using the direct detector compared to the CCD.

In situ TEM and ultrafast TEM
In situ TEM is a broad term that may describe a number of different types of experiments. In situ TEM typically involves a specimen that is either held in a gas [52] or liquid [53] environment, or subjected to a type of external stimulus such as temperature change [54], electrical biasing [55], mechanical deformation [56], or may simply involve the observation of radiation induced effects from exposure to ions or the electron beam [57].
Some in situ TEM experiments may incorporate more than one of the factors described above. The common goal of these different types in situ TEM experiments is typically to observe the morphological, chemical or crystallographic evolution of the specimen over time.
For many in situ TEM experiments, a key challenge is to differentiate changes that occur in the specimen due to the environment or external stimulus from changes that result from unintended electron beam damage [58]. This means that in situ experiments often require as low an electron beam dose as practical to minimize the potential effects of electron beam damage, whilst still achieving sufficient resolution to observe interesting changes in the specimen. For other in situ TEM experiments, the goal is to achieve high temporal resolution, which requires high detector frame rates, typically 100 frames per second or faster. Operating a detector at high frame rates necessarily means operating in a regime where the electron dose per frame is relatively low [59]. In both cases, sensitivity of direct detectors offers an advantage over scintillator coupled cameras. As with low dose static TEM imaging, MAPS detectors are typically preferred for in situ imaging due to their high pixel count, which allows a larger field of view to be imaged. With MAPS based direct detectors, an important consideration for achieving high temporal resolution is that operating the detector in integrating mode will deliver usable images at a much greater frame rate than counting mode, as explained in section 1.3 above. For this reason, in situ experiments that aim to maximize temporal resolution have typically made use of direct detectors in integrating mode [60,61].
Examples of materials processes that have been studied with in situ TEM and direct detectors include electrochemical lithiation [62], particle coalescence [63], nanoparticle self-assembly [64,65], atomic resolution observations of deformation under strain [66], and the nucleation of metal organic framework nanocrystals [67]. Materials applications of high temporal resolution in situ TEM with direct detectors have included studying nanoparticle dynamics [60], and studying atomic structural fluctuations in nanocatalyst particles [61,68,69]. Electrostatic subframing systems are being explored in conjunction with MAPS detectors as a method of increasing the temporal resolution of high speed in situ TEM experiments by more than an order of magnitude at the cost of a reduced field of view [70,71].
A different approach to performing high temporal resolution experiments is ultrafast TEM (UTEM). In UTEM, a laser pulse is used to excite electron pulses from an electron gun, and a second laser can excite a specimen with a femto to picosecond time delay between the gun laser and specimen laser [72]. By varying the time delay, and repeating the experiment numerous times, a movie can be constructed to show the response of the specimen to the excitation laser pulse with femto to picosecond temporal resolution [73]. Since the intensity of each individual pulse can be very low, the high sensitivity of direct detectors allows UTEM experiments to be performed more efficiently, with lower repetition rates [74]. Recent materials applications of UTEM with direct detectors have included the visualization of acoustic vibrations in nanorods [75], and tracking the evolution of charge density wave domains [73].

EBSD
Electron backscatter diffraction (EBSD) is an established SEM technique that involves the acquisition of a two-dimensional (2D) EBSD pattern at each position of a 2D SEM scan [76]. This yields a 4D dataset that can be analyzed to extract information that can then be used for crystal grain size and orientation determination, lattice strain measurement, and phase identification. There are ongoing efforts to advance the capabilities of EBSD on multiple fronts, including high resolution EBSD pattern acquisition to more accurately measure lattice strain [77,78], and high speed acquisition of EBSD patterns to improve throughput for the use of EBSD with three-dimensional (3D) serial sectioning [79] and in situ EBSD [80]. Another area of development involves acquisition of EBSD patterns at low electron beam energies (which for SEM means <5 keV), which can increase spatial resolution in EBSD mapping by reducing the electron beam interaction volume in the specimen [81], and may also limit charging effects in poorly conducting specimens [82]. Direct detectors are being investigated for high resolution, high speed, and low beam energy EBSD applications.
The use of a direct detector for EBSD was first reported in 2013 by Wilkinson et al [20]. In this case a 1024 × 1024 pixel MAPS based detector was employed. As stated in section 1.2 above, conventional MAPS based direct detectors had typically performed better at higher electron beam energies (i.e. 200-300 keV). However, Wilkinson et al were able to develop a mechanically thinned MAPS sensor that was sensitive to low energy electrons when back-illuminated [20]. The detector showed promise, even at a beam energy of 5 keV, which is below the energy at which conventional EBSD detectors perform well, but the direct detector was limited to a relatively low frame rate of 28 frames per second [20].
Hybrid pixel direct detectors based on Medipix and Timepix sensors were first reported for EBSD in 2015 [83], demonstrating acquisition of EBSD data with beam energies as low as 5 keV. HPADs have since been employed for applications including 3D EBSD [84], and the measurement of halide perovskite crystal grain size and orientation [85].
More recently, researchers at the University of California at Santa Barbara collaborated with Direct Electron to develop a 2048 × 2048 pixel MAPS detector for EBSD, known as the DE-SEMCam, which has been demonstrated at beam energies as low as 3 keV [21]. The high pixel count of the sensor is intended to allow high resolution EBSD patterns to be recorded. The detector operates at 281 frames per second if all rows of pixels are read. However, the detector is also capable of operating in an acquisition mode known as arbitrary kernel row addressing (AKRA), which allows data acquisition at up to almost 6000 patterns per second [21]. AKRA allows a fraction of rows of pixels on the detector to be read out, while other rows are skipped. The rows that are read out can be distributed across the whole field of view of the sensor [86], effectively subsampling the EBSD pattern as illustrated in figure 4. AKRA is intended to give the detector flexibility to perform high throughput experiments in addition to the high resolution experiments that can be performed at full frame readout. At the fastest reported frame rate, only, only 4.3% of pixels are read out, and frame time is only 4.7% of full frame acquisition time, meaning that the number of electrons used to index each pattern is a factor of ∼500 lower than at full frame readout. For the nickel specimen studied by The same EBSD pattern as in (a), but with only a quarter of the rows read out. (c) The same EBSD pattern as in (a), but with only a sixteenth of the rows read out. (d) An inpainted EBSD pattern produced from data acquired with one quarter of detector rows read out as in (b). (e) An inpainted EBSD pattern produced from data acquired with one eighth of rows read out. (f) An inpainted EBSD pattern produced from data acquired with one sixteenth of rows read out, as in (c). Inpainting was performed using the Telea algorithm in OpenCV [87]. The detector frame rate increases as the number of rows read out is reduced, but the amount of information in the subsampled EBSD patterns is also reduced compared to the full pattern, as evidenced by the degree of pixilation of the inpainted images (d)-(f). However, sufficient information can be retained in the subsampled patterns to perform crystallographic indexing, either on an inpainted pattern, or, in principle, directly on a sub-sampled pattern [21].
Wang et al subsampling each EBSD pattern by a factor of 12.5%, appeared to have relatively little effect on indexing of the EBSPs, and therefore on the spatial resolution in the crystal grain map [21]. Subsampling by a factor of 4.3% led to many patterns being indexed incorrectly. However, the authors were able to demonstrate improved accuracy (92%) using post-collection techniques such as neighbour pattern averaging and reindexing, with grain interiors correctly indexed, and only grain boundaries incorrectly indexed [21]. The effects of the AKRA compressive sensing technique on the spatial resolution of EBSD maps for other materials are still being explored. In addition to the electron optics of the microscope, spatial resolution in EBSD depends upon the success rate of a pattern indexing algorithm in correctly identifying crystal grain orientation. This in turn depends on the probe current that can be applied to the specimen, the strength of the backscatter signal produced by the specimen, and the quality of the indexing algorithm used. Each of these factors may impact the spatial resolution attainable in the AKRA readout mode.

4D STEM and SPED
Conventional STEM detectors, such as bright-field (BF), and annular dark-field detectors, are monolithic. At each probe position of a 2D STEM scan, the total signal received by each of these detectors is integrated and corresponds to the intensity of a pixel in the resulting STEM image. Instead of recording one value of intensity per probe position, 4D STEM, uses a pixelated detector to record a 2D convergent beam electron diffraction pattern, for every position, in the 2D STEM raster over the specimen [88]. 4D STEM bears some similarity to EBSD in that a 4D dataset of diffraction patterns is recorded; however, there are some key differences. In 4D STEM, the electron beam is transmitted through a thin specimen, and the the detector is placed directly in the path of the transmitted beam, on the optic axis of the microscope. Although a similar type of experiment known as on-axis transmission Kikuchi diffraction can be performed in an SEM [89], much of the development of direct detectors for 4D STEM to date has been performed in STEMs.
Direct detectors are attractive for 4D STEM due to their readout speed and single electron sensitivity, allowing them to record data more rapidly than scintillator-coupled TEM cameras. There are a range of techniques that can be applied to analyze 4D STEM datasets generated by direct detectors to extract Here, the electron beam is being deflected towards the centre of atomic columns due to internal atomic electric fields. Colour and arrow size represent the magnitude, and arrow orientation represents the direction of the deflection of the electron beam due to electric fields. information about the specimen. These techniques include generating different types of images of the specimen [12,[90][91][92], (example shown in figure 5(a)), and visualizing electric fields [93][94][95][96][97], (example shown in figure 5(b)), magnetic fields [14,98,99], crystal grain orientations [100][101][102] and strain [103][104][105]. Computational methods may also be applied to 4D datasets to generate images with a spatial resolution that exceeding that of traditional STEM imaging techniques [106][107][108][109].
To maximize the information available in a 4D STEM dataset, it is often desirable to be able to simultaneously record the intense BF disk and the weaker signal in the dark field region of the diffraction patterns. HPADs have therefore been widely used for 4D STEM due to their inherent high dynamic range. For MAPS detectors, the desire to simultaneously image intense and diffuse features has driven the development of techniques such as Direct Electron's 'HDR Counting' method [18]. One present limitation of 4D STEM is that current direct detector readout speeds (1-10 kHz) remain around two orders of magnitude lower than those of conventional 2D STEM detectors (100-1000 kHz). The desire to improve the speed of 4D STEM is driving the development of new MAPS detectors with readout speeds approaching those of conventional 2D STEM detectors [31,37]. For HPADs an order of magnitude increase in frame rate has been demonstrated at the cost of a significant reduction in dynamic range by reducing the bit depth of data recorded to as low as 1 bit, which remains sufficient for some types of 4D STEM experiment [110]. The versatility of 4D STEM has allowed the technique to be applied to a number of different materials to date, including 2D materials [94,102,104,111], metallic glasses [112,113], battery cathodes [101,114], ferroelectrics [97,115], and perovskite oxides [116].
Scanning precession electron diffraction (SPED) is a different TEM based technique, that can also be used for crystal grain orientation and phase mapping [117]. Conventional SPED employs a tv-rate CCD camera to record videos of electron diffraction patterns displayed on the phosphor screen of a TEM, but direct detectors are now being explored as an alternative [117]. Direct detectors have recently been used for SPED analysis of a halide perovskite [118], and a dual phase alloy [117].

EELS and EFTEM
Electron energy loss spectroscopy (EELS) is a technique in which a spectrometer is used to disperse the electron beam according to energy along the horizontal axis of a detector placed behind the spectrometer. 2D images of the dispersed electron beam are recorded and then summed along the vertical (non-dispersive) axis of the detector to form a 1D spectrum [119]. The energy-loss spectrum may be broadly grouped into three regions, an intense zero-loss peak (ZLP), comprised of electrons that did not interact strongly with the specimen, a much less intense core-loss regions, comprised of electrons from the beam that lost energy due to ionization of core-shell electrons in the specimen [119], and a low-loss region, comprised of electrons from the beam that lost energy due to excitations of plasmons [119], vibrational modes in the specimen [120], and other phenomena. Analysis of different phenomena in EELS can provide information about the elemental composition of the sample, in addition to chemical and bonding information [119]. EELS may involve the acquisition of single spectra, or the acquisition of a 3D dataset consisting of a spectrum acquired at every probe position of a STEM scan. Similarly to 4D STEM, the high sensitvity and fast readout speed of direct detectors can allow EELS data to be recorded more rapidly than with conventional cameras, meaning improved throughput, reduced electron dose to the specimen, and particularly in the case of spectral mapping, reduced specimen drift during acquisition time [121]. Both MAPS and HPADs have been applied to EELS, with initial studies using the detectors to acquire spectra from metal oxides [27,[121][122][123][124]. MAPS based direct detectors have exhibited high sensitivity to the relatively weak electron signal in the core-loss region of electron energy loss spectra [122,125]. The high pixel count of MAPS detectors allows them to record electron energy loss spectra covering a large range of electron energy loss, and comparisons with scintillator-coupled cameras have demonstrated that MAPS detectors can achieve a superior energy resolution to scintillator coupled detectors for spectra recorded over a given energy loss range [121,122]. HPADs have shown promise given that their inherently high dynamic range allows them to image very simultaneously image intense features like to ZLP and relatively weak features such as core loss edges in the same electron energy loss spectrum [27,123].
A related technique to EELS is energy-filtered TEM (EFTEM). Instead of acquiring a spectrum at each pixel of a STEM scan as in EELS, EFTEM involves the acquisition of a TEM image at each energy loss selected by an energy filter. Electron beam dose is particularly low when trying to acquire EFTEM images at electron energy losses associated with core-loss ionization, which has led to the exploration of MAPS detectors for use in EFTEM, particularly for imaging biological specimens labelled with heavy elements [126][127][128]. Zero-loss filtering in EFTEM can be employed to filter out inelastically scattered electrons from images of thick specimens to improve contrast, and this technique has been used with direct detectors to image virtified specimens in cryo-TEM [125], and image copper nanocatalysts in an in situ liquid TEM experiment [129].
The recent development of the CEFID spectrometer from CEOS [130], which is designed to be compatible with detectors from different manufacturers offers micoscopists greater flexibility in choosing the type of direct detector that they may want to use for EELS or EFTEM experiments, which may help facilitate the adoption of a broader range of different direct detectors for EELS and EFTEM in the future.

Challenges and opportunities
As described above, direct detectors are now being applied in a range of applications in TEM, SEM, and STEM. Great strides have been made in direct detector technology since their introduction to electron microscopy, but there remain challenges and opportunities for further advances.
One challenge involves development of new direct detectors for TEM that perform consistently well over a broad range of electron beam energies. As described in section 1.2, both HPADs and MAPS based detectors have to date typically performed optimally over a limited range of electron beam energies. Extending the range of beam energies for which a single direct detector performs optimally would offer microscopists greater experimental flexibility. Another challenge is to develop detectors with faster frame rates, which would offer benefits in terms of experimental throughput for 4D STEM as discussed in section 2.4, and would offer simular benefits for in situ TEM, EELS, and EBSD. Another challenge that affects all of the techniques described in this article is how to effieciently analyze and store the large quantities of data that can be rapidly generated by direct detectors. A typical 4D STEM or EBSD dataset may contain tens to hundreds of thousands of frames of data or more, potentially occupying tens of gigabytes to many terabytes of memory. An in situ TEM movie of 16 bit, 16 megapixel frames acquired at 100 frames per second involves data rates exceeding 3 gigabytes per second, or 10 terabytes per hour of continuous acquisition. Researchers are exploring new computational methods such as machine learning as a potential solution to more rapidly analyze the large datasets generated by direct detectors [131][132][133], file storage in an HDF5 format that supports compression has been described as one potential solution for storing these large datasets [134]. Continued investigation by both detector manufacturers and microscopy facilities into methods of analyzing and storing large microscopy datasets will be important, particularly as the capabilities of direct detectors, and quantities of data they can potentially generate, continues to advance over time.

Summary
In summary, direct detectors are an important innovation in electron microscopy. Direct detectors offer improved signal to noise at a given electron beam dose compared to conventional detectors, meaning that the use of direct detectors greatly enhances a wide variety of different SEM, TEM and STEM techniques, including in situ TEM, 4D STEM, EBSD and low-dose TEM imaging. Continued innovation in direct detector technology promises to further benefit electron microscopy in the future.

Data availability statement
The data that support the findings of this study are available upon reasonable request from the authors.