Introduction

The organization of the genetic material inside a cell nucleus is structured.1,2 Evidence has suggested that the different subnuclear structures (e.g., nucleolus, nuclear bodies, transcription factories)3,4 coordinately regulate the positioning and expression of genes.5 The disruption of this organization contributes to the aberrant readout of the DNA's information, and thus, abnormalities of nuclear structure are seen accompanying certain disease states.6,7,8,9,10

To understand how genes coreside around preferred subnuclear structures, it is essential to identify the genes located around these subnuclear structures. We recently developed a new technique capable of extraction and identifying genes located in the proximity of a specific subnuclear structure.11 The technique involves nanomanipulation inside a scanning electron microscope (SEM) to physically extract a minute amount of biomaterial from a desired location inside a cell nucleus, and biochemically amplify and sequence the DNA contained within. We did not choose to use atomic force microscopy12,13,14 for manipulating the subnuclear structures because in atomic force microscopy, the same atomic force microscopy cantilever tip is used for both imaging and manipulation, which makes simultaneous imaging and manipulation difficult, and more importantly can lead to DNA cross-contaminations. Laser ablation and sample catapulting were also used for DNA extraction;15,16 however, the manipulation resolution is limited. Although optical microscopy platforms could be extended for nano-imaging,17 the integration of nanomanipulators can be difficult.

In SEM-based nanomanipulation of subcellular structures, electron beam-induced damage to biomaterials such as DNA must be minimized in order to preserve biochemical integrity. A key step in DNA extraction using nanomanipulation is to correlate SEM images with fluorescence images for identifying targets of interest (Figure 1). Fluorescence imaging permits convenient visualization of subcellular structures, many of which are not contrasted by SEM imaging. SEM images, however, provide higher imaging magnification and resolution provided that attention is paid to accurate sample preservation, and that this can be evaluated objectively. We conducted image correlation manually using commercially available software; however, the process demands careful manual adjustments of image size, position, rotation and distortions to achieve reasonable correlation accuracy.

Figure 1
figure 1

Correlation between SEM and fluorescence imaging for visualizing subcellular structures to identify manipulation targets. SEM, scanning electron microscopy.

Full size image

To improve the speed and accuracy of image correlation, image processing algorithms are developed in this work to automatically establish correspondence between images acquired by different microscopy techniques. Correlative imaging is a challenging multimodal and multiscale image registration problem. The conventional microscopy image correlation techniques require the physical addition of fiducial marks to the sample as landmarks. These landmarks can be easily identified in images captured under different modes of microscopy (e.g., SEM and fluorescence).18,19,20,21 However, our DNA extraction task must be conducted under specific SEM imaging conditions with limited imaging resolution, imposed by the necessity of preserving the biochemical integrity of the sample. Such poor imaging conditions make the identification of nanometer-sized fiducial marks difficult. Although alternative methods also exist that require no fiducial marks22,23 for image correlation, these methods are time-consuming due to the requirement of significant manual interventions (e.g., manually select features, data training, etc.), or require special reference marks on the sample substrate.

To cope with translation, rotation and scaling differences between images, scale invariant feature transform (SIFT) descriptor24 is a powerful technique for detecting a set of local feature vectors for image registration. Mikolajczyk et al.25 evaluated the performance of various local invariant feature descriptors and concluded that the SIFT descriptor outperformed other descriptors. The SIFT algorithm, however, cannot extract the affine transformed (i.e., distorted) features. This challenge was overcome by the fully affine SIFT (ASIFT) approach,26 with the drawback of being significantly more computationally demanding than SIFT.

In this paper, the ASIFT algorithm is used to correlate SEM and fluorescence images for identifying subcellular structures. It was accelerated by a graphic processing unit (GPU) to enhance the speed of SEM-fluorescence image correlation. The correlation results under different imaging conditions were quantitatively evaluated. The suitability of the approach for SEM-based nanomanipulation of subcellular structures is discussed. Based on our previous work,27 this paper provides details on technique implementation and nanomanipulation results. Furthermore, comparison experiments were conducted to demonstrate the advantages of the technique.

Materials and methods

DNA nano-extraction

For the completeness of this paper, a brief overview of our DNA nanomanipulation process is introduced in this section. In sample preparation, cells were sliced into thin cross-sections using cryomicrotome and embedded in glycerol as a cryoprotectant. Sections were then washed in PBS, restriction digested and ligated with a DNA oligo.11 Nuclear structures of interest were then immunolabelled, and fluorescent images were taken. After transferring the sample into SEM, the fluorescent images were correlated with the SEM images to identify targets of interest for manipulation. Figure 2 shows the nanomanipulation set-up inside SEM. Cell sample is mounted on the nanomanipulator, which provides close looped, nanometer motion resolution along XYZ axes.28 The custom fabricated end tool, termed nano spatula, is mounted horizontally on a stationary support, facing the slanted sample substrate. The nanomanipulation system and the nano spatula are controlled to physically ‘scoop up’ the target within a single cell nucleus (Figure 3). Finally, the extracted material is removed from the SEM, amplified by polymerase chain reaction and sequenced to identify the genes.

Figure 2
figure 2

DNA nanomanipulation set-up inside SEM. Cell sample is mounted on an XYZ nanomanipulator, facing the stationary end tool (nano spatula). SEM, scanning electron microscopy.

Full size image
Figure 3
figure 3

DNA extraction from within a single cell nucleus. (a) Locating the cell of interest inside SEM, guided by correlated fluorescence image (not shown). (b) Landing the nano spatula tip onto the target of interest, and slide the cell fragments onto the beveled surface. (c) Lifting up the nano spatula along with the extracted cell fragment. (d) SEM image taken after DNA extraction. The SEM images were taken under higher electron energy (1 kV) and slower scan rate to enhance image quality. In reality, lower electron energy is used (0.4 kV) to ensure survival of DNA. SEM, scanning electron microscopy.

Full size image

To minimize electron beam damage to DNA, the accelerating voltage of SEM imaging must be kept low, at the expense of significantly increasing SEM image noise level. Moreover, the cell sample is tilted to minimize electron penetration depth into the sample, which causes distortions to SEM images. Image distortion caused by sample tilting can be largely compensated by the SEM itself; however, this requires accurate knowledge of sample tilting angles. Our samples are mounted on an adjustable slanted surface. Conventionally, each time when a new sample is mounted, its tilting angle needs to be manually recalibrated through trial and error. In this work, a tilting angle estimation method is developed to avoid trial and error and achieve high estimation accuracies.

The necessary low accelerating voltages and sample tilting, for preserving the integrity of DNA during SEM imaging and nanomanipulation, result in noisy and distorted images. In addition, cells' fluorescence images are taken in the wet state, while the SEM images are taken in the dry vacuum environment. Experimental observations reveal that the cell sample shrinks inside SEM (vs. wet state under fluorescence imaging) because moisture is largely removed by the vacuum environment. This complication further distorts the images.

SEM-fluorescence image correlation

This section describes an image correlation procedure developed for nano-extraction of DNA. The first step is to estimate the cell sample tilting angle. It is input into the SEM to automatically compensate for most of the image distortions caused by sample tilting. SEM images are then denoised in real-time using GPU accelerated non-local (NL) means29 filtering and are correlated with fluorescence images using the ASIFT method.

Most SEMs has an integrated function to compensate for image distortions caused by sample tilting, given that the tilting angle is accurately known. For each DNA nanomanipulation experiment, the sample tilting angle varies slightly due to mechanical misalignments of the set-up. For calibration, a sample containing micrometer-sized reference object (square shown in Figure 4) is placed adjacent to the cell sample. Depending upon the tilting angle (φ) of the cell sample, the sides of the square reference would appear shorter in proportion according to trigonometry. Because the SEM image is nearly an orthogonal parallel projection,30 the tilting angle is

Figure 4
figure 4

Sample tilting angle estimation under SEM imaging. SEM, scanning electron microscopy.

Full size image

Using Canny edge detection and Hough transform, lines are detected and colored in red shown in Figure 4

The NL-means algorithm31 filters a noisy image v(i) by computing a weighted average of all the pixels in a search window.

In our previous work,29 we demonstrated that as a preprocessing step for SEM image analysis, NL-means filtering outperforms other popular denoising algorithms. In this work, we apply NL-means filtering to SEM images before performing image registration for increasing image correlation accuracy.

The ASIFT algorithm introduces an affine model to the SIFT descriptor for effective extraction of distorted image features. Assuming image distortions are caused by viewpoint changes, image distortions can be locally expressed by affine planar transforms.26 The deformation model of image u(x,y) is u(x,y)→u(ax+by+e, cx+dy+f). is a linear planar map having the following decomposition

where α denotes the rotation angle and λ is the scale factor, both of which can be estimated with the SIFT algorithm. φ is latitude angle and β is longitude angle, and the two define the imaging axis orientation.

The ASIFT algorithm involves two steps. Firstly, an image pair (one SEM image and one fluorescence image) to be registered is individually transformed by simulating a large set of linear distortions caused by the change of the longitude angle β and latitude angle φ. Secondly, it performs SIFT comparison of all the simulated images generated to find correct matching features. Since every three pairs of matching point features determine a unique affine transformation, least-squares fit is used to compute a refined affine transformation. With the computed affine transformation, our system overlays the fluorescence image on top of SEM image to identify the target region in the SEM image.

For a typical SEM image with a size of 640×480, the traditional CPU implementation of ASIFT algorithm costs more than 2 min. We accelerated this computation via GPU since the problem can be expressed as data-parallel computations. The first step of ASIFT, simulating distortions by varying imaging axis, can be fully parallelized because the transformations applied to the different pixels are mutually independent. The second step of ASIFT, although not parallelized in nature, can still be accelerated by exploiting GPU memory architectures such as shared memory.

Mean absolute error (MAE) is used in this work to measure the accuracy of image correlation with the ASIFT algorithm. To do manual benchmarking, first a skilled operator carefully aligns more than 15 features over the entire image. Then manual alignment parameters such as scaling ratios and translations are recorded followed by affine transformation matrix calculation. After applying both the benchmarking matrix and the matrix produced by the ASIFT method, each feature point in the fluorescence image receive two different matching points in the corresponding SEM image. MAE is then determined by averaging all errors.

Experimental results

A calibration sample that contains micrometer-sized square features was used to evaluate the effectiveness of sample tilt angle estimation. The dimensions of a tilted square were measured through SEM imaging, and the sample tilting angle was estimated via the method described in previous section. The distortion compensated image based on the estimated sample tilting angle was compared to the actual dimensions of the square feature, measured from the top view. The differences in the square's side length were within 10 pixels (corresponds to about ±2.5° sample tilt). This effective distortion compensation significantly facilitates SEM-fluorescence image correlation.

Experiments were conducted to evaluate the performance of NL-means filter and ASIFT method in terms of accuracy and execution time. The correlation accuracy is quantified through MAE, as described in the previous section. Two main factors influence the image correlation accuracy, including SEM image noise level and SEM magnification. Fluorescence images were taken at the highest magnification with an inherent low noise level. A total of 75 SEM images under different imaging magnifications and different noise levels were correlated with their corresponding fluorescence images.

Figure 5 shows quantitatively how different noise levels in SEM imaging affect image correlation accuracy. Different SEM image noise levels were simulated by choosing different amounts of image frame integration. With higher frame integration, the frame rate became lower. Lower frame rates results in lower noise levels. As shown in Figure 5, the accuracy of image registration is highly dependent upon the noise level of SEM images.

Figure 5
figure 5

(a) Effect of image noise level on correlation accuracy. The highest frame rate of the SEM is ∼20 Hz; thus, 2 frames integration corresponds to 10 Hz frame rate. Under each frame integration setting, ten SEM-fluorescence image pairs were computed, and the error bars represent ±1 standard deviation. (b and c) A sample SEM image taken with two-frame integration, before denoising and after denoising, respectively. SEM, scanning electron microscopy.

Full size image

For nano-extraction of DNA inside the SEM, high frame rates (i.e., lower frame integration) allow more real-time and better visualization of the nanomanipulation process. To improve the correlation accuracy at high frame rates, the system performs NL-means denoising preprocessing on SEM images before correlation is made. This preprocessing step was found to significantly improve the MAE of correlation at high SEM frame rates. In experiments, nanomanipulation was typically conducted under 10 Hz SEM imaging (i.e., ‘frame integration’=2 in Figure 5a). In this case, NL-means denoising improved the MAE of SEM-fluorescence image correlation by 62%, compared to directly performing correlation without denoising SEM images. At low frame rates (e.g., 32 frames integrated), little noise existed; hence, the denoising algorithm undesirably reduced image contrast slightly on these low-noised SEM images, resulting in slightly reduced correlation accuracy.

Since image magnification affects the number of identifiable features in the field of view, it plays a key role in the accuracy of SEM-fluorescence image registration. Under each SEM imaging magnification, 10 random regions of the cell sample were correlated with its corresponding fluorescence images, and the correlation accuracy was computed and summarized in Figure 6a. The results show that correlation accuracy (i.e., reduced MAE) increased with image magnification, up to a magnification of ∼2400×, after which the trend reversed. At 2400× imaging magnification, the MAE of correlation was 68±34 nm.

Figure 6
figure 6

(a) Correlation accuracy under different SEM imaging magnifications. (b and c) Two SEM magnification extremes, where ASIFT was found to have difficulty in identifying matching features needed for image correlation. ASIFT, affine scale-invariant feature transform; SEM, scanning electron microscopy.

Full size image

Under the SEM imaging conditions that ensure low electron beam-induced DNA damage, only the cell membrane contour is easily visible in SEM images. As the image magnification increases, the cell membrane contour becomes better defined from the increase in image resolution, at the expense that the number of cells within the field of view decreases. At magnifications lower than 2400×, the effect of higher image resolution outweighs the loss in having fewer cells/features within the field of view. Therefore, the correlation accuracy improves with increasing magnification. This trend reverses at magnifications higher than 2400× because the actual SEM imaging resolution under the poor, DNA viable imaging conditions no longer improve essentially. At the same time, the number of cells in the field of view continues to decrease at higher magnifications, reducing the number of identifiable features.

Image correlation failure occurred at the two magnification extremes. For magnification <500× (Figure 6b), the poor imaging resolution often does not provide an adequate number of identifiable features for completing the affine transformation. At the other extreme, with magnifications >4000× (Figure 6c), the number of features within the field of view again became too low to achieve high correlation accuracy.

During the process of DNA extraction, several factors caused SEM imaging drift over time. These factors include external electromagnetic interference, sample negative charging and thermal drift of the nanomanipulator. This drift in SEM imaging makes the image correlation process time-sensitive, demanding the rapid completion of image correlation. If the computation of ASIFT takes too long, the transformed fluorescence image would not overlay well with the SEM image.

GPU parallel computing was utilized in our system to speed up the computation of ASIFT. The GPU-accelerated ASIFT correlation was implemented on a standard PC (Intel Core i7 3.6 GHz CPU, 3 GB DDR3 RAM, NVidia GTX560 GPU with 1 GB global memory and 48 kB on-chip shared memory per block). For correlation of image pairs that are 640×480 pixels in size, the GPU-accelerated ASIFT took an average of 15.3±5.7 s to correlate one image pair (n=20). This speed is approximately nine times faster than CPU correlation (averagely 140.5±30.3 s per correlation), and about eleven times higher than manual operation (averagely 174.9±56.4 s per correlation). The average drift rate of our nanomanipulation set-up was 1.24 nm s−1, determined with the method we previously reported.29 With GPU-accelerated ASIFT, a 15.3 s computation time translates to SEM image drift of ∼19 nm.

The proposed method was also evaluated by comparing the correlation results of the existing SIFT and MSER methods. Thirty SEM images captured under different SEM imaging magnifications with two-frame integration were correlated with their corresponding fluorescence images. In all cases, neither SIFT nor MSER found three or more correct matching pairs. Since a minimum of three matching pairs is required to calculate an affine transformation, both methods failed in the image correlation and MAE cannot be calculated.

Figures 7 and 8 illustrate examples of SEM-fluorescence image correlation results. The SEM images were taken under SEM conditions that ensure DNA viability, at a frame rate of 10 Hz. They were denoised in real time by GPU accelerated NL-means filtering. The fluorescence images were converted to grayscale images followed by intensity inversion. Figure 7 shows the registration of the images, where the lines connect matching feature points in the SEM and fluorescence images. Originally 26 correspondences were found in this image pair, only four of which are illustrated in Figure 7. Figure 8 shows the transformed fluorescence image overlaid on the SEM image. The blue rectangle area in Figure 8a shows the location of manipulation target. As shown in Figure 8b, the manipulation target (NB4 cell nuclear body, ∼0.8 µm in size) is the overlapping region between the red and green fluorescence spots highlighted by blue arrow. Figure 8c shows the corresponding location (red star labeled) in the SEM image. The MAE value was quantified to be 91 nm, which is the accuracy of SEM-fluorescence correlation of this image pair.

Figure 7
figure 7

SEM and fluorescence images. Color lines connect the identified matching feature points. SEM, scanning electron microscopy.

Full size image
Figure 8
figure 8

(a) Overlaid SEM and fluorescence images at low magnification. Zooming into a single cell level reveals well correlated features between (b) fluorescence image and (c) SEM image. SEM, scanning electron microscopy.

Full size image

Conclusion

In SEM-based nanomanipulation of biomaterials within subcellular structures, electron beam induced damage to the biomaterials such as DNA must be minimized in order to preserve biochemical integrity. The SEM imaging conditions suitable for this purpose, however, result in noisy and distorted images. This paper described the use of NL-means denoising and ASIFT image correlation for locating a target of interest within SEM and fluorescence images. Through accelerated GPU implementations of the algorithms, denoising of SEM images was achieved in real time, and image correlation was shown to be 11 times faster than manual operation. Various factors influencing the performance of the proposed SEM-fluorescence image correlation process were evaluated. Our SEM-fluorescence correlation approach requires no fiducial marks and no manual feature selection or data training, enabling targeted nanomanipulation of subcellular structures in the SEM.