Correction of scan line shift artifacts in scanning electron microscopy: An extended digital image correlation framework

High resolution scanning electron microscopy (HR-SEM) is nowadays very popular for different applications in different fields. However, SEM images may exhibit a considerable amount of imaging artifacts, which induce significant errors if the images are used to measure geometrical or kinematical fields. This error is most pronounced in case of full field deformation measurements, for instance by digital image correlation (DIC). One family of SEM artifacts result from positioning errors of the scanning electron beam, creating artifactual shifts in the images perpendicular to the scan lines (scan line shifts). This leads to localized distortions in the displacement fields obtained from such images, by DIC. This type of artifacts is corrected here using global DIC (GDIC). A novel GDIC framework, considering the nonlinear influence of artifacts in the imaging system, is introduced for this purpose. Using an enriched regularization in the global DIC scheme, based on an error function, the scan line shift artifacts are captured and eliminated. The proposed methodology is demonstrated in virtually generated and deformed images as well as real SEM micrographs. The results confirm the proper detection and elimination of this type of SEM artifacts. © 2018 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license. ( http://creativecommons.org/licenses/by/4.0/ )


Introduction
Scanning electron microscopy (SEM) has proven itself to be one of the most powerful microscopy methods available. It offers a high spatial resolution (e.g. compared to light microscopy) and relative ease of use (e.g. with respect to transmission electron microscopy). Besides qualitative studies based on SEM images of different materials, quantitative information can be extracted from them as well. Examples of quantitative measurements of in-plane geometrical properties may be found in different fields, such as nanocomposites [1,2] , micro residual stress measurements [3] , biomedical engineering [4,5] and microscopy methods [6,7] . Comparative studies between more than one image introduce another level of quantitative analysis on SEM images. Advanced quantitative measurements go as far as full field displacement measurements based on digital image correlation (DIC) using SEM images in experimental micromechanics [8][9][10][11][12][13] . However, using SEM images for a quantitative geometrical measurement becomes problematic if several complicated imaging artifacts occur [14,15] . These artifacts result in distortions in the image, leading to errors in measurement of the underlying geometrical or kinematical fields. In particular, such distortions induce artifactual deformations and strains in DIC measurements, constituting significant errors if ignored. Due to the sensitivity of full field displacement measurements to these errors, caused by artifactual displacement fields, the DIC community is very concerned with the treatment of such imaging artifacts. The same concern is justified for other quantitative geometrical measurements based on SEM images. SEM imaging artifacts can be categorized into three types [10] . The first type is non-random, time-independent spatial distortion, which is similar to distortions observed in optical systems. A number of papers propose different methods for dealing with this type of artifact [8,16,17] . The second category is nonrandom, time-dependent distortion, referred to as drift, which triggers stretch/compression and/or shear distortions in images. This is a direct result of the scanning involved in the SEM imaging process [18] . Different methods for correcting this artifact are proposed in the literature, for SEM images [8,16,19,20] , and for other microscopy methods involving a similar scanning process [21,22] . The third type is a random, time-dependent distortion which is due to positioning errors of the electron beam during scanning, referred to as "scan line shifts". Lagattu et al. [23] reported the presence of such artifacts, in the form of local peaks parallel to the scan lines in strain maps determined SEM-DIC. Sutton et al. [9] conducted an extensive study on this artifact, which induces jumps in the dis- placement maps obtained by DIC. The effect of scanning parameters in four SEMs from different manufacturers was studied in their work. Integrating a collection of images was a solution, proposed to limit the influence of the scan line shifts. Integration of eight images reduces the artifact to a good extent, for scan line shifts upto one pixel. However, scan line shifts have an amplitude ranging up to 5 pixels, for which a significant artificial strain localization band remains after averaging. This paper proposes a systematic method to resolve the scan line shift artifact for SEM images, applicable to any imaging technique involving a scanning process.
Let us consider the scan line shift artifact in more detail. Fig. 1 exhibits a clear scan line shift in an SEM image. On the left, in Fig. 1 (a), an SEM image of a dual phase steel with a field of view of 30μm is presented. On the right, a zoomed view of the same image is compared to the zoomed view of another SEM image, scanned immediately after the first one (showing the same region). No mechanical displacement was applied to the specimen between the two consecutive scans. The first image contains a scan line shift. This scan line shift is highlighted in Fig. 1 (a) by the in-dicators showing the lower half of the feature moving downwards in the zoomed image on the right. Fig. 1 (b) shows the displacement in the vertical direction found by standard local DIC between the two aforementioned SEM images. Two distinct jumps in the displacement field, one positive and one negative, are visible, occurring at certain y values and constant for all x values ( y is the vertical coordinate, perpendicular to scanning direction, and x is the horizontal, scanning direction).
The occurrence of scan line shifts originates from the SEM imaging process. An SEM image is generated by scanning an electron beam point by point on the surface of the specimen and gathering the electrons that are emitted from the surface, due to the interaction of the electron beam and the specimen, by different sensors. Since the positioning of the beam during the scanning cannot be controlled by a closed loop, positioning errors are inevitable. The underlying reason for the scan line shifts is not reported in literature. A speculative cause of the incidental mispositioning may be a single dust particle in the electron column, which gradually charges up, but then suddenly releases the charge through an electric discharge which causes the sudden mispositioning of the electron beam. This error is a high frequency phenomenon if occurring between pixels in the same row in an image, thus contributing to the noise. However, if such a positioning error occurs between two consecutive scanning lines, a low frequency phenomenon, it becomes detectable as a shift in the image, see e.g. in Fig. 1 (b). The repositioning of the electron beam in the direction perpendicular to the scanning direction is done once per scan line. Hence, an error in this direction persists until the end of that scan line and subsequently propagates to the next line. This type of artifact therefore reveals an artificial band of localized distortion along the scanning direction in the image.
The aim of the current work is to develop a framework to deal with the scan line shifts in SEM images using a systematic approach. This enables: (i) direct removal of scan line shifts from images, (ii) integration with other methods available in the literature, for the correction of drift and spatial distortion artifacts, (iii) future extension of the method to a general framework dealing with all types of SEM artifacts in a unified manner. To this end, global digital image correlation (GDIC) is used as a general framework in order to deal with the scan line shift artifact.
In GDIC the deformation field between two images is directly determined in the whole region of interest of the images. This is done by parameterizing the spatial variation of the displacement field, possibly using prior knowledge on the kinematics characterizing the deformation, i.e. regularizing the displacement field. Using a smooth step function, e.g. error function, for each scan line shift occurring in the images, the artifact field needed for correcting the images can be determined. Contrary to local DIC, the GDIC framework allows for direct insertion of these step function fields in the brightness conservation algorithm. Note that this GDIC framework can be extended in the future to correct for drift and spatial distortion as well.
In order to achieve the aim mentioned above, the standard GDIC formulation is modified for incorporating imaging artifacts ( Section 2.1 -2.3 ). The new formulation is included in a proper procedure for accurately detecting and removing scan line shifts ( Section 2.4 ). The methodology is applied to different virtual experiments to asses the effectiveness, accuracy and convergence behavior of the extended GDIC algorithm ( Section 3 ). Finally the method is applied to real SEM images for two cases, one with mechanical deformation and one without ( Section 4 ).

Brief review of conventional GDIC
The algorithm used in GDIC is based on the minimization of the gray scale residual : which is the difference between the gray value in the reference image f , at original position x , and the deformed image g at deformed position φ( x ). The mapping function φ implicitly defines The Pattern The Image The Pattern The Image the displacement field u ( x ). The notational convention of Neggers et al. [24] is used, where the operator • indicates the composition of two functions. The mapping function is parametrized by a set of degrees of freedom (dofs) stored in the column matrix a = [ a 1 , a 2 , . . . , a n ] T . Due to noise in the images and the inaccuracy of the regularization used in φ(x , a ) , the gray scale conservation leads to a non-zero residual field. This residual field is minimized with respect to a using a least squares approach. As explained in detail in [24] , this results in a nonlinear system of equations which is linearized to yield an iterative Newton-Raphson scheme for updating the dofs, δa , in each iteration: The elements of M and b in eq. (2) are defined as: where i, j = 1 , 2 , . . . , n (with n the number of degrees of freedom) and the domain on which the problem is solved i.e. the region of interest in the images. Note that the matrix M results from the Hessian matrix of a full-Newton method after neglecting two higher order terms, see [24] . The function L i , in Eqs. (3) and (4) is given by: i.e. L i is the inner product of ∇( g • φ( x )), the gradient of the deformed image assessed at position φ( x ) with respect to the de-   on the degrees of freedom, the sensitivity functions can be taken as basis functions of the displacement field. This is often the case, though not always. Different approximations can be made for the image gradient, affecting the resulting convergence behavior, as discussed by Neggers et al. [24] . It is common to choose the gradient of the reference image as an approximation of the image gradient in case of linearly independent basis functions and small deformations. This implies that the L i ( x ) for all dofs (and thus M ) is calculated once at the start of the correlation and used for all subsequent iterations.

New GDIC formulation for imaging artifacts
As discussed in the introduction, if SEM images are used for DIC or any other quantitative measurement, the imaging artifacts should be corrected for, to avoid significant errors in the results. Global DIC can be used to eliminate SEM imaging artifacts in a systematic way, by incorporating them via mapping functions akin to the function φ( x ) in Eq. (1) . However, note that the imaging artifacts may be present in both the reference image f and the deformed one, g . Fig. 2 (a) shows the sequential mechanical and ar-tifactual mappings, where F is the reference pattern free of artifacts, G is the true deformed pattern (also free of artifacts) and f and g are undeformed and deformed images with artifacts, respectively. Fig. 2 (b) depicts the true, artifact-free reference pattern and the position vector of a specific material point, x . In order to identify the corresponding gray scale value in image f , the position vector has to be corrected through the mapping function φ f ( x ) representing the artifacts in the reference image. The mapping function φ f ( x ) provides the pixel position where the gray scale value from the material point x was registered, while scanning image f . This is illustrated in Fig. 2 (c). The same material point in x is mapped onto a new true position by the deformation map φ m (x ) = x + u (x ) . The deformed pattern G thus obtained, is subsequently imaged, and also possibly affected by artifacts, which are characterized by φ g ( x ). The composition of these two mapping functions in image g incorporates both the mechanical deformation and the correction needed for the imaging artifacts. The extended residual in gray scale conservation now reads: Step 1 Step 2

Fig. 9.
Iterative error evolution for virtual images containing one scan line shift only, without noise, and with 10 pixels of perturbation in initial guess, (iterations 1-10: first correlation step; iterations 11-15: second correlation step). Note that in the absence of imaging artifacts, i.e. φ f = x and φ g = x , the conventional definition (1) is recovered, with only mechanical mapping φ = φ m remaining. Note also that, a represents the dofs of all the mapping functions, mechanical and imaging related, as a column matrix. Evidently, each mapping function depends only on the degrees of freedom associated with it. However, to avoid confusing notations, the complete array of dofs, i.e. a , is used everywhere. Starting from this extended definition of the residual field, Eq.
where L f i (x ) , L g i (x ) and L m i (x ) are defined as: The present paper focuses on the scan line shift artifacts. This requires a proper definition of the artifact mapping functions φ f and φ g , capturing the scan line shifts mathematically. Note that the proposed framework has the potential to be extended in the future to incorporate the two other types of SEM imaging artifacts, i.e. drift and spatial distortion, if proper sensitivity functions are included in the artifact mapping functions, which would enable to determine the mechanical deformation field corrected for any artifacts. By focusing on the scan line shift artifacts only, as is done here, the influence of drift and spatial distortion is not eliminated. However, after the scan line shifts have been detected and corrected properly, the procedure introduced in the work by Sutton et al. [8] can be used to correct for drift and spatial distortion.

The new GDIC formulation for the scan line shift artifact
In order to define a proper mapping function to describe the scan line shift artifact, the underlying cause of the artifact and the trace it leaves in the images should be better understood. Fig. 3 illustrates the mispositioning of the electron beam and the image resulting from it. On the left side of Fig. 3 (a) the horizontal lines represent the scanning lines if no positioning error would occur. The dots represent the actual positions where the intensities are registered. In this sketch, a gradual increase of mispositioning from zero to one pixel occurs in four scan lines. Indeed, in real SEM images the scan line shift typically has a width of a few scan line spacings. In the image on the right, the measured intensity is attributed to the presumed pixel position. This results in an artifactual localized stretching of the pattern in the image. A localized compression results in the opposite case, in which the actual scan lines have a larger spacing than the presumed ones, as depicted in Fig. 3 (b). The same mispositioning depicted in Fig. 3 can also occur in the scanning direction ( x direction), leading to an artificial shear distortion in the images.
Based on the description above, the scan line shifts are efficiently described by mapping functions based on the error function, as: where:  cates that the mapping function belongs to the artifacts present in the reference image, while subscript g would refer to the mapping for artifacts in the deformed image. Note that the artifact mapping function is not linearly dependent on the degrees of freedom. This implies that L i ( x ) needs to be updated in every iteration for these dofs. Hence, it is not numerically convenient to adopt the gradient of image f as an approximation of the complete form of the image gradient. Accordingly, the complete image gradient, as described in Eqs. (8) -(10) , is used instead. Fig. 4 shows the graphical representation of the distortion field ( φ f (x ) − x ) based on the error function, where the color corresponds to the norm of the distortion vector and the dashed rectangle shows the undistorted configuration.

Detection and correction of the scan line shifts
In order to solve the nonlinear least squares problem in global DIC, it is common to use a Newton-Raphson scheme, whereby the robustness depends on the initial guess of the degrees of freedom. This initial guess is particularly important for the position of the scan line shift, y 0 , due to the small support of the corresponding sensitivity function. In this section the procedure to detect the presence of scan line shifts in images and the subsequent steps to accurately estimate the corresponding degrees of freedom are discussed.
To detect possible scan line shifts in the images, a precorrelation between two images is performed. The pre-correlation is a simple conventional global DIC procedure using only a set of linear polynomials as basis function, i.e. six dofs. The precorrelation obviously neglects the existence of a scan line shift, if present, which is therefore reflected in residual field. The few degrees of freedom defining the deformation field enable the polynomial to match the displacements on both sides of the scan line shift only in an average sense. This results in a strong gradient in the residual in the direction perpendicular to the scan lines. This pronounced spatial variation in the residual field of a single corre- In order to be able to allocate each scan line shift to the correct image, the pre-correlation should be performed between all three pairs of a set of three images, i.e. three separate pre-correlations on three images of the same area of the specimen. Fig. 5 shows an example (case of Section 4.1 ) of the row average of the residual fields for three pre-correlations between Images 1-2, 2-3 and 1-3, in blue, orange and green, respectively. The horizontal axis is the y-coordinate of each row, along the direction perpendicular to the scan lines. A high gradient in the residual field that occurs in the same position in two pre-correlations represents a scan line shift that exists in the image that is present in both pre-correlations. Thus, by comparing the average residual of three pre-correlations all scan line shifts in the images can be allocated to their corresponding images. A convenient algorithm has been devised to automatically perform the procedure described above and allocate each scan line shift to the corresponding image for each image triple. For more details see Appendix A .
When approximate scan line shift positions ave been established for each image through the pre-correlations, the artifact mapping functions, i.e. φ f and φ g , can be assigned for the final correlation between two images. The resulting initial guess for the position is accurate to within ± 10 pixels. It will be shown in the next section that this accuracy is sufficient to guarantee convergence. Subsequently, the correlation to correct the scan line shifts is performed in two steps. In the first step, the width of each scan line shift is fixed to a large value (here taken as 20 pixels) and only the amplitudes in x and y direction and the position of the scan line shifts are taken as degrees of freedom. The second step is carried out with all four degrees of freedom for each scan line shift, using the results of the previous step as initial guesses. Fixing a large width in the first step causes the support of the sensitivity map related to the position degree of freedom to be large, which relaxes the dependence on the scan line shift position's initial guess and thus improves robustness. This point will be discussed in more detail in the following section.
An overview of the whole process described above is presented in Fig. 6 . distortions associated with the scan line shift artifact are applied. The resulting scan line shift artifact fields are computed from the images using the GDIC approach presented in the previous section. The error in the correlated distortion field is obtained by comparing the computed field with the exact field used in the generation of the deformed/distorted images.

Validation of the method
The generation of the images should not introduce any errors in the virtual displacement field in the images, as these would be attributed to the error in the GDIC method. One of the most common sources of error in DIC in general is the error related to the gray scale interpolation [10] . This bias error is inevitable in the correlation process. When generating virtual images the same inter- polation errors may occur. For this reason discrete interpolation is avoided in the generation process by defining the images as continuous mathematical functions. They are based on the superposition of a number of randomly placed circular Gaussian peaks: where a is the amplitude of each peak, μ x and μ y are the center coordinates of each peak (chosen randomly), and σ x and σ y are standard deviations in x and y direction, respectively. The standard deviations, taken equal here, define the width of each speckle peak. The image is made with two layers of Gaussian peaks corre-  sponding to two different speckle sizes. The addition of the larger speckles (the second layer) improves the robustness of the correlation in terms of initial guess. One of the virtual experiments of Section 3.2 is repeated with only one layer of Gaussian peaks (the finer speckles) to evaluate the influence of the second layer. Gaussian noise with a standard deviation of either 0.5, 1, or 2.5% of the dynamic range has been added to each image. All the images are normalized in gray scale to have a dynamic range equal to 1. The virtual pattern, F , represents the physical pattern on the specimen in a real experiment. Both images f and g are generated from F and are 513 × 513 pixels in size. The deformed image, g , is generated by mapping the pattern F based on ψ, the generation mapping function. In order to find the generation mapping function, let us consider a general case of virtual generation of a deformed image g ( x ) with deformation described by the mapping function φ(x ) = x + u (x ) , from the reference pattern F ( x ). In the absence of noise we have F (x ) = g( φ(x )) for each x . Inserting x = φ −1 (x * ) in the previous equation yields: F ( φ −1 (x * )) = g(x * ) . Thus, the generation mapping function is the inverse of the mapping function that describes the deformation/distortion field that will eventually be present in image g , i.e. ψ ( fined as in Eq. (11) . The virtual experiments are performed on three image categories: (i) one scan line shift in an image pair, (ii) multiple scan line shifts per image in a set of three images, to study the limitations of the method in terms of number of scan line shifts per image and their spacings, and (iii) multiple scan line shifts per image in the presence of mechanical deformation fields. The images of the two later categories are also distorted by other SEM artifacts, i.e. drift and spatial distortion. Note that the first two categories represent the geometrical correction needed when using SEM images, whereas the last category focuses on the measurement of mechanical deformation.

Virtual experiments with a single scan line shift
In the first set of virtual experiments, the robustness of the proposed method in terms of initial guess as well as the influence of noise is evaluated. A pair of virtual images is generated, one without any artifact and one with a single scan line shift embedded. The scan line shift is placed at position y = 3 pixels (the origin of the coordinate system being at the center of the image) and has a width of 4 pixels with amplitudes of 1 and 2 pixels in x and y direction, respectively. The virtual images are then correlated using the approach proposed in Section 2 . In order to assess the influence of the initial guess, a series of correlations are performed with the same set of images, but with a different initial guess for the degrees of freedom of the scan line shift. Since the pre-correlations only serve to provide an initial guess for the position of the scan line shifts, no pre-correlation is performed in this case. The convergence criterion is based on the mean of the absolute value of change in the degrees of freedom, for which a tolerance of 1e −5 is adopted. Fig. 8 shows the average convergence success against random perturbations of a certain amplitude in the initial guess with respect to the exact values. Each point in the plot represents ten correlations for a certain perturbation in the initial guess of the degrees of freedom. A success rate of 100% means that all the ten correlations with that specific perturbation amplitude have converged, while 0% means that none have converged. The only difference between the different curves shown is the noise level added to the images. The standard deviation of the noise implemented in the virtual images has been varied from 0 to 2.5% of the dynamic range. For one noise level, i.e. 1%, also a direct (1-step) correlation with all four dofs is shown, in contrast to the 2-step correlation procedure described in the previous section. The large sensitivity to the initial guess of the 1-step correlation, compared to the 2step correlation, justifies the use of the latter. In case of 0.5% and 1% noise, the convergence success rate is 100% upto more than 90 pixels of perturbation in the initial guess, while this value equals 80 pixels for a noise level of 2.5%, which shows that the approach is very robust. Fig. 9 shows the error in the degrees of freedom characterizing the scan line shift, i.e. the difference of the dofs obtained by the correlation and the reference values used to generate the images (amplitude in x and y direction, position and width), for each iteration during both steps of the correlation. The error for the width is meaningful only for the second correlation step, where it is used as a degree of freedom; therefore it is plotted only for the iterations related to the second correlation step. The final error is negligibly small (bellow 5e −4 pixel) for both amplitudes and relatively small for the position ( 1 . 2e −1 pixel) and width ( 3 . 1e −1 pixel) at final convergence. The higher value of error associated with the position and width, compared to the amplitudes, is due to the fact that an error in the evaluation of position and width increases the residual field only at the location of the scan line shift, whereas the same relative error in the amplitude of the scan line shift affects a large area of the residual. This results in the observed lower sensitivity to the width and position. In summary, the method reveals and adequate robustness for a wide variation in the initial guess and in the presence of noise.

Virtual experiments with multiple scan line shifts
SEM images may exhibit multiple scan line shifts, and may be affected by drift and spatial distortions as well. The limitations of the proposed method in terms of the number of scan line shifts per image and the distance between occurring scan line shifts are investigated in this section. This is done in a more realistic case, i.e. in the presence of drift and spatial distortion and without including an accurate description of them.
Virtual experiments with different numbers of scan line shifts per image, ranging from two to six, are studied. For each case, three images are generated with equally spaced scan line shifts distributed among the images.  shifts in the first, second, and third image, respectively. Note that the minimum distance between scan line shifts in the case of 2, 3, 4, 5 and 6 scan line shifts per image is 73, 51, 40, 32 and 27 pixels, respectively. Each case (with a certain number of scan line shifts per image) is repeated a number of times while keeping the spacing constant but modifying the amplitudes and the widths randomly. The amplitudes are taken from a normal distribution with a mean value equal to 1.5 pixels and a standard deviation of 1.75 pixels, motivated by the more frequent observation of positive amplitudes in practice. The widths are taken from a normal distribution with a mean value equal to 7 pixels and a standard deviation of 1.5 pixels. The cases with two to six scan line shifts include 6, 9, 12, 15 and 18 scan line shifts in total for each image triple, while 8, 5, 4, 3 and 3 image triples are tested, respectively. In this way, a total of at least 45 scan line shifts are included for each case.
To each image triple, additional distortion fields, i.e. drift and spatial distortion, are added as well, Fig. 11 . A radial polynomial of order five ( U s (ρ) = k 3 ρ 3 + k 5 ρ 5 where ρ is the radial distance from the center of the image, and k 3 and k 5 are the degrees of freedom) is chosen to represent the spatial distortion, which resembles the radial distortion of an optical lens [25] . Fig. 11 (a) and (b) show the spatial distortion field in x and y , respectively. Drift in SEM is a continuous phenomenon over time, which progresses while each image is being scanned. The evolution of drift in time is described by a third order polynomial (similar to [9] ), as in Fig. 11 (c). This results in a drift field with a form similar to shear in horizontal direction and stretch in vertical direction. Fig. 11 (d) and (e) show the drift field in x and y in the third image of the triplets, respectively.
Each image triple is used to perform the pre-correlations to identify an initial guess for the positions of scan line shifts and allocate each one to its corresponding image, as explained in Section 2.4 . Fig. 12 shows the results of the pre-correlations, where the success rate in allocating the scan line shifts is the percentage of the scan line shifts allocated correctly. The criteria for assessing the allocation of a scan line shift are: (1) it is allocated to the correct image; (2) the initial guess for its position is within ± 10 pixels of the correct position. Note that the latter tolerance is well below the limit value found in the previous section for the acceptable error in the initial guess (i.e. 80 pixels). Each number of scan line shifts per image corresponds to a certain minimum distance between the scan line shifts, as indicated by the horizontal axis on top of Fig. 12 . It is observed that upto three scan line shifts per image with a spacing of 51 pixels, all scan line shifts are allocated correctly. Beyond this number, the success rate drops bellow optimal, but it remains well above 60% even for six scan line shifts per image, at a spacing of less than 30 pixels.
The result of the pre-correlations, i.e. the allocation of the scan line shifts, is subsequently used to correlate the first two images in each image triple in order to measure the scan line shift artifacts present in them. The artifact mapping functions, φ f and φ g , are defined based on the scan line shifts detected in the pre-correlations. φ m is regularized by first order polynomials. Please note that this mapping function includes, next to mechanical deformation, drift and spatial distortion. These are generally unknown; to mimic this, we use a suboptimal regularization for φ m , to account for these other artifacts in an average sense, in order to get a better accuracy in scan line shift artifact correction. Fig. 13 (a) shows the mean absolute error in the measured scan line shift mapping function, defined as Ē L = 1 where is the region of interest, A is the area of the region of interest, and φ g,ref ( x ) is the reference scan line shift artifact mapping function used to generate the virtual images, in this case images g .
The blue crosses in the graph depict the errors obtained for the image pairs. For each case where all the scan line shifts are allocated correctly (i.e. the cases with two or three scan line shifts per image and one of the cases with four), all the scan line shift artifact fields are measured with a high accuracy as can be expected from global DIC (below or at 0.01 pixels). The cases including initial scan line shift allocation errors finally result in errors in the range of 0.4 to 2.7 pixels depending on the number of wrongly allocated scan line shifts. The brown circles in the graph represent the averaged values. Hence, the method works reliably up to three scan line shifts per image, or 51 pixels of spacing between scan line shifts, and looses accuracy beyond this number.
To put errors in Fig. 13 (a) in perspective, we compare them against the distortions present in the uncorrected images. Therefore, Fig. 13 (b) shows the relative correction of scan line shifts, which is the ratio of the mean absolute error in scan line shift artifact fields and the mean absolute value of the reference scan line shift fields, used to distort the virtual images, subtracted from one (complete correction) for each case, i.e. 1 −Ē L φ g,re f (x ) −x . This value represents the percentage of correction the current method provides, relative to the required correction. In the cases with two and three scan line shifts per image, 99.5% correction is achieved. For the cases where the method is less accurate, i.e. with four, five and six scan line shifts per image, 85.6%, 55.3% and 54.9% correction is still achieved, respectively.
Finally, in order to evaluate the influence of the virtually generated speckle pattern, two extra cases with three line shifts per image are done with two different speckle patterns. In the first case a virtual pattern of only one layer of Gaussian peaks (the finer speckles) is considered, whereas in the second case a pattern from a real SEM image ( Fig. 18 (a)) is distorted virtually, both confirming the obtained accuracy for the case with two layer Gaussian peak pattern.
The proposed method appears to be reliable and accurate up to three scan line shifts per image, and in the presence of drift and spatial distortion, even though a suboptimal approximation was used for the latter. Thus the method is robust removing scan line shift artifacts for the purpose of geometrical image correction.

Virtual experiments with multiple scan line shifts in the presence of mechanical deformation
Another set of virtual experiments is conducted to evaluate the performance and robustness of the proposed method in the presence of mechanical deformation, drift and spatial distortion. Note that, similar to Section 3.2 , again a suboptimal description for drift and spatial distortion is used in this section. Fig. 14 (a) depicts a typical strain field, measured using SEM images and local DIC [26] , exhibiting high strain gradients and localization bands. The figure depicts the ε xx field, measured from images with a horizontal field width of 85 μm after 0.98% of macroscopic strain on a René 88DT (a commercial polycrystalline nickel-based super-alloy) specimen. The strains were obtained by local DIC analysis with 21 pixel (0.4 μm) subset size, a 3 pixel step size and a strain window of 15 pixels [26] . The virtual experiments of this section are performed with mechanical deformations generated virtually but representing the main features of the field shown in Fig. 14 (a). i.e. parallel localization bands. These bands are furthermore assumed to span the entire width of the image. This represents the most challenging case for the correction of scan line shift artifacts, since the mechanical deformation field then looks remarkably similar to the scan line shift artifact field particularly at small angles between the localization bands and the scanning direction (horizontal). The effect of orientation of the localization bands will be studied below. Fig. 14 (b), (c) and (d) show the distribution of the three relevant strain components ( ε xx , ε yy , ε xy ) for the orientation of θ = 30 • . The strain fields represent a background tension of 0.5% with a Poisson's ratio of 0.5 (no volumetric strain) and shear bands that make a 45 • with the tensile axis. Note that the whole strain state is rotated for each case, keeping the angle between the tension and the localization bands constant. The strain amplitudes, the width (30 pixels), and the spacing (150 pixels) of the bands are close to Fig. 14 (a). A range of θ = 90 • (vertical bands) to θ = 0 • (horizontal bands) is considered, of which the latter is the most challenging case when the SEM images are captured using horizontal scan lines.
Note that for the mechanical strain measurement with SEM images, after correcting the scan line shift, it is necessary to correct for the drift and spatial distortion as well. As mentioned before, the method proposed by Sutton et al. [8] can be used for this purpose. In order to correct for the drift artifact with this method, two images need to be acquired at each load step. Even though the focus of the current study is not on the correction of drift or spatial distortion, in order to consider a realistic case, the virtual experiments of this section are conducted with the same strategy of using image pairs for each load step. Fig. 15 depicts the structure of the virtual experiments performed for each localization orientation, including six images (three image pairs). The first row in the figure shows the virtual images while the second row sketches the mechanical displacements incorporated in each image pair. The first image pair (load step one) has no mechanics and the next two pairs incorporate the mechanical deformation in two increments. The third row represents the drift and spatial distortion, and the last row represents the scan line shift artifact fields included in the virtual experiments. Three scan line shifts per image are distributed among Images 1, 2 and 3, with equal spacing, as depicted in the figure. The exact same scan line shifts are repeated in Images 4, 5 and 6.
In order to allocate all the 18 scan line shifts in these six images to the correct images and attain the initial guess values for their positions, two pre-correlation steps are performed. The first pre-correlation is done on Images 1, 2 and 3, while the second one is done on images 4, 5 and 6, as shown in Fig. 15 . Using this procedure, the influence of the mechanical deformation on the precorrelations is minimized.
Once all the scan line shifts are allocated, three correlations are performed to correct the scan line shift artifacts in image pair 1, 2 (no mechanical deformation present, load step 1), image pair 3, 4 (in the presence of the mechanical deformation of load step 2), and image pair 5, 6 (in the presence of the mechanical deformation of load step 3). Note that these three correlations are performed on image pairs with both images containing the same mechanical deformation field. Accordingly, each correlation can be performed as if no mechanical deformation were present. The only difference that the presence of mechanics is imposing, is in the pre-correlations, since there is one image in the pre-correlations of each of the image triples that contains a different mechanical displacement field, see Fig. 15 . Fig. 16 shows the results of both pre-correlations mentioned above as a function of the localization band orientation θ . The suc-  cess rate in allocating scan line shifts is the ratio of scan line shifts allocated correctly (using the same criteria as in Section 3.2 ) versus the total number of scan line shifts, which is 18 here. It is observed that the localization bands start to cause problems in the pre-correlations only when the orientation is almost horizontal, i.e. less than 10 • . This was anticipated since the horizontal localiza- The results of the pre-correlations are used to perform the actual correlations to correct the scan line shifts in the image pairs at constant load. In here, only the results of the correlations on the first image pair (no mechanics) and third image pair (maximum mechanical deformation) are discussed. The results of the second image pair are fully consistent with the ones from the other two pairs. The artifact mapping functions, φ f and φ g , are defined based on the scan line shifts detected in the pre-correlations. The mechanical deformation has no influence on the correlations and a suboptimal parametrization for drift and spatial distortion is used, i.e. φ m is regularized with only first order polynomials to capture the influence of drift and spatial distortion, which is sufficient for an accurate correction of the scan line shift artifacts. Fig. 17 shows the mean absolute error in the measurement of scan line shift artifact fields, Ē L , for different localization orientations. For localization orientations down to θ = 10 • the scan line shift artifact fields are measured with the expected DIC accuracy (approximately 0.01 pixels) for all images (1, 2, 5 and 6). This means that the scan line shift artifact is corrected more that 99% for all these cases. The error is in the same range for Images 1 and 2 in the case of θ = 5 • .
Similar to the results of the previous section, it is confirmed that as long as all the scan line shifts are allocated correctly in the precorrelations, the correction is performed with high accuracy.
It is observed that the proposed method is robust in the presence of almost horizontal ( 10 • and more) localization bands spanning the width of the image, even with presence of drift and spatial distortion, that do not need to be accurately captured.

Application to real SEM images
The method for detecting and removing scan line shift artifacts is next applied to real SEM images. This study is divided into two parts. First, a case without mechanical deformation is discussed, followed by a second case using SEM images taken during an insitu tensile test inside an SEM chamber.

SEM images with no mechanical deformation
A set of three SEM images are taken of a Dual Phase (DP) steel specimen in an FEI Quanta 600F microscope in Secondary Electron (SE) imaging mode. The images are taken one after another without any deformation or translation of the specimen. The field of view of the captured images is about 30 × 30μm, see Fig. 18 (a). The natural pattern present in the images results from etching, resulting in a surface topology that uncovers the microstructure of the material.
A local digital image correlation (LDIC) between pairs of images is performed using the commercial LDIC package VIC-2D TM . A subset size of 41 pixels and a step size of 6 pixels is used for this correlation. Fig. 19 (a) shows the vertical displacement map obtained from the LDIC for one pair of images. Two distinct scan line shifts can be observed, with roughly half a pixel amplitudes (in y direction), one in positive, and the other in negative direction.
Next, the same images are corrected for the observed scan line shifts using the proposed method. Firstly, a series of precorrelations between all three images is performed, resulting in detection of two scan line shifts, one in the first image and the other in the second image. After the allocation of the scan line shifts, the mapping functions of artifacts in f (the first image) and g (the second image), i.e. φ f and φ g respectively, are assigned. First order polynomials define the mechanical deformation mapping function, φ m . Since there is no mechanical deformation in this case, this mapping function only captures the difference in drift and spatial distortion artifacts possibly present in the images, enabling a precise detection of the scan line shift artifacts (i.e. the field identified by φ m is discarded afterwards). The correlation is performed and both images are corrected for the scan line shifts, thereby effectively eliminating the scan line shifts from the images.
To assess the effectiveness of this methodology, LDIC, with identical settings as used for the uncorrected images, is performed on the corrected images. Local DIC is chosen since it requires no parametrization of kinematic fields, and thus provides a nonbiased evaluation of the correction. Note that none of the steps of the proposed method uses local DIC, i.e. all the steps from precorrelations to the main correlation are based on global formulation of DIC. LDIC is only used here as a method for evaluation of the level of correction attained in real SEM images. Fig. 20 presents this difference in an overview: on the left the steps for correcting scan line shift artifacts based on the proposed method (details in Fig. 6 ); and on the right the two LDIC steps for the evaluation of the accuracy of the proposed method. The map of the displacement in y direction thus obtained is depicted in Fig. 19 (b). The effect of the two scan line shifts is completely eliminated from the displacement field. Fig. 21 shows the average row values of U x and U y as a function of the vertical position of the row for the local DIC results before and after elimination of the scan line shifts. In Fig. 21 (b), both scan line shifts, with amplitudes in y direction equal to 0.55 and −0 . 49 pixels, respectively, are eliminated completely, i.e. there is no trace left of the sudden increase or decrease of U y after elimination. The x components of the scan line shifts are eliminated with the same accuracy, as evidenced in Fig. 21 (a).

SEM images of a in-situ tensile test
A set of three SEM images with a large scan line shift were available of a bainitic steel specimen, taken during an in-situ tensile test in an FEI Quanta 600F microscope in SE imaging mode. The images are from three subsequent deformed states of the specimen, which was subjected to tension in the vertical, y direction. Note that in these prior experiments, no image pairs were taken per loading step, increasing the complexity of the problem. The field of view of the captured images is close to 25 × 25μm, see Fig. 18 (b). The natural pattern in the image again results from etching, resulting in a surface topology disclosing the microstructure of the material.
LDIC between the first two images is again performed using VIC-2D TM . The same settings used before (subset size of 41 pixels, step size of 6 pixels) are adopted for this correlation. Fig. 22 (a) shows the vertical displacement map resulting from LDIC. A distinct jump is visible, of a magnitude close to 5 pixels. Note that this is one of the extreme cases of a scan line shift artifact observed with this electron microscope in terms of amplitude and width, making it a relevant test case for validation of scan line shift artifact elimination on real SEM images in the presence of drift, spatial distortion and mechanical deformation.
The proposed procedure is applied to the three images. The precorrelations revealed that the scan line shift occurs in the second image, thus the correlation for the correction of this line shift artifact is performed with the first two images among the three. Since there is only a single scan line shift corresponding to Image g (the second image), the artifacts mapping function of the reference image, φ f , can be ignored. One scan line shift is allocated to φ g with the proper initial guess, obtained from the precorrelations. First order polynomials define the mechanical deformation mapping function, φ m . In this case, the mechanical mapping function represents the mechanical deformation in addition to drift and spatial distortion artifacts affecting the images. Again, the purpose is to secure a proper detection and correction of the scan line shift artifact, not the correct measurement of drift, spatial distortion, and mechanical deformation.
The Image g is subsequently corrected for the scan line shift and local DIC is performed on the corrected image with respect to the same reference image. The corresponding vertical displacement map of this correlation is depicted in Fig. 22 (b). An acceptable elimination of the scan line shift artifact is observed, despite the presence of such a strong artifact and mechanical deformation. Note also that the scan line shift is located close to the top edge of the image, which poses no problem for the methodology. Fig. 23 reveals the row average of U x and U y for the LDIC results, before and after elimination of the scan line shift. The scan line shift of 4.9 pixels amplitude in y direction is properly eliminated. The ratio of the amplitude of the residual of the sudden change in the displacement after elimination to that before elimination is 0.002. The elimination of the scan line shift in the x direction is performed with the same level of precision, as presented in Fig. 23 (a).
Note that in this case, in contrast with the virtual experiments of Section 3.3 , the mechanical deformation field is not complex. The limited complexity in the mechanics makes it possible to perform the pre-correlations and correlation successfully without the need for image pairs for each load step.

Conclusions
Images taken by scanning electron microscopy exhibit artifacts that may result in considerable errors if used for quantitative measurement of, e.g. in-plane geometrical properties or determination of mechanical deformation fields by DIC. These artifacts can be divided into three categories: spatial distortion, drift and scan line shifts. This paper proposes a robust method to deal with scan line shifts, using a systematic approach that does not rely on averaging of images. The nonlinear composition of the effect of mechanical deformation and the distortions caused by the artifacts is taken into account, leading to an extended global DIC framework suitable for dealing with artifacts. A regularization enrichment based on error functions is proposed to identify scan line shift artifacts and to eliminate them from the affected images.
The method is assessed using a series of virtual experiments based on artificially generated and deformed patterns. The influence of the initial guess of the scan line shift parameters on the robustness and convergence is tested, enabling convergence even with an initial guess error of 80 pixels for the considered virtual test cases. The resulting amplitude of the scan line shifts present an error in the order of 5e −4 pixel in this virtual experiment. The position and the width of the scan line shifts are evaluated with ∼ 0.1 and ∼ 0.3 pixel accuracy in all virtual experiments, respectively.
The influence of number of scan line shifts and their spacing is studied in a second series of virtual experiments. It is shown that images exhibiting up to three scan line shifts per image can be corrected with an accuracy expected from GDIC (order of 0.01 pixels). The success rate of the scan line shift artifact correction drops to 92%, 84% and 65% for cases of 4, 5 and 6 scan line shifts per image, respectively.
Another set of virtual experiments reveals the influence of a complex mechanical deformation field, exhibiting localization bands in different orientations, on the correction of multiple scan line shift artifacts per image. The correction of three scan line shifts per image is still successfully performed, with ∼ 0.01 pixel accuracy, when the orientation of the localization bands is > 10 °d ifferent from that of the SEM scanning lines. The proposed methodology is applied to two sets of real SEM images, one without mechanical deformation and the other taken from an in-situ uni-axial tensile test. The images taken reveal scan line shift artifacts up to 5 pixels amplitude. In both cases, all scan line shifts are eliminated accurately. The LDIC maps of the images before and after removal of the scan line shifts emphasize the need for the proposed elimination method and demonstrate the effi-ciency of the method. The proposed method relies on minimal data acquisition for artifact detection and removal, i.e. no averaging over multiple images is required. The adopted deterministic regularization for this type of artifact results in a high robustness, even when applied to extreme cases of scan line shift artifacts (high amplitude of shifts). The newly proposed extended GDIC general framework based on nonlinear mapping functions for the artifacts in principle can be extended to simultaneously deal with other types of artifact as well, e.g. drift and spatial distortion, which will be discussed in future publications. mized. In spite of this, it proves to be reliable in a good range of number and spacing of scan line shifts, as discussed in Section 3.2 . Obviously, further optimization might still be possible.