2.1.

Correlation Mapping Microscopy Processing Technique

The correlation mapping algorithm is applied to consecutive pairs of microscopy images. The first step of the algorithm is to suppress noise, which will cause decorrelation, by applying a “structural mask” to the images. The structural mask includes regions that contain structural information in the microscopy image. The mask is generated by performing a kernel average followed by a binary threshold on the source microscopy images. A kernel is a two-dimensional (2-D) array of image pixels, the size of which is determined by the user.

The next step is determination of the correlation between two temporally consecutive microscopy images captured at the same location. This is calculated by using a Pearson correlation to cross correlate a grid from microscope image A ($I_{\mathrm{A}}$) to the same grid from image B ($I_{\mathrm{B}}$):

The grids $I_{\mathrm{A}}$ and $I_{\mathrm{B}}$ are stepped across the whole image, and a correlation value is calculated at each pixel resulting in a 2-D correlation map. The resultant map has correlation values in the range of $0\pm 1$. This allows for immediate determination of flow regions as higher correlation represents static regions of the sample, and lower correlation values represent regions of flow within the sample. Noise present in the images generated by the system could lower the correlation value of static features, as noise would cause decorrelation. In our image processing, we assume correlation values of $>0.6$ are static features.

In the phantom studies, the ability of the correlation mapping algorithm to contrast the flow region is validated using the correlation signal-to-noise ratio (CSNR). Metrics such as the CSNR have previously been used to evaluate the quality of angiography algorithms.¹⁵ CSNR is defined as

The optimal kernel size must be selected by the user. The kernel size used in this work was arbitrarily chosen for optimal image quality with the trade-off of processing time, spatial resolution, and CSNR. Larger kernels require greater processing times but are less susceptible to noise than smaller size kernels. However, very large kernel sizes causing “blurring” in the correlation map image, which will make smaller regions of flow undetectable from an in vivo perspective. If the aim is to detect smaller capillaries or to measure capillary diameter, this will be a significant issue. If a very small kernel is used, such as $3\times 3$, the resulting data set can be quite noisy as small changes in the structural signal can result in decorrelation and may be mistaken for regions of flow. An optimum kernel size must be determined. In this work, a kernel size of $11\times 11$ was used as it delivered a good quality image with an acceptable signal-to-noise ratio while providing sufficient spatial resolution to avoid incurring any image blurring, which larger kernel sizes would cause.

Implementation of the correlation mapping microscopy algorithm is an “in-house” developed multithreaded code written using the Java programming language. The program can process 30 adjacent microscopy images of size $1000\times 1000\text{pixels}$ in $<1.5\mathrm{s}$ using eight threads. Further speedup could be achieved using optimized code or GPU-based techniques, enabling implementation of the algorithm in real time.

The dynamic range of velocity depends on a variety of factors, including frame rate and resolution. There must be enough time between frames so that the movement of the particles in the liquid can be detected within a given resolution. Confocal microscopy has better resolution, and because of this it should be able to operate at a higher frame rate and give the best dynamic range of velocities.

2.2.

Widefield/Confocal Microscopes and Optical Coherence Tomography System

To validate the correlation mapping microscopy algorithm, a variety of microscope systems was used in this study. The widefield microscope used was an Olympus measuring microscope (STM-MJS2), using a broadband white light source with an Olympus MS Plan $10\times$ objective lens with a numerical aperture (NA) of 0.3, giving the system a lateral resolution of $1.1\mu \mathrm{m}$. Images were captured using a Lumenera Infinity 2 digital CCD camera operating at $\sim 1$ frame per second (fps).

Reflectance confocal microscopy images were acquired with an Andor Revolution spinning disk (Andor Technology Ltd., Belfast, UK) system operating at a wavelength of 640 nm and capturing images at 15 fps. Using a $10\times$ objective lens of NA 0.3, the lateral resolution of this system was $0.85\mu \mathrm{m}$, and the axial resolution was $9.31\mu \mathrm{m}$.

In vivo images were acquired using a reflectance confocal microscope, Vivascope 1500^® (Lucid Inc., Rochester, New York), a commercial clinical research system using an 830 nm (near-infrared) laser. This operating wavelength of 830 nm is at an “optical window” of the skin (mean free path for scattering $\sim 100\mu \mathrm{m}$), and thus the system facilitates investigation of the epidermis and papillary dermis to a depth of $\sim 350\mu \mathrm{m}$ with a lateral resolution of $0.4\mu \mathrm{m}$ and an axial resolution of $1.9\mu \mathrm{m}$ in the center of the image field. The laser power is relatively low at 30 mW, therefore avoiding any tissue damage. An imaging frame rate of 9 fps is achievable with a field of view of $1000\times 1000\mu \mathrm{m}$.

For evaluation of in vivo correlation mapping microscopy images, a comparison was performed with images taken from a commercial OCT system (OCM1300SS, Thorlabs Inc., Newton, New Jersey). This is a Fourier-domain system using a swept source laser (SL1325-P16 Thorlabs Inc.) operating at a center wavelength of 1325 nm and at a scanning rate of 16 kHz, providing an axial resolution of $\sim 12\mu \mathrm{m}$ and a lateral resolution of $25\mu \mathrm{m}$.

3.1.

Widefield Microscopy Phantom Imaging

To demonstrate the ability of correlation mapping microscopy to extract flow with low velocities, a capillary tube sample was fabricated. The sample consisted of a glass capillary tube, mounted on a piece of black card and filled with 3% intralipid solution, which was free to move under Brownian motion. This tube appears on the right-hand side in Figs. 2(a) and 2(b). A separate empty glass capillary tube was also mounted on the black card. This tube appears on the left-hand side in Figs. 2(a) and 2(b). Both capillary tubes had an internal diameter of 100 μm. The widefield microscope focus was set to the center of the capillary tubes, and a series of images were taken as shown in Figs. 2(a) and 2(b). Movie clips are provided for Figs. 2(a) and 2(b). It is very difficult to differentiate any flow/Brownian motion regions from static regions by examining these images and movie. The correlation map resulting from applying the correlation mapping microscopy algorithm to these two temporally consecutive images is shown in Fig. 2(c). A movie is also provided for Fig. 2(c). In this image and in the corresponding movie, it can be seen that the capillary tube on the right-hand side containing the intralipid solution moving under Brownian motion is highlighted by brighter colors, representing lower correlation values. To calculate the CSNR value, a region of interest in both a static area and an area of flow/Brownian motion needed to be analyzed. A $50\times 50\text{pixel}$ region within both the empty capillary tube [blue square in Fig. 2(c)] and capillary tube filled with intralipid [green square in Fig. 2(c)] was selected. The resulting CSNR value was 10.89. This illustrates the ability of the correlation mapping microscopy algorithm to clearly visualize the region of flow within the image.

Fig. 2

Correlation mapping microscopy applied to two temporally consecutive widefield microscopy images, (a) and (b). Each image shows two $100\text{-}\mu \mathrm{m}$ inner diameter capillary tubes. In both images, the tube on the left is empty and the tube on the right contains intralipid solution moving under Brownian motion. The resulting correlation map is shown in (c) with higher correlation values shown as a darker color and lower correlation values as a brighter color. CSNR was calculated between the flow region within the green square and the static region within the blue square. The scale bar in parts (a)–(c) is $100\mu \mathrm{m}$. The correlation map color scale is shown in (d). A movie corresponding to (a)–(c) is provided. Video 1: Example of a widefield microscopy image of phantom capillary tube sample. Video 2: Correlation mapping microscopy movie generated from widefield microscopy frames of a capillary tube flow phantom, as in Video 1. (Video 1, MOV, 530 KB [URL: http://dx.doi.org/10.1117/1.JBO.21.4.046004.1]; Video 2, MOV, 529 KB [URL: http://dx.doi.org/10.1117/1.JBO.21.4.046004.2]).

3.2.

Widefield Microscopy Phantom Imaging Through Scattering Medium

The aim of the next experiment was to test the ability of correlation mapping microscopy to detect flow when imaging through a scattering medium. To test this, the sample used in Sec. 3.1 was covered with a layer of semitransparent scotch tape from the 3M company. This tape has a mean thickness of $60\mu \mathrm{m}$ and a scattering angle of about 5 deg. The widefield microscope was focused to the center of the capillary tubes, and a series of images was taken, an example of which is shown in Fig. 3(a). A movie is provided corresponding to Figs. 3(a) and 3(b). The capillary tube in the upper part of Fig. 3(a) contains 3% intralipid solution moving under Brownian motion; the lower capillary tube is empty. Figure 3(b) clearly shows the upper capillary tube containing the intralipid solution highlighted by brighter colors representing lower correlation values. To calculate the CSNR value, a region of interest in both a static area and an area of flow/Brownian motion needed to be analyzed. A $50\times 50\text{pixel}$ region within both the empty capillary tube [blue square in Fig. 3(b)] and capillary tube filled with intralipid [green square in Fig. 3(b)] was selected. The resulting CSNR value was 5.12. This illustrates the ability of the correlation mapping microscopy algorithm to clearly visualize the region of flow within the image, even when imaging through a scattering medium.

Fig. 3

Correlation mapping microscopy applied to temporally consecutive widefield microscopy images of a capillary tube sample covered with scattering sticky tape. (a) A conventional microscope image of two $100\text{-}\mu \mathrm{m}$ inner diameter capillary tubes; the tube on the bottom is empty, and the tube on the top contains intralipid solution moving under Brownian motion. The resulting correlation map is shown in (b). CSNR was calculated between the flow region within the green square and the static region within the blue square. The scale bar is $100\mu \mathrm{m}$. The correlation map color scale is shown in (c). A movie corresponding to (a) and (b) is provided. Video 3: Example of a widefield microscopy image of phantom capillary tube sample when imaging through scattering medium (MOV, 164 KB). Video 4: Correlation mapping microscopy movie generated from widefield microscopy frames of a capillary tube flow phantom imaged through a scattering medium, as in Video 3 (Video 3, MOV, 296 KB [URL: http://dx.doi.org/10.1117/1.JBO.21.4.046004.3]; Video 4, MOV, 295 KB [ http://dx.doi.org/10.1117/1.JBO.21.4.046004.4]).

3.3.

Confocal Microscopy Phantom Imaging

We next tested the applicability of the correlation mapping microscopy algorithm to confocal microscopy. A capillary tube sample containing three tubes filled with 3% intralipid solution was prepared. The three tubes, with inner diameters 50, 100, and $200\mu \mathrm{m}$, were mounted on a glass microscope slide as shown in Fig. 3(a). Imaging was performed with the Andor Revolution spinning disk confocal microscope, described in Sec. 2.1, with the imaging depth set to approximately the center of the $50\text{-}\mu \mathrm{m}$ inner diameter capillary tube. A movie is provided corresponding to Figs. 4(a) and 4(b). From the conventional microscopy images, it is very difficult to differentiate static regions from regions of flow. The correlation mapping microscopy algorithm, described in Sec. 2.1, was applied to two temporally consecutive confocal microscope images of the sample. The resulting correlation map is shown in Fig. 4(b). Once again, the internal cavity of the capillary tubes is highlighted in brighter colors corresponding to low correlation values, differentiating these regions as containing flow/liquid moving under Brownian motion as opposed to the darker colored static regions between the tubes. To calculate the CSNR value, a region of interest in both a static area and an area of flow/Brownian motion needed to be analyzed. A $50\times 50\text{pixel}$ region within the $50\text{-}\mu \mathrm{m}$ capillary tube containing intralipid was selected to represent a region of Brownian motion [green square in Fig. 4(b)]. Additionally, a $50\times 50\text{pixel}$ region between the 50- and $100\text{-}\mu \mathrm{m}$ capillary tubes was selected to represent a static region [blue square in Fig. 4(b)]. The resulting CSNR value was 6.30. This result demonstrates possibility of the correlation mapping confocal microscopy to clearly visualize regions with flow.

Fig. 4

Correlation mapping microscopy applied to temporally consecutive confocal microscopy images of a sample with capillary tubes of three sizes (inner diameter of $50\mu \mathrm{m}$ at the top, $100\mu \mathrm{m}$ in center, and $200\mu \mathrm{m}$, the upper portion of which is visible at the bottom). Conventional confocal microscope image is shown in (a). The resulting correlation map is shown in (b). CSNR was calculated between the flow region within the green square and the static region within the blue square. The scale bar is $100\mu \mathrm{m}$. The correlation map color scale is shown in (c). A movie is provided corresponding to (a) and (b). Video 5: Example of a confocal microscopy image of phantom capillary tube sample. Video 6: Correlation mapping microscopy movie generated from confocal microscopy frames of a capillary tube flow phantom, as in Video 5 (Video 5, MOV, 4.6 MB [URL: http://dx.doi.org/10.1117/1.JBO.21.4.046004.5]; Video 6, MOV, 4.47 MB [URL: http://dx.doi.org/10.1117/1.JBO.21.4.046004.6]).

3.4.

In Vivo Imaging

To demonstrate the clinical suitability of correlation mapping microscopy and compare it to the established microcirculation imaging modalities, in vivo imaging was performed. Both in vivo confocal microscopy using the Vivascope 1500 and OCT using the Thorlabs systems described in Sec. 2.1 were performed on the volar forearm of a 27-year-old male with informed consent. The OCT image together with its derived cmOCT image of the volar forearm at a depth of $\sim 250\mu \mathrm{m}$ is shown in Figs. 5(a) and 5(b), respectively. These images have a field of view of $3000\times 3000\mu \mathrm{m}$. A zoomed-in portion of the OCT image is shown in Fig. 5(d), and of the cmOCT image is shown in Fig. 5(e). These images have a field of view of $500\times 500\mu \mathrm{m}$, the same as that of images generated by the Vivascope 1500 reflectance confocal microscopy system. It is difficult to make out vessel structure and location from low-resolution OCT and cmOCT images in Figs. 5(a)–5(e).

Fig. 5

In vivo correlation mapping microscopy compared to OCT. An OCT image of the volar forearm at depth $\sim 250\mu \mathrm{m}$ is shown in (a), with a cmOCT image derived from this shown in (b). Part (d) shows a zoomed-in region of (a) of size $500\times 500\mu \mathrm{m}$, the same dimensions of images provided by the Vivascope 1500. (e) cmOCT image related to (d). An image of the volar forearm at depth $\sim 250\mu \mathrm{m}$ provided by the Vivascope is shown in (f). (g) The correlation mapping microscopy image corresponding to (f). CSNR was calculated between the flow region within the green square and the static region within the blue square in (g). Scale bars in (a) and (b) are $1000\mu \mathrm{m}$. Scale bars in (d) to (g) are $100\mu \mathrm{m}$. Correlation scale bar is shown in (c). Video 7: In vivo confocal microscopy of volar forearm. Video 8: In vivo correlation mapping microscopy movie generated from confocal microscopy frames of volar forearm, as in Video 7 (Video 7, MOV, 0.1 MB [URL: http://dx.doi.org/10.1117/1.JBO.21.4.046004.7]; Video 8, MOV, 111 KB [URL: http://dx.doi.org/10.1117/1.JBO.21.4.046004.8]).

An example image of the volar forearm generated by the Vivascope 1500 confocal microcopy system is shown in Fig. 5(f). The dermal papillae are visible in the confocal microscopy image of Fig. 5(f) as bright circular basal layers with a dark center. Inside these circles, the lumina of the capillary loops are apparent as black holes and appear in pairs, with an arterial and a venous capillary located next to one another. After applying the correlation mapping algorithm to these images, the microcirculation is clearly visualized with high contrast [as shown in Fig. 5(g)], allowing us to investigate vessels of size 10 to $20\mu \mathrm{m}$ located in the dermal–epidermal junction layer. The correlation maps can be averaged across a number of frames (five in this case) to give a clearer image. This averaging could be reduced or eliminated, if sample motion is reduced either with improved mounts or motion correction image processing algorithms. Averaging is not performed on the OCT data; however, a motion compensation algorithm is applied to reduce any effect of axial or lateral motion. It is impossible to visualize vessels in the dermal papillae using most currently available commercial OCT systems. The observed locations of vessels in Fig. 5(g) match well with the locations of vessels one would expect from an examination of Fig. 5(f). Note that while the dermal papillae are easy to identify in the confocal reflectance microscopy image of Fig. 5(f), not all of these structures contain currently dilated vessels with blood flow. After application of the correlation mapping algorithm, the regions of blood flow are easily identified. To calculate the CSNR value, a region of interest in both a static area and an area of flow/Brownian motion needed to be analyzed. A static region was identified as a region of tissue outside the dermal papillae, where no blood vessels are present [blue square in Fig. 5(g)]. Within the dermal papillae, blood vessels are present, and a region in this area was chosen as the “flow” region. This is identifiable as the region within the green square in Fig. 5(g). A $50\times 50\text{pixel}$ area of both these static and flow regions was selected, and the CSNR was calculated. The resulting CSNR value was 4.95.

It is noted that it is possible to calculate additional information, such as capillary loop density and vessel diameter, from Fig. 5(g). Calculation of these markers was performed by first loading the in vivo confocal images into ImageJ, which is a public domain image processing tool (version 1.49, available in Ref. 17). The ImageJ software was calibrated with the known pixel-distance relationship. The area within the field of view of the image could then be estimated. The capillary loop density can be calculated by manually identifying and counting the number of detected vessels within this area. The capillary density of the volar forearm has been calculated to be $\sim 28\text{loops}/{\mathrm{mm}}^2$; this value is within the 14 to $30\text{loops}/{\mathrm{mm}}^2$ range previously reported.¹⁶ Additionally, the average capillary loop diameter can be estimated. Capillary size measurements between two reference points were performed by tracing a horizontal line between clearly distinguishable inner borders of capillaries. The ImageJ software calculated the intracapillary distance in micrometers according to the previous calibration. We estimate the average capillary loop diameter to be $\sim 9.50\mu \mathrm{m}$, which is close to the value reported in previous studies.⁶ Such parameters are relatively easy to determine from correlation mapping images as the patent capillaries are immediately identified. Capillaries in the dermal papillae have diameters that are below the resolution of typical commercially available OCT systems and therefore cannot be detected by such systems. Correlation mapping confocal microscopy can complement OCT in scenarios requiring imaging of small blood vessels in the dermal–epidermal junction.