Deep-learning-assisted microscopy with ultraviolet surface excitation for rapid slide-free histological imaging

: Histopathological examination of tissue sections is the gold standard for disease diagnosis. However, the conventional histopathology workflow requires lengthy and laborious sample preparation to obtain thin tissue slices, causing about a one-week delay to generate an accurate diagnostic report. Recently, microscopy with ultraviolet surface excitation (MUSE), a rapid and slide-free imaging technique, has been developed to image fresh and thick tissues with specific molecular contrast. Here, we propose to apply an unsupervised generative adversarial network framework to translate colorful MUSE images into Deep-MUSE images that highly resemble hematoxylin and eosin staining, allowing easy adaptation by pathologists. By eliminating the needs of all sample processing steps (except staining), a MUSE image with subcellular resolution for a typical brain biopsy (5 mm × 5 mm) can be acquired in 5 minutes, which is further translated into a Deep-MUSE image in 40 seconds, simplifying the standard histopathology workflow dramatically and providing histological images intraoperatively.


Introduction
Histopathological examination is essential for tissue-based disease diagnosis. The gold standard is to obtain histologically stained images of thin tissue sections under a brightfield microscope, which requires lengthy and laborious tissue preparation steps: (i) the tissue sample should be formalin-fixed and paraffin-embedded (FFPE), (ii) sectioned by a microtome to get optically thin slices (4-7 µm), (iii) picked up onto slides in the floatation bath and deparaffinized (iv) stained with histological stains (e.g., hematoxylin and eosin (H&E)), and (v) mounted on a glass slide ( Fig. 1(A)). The entire procedure takes at least a day, and usually requires 3-5 days [1], causing a delay for the disease diagnosis, thus, imposing an additional physical and mental burden on patients. The alternative workflow for intraoperative histology can be achieved by frozen section, which takes ∼20-30 mins to process the specimen (including freezing, cutting, and staining). However, the image quality of the frozen section is often unsatisfactory compared to standard histology provided by the FFPE tissue section due to the artifacts caused by poor sectioning and staining qualities [2]. To address this pressing clinical need, it is important to develop a new approach that could provide fast and slide-free histological imaging with high image quality and molecular specificity for accurate intraoperative assessment.
In recent years, various slide-free imaging modalities have been proposed to eliminate the need for a lengthy sample preparation procedure. These approaches include confocal microscopy [3], photoacoustic microscopy [4,5], nonlinear microscopy [6][7][8], structured illumination microscopy [9], and light-sheet microscopy [10]. To improve the interpretability of the images acquired by these slide-free imaging techniques, different models have been utilized to transform these images into histologically stained images, which can be divided into two categories, including analytical-based and deep-learning-based approaches. For the analytical-based approach, the that is comparable to the conventional H&E-stained image after inputting a MUSE image acquired from fresh mouse brain tissue, bypassing the complex sample preparation procedures. (C) The basic structure of CycleGAN to learn the mapping function between the MUSE image (e.g., FFPE mouse brain slices) and any H&E-stained image. Two generator-discriminator pairs are used for model training using unpaired image data. Rec: recovered; SSIM: structural similarity.

Sample preparation and staining
In this study, FFPE thin slices, formalin-fixed and thick, fresh and thick mouse brain, and formalin-fixed thick mouse liver samples were used. Mice were supplied from the Animal and Plant Care Facility at the Hong Kong University of Science and Technology (HKUST). The experiments were performed with the approval of the Animal Ethics Committee and the medical surveillance of the Health, Safety & Environment Office at HKUST.
For FFPE sample preparation (Fig. S1), the harvested mouse brain was fixed with 4% formalin solution for one day. Then, the fixed mouse brain was embedded in a paraffin block, cut into thin sections (7-µm thick) by a microtome (HistoCore AUTOCUT, Leica Biosystems), and placed on a quartz slide. After deparaffinization of the tissue slice with xylene, Hoechst 33342 (500 µg ml −1 in phosphate-buffered saline (PBS)) solution was used to stain the sample for about 20 s. After imaging by the MUSE system, the same slice was stained with H&E following the conventional protocol to get the ground truth histological image.
For fixed and thick mouse brain sample preparation, after embedding the fixed mouse brain into 2% agarose gel, vibratome was used to shave off several 100-µm thick slices until the remaining thick brain sample has a flat surface. Afterward, we used the remaining thick brain with a thickness of several millimeters for the experiment. A low concentration Hoechst 33342 (50 µg ml −1 in PBS) solution was used to stain the sample for about 2-3 mins, which ensures only the tissue surface was labeled. After imaging with the MUSE system, the thick sample was paraffin-embedded, sectioned for a series of layers adjacent to the surface of the block, and stained with H&E to get the reference histological image. For fixed and thick mouse liver sample preparation, the whole procedure is the same except mouse liver sample was stained with DAPI (5 µg ml −1 in PBS) solution.
For fresh and thick mouse brain sample preparation, we cut the brain into halves using a typical blade by hand. Then, we stained the brain with a low concentration Hoechst 33342 (50 µg ml −1 in PBS) solution for about 2-3 mins before MUSE imaging. After that, we put the stained brain into a 4% formalin solution for fixation, embedded it in a paraffin block, and sectioned it into thin slices (7-µm thick). Finally, we acquired the reference bright-field histological image with H&E staining. All the above H&E histological images were obtained with a digital slide scanner (NanoZoomer, Hamamatsu Photonics K.K.).

Data acquisition and processing
The slide-free MUSE images were captured using our lab-built inverted fluorescence microscope with a motorized stage (Fig. S2). The specimen was put on a quartz slide which is UV transparent, and the slide was placed on the sample holder connected to a three-axis motorized stage. A UV light from an obliquely oriented 265-nm LED (M265L4, Thorlabs Inc.) was focused on the Hoechst-stained sample. Subsequently, the excited fluorescence signals were collected using a 10× objective lens (RMS10X-PF, NA = 0.3, Thorlabs Inc.), and focused on a color camera (DS-Fi3, 5.9 megapixels, Nikon Inc.) using an infinity-corrected tube lens (TTL180-A, F = 180 mm, Thorlabs Inc.).
We developed a LabVIEW-based software user interface to acquire data by delivering instructions to a single-chip microcomputer (SCM, Mega 2560, Arduino) in order to trigger the camera and motorized stage to scan different areas along x and y directions. The exposure time of the camera for each FOV (∼691 µm × 491 µm) is ∼500 ms. The mosaic FOVs would be stitched with a 20% overlapping area in MATLAB after scanning the whole slice.
For FFPE samples, the MUSE and the corresponding H&E-stained images of the same thin section can be provided. Therefore, it is possible to implement both pix2pix and CycleGAN models, and then evaluate their performance by comparing them with the ground truth H&Estained images. The ROI between these two imaging modalities was registered in MATLAB through a complex image registration process. First, the global registration process scaled the pixel size, achieving rotation and shift corrections between the same ROI of the MUSE and H&E-stained images [26]. Then, the local registration process was performed to accurately register the input and target image pairs at the patch level by applying an elastic image registration algorithm [27]. These image pairs were used for pix2pix model training and evaluation. The size of the image patch was 256 × 256. The training datasets include 1011 paired MUSE and H&E-stained image patches from the same FFPE mouse brain slice. The validation datasets include 479 paired MUSE and H&E-stained image patches from another FFPE mouse brain slice. For the CycleGAN model, the unpaired training datasets include 5883 MUSE and 5846 H&E-stained image patches.
For fixed mouse brain samples, the CycleGAN model was trained with unpaired 1505 MUSE and 1502 H&E-stained image patches. The validation datasets include 3600 MUSE image patches from another fixed mouse brain sample. For fixed mouse liver samples, the CycleGAN model was trained with unpaired 1288 MUSE and 1481 H&E-stained image patches. The validation datasets include 805 MUSE image patches. For fresh and thick mouse brain samples, the CycleGAN model was trained with unpaired 3177 MUSE and 2574 H&E-stained image patches. The validation datasets include 2500 MUSE image patches from another fresh mouse brain sample.

Network architecture
The GAN [13] architecture is applied to learn the style transformation from slide-free MUSE images to their equivalent bright-field H&E-stained images. A standard DCNN model uses an encoder-decoder network [28] for image translation tasks. For example, the popular encoderdecoder model "U-Net" [29] adds skip connections to transfer the low-level information directly across the network. The L1 loss function (mean absolute error) or L2 loss function (mean squared error) are the general option to minimize the distance between network output image and target image. However, these two loss functions tend to produce blurry images caused by the pixel-wise average of plausible results in the pixel space [30]. To address this problem, GANs can output sharp and realistic images through the adversarial training of two DCNN models, including a generative model (Generator) and a discriminative model (Discriminator). The generator G aims to fool the discriminator D by producing fake images similar to target images, while D intends to classify the generated fake image and real target image. In our case, the generator G learns the statistical transformation between MUSE and H&E-stained images, whereas the discriminator D aims to distinguish Deep-MUSE images (i.e., fake generator output) from real bright-field H&E-stained images. We used two types of GAN models, including pix2pix [16] for paired image translation tasks and CycleGAN [18] for unpaired image translation tasks.
Taking the FFPE samples as a demonstration, we can use these two GAN models. The architecture of pix2pix ( Fig. S3(A)) includes one generator G and one discriminator D. The generator G translates image domain X (MUSE images) into image domain Y (H&E-stained images). The MUSE image x is fed into the generator G and output the fake virtual H&Estained image (Deep-MUSE image)ŷ = G(x). The discriminator D makes a judgment about the concatenation of the MUSE image x with the fake virtual H&E-stained imageŷ or real H&E-stained image y. The loss function of GAN can be expressed as: images: The full objective of pix2pix is: For many applications, it is hard to acquire paired data for pix2pix model training. To overcome this challenge, the architecture of CycleGAN ( Fig. 1(C)) was designed for performing image translation with unpaired data. From the preliminary experiment, we found that the model trained by original MUSE images and H&E-stained images will output low-quality Deep-MUSE images (Fig. S4). The model converted bright cell nuclei and dark background in MUSE images into a white background and purple nuclei in Deep-MUSE images, respectively. The general tendency is that the CycleGAN model will map the dark color (black) in MUSE images into the dark color (purple) in H&E-stained images. To address this problem, we have inverted the color of MUSE images for model training and inference, which leads to an accurate style transfer function.
We assigned the inverted MUSE image x to image domain X, and the H&E-stained image y to image domain Y. The generator G maps image domain X to image domain Y by transforming the real MUSE image x into fake virtual H&E-stained imageŷ = G(x). The generator F maps image domain Y to image domain X by transforming real H&E-stained image y into fake MUSE imagex = F(y). The discriminator D X classifies real MUSE image x and fake MUSE imagex. The discriminator D Y classifies real H&E-stained image y and fake virtual H&E-stained imageŷ. The objective of the adversarial loss function is: With this GAN loss only, it is not possible to guarantee successful transformation between the MUSE image and H&E-stained image without pair images for supervised training. It is important to make sure that all the content in MUSE images can be preserved (e.g., the nuclear size and count), and the style should be visually similar to H&E-stained images during the transformation. By combining the mapping function of the generators G and F, we can impose an extra constraint named cycle consistency loss [18]. A MUSE image x from domain X will become recovered imagex = F(G(x)) after passing the generators G and F, i.e., F(G(x)) = x. Similarly, an H&E-stained image y from domain Y is converted to recovered imageŷ = G(F(y)) after going through the generators F and G, i.e., G(F(y)) = y. Therefore, we used L1 loss between original images and recovered images as this cycle-consistency loss: To further improve the perceptual quality of the recovered images, we also calculated the structural similarity (SSIM) index [31] between the original image x and recovered image y as the perceptual distance. The SSIM index is composed of three functions including luminance l(x, y), contrast c(x, y), and structure s(x, y): where µ x and µ y refer to the mean values of image x and image y, respectively. σ x and σ y refer to the standard deviations of image x and image y, respectively. σ xy stands for the covariance between image x and image y, and c 1 and c 2 stand for constants. The expression can be simplified by setting α = 1, β = 1, γ = 1 and c 3 = c 2 /2. The value of the SSIM index is ranged from zero to one. Then, the loss function is constructed by one minus the SSIM index. In our experiment, the CycleGAN model trained with SSIM loss outputs smooth and authentic Deep-MUSE images (Fig. S5). SSIM loss is calculated between original images x, y, and the recovered imageŝ The generators G and F should be identity mapping for input H&E-stained image y and MUSE image x, respectively [32], i.e., G(y) = y and F(x) = x. The L1 loss function can be used to measure the image distance: The final objective of CycleGAN becomes: The generator and discriminator architectures ( Fig. S3(B)) are the same in pix2pix and CycleGAN. Only the input images of the generator and discriminator are different as mentioned above. The generator is adopted from the image transformation network that contains convolutional layers for image downsampling and upsampling [33]. Several residual blocks [34] connect the downsampling and upsampling blocks. The discriminator networks use PatchGAN [16] to preserve the high-frequency structures in the images. The real or fake images will be classified at the local image patch level after passing through the convolutional layers.
For pix2pix experiments, we set λ = 10 in Eq. (3) and train the model with 155 epochs. For CycleGAN experiments, we set λ 1 = 10, λ 2 = 5, λ 3 = 1 in Eq. (1)3 and train the model with 70 epochs. The networks were trained from scratch with a learning rate of 0.0002. The Gaussian distribution with a mean = 0 and standard deviation = 0.02 was used to initialize the weights of the network. An Adam optimizer was used to minimize the overall loss functions.

Novel workflow
By the integration of the MUSE system and deep neural network, we enable a new histopathology workflow ( Fig. 1(B)) that can provide rapid slide-free histological imaging of unprocessed specimens with various thicknesses. The MUSE system (Fig. S2) is applied to generate fluorescence images of a fresh tissue specimen by UV-LED illumination directly. This system can achieve high resolution and high imaging contrast with the visible light emission of fluorescent dyes excited by UV light, which is sufficient to reveal the morphology of individual cell nuclei.
The MUSE image quality is comparable to the bright-field H&E-stained images obtained from a digital slide scanner using an objective lens with the same numerical aperture (NA). The unsupervised deep learning model ( Fig. 1(C)) learns the style transfer function between the MUSE and H&E-stained images after the bidirectional training of unpaired image data. In whole-slide imaging, we have considered an imaging FOV of 5 mm × 5 mm, which is a typical size of a brain biopsy for comparison. The MUSE imaging system takes ∼6 mins only to generate Deep-MUSE images for the entire workflow, whereas the NanoZoomer takes ∼2-3 mins to obtain a brightfield image with the same FOV after lengthy sample preparation steps. We validate the proposed workflow using mouse brain tissue samples with different thicknesses and preparation protocols including FFPE thin sections, fixed thick mouse brain sections cut by vibratome, and fresh thick mouse brain sections cut by hand. The results show that the CycleGAN model is capable of transforming MUSE images into Deep-MUSE images that are comparable to the brightfield H&E-stained images regardless of the sample thickness.

Evaluation metrics
To evaluate the Deep-MUSE image quality of different mouse brain sample, we use three quantitative metrics including maximum mean discrepancy (MMD), Pearson correlation coefficient, and peak signal-to-noise ratio (PSNR).
The simplified mathematic form of MMD can be represented as: where image data distribution X has n samples, image data distribution Y has m samples, and the H term refers to kernel embedding of distributions.
Pearson correlation coefficient is defined as: where cov(X, Y) stands for covariance between data distributions X and Y, σ X and σ Y refer to the standard deviations of data distributions X and Y, respectively. In our case, a Deep-MUSE image x and an H&E-stained image y were used as data distributions X and Y. PSNR is defined as: where MAX I equals to 255 for 8-bit image data, and MSE represents the mean squared error between an input image X and a target image Y with an image size of m × n. For color images, the MSE of R, G, and B channels were averaged.

Imaging of FFPE thin mouse brain sections without and with H&E staining
To initially validate the performance of our MUSE imaging system and the associated deep learning algorithm for style transformation, several 7-µm FFPE mouse brain sections with Hoechst staining are imaged by our MUSE system. The MUSE images (Fig. 2(A), 2(E), 2(I), and 2(M)) are subsequently transformed into Deep-MUSE images by both pix2pix (Fig. 2(B), 2(F), 2(J), and 2(N)) and CycleGAN ( Fig. 2(C), 2(G), 2(K), and 2(O)) models. We scanned the FFPE thin mouse brain section over a region of interest (ROI) with 7.4 mm × 5.2 mm along the x and y axes, respectively. The scanning time was ∼3 mins. The entire MUSE image of this section is presented in Fig. 2(A). As a result of the excited fluorescence signal from the FFPE sample with Hoechst staining, cell nuclei appear to be green and the cytoplasm is dark blue in MUSE images. After MUSE imaging, the same slice was stained with H&E, and then imaged with a digital slide scanner for comparison (Fig. S1). The cell nuclei (purple) and cytoplasm (pink) are shown in the corresponding bright-field H&E-stained image (Fig. 2(D)). Since these two image modalities were performed on the same slice, the cell distribution and structure should match well between the MUSE images ( Fig. 2(A), 2(E), 2(I), and 2(M)) and the corresponding H&E-stained images ( Fig. 2(D), 2(H), 2(L), and 2(P)). The Deep-MUSE images are generated from the same MUSE image (Fig. 2(A)) by the pix2pix model ( Fig. 2(B)) and CycleGAN model (Fig. 2(C)). Note that this MUSE image dataset was only used for validation, and the well-trained models were built on other training datasets. For pix2pix model training, the MUSE and H&E-stained image pairs of the same FFPE slice were obtained after the rigid image registration process. For CycleGAN model training, the same MUSE image for pix2pix model training was selected. However, the H&E-stained image was chosen from our digital H&E-stained image database because the CycleGAN model does not require paired data. Both GAN models learned to translate the MUSE images into Deep-MUSE images that are equivalent to the conventional H&E-stained images after training. The MUSE image (Fig. 2(E)) and the corresponding H&E-stained image (Fig. 2(H)) of the hippocampus region have a great similarity in the cell distribution. The Deep-MUSE images by the pix2pix model (Fig. 2(F)) and CycleGAN model (Fig. 2(G)) preserved the cell distribution from the MUSE image (Fig. 2(E)), and possessed the H&E color style that is comparable to the ground truth H&E-stained image ( Fig. 2(H)). The two zoomed-in regions (yellow (Fig. 2(I)) and blue (Fig. 2(M))) marked in the MUSE image (Fig. 2(E)) show subcellular features such as nucleoli, marked with orange and blue arrows. The corresponding regions (Fig. 2(L) and 2(P)) in the H&E-stained image (Fig. 2(H)) have similar features and cellular details, marked with orange and blue arrows, which indicates that the original MUSE images can provide high imaging contrast and high resolution as the conventional bright-field images with H&E staining. To further improve the interpretability and applicability of our MUSE images by pathologists, Deep-MUSE images can be used to provide a similar style as conventional H&E-stained images, which can be generated by the pix2pix model ( Fig. 2(J) and 2(N)) and CycleGAN model (Fig. 2(K) and 2(O)). Visualization 1 shows a series of closeup images of a column in the ROI of an FFPE mouse brain section imaged by MUSE and the corresponding H&E-stained images using different methods. The cellular structures and cytoplasm can be well identified in the closeup scanning images. The ground truth H&E-stained images (Fig. 2(L) and 2(P)) show many white cracks and suffer from information loss due to the subsequent sample preparation in the H&E staining process, e.g., washing by distilled water, dehydration using graded ethanol, vitrification by xylene. Deep-MUSE images (Fig. 2(K) and 2(O)) preserve the cellular and cytoplasm information from the original MUSE images (Fig. 2(I) and 2(M)).
To evaluate the performance of the pix2pix and CycleGAN models, we used two quantitative metrics -Pearson correlation coefficient and PSNR, to further analyze the Deep-MUSE image quality of the FFPE thin mouse brain section ( Fig. 3 and Table 1). The corresponding ground truth H&E-stained images were used as the reference images in the evaluation. Pearson correlation coefficient is a statistical measure reflecting the linear correlation between two data variables. The value ranges from zero to one. The higher the value (i.e., closer to one), the higher the image similarity between the Deep-MUSE image and the ground truth H&E-stained image. PSNR is a commonly used indicator to measure image quality, especially in image super-resolution and reconstruction tasks [35]. The higher the PSNR value, the better the image quality.

Table 1. Evaluation of Deep-MUSE image quality generated by different GAN models using the FFPE thin mouse brain section
For the training dataset, the distributions of Pearson correlation coefficient and PSNR show that the quality of the Deep-MUSE images generated by the pix2pix model is higher than that of the Deep-MUSE images generated by the CycleGAN model (Fig. 3(A) and 3(C)). However, we should consider the performance of the models on the validation dataset for practical applications. The comparison of the training dataset is not favorable to the CycleGAN model because the pix2pix model learns directly from the ground truth H&E-stained image data with paired training. In practice, the quality of Deep-MUSE images generated by the pix2pix model drops significantly on the validation dataset, which becomes comparable to the Deep-MUSE image quality generated by the CycleGAN model ( Fig. 3(B) and 3(D)). Table 1 shows the detailed statistical values with different models and datasets. The pix2pix model obtained higher values in the Pearson correlation coefficient, PSNR, and SSIM than that of the CycleGAN model on the training dataset. However, for the validation dataset, the Deep-MUSE images generated by the two models have nearly identical image quality with similar values of Pearson correlation coefficient and PSNR. The SSIM of Deep-MUSE images on the validation dataset shows that the CycleGAN model has better generalization than the pix2pix model with a higher value. These qualitative evaluation metrics prove that the Deep-MUSE image generated by the CycleGAN model is stable on both training and validation datasets.

Deep-MUSE image of a fixed thick mouse brain and liver versus conventional H&Estained image
To demonstrate the possibility of the proposed histopathology workflow using Deep-MUSE images with the CycleGAN model for thick tissue, we imaged a formalin-fixed thick mouse brain and liver tissues. The MUSE image was then transformed into a Deep-MUSE image using the well-trained CycleGAN model with minimal and stable loss (Fig. S6). Subsequently, the imaged sample was further processed following the standard histopathology workflow to get the H&E-stained image of the superficial layer as a reference. Note that the thick specimen surface imaged by the MUSE system is not exactly the reference H&E-stained thin slice due to the unavoidable difference in orientation and imaging thickness. Therefore, only the CycleGAN model can be utilized for a thick tissue style transformation, further demonstrating the importance of unsupervised learning in this application.
We have imaged a fixed mouse brain specimen. The scanning time was ~7 mins to image the entire cross-section with an area of 10.9 mm × 7.0 mm (Fig. 4(A)). Fig. 4 shows this validation dataset using the well-trained CycleGAN model. The Deep-MUSE image generated by the CycleGAN model ( Fig. 4(B)) and the reference H&E-stained image (Fig. 4(C)) provide similar cellular features, which can be easily identified in the hippocampus regions. The images of the left hippocampus region (Fig. 4(D) to 4(F), red regions) and right hippocampus region (Fig. 4(G) to 4(I), purple regions) show similar distributions of cell nuclei. The yellow (Fig.  4(J) to 4(L)) and blue regions (Fig. 4(M) to 4(O)) show the individual cell nuclei including their nucleoli. Both the Deep-MUSE images (Fig. 4(K) and 4(N) For the training dataset, the distributions of Pearson correlation coefficient and PSNR show that the quality of the Deep-MUSE images generated by the pix2pix model is higher than that of the Deep-MUSE images generated by the CycleGAN model (Fig. 3(A) and 3(C)). However, we should consider the performance of the models on the validation dataset for practical applications. The comparison of the training dataset is not favorable to the CycleGAN model because the pix2pix model learns directly from the ground truth H&E-stained image data with paired training. In practice, the quality of Deep-MUSE images generated by the pix2pix model drops significantly on the validation dataset, which becomes comparable to the Deep-MUSE image quality generated by the CycleGAN model (Fig. 3(B) and 3(D)). Table 1 shows the detailed statistical values with different models and datasets. The pix2pix model obtained higher values in the Pearson correlation coefficient, PSNR, and SSIM than that of the CycleGAN model on the training dataset. However, for the validation dataset, the Deep-MUSE images generated by the two models have nearly identical image quality with similar values of Pearson correlation coefficient and PSNR. The SSIM of Deep-MUSE images on the validation dataset shows that the CycleGAN model has better generalization than the pix2pix model with a higher value. These qualitative evaluation metrics prove that the Deep-MUSE image generated by the CycleGAN model is stable on both training and validation datasets.

Deep-MUSE image of a fixed thick mouse brain and liver versus conventional H&E-stained image
To demonstrate the possibility of the proposed histopathology workflow using Deep-MUSE images with the CycleGAN model for thick tissue, we imaged a formalin-fixed thick mouse brain and liver tissues. The MUSE image was then transformed into a Deep-MUSE image using the well-trained CycleGAN model with minimal and stable loss (Fig. S6). Subsequently, the imaged sample was further processed following the standard histopathology workflow to get the H&E-stained image of the superficial layer as a reference. Note that the thick specimen surface imaged by the MUSE system is not exactly the reference H&E-stained thin slice due to the unavoidable difference in orientation and imaging thickness. Therefore, only the CycleGAN model can be utilized for a thick tissue style transformation, further demonstrating the importance of unsupervised learning in this application. We have imaged a fixed mouse brain specimen. The scanning time was ∼7 mins to image the entire cross-section with an area of 10.9 mm × 7.0 mm (Fig. 4(A)). Figure 4 shows this validation dataset using the well-trained CycleGAN model. The Deep-MUSE image generated by the CycleGAN model (Fig. 4(B)) and the reference H&E-stained image (Fig. 4(C)) provide similar cellular features, which can be easily identified in the hippocampus regions. The images of the left hippocampus region (Fig. 4(D) to 4(F), red regions) and right hippocampus region (Fig. 4(G) to 4(I), purple regions) show similar distributions of cell nuclei. The yellow (Fig. 4(J) to 4(L)) and blue regions (Fig. 4(M) to 4(O)) show the individual cell nuclei including their nucleoli. Both the Deep-MUSE images (Fig. 4(K) and 4(N)) and original MUSE images (Fig. 4(J) and 4(M)) can distinguish nucleoli from dark purple color and bright green color, respectively, which are marked with orange arrows (Fig. 4(J) and 4(K)) and blue arrows (Fig. 4(M) and 4(N)). Visualization 2 shows a series of closeup MUSE images of a column of the ROI, the corresponding Deep-MUSE images, and the H&E-stained images of the adjacent layer. The style and the cellular information of Deep-MUSE images generated by the CycleGAN model are comparable to the conventional H&E-stained images. We also imaged a fixed and thick mouse liver specimen to explore the applicability of Deep-MUSE on other organs. Figure 5 shows MUSE images (Fig. 5(A) and (B)), Deep-MUSE images (Fig. 5(C) and (D)), and reference H&E-stained images (Fig. 5(E) and (F)) from two regions of the same mouse liver sample. The Deep-MUSE images (Fig. 5(C) and (D)) generated by the CycleGAN model and original MUSE images (Fig. 5(A) and (B)) have similar structures such as hepatocytes and blood vessels marked with green and orange arrows, respectively. These common cellular features are comparable in the reference H&E-stained images (Fig. 5(E) and (F)). The results demonstrate that Deep-MUSE imaging can also be applied to other tissues and organs.

Deep-MUSE image of a fresh thick mouse brain versus traditional H&E-stained image
To illustrate the full potential of our novel approach for clinical and intraoperative applications, our proposed histopathology workflow using Deep-MUSE images with the CycleGAN model has been applied to fresh mouse brain specimens. Similarly, the bright-field H&E-stained image of the adjacent thin section was obtained for validation. For the fresh mouse brain specimen, the ROI was 10.7 mm × 7.3 mm, and the scanning time was ∼6.5 mins. Figure 6 shows this validation dataset using the well-trained CycleGAN model. Note that the color tone of the fresh thick mouse brain image (Fig. 6(A)) is different from the fixed thick mouse brain image (Fig. 4(A)) due to different biomolecular properties with/without fixation and different white balance ratio of the camera. Therefore, the parameters of the CycleGAN model are unique for samples with different preparation protocols. The models were trained separately. The MUSE (Fig. 6(A)) and Deep-MUSE images (Fig. 6(B)) show identical features in the hippocampus region. The red (Fig. 6(C)) and yellow regions (Fig. 6(F)) are from the right hippocampus in the MUSE image ( Fig. 6(A)). The corresponding Deep-MUSE images output by the CycleGAN model ( Fig. 6(D) and 6(G)) and the reference H&E-stained images (Fig. 6(E) and 6(H)) show similar cellular features and distributions. Individual cells can be observed in the original MUSE images (Fig. 6(C) and 6(F)), the Deep-MUSE images (Fig. 6(D) and 6(G)), and the reference H&E-stained images (Fig. 6(E) and 6(H)).
To evaluate the Deep-MUSE image quality, we adopted qualitative and quantitative methods for both fixed or fresh thick tissue samples. First, we used the pre-trained deep neural network VGG16 [36] to extract the 4096 dimension feature vector of the input images for further analysis, which can reduce data volume and reserve the discriminative information. For qualitative measurement, we applied t-SNE [37] to visualize the data on a two-dimensional graph. We chose three image datasets for visualization, which include MUSE images, H&E-stained images for model training, and Deep-MUSE images generated by the model. The visualization diagrams for the fixed mouse brain specimen (Fig. 7(A)) and fresh mouse brain specimen (Fig. 7(B)) show that the clear separation of the MUSE images from the Deep-MUSE and the H&E-stained images. The Deep-MUSE images mixed with the conventional H&E-stained images, which indicates that the Deep-MUSE images generated by the deep learning model mimic the style of the specified H&E-stained images well. To analyze the Deep-MUSE image quality quantitatively, we calculated the maximum mean discrepancy (MMD) [38] to measure the distance between the generated image distribution and the target image distribution [39]. In our case, we computed the MMD distance from Deep-MUSE image features to conventional H&E-stained image features. To provide a reference, the MMD distance from the original MUSE image features to the conventional H&E-stained image features was also computed for comparison. The value of MMD is based on every batch with 100 images sampled from each data distribution. The MMD distance for the fixed mouse brain specimen ( Fig. 7(C)) and fresh mouse brain specimen (Fig. 7(D)) reveals a similar conclusion that the distance from the generated Deep-MUSE images to the conventional H&E-stained images is much shorter than the distance from the original MUSE images to the conventional H&E-stained images. This result is in line with our expectation as the CycleGAN model learns the complex style mapping function between the MUSE images and H&E-stained images during the training stage. The Deep-MUSE images output by our model is close to the conventional H&E-stained images. Table 2 shows the statistical data of MMD distances for both fixed and fresh thick mouse brain samples, corresponding to the Fig. 7(C) and 7(D). Table 2. MMD distances of MUSE and Deep-MUSE images to training H&E-stained images using different thick mouse brain samples distribution and the target image distribution [39]. In our case, we computed the MMD distance from Deep-MUSE image features to conventional H&E-stained image features. To provide a reference, the MMD distance from the original MUSE image features to the conventional H&Estained image features was also computed for comparison. The value of MMD is based on every batch with 100 images sampled from each data distribution. The MMD distance for the fixed mouse brain specimen ( Fig. 7(C)) and fresh mouse brain specimen (Fig. 7(D)) reveals a similar conclusion that the distance from the generated Deep-MUSE images to the conventional H&E-stained images is much shorter than the distance from the original MUSE images to the conventional H&E-stained images. This result is in line with our expectation as the CycleGAN model learns the complex style mapping function between the MUSE images and H&E-stained images during the training stage. The Deep-MUSE images output by our model is close to the conventional H&E-stained images. Table 2 shows the statistical data of MMD distances for both fixed and fresh thick mouse brain samples, corresponding to the Fig. 7(C) and 7(D).

Discussion
We demonstrated that the MUSE imaging system integrated with a deep learning algorithm is able to provide rapid slide-free histological images (i.e., Deep-MUSE) which are equivalent to standard H&E-stained images. This data-driven approach has great potential to become an alternative intraoperative histopathology workflow, replacing the needs of any sample processing (except staining) while providing high-quality images that are close to FFPE thin slices with H&E staining. To prove the reliability of our proposed workflow, we compared the performance of the CycleGAN model and the state-of-the-art pix2pix model on FFPE thin slices with the ground truth H&E-stained images as the reference (Fig. 2 and Visualization 1). Afterward, the presented workflow was tested on unsectioned fixed mouse brain and liver specimens to further explore the rapid and slide-free advantages of the MUSE imaging technique (Fig. 4, Fig. 5 and Visualization 2). Finally, we directly apply our workflow to fresh mouse brain samples (Fig. 6).
The correspondence between Deep-MUSE images and standard H&E-stained images shows that our approach can assist pathologists for histopathological examination. The superiority of the proposed histopathology workflow with Deep-MUSE images can be elaborated in two aspects. First, it is rapid to acquire Deep-MUSE images with similar molecular contrast compared with the conventional H&E-stained images. For the MUSE imaging system, its slide-free imaging capability allows imaging of thick tissue directly with UV excitation light, bypassing time-consuming sample preparation steps that are needed to obtain optically thin sections in standard histopathology workflow. It will only take ∼30 seconds for hand cutting, ∼2 mins for Hoechst staining, and ∼2 min for acquiring a MUSE image with subcellular resolution for a typical brain biopsy (5 mm × 5 mm). As for training the deep neural network, dedicated hardware (e.g., graphics processing units (GPUs)) can be used for acceleration. For example, the model training of the fresh mouse brain network accelerated by a single GPU requires only 20 hours. More importantly, once the model obtains optimal parameters after training, the feed-forward network outputs the Deep-MUSE images rapidly with the speed of 1.6 s mm −2 by a single GPU acceleration. It will take ∼40 seconds to generate a 5 mm × 5 mm Deep-MUSE image from a 5 mm × 5 mm MUSE image. Therefore, our approach will require ∼6 mins in total to obtain a 5 mm × 5 mm Deep-MUSE image that is comparable to a standard H&E-stained image. With the proposed Deep-MUSE workflow, it is possible to provide rapid histology ∼6 mins. Second, the deep learning model outputs virtual H&E-stained images with a consistent style, and reduces the color variations influenced by different lab workers and staining conditions, thus providing a unified color tone for accurate diagnoses. We compared the performance of pix2pix and CycleGAN models, finding that the CycleGAN model provides images as high quality as the pix2pix model, while offering higher flexibility. For instance, for FFPE mouse brain thin sections, the CycleGAN model has comparable performance with the pix2pix model on the validation dataset. However, the pix2pix model requires paired MUSE and H&E-stained images with a high correlation for training. The correlation between these image pairs will be easily affected by the physical staining process and image registration quality. On the contrary, the H&E-stained image data for CycleGAN model training does not have such strict requirements due to the unpaired training mode. For thick mouse brain samples, the pix2pix model cannot be applied as there is no paired image data for the model training. It is also not possible to directly test MUSE images of the thick tissue on a well-trained pix2pix model trained by FFPE thin section because the color tone of the MUSE images of thin section and thick tissue is different (Fig. S7). The CycleGAN model can easily select H&E-stained images from the database for model training, which simplifies the data pre-processing procedures by eliminating the need for image registration.
The limitations of the proposed workflow can be stated in the following ways. First, three models corresponding to mouse brain samples prepared by three different sample preparation protocols must be trained separately because of the color tone variation (Fig. 2(A), 4(A), and 6(A)). One possible solution is to modify our system with a monochrome camera to acquire grayscale images. The unified CycleGAN model can be trained with the mixed grayscale image data of thin and thick tissue samples. Second, our current model can only be applied to obtain virtual H&E staining because other staining methods, like Masson's trichrome stain and Jones' silver stain, have not been added to the training dataset. To achieve the multi-staining network and extend the application scenarios, Deep-MUSE images with individual staining or a mixture of multiple staining can be generated by simply adding the condition label to different stained image domains, e.g., a digital staining matrix to the input image for paired training [40], or domain labels for unpaired multi-domain image-to-image translation [41]. Finally, we have focused on normal mouse brain and liver tissues in this study. More investigations are needed to test for the style transformation accuracy on human specimens, especially for cancerous tissues from different organs.
The potential applications for the presented workflow are clear. Deep-MUSE images provide high molecular contrast with subcellular resolution without laborious sample preparation (e.g., staining only), which can enable intraoperative pathologic assessment. For example, during a surgical operation, a fresh human tissue excised by a surgeon in the operating room can be placed into our system after staining. Subsequently, a high-resolution Deep-MUSE image for a typical brain biopsy (5 mm × 5 mm) will be obtained ∼6 mins, which can provide an alternative to intraoperative frozen section histology which takes ∼30 mins. Different cells in Deep-MUSE images can also be automatically classified and graded by deep learning algorithms [42][43][44]. We believe that our proposed novel workflow holds great promise as an accurate and intraoperative assessment tool used by surgeons and pathologists in hospitals.