Translation of atherosclerotic disease features onto healthy carotid ultrasound images using domain-to-domain translation

Objective: In this work, we evaluated a model for the translation of atherosclerotic disease features onto healthy carotid ultrasound images. Methods: An un-paired domain-to-domain translation model – the cycle Generative Adversarial Network (cycleGAN) – was trained to translate between carotid ultrasound images of healthy arteries and images of pronounced disease. Translation performance was evaluated using the measurement of wall thickness in original and generated images. In addition, we explored disease translation in different tissue segments (subcutaneous tissue, muscle, lumen, far wall, and deep tissues), using structural similarity index measure (SSIM) maps. Results: Features of pronounced disease were successfully translated to the healthy images (1.2 (0.33) mm vs 0.43 (0.07) mm, p < 0.001), while overall anatomy was retained as SSIM value was equal to 0.78 (0.02). Exploration of translated features showed that both arterial wall and subcutaneous tissues were modified in the translation, but that the subcutaneous tissue was subject to distortion of the anatomy in some cases. The image quality influenced the disease translation performance. Conclusion: The results show that the model can learn a mapping between healthy and diseased images while retaining the overall anatomical contents. This is the first study on atherosclerosis disease translation in medical images. Significance: The concept of translating disease onto existing healthy images may serve purposes such as education, cardiovascular risk communication in health conversations, or personalized modelling in precision medicine.


Introduction
Atherosclerosis leads to thickened arterial walls and focal plaques, and it is the major cause of cardiovascular disease (CVD), with a very high incidence of related deaths [1].Ultrasound imaging of the carotid arteries is commonly used to assess disease stage, progression, and CVD risk, based on measurements of the carotid arterial wall thickness (cIMT) and the presence and composition of focal plaques [2].
In this work, we aim to evaluate a model to simulate this disease on a personalized level.We train a model to take images of a healthy person's carotid artery and generate a new image with features of pronounced disease but with preserved anatomical features.In general, such generated images could be used for educational purposes and simulations.In specific, recently, our group showed that presenting an image of a patient's carotid artery disease severity increases a patient's adherence to treatment and lowers risk factors [3,4].Therefore, such generated images could be used as a tool in primary prevention and CVD risk communication within a health conversation [4].
The problem we want to solve is equivalent to translating features from one image domain to another.So-called domain-to-domain translation methods (deep learning models) have been proposed for similar tasks in medical imaging applications.E.g., the cycle generative adversarial network (cycleGAN) has been used to generate authentic speckle pattern features in simulated B-mode images [5], and to generate authentic musculoskeletal images [6], both from simple simulated images.GANs have also been used to translate features of super-resolution images onto images of low-resolution [7] and for the generation of different stages of tumour growth in ultrasound mammographic images [8].Moreover, our group also recently applied a modified 3D cycleGAN model to generate authentic ultrasound image sequences of contracting skeletal muscles from simplified simulated sequences (video) [9].To the best of our knowledge, the idea of disease injection to existing ultrasound images using deep learning is not yet explored.
In this work, we evaluate the 2D cycleGAN model [10] to solve this task, where translation between images of two domains with different feature distributions was obtained while retaining content of the original images [11][12][13].

Datasets
Longitudinal ultrasound B-mode images of the common carotid arteries were retrospectively included in this work from the VIPVIZA study [4].The VIPVIZA study is an intervention study for primary cardiovascular disease prevention.The study enrolled 3532 healthy individuals with sub-clinical atherosclerosis of ages 40, 50, and 60 s.For each subject, multiple carotid ultrasound images were acquired at enddiastole (approximately-five consecutive heartbeats) from four different standardized projections (120-, 150-, 210-and 240-degrees Meijer arc) from both the left and right arteries.In total, approximately 140 000 (5 heartbeats × 4 projections × 3500 subjects × 2 examinations) images were available within the VIPVIZA study.The study conformed to the Declaration of Helsinki and was approved by the Swedish Ethical Review Authority.
The ultrasound images were acquired using the CardioHealth Station ultrasound system with a linear 7 MHz probe (Panasonic Healthcare Corporation of North America, Newark NJ, USA).Images had 8-bit resolution and size was 464x435 (field-of-view 40x38 mm, depth × width) and were down-sampled using cubic-splines interpolation to 400x400 for training purposes.The equivalent resolution was approximately 0.1 mm/px.

Selection of images
Healthy images (defined as Domain A) had thin cIMT (range 0.3-0.5 mm, mean 0.45 mm, N = 2319).Images with pronounced disease (defined as Domain B) had thick cIMT (range 1.1-2.0mm, mean 1.2 mm, N = 2071).Images for training and testing were randomly selected from the two domains at an 80:20 ratio.The clinical characteristics of the two domain data sets are given in Table 1.Fig. 1A and B show example images of both domains.

The CycleGAN model
Generative adversarial networks (GANs) have gained popularity for their ability to generate realistic-looking synthetic medical image data [10,[14][15][16].Domain-to-domain translation models are a branch of GANs that allow the translation of images from one domain to another with different feature distributions while retaining content [11][12][13]17].
When the images of two domains are not paired, the cycleGAN model offers a solution for this task.In this work, a cycleGAN model [10] was trained to translate between domain A (no atherosclerosis) and domain B (pronounced atherosclerosis) datasets.
The architecture of cycleGAN [10] comprises two GANs working in the opposite direction i.e., transformation from G 1 (x A )→x B and transformation G 2 (x B )→x A .In this study, the cycleGAN was trained for epochs (determined empirically based on a stopped reduction of the training loss).We experimented with ResNet [18] as well as U-Net architecture as the generator of the cycleGAN.We found that the ResNet architecture gave a superior performance, and hence we used a generator with the ResNet architecture.
The default setting for up-sampling in the generator part uses transpose convolution.The default method down-samples the image times and then extracts the feature using the ResNet18 model.Following this, the model up-samples the features by a factor of 4 using transposed convolution.In our implementation, we replaced the transpose convolution by sub-pixel convolution layer [19].The benefit of the sub-pixel convolution layer is that it replicates the channels of the image and shuffles the pixels in each replica.It has been demonstrated to give an advantage in the case of image super-resolution [19].

Evaluation and exploration of disease translation
The disease translation was evaluated (on the test data) by comparing the cIMT of the input and the generated output image pairs.The cIMT was manually measured in the input and output images (randomly presented and blinded to the operator).Next, features of the disease translation were explored by quantifying the structural similarity index measure (SSIM) map between the input-output image pairs [20].The SSIM ranges from 0 to 1 (low to high similarity).The authenticity of generated images was assessed by two carotid ultrasound experts (CG and EN) who, presented blindly and in random order, classified 50 real and 50 generated images as real or generated.
Association between image quality and SSIM was quantified by Pearson's correlation coefficient.Image quality was estimated as contrast tissue-to-lumen, CTL =(|I lumen − I tissue |)/|I tissue |, where the lumen and tissue intensities were computed as the mean intensities in the corresponding muscle and lumen ROIs, respectively.

Evaluation of disease translation
The cycleGAN model was trained to translate between two carotid ultrasound image sets: domain Ahealthy (thin arterial wall, carotid intima-media (cIMT) < 0.5 mm, N = 1855), and domain Bpronounced atherosclerotic disease (thick cIMT, >1.1 mm N = 1656) (Fig. 1A, B).Fig. 1C and 1D show an example of one originally healthy and the corresponding generated image with pronounced disease.The cIMT was 0.43 (0.07) mm and 1.2 (0.33) mm in the test images of the original input and generated output images (Fig. 1F, p < 0.001).Thus, the healthy images with thin cIMT were successfully translated to pronounced disease with thick cIMT (similar to that of the training data for domain B, 1.12 (0.18) mm).cIMT was not possible to measure in N = (15 %) images due to poor image quality.The domain experts classified the generated images as real images in 67 % and 75 % of the cases.

Exploration of disease translation
The translated features of the disease were explored using the SSIM between original and generated image pairs (Fig. 1E) in the full image, subcutaneous, muscle, lumen, arterial wall, and deep tissues, respectively (Fig. 1G, H).The SSIM showed a low similarity in the far wall, indicating that the model had modified this region in the generated image as compared to the original.The SSIM for the entire image was 0.78 (0.02).The far carotid wall and the subcutaneous tissue had lower SSIM as compared to the muscle, carotid artery lumen, and the deep tissue (Fig. 1H, p < 0.001).Fig. 2A-C shows additional examples of original and generated images with successfully translated disease.
There were a few cases where the underlying anatomy was distorted in the subcutaneous tissues of the generated images (Fig. 2D-F).Two regions with distortions could be observed: 1) the outer left and right tissues had reduced similarity for all generated images, and 2) in a few cases, the anatomy of the original image was distorted in the subcutaneous tissue segment (N = 22, ~5% of the test images).

Influence of image quality on disease translation
Motivated by the large range of SSIM of the far wall (0.25-0.70,Fig. 1H) and the observed 15 % dropout in cIMT measurements, we analysed if the image quality could account for this variation.Image quality was quantified using the ratio between wall echogenicity (average pixel intensity of the wall tissue) and the echogenicity of the lumen -Contrast tissue-to-lumen (CTL).The correlation between CTL and SSIM in the different tissues was: subcutaneous 0.08 (p = 0.16), muscle − 0.11 (p = 0.073), blood − 0.11 (p = 0.61), far wall − 0.46 (p < 0.001), deep tissue − 0.10 (p = 0.09), respectively.These results show that image quality only influenced the translation of disease features in the far wall tissues significantly.Thus, high noise in the lumen (low CTL) resulted in less translated disease features (high SSIM).Fig. 2G-I shows an example of how poor contrast influenced the translation.

Discussion
This study showed that the cycleGAN model can translate image features of pronounced atherosclerotic disease (thickened cIMT) onto images of healthy carotid arteries.The overall high similarity (SSIM) indicates that the main image features, i.e., the underlying anatomy, were retained.In contrast, the lower SSIM of the subcutaneous and far wall tissues indicate that these were adapted with the disease.The disease adaptation of the far wall was expected due to the selection of the datasets.The slight modification to the subcutaneous tissue was likely related to artifact generation by the method (see Fig. 2E-F).Noise in the lumen of the images (low CTL) was found to reduce the translation of disease features (higher SSIM).Such noise is commonly seen in carotid   ultrasound images but can be reduced using harmonic imaging sequences [21].The classification by the domain experts showed that the generated images were visually perceived as real images in a majority of the cases.
The performance of the reverse generator of the cycleGAN that is trained to generate healthy images from images with pronounced disease was also assessed (Fig. 3).The cIMT was 0.56 (0.12) and 1.36 (0.29) mm for the input and generated images (Fig. 3AB, p < 0.001).The SSIM for the entire image was 0.64 (0.07).Thus, the performance of the reverse generator for disease removal had similar performance as the disease generation generator.
The concept of translating disease onto existing ultrasound images is new, but in other medical image modalities it has recently been demonstrated.E.g. cycleGAN models were used to generate pathological images from normal optical coherence tomography (OCT) images of the eye [22], and the Ad cycleGAN model was proposed to generate COVID-19 positive chest X-ray images from normal images [23].
The cycleGAN model was trained on relatively limited data, which could influence the performance.In addition, translation between only two stages of the disease was adapted.While the chosen model only can perform one-to-one mappinghere between healthy and diseased states, models that support the generation of different stages of disease or outcomes could be achieved by adding data constraints and allowing one-to-many mapping models e.g.[24,25].
This work presents a proof-of-concept on the generation of disease into healthy carotid ultrasound images.There are several potential applications of this concept.First, it may be used to simulate images of disease for education or for injection of disease into an existing image of a healthy subject for risk communication in a health conversation -to improve adherence to prescribed treatment [4].Second, the current understanding of image features of atherosclerotic disease concerns mainly morphological features (e.g., cIMT), and a data-driven disease generation model may provide new insights on additional image features in different stages of the disease beyond the currently used measurements.Third, models that can learn a mapping between healthy and diseased domains, open new dimensions for personalized disease prediction modelling [26] and should be evaluated in future studies.Finally, the model could also be used to generate diseased or healthy images for data augmentation purposes, to improve the training of deep learning models on atherosclerosis and ultrasound images.

Conclusion
Translation of atherosclerotic disease onto existing ultrasound images of healthy carotid arteries was demonstrated.The proposed concept may be useful in education, simulations, or primary prevention to boost risk communication in a health conversation on CVD risk.

Fig. 1 .
Fig. 1.A: Examples of training images with no atherosclerotic disease (domain A, cIMT < 0.5 mm and no plaques).B: Examples of training images with pronounced disease (domain B, cIMT > 1.1 mm and plaques present).C: Original image (healthy, thin wall, Domain A).D: Generated image with translated disease (thick wall) with overall anatomy retained.E: Similarity map (SSIM) between the two images.A low value indicates disease adaptation to this segment.F: Wall thickness (cIMT) of the original and generated images.G: Illustration of segmented regions.H: Similarity (SSIM) in segmented regions between original and generated image pairs (N = 460 images).

Fig. 2 .Fig. 3 .
Fig. 2. A-C: An example with successful disease translation and retained anatomy, as indicated by the horizontal red region of the far wall in C. D-F: An example where the generator model has distorted the underlying anatomy of the subcutaneous tissue as indicated by red regions in the F. G-I: An example of how contrast influenced the disease translation.The original image G had two regions with low and high contrast (contrast tissue-to-lumen, CTL).It can be seen in the SSIM map I that no disease was translated to the low-contrast region.

H
.Ali et al.

Table 1 Subject and image characteristics of the training data domains A and B
Values represent mean and standard deviations.