Mimicry Embedding Facilitates Advanced Neural Network Training for Image-Based Pathogen Detection

In biology, the use of deep neural networks (DNNs) for analysis of pathogen infection is hampered by a lack of large verified data sets needed for rapid network evolution. Artificial neural networks detect handwritten digits with high precision thanks to large data sets, such as MNIST, that allow nearly unlimited training. Here, we developed a novel strategy we call mimicry embedding, which allows artificial intelligence (AI)-based analysis of variable pathogen-host data sets. We show that deep learning can be used to detect and classify single pathogens based on small differences.

tectures have been proposed (16,17). These architectures are nested, allowing the retention of image feature positional information and optimization of CNN performance on images with a larger field of view.
However, these architectures remain data-hungry and often perform poorly on small host-pathogen data sets of high complexity (18). One major reason for this is the lack of large, balanced well-verified biological data sets (19), akin to MNIST (20) and ImageNet (21), that allow rapid algorithm evolution. To circumvent this, ANN analysis of pathogen images can be aided through transfer learning (22,23). For this, weights of a network trained on one data set are transferred onto a fully or partially identical untrained network, which is then trained on a host-pathogen data set of a similar nature (23). This approach shortens training time and is generally considered more efficient than random weights initialization strategies (22).
Here, we describe a novel data embedding strategy, which we term "mimicry embedding," that allows researchers to utilize ANNs in their analysis. Mimicry embedding involves transforming data sets such that they mimic verified nonbiomedical data sets, thereby allowing mimicry weights transfer from the latter. While most hostpathogen biomedical data sets are not large enough for deep learning, advances in high-content two-dimensional (2D) and 3D fluorescence imaging (24) can serve to increase the size of data sets for ANN analysis (14). By embedding 3D, fluorescentimage-based vaccinia virus and 2D/3D Toxoplasma gondii host-pathogen interaction data sets to mimic gray-scale handwritten digits, we demonstrate that mimicry weights transfer from MNIST (20) allows one to harness the performance of cutting-edge ANN architectures for the analysis of host-pathogen data sets.

RESULTS
Image acquisition and pathogen detection. To classify single-pathogen data in 3D biomedical images, we developed ZedMate, an ImageJ-Fiji (25) plugin that uses the Laplacian of Gaussian spot detection engine of TrackMate (26). We challenged ZedMate with multichannel, 3D fluorescent images of late-time-point vaccinia virus (VACV)infected cells ( Fig. 1A and Fig. S1). Owing to its large size, well-defined structure, and multiple layers of resident proteins that distinguish different virus forms, VACV has the features needed for complex fluorescence microscopy-based pathogen particle analysis (Fig. 1B). By detecting and linking individual virions within an image across the Z-dimension, ZedMate transforms a series of 2D images into a 3D data set (Fig. 1B). From the original four-fluorescent-channel composite, ZedMate generates grayscale images that preserve the intensity distribution across the Z-dimension of each detected channel (Fig. 1C, top). From this, fluorescence intensity matrices of each channel per Z-plane are then generated for individual particles (Fig. 1C, bottom). Using these matrices and accounting for the 3D positional information of the detected particles, ZedMate reconstructions can be plotted (Fig. 1D). Intensity analysis across all channels allows for binning (e.g., using a manual or automated threshold selection strategy) of virions into three categories consistent with their biological readouts (Fig. S1B).
Mimicry embedding assists deep neural network training for virus particle analysis. Initial reconstructions indicated that ZedMate cannot distinguish between incoming cell-free virions and newly replicated cell-associated virions based solely on channel 1 (c1) and c2 intensities ( Fig. 1D and Fig. S1). To improve the precision of ZedMate-based binning, we devised a binary machine learning/deep learning (ML/DL) strategy relying on manual annotation to separate cell-free from cell-associated virions. Using random sampling, we generated a data set of 36,492 images containing equal numbers of cell-associated and cell-free virions. To maintain the spatial information acquired in ZedMate, we attempted to train the capsule ANN (CapsNet) (16) on this annotated data set. During initial attempts, CapsNet failed to converge, with accuracy around 0.5 (Fig. S2A). As CapsNet is known to converge well on the relatively simple grayscale data set, MNIST (16) (Fig. S2B), we asked how CapsNet would converge on a binary classification problem similar to ours. We tested this by splitting MNIST into balanced (Ͻ5 or Ն5; evens or odds) or unbalanced (one digit versus all) data sets. With no changes to CapsNet other than loading binary relabeled MNIST weights (Ͻ5 and Ն5) for initialization, CapsNet converged on MNIST with 99.6% accuracy ( Fig. 2A). We found that all binary relabeled MNIST weights variants performed comparably (Fig. S2C).
To allow transfer learning from this network to our VACV data set, we designed a vector-embedding strategy we term "mimicry embedding." For this, the tensors of each virion's multichannel, fluorescence Z-profiles from ZedMate are assembled across the x axis. This is followed by linear interpolation and padding, which serve to center the virion in a 28-by 28-pixel image such that the resulting data mimic the grayscale MNIST data set (Fig. 2B). With this approach, we aimed to preserve the weights of early CapsNet layers by maintaining the binary MNIST-trained CapsNet architecture and performing weights transfer. Training on our mimicry-embedded real-world data set achieved 96.5% accuracy (96.2% precision, 96.2% recall) at separating cell-free from cell-associated virions ( Fig. 2B; see Fig. S3A to D for classifier training). Data set augmentation (10% horizontal and vertical shift), with mimicry embedding and MNIST weights transfer, resulted in only modest improvement to generalization, and there was no difference without mimicry embedding and MNIST weights transfer (data not shown).
To visualize how the trained ANN distinguished between cell-free and cellassociated virions with such accuracy, we used the CapsNet generator. The reconstruc- tions indicated that cell-free virions were elongated, with moderate intensity profiles, while cell-associated virions were compact and very bright (Fig. 2C). The reconstructions were in agreement with mimicry-embedded virions, suggesting that these properties yielded the base for the high classification accuracy (Fig. 2D).
Given its ability to improve CapsNet-based classification of our data set, we wondered if mimicry embedding would also enhance the performance of other neural network architectures with increasing expressive capacity (Table 1). To this end, using a conventional feed-forward network (1,570 trainable parameters) as a baseline, we compared LeNet CNN (30,472 trainable parameters) (7), ResNet-101 (42,542,978 trainable parameters) (27), and CapsNet (7,281,680 trainable parameters). We found that mimicry embedding did not improve the performance of low-expressive-capacity architectures, such as the small feed-forward network, but it has a tendency to improve the performance of high-expressive-capacity architectures on small data sets (Table 1). Furthermore, in the case of CapsNet architecture, it is likely that the random initialization of weights caused the issue with our VACV data set, as it is small and complex, two things CapsNet struggles with (18). To verify the impact of mimicry embedding on CapsNet, we performed inference on an unseen (separate from training and validation sets) experimental data set. Figure 3 shows the workflow, from an input four-channel image ( Fig. 3A and Fig. S1A and B) to detection and binning of virions (Fig. 3B), followed by mimicry embedding and CapsNet separation of cell-free versus cell-associated virions (Fig. 3C). The results indicate that our model and binning combined allow accurate classification of virions into four biologically relevant classes within unseen data sets (Fig. 3D).  Statistical validation of machine learning models is provided in Fig. S3.
We have established that mimicry embedding and weights transfer allows us to distinguish between incoming cell-free and newly assembled cell-associated virions at late time points after infection. Next, we asked if this approach could also be used to classify extracellular versus intracellular virions during virus entry, a single-cell assay that often requires specific antibodies or labeling strategies and labor-intensive manual annotation (see "Superresolution imaging of VACV intracellular virions" in Materials and Methods). Considering these common limitations, we generated a training data set consisting of 30,537 images that would allow generalization of this approach. Early infected cells, with virions seen in c2, were stained with common fluorescent DNA (c1) and actin (c3) dyes. To circumvent hand-labeling of the training data, immunolabeling to distinguish between intra-and extracellular virus (c4) was used as a weak labeling (28) strategy ( Fig. 4A and Fig. S4A). After ZedMate detection and transformation of individual particles, intra-and extracellular virus weak labeling (c4), which distinguished 21,566 extracellular and 8,971 intracellular virions, was removed for mimicry embedding. By maintaining our binary MNIST-trained CapsNet and performing weights transfer, we could achieve 82% accuracy (81.3% precision, 81.4% recall) in differentiating between intra-and extracellular virions in the absence of specific-antibody labeling and manual annotation (See Fig. S4B to E for classifier training).
To estimate accuracy, inference was performed on an unseen data set in which intraand extracellular virions were quantified using c1 to c4 (measured), inclusive of extracellular virion weak labeling, or only c1 to c3 (predicted) (Fig. 4B). An 86% match between measured and predicted quantification of intracellular particles was seen (Fig. 4B, inset). This indicates that weak labeling can effectively substitute for manual annotation of training data sets when intra-and extracellular virion signals are being classified. As an additional test of the ANN, we generated a data set skewed for extracellular virions by blocking virus entry with IPA-3 (29, 30) (Fig. 4C). Consistent with its performance (Fig. S4B to E), a 93% match between measured and predicted quantifications of intracellular particles was seen (Fig. 4D). Finally, when we visualized the reconstructions of intra-and extracellular virion classes, extracellular virions appeared brighter and more elongated in the Z direction than intracellular ones (Fig. 4E). This was in agreement with their mimicry-embedded counterparts (Fig. 4F), explaining the ANNs' ability to accurately predict between and quantify these two virion classes.
Mimicry embedding enables deep neural network training for Toxoplasma gondii viability analysis. To assess the general applicability of our mimicry embedding approach, we acquired an imaging data set of cells infected with an enhanced green fluorescent protein (EGFP)-expressing version of the parasite Toxoplasma gondii. While T. gondii-EGFP is readily visualized by conventional microscopy, detecting and quantifying intracellular viability at the single parasite level are challenging (14). To generate a T. gondii viability training data set, cells infected with T. gondii-EGFP (c1) were fixed and stained with fluorescent markers of DNA (c2) and host cell ubiquitin (c3), which was used as a weak label to annotate the subset of "unviable" parasites (14, 31) (Fig. 5A). Individual particle detection and transformation in ZedMate were followed by mimicry embedding in the absence of c3 weak labeling. This training data set, generated through random sampling, was composed of 2,694 images containing equal proportions of viable and unviable examples. After weights transfer from the CapsNet trained on binary MNIST (Fig. 2B) and fine tuning on T. gondii-EGFP, we achieved 70% accuracy (precision, 72.5%; recall, 70.7%) in the absence of specific viability labeling (see Fig. S5A to D for classifier training).
To ensure that the ANN could accurately distinguish between viable and unviable parasites, we generated a data set of cells infected with T. gondii-EGFP using a specific viability label (c3) as ground truth (Fig. 5A). To further assess viability, experiments were performed in the absence or presence of gamma interferon (IFN-␥), which drives parasite killing. Upon model training and validation, test inference on this data set using c1-c2 resulted in 84% and 80% matches between measured (c3) and predicted (c1-c2) viability in the absence or presence of IFN-␥, respectively (Fig. 5B). CapsNet generator reconstructions showed that "viable" T. gondii-EGFP organisms appear larger and brighter than "unviable" parasites in both c1 and c2 (Fig. 5C). This likely explains the ability of the model to accurately predict T. gondii-EGFP viability in the absence of specific c3 viability labeling.
In an attempt to train a general model for in vivo parasite viability assessment using our in vitro data set, we performed mimicry embedding on T. gondii-EGFP (Fig. 5A, c2  panel). This resulted in a Ͼ10% drop in prediction accuracy when training on CapsNet. This suggested that single-channel mimicry embedding does not provide enough context for training of complex algorithms. However, we reasoned that as our mimicry embedding is based on MNIST, we could use any algorithm that performs well on this data set. By switching to DropConnect (32) architecture, which performs among the best on MNIST, our classifier achieved 65% accuracy (precision, 65.9%; recall, 64.3%) in differentiating between viable and unviable parasites using a single channel (see Fig. S5E to H for classifier training). To test this classifier on an in vivo data set, we infected zebrafish (Danio rerio) larvae with T. gondii-EGFP in the hindbrain ventricle. Infected larvae were imaged at 0, 6, and 24 h after infection by fluorescent 3D stereomicroscopy (Fig. 5D). ZedMate was used to detect and quantify T. gondii-EGFP numbers over time (Fig. 5E). A dramatic drop in parasite count was seen between infection at 0 h and 6 h, followed by increased numbers of T. gondii-EGFP by 24 h. Next, the T. gondii-EGFP Z-profiles were mimicry embedded and normalized, and viability was inferred using the in vitro infected-cell model previously trained on DropConnect. At high pathogen load (0 h), 48% of T. gondii-EGFP organisms were scored as viable (Fig. 5F). By 6 h, this increased to 95%, and there was no significant change within 24 h. These results are consistent with an initial clearing of unviable parasites between 0 and 6 h and replication of the remaining viable ones (33).
Finally, we sought to determine the utility of mimicry embedding for analysis of 2D data sets. To this end, we generated a 3-channel, multiclass 2D T. gondii-EGFP vacuole data set using maximum intensity projections (MIPs) of high-content microscopy images. For this, T. gondii-EGFP-containing vacuoles were detected and cropped into individual images. The vacuole MIPs were split into DNA (c1), EGFP (c2), and host cell ubiquitin (c3) channels. Here, c3 was used as a weak label to annotate the subset of "unviable" parasites, thereby defining the ground truth for parasite viability (14,31). Lastly, vacuole images were sorted into single-and multiple-pathogen subclasses, resulting in a complex 2D data set of 7,021 images annotated into 4 classes: single viable (3,922 images), single unviable (1,510 images), multiple viable (840 images), and multiple unviable (749 images) (Fig. 6A).
Using this data set, we compared the performance of various architectures with and without mimicry embedding (Fig. 6B). As with low-complexity binary data sets (Table 1), mimicry embedding did not enhance the performance on either c1 or c2 data sets with the baseline low-expressive-capacity feed-forward network. However, it did provide marked improvement in both c1 and c2 when networks with high expressive capacity were used. This was most pronounced in the case of ResNet-101, which performed poorly when confronted with four classes versus two, and CapsNet, which again failed to initialize. These results support our conclusion that mimicry embedding is useful for higher-expressive-capacity architectures, like Res-Net and CapsNet, when training on small and unbalanced data sets, such as those often found in biomedical imaging.

DISCUSSION
ANN analysis of fluorescent host-pathogen data sets has trailed behind the unprecedented advancement of artificial intelligence (AI) analysis seen in other fields. This is largely due to the lack of open-source, verified pathogen data sets comparable to MNIST and ImageNet (20,21). Here, we present ZedMate and mimicry embedding as a strategy to harness the power of verified data sets like MNIST and transfer learning to train highly accurate models for analysis of biomedical images.
Other strategies that aim to improve the performance of deep neural networks on real-world data sets include image augmentation (15,34), self-supervised learning (35,36) and semisupervised learning (37). Image augmentation aims to boost the size of the data set by applying linear changes to it. While routinely used in deep learning systems, only limited augmentation types remain sensible for strictly controlled domains like microscopy. Both semisupervised and self-supervised approaches relieve the need for labeled data but do not eliminate the requirement for large unlabeled data sets. This leaves a void in the biomedical image analysis domain, where often only shallow learning approaches (e.g., feed-forward networks) that do not fully appreciate the phenotypic complexity in biology are applicable. Mimicry embedding aims to fill this void, allowing researchers to utilize large-expressive-capacity models capable of learning complex data representations. While the representations learned from an unrelated verified data set, like MNIST, might not be optimal for all scenarios, our results show that this may prove useful for network initialization.
By testing a variety of neural network architectures of increasing expressive capacity, we found that mimicry embedding does not change the ability of low-expressivecapacity feed-forward networks but improves the performance of high-expressivecapacity neural networks, including ResNet and CapsNet. This intuitively suggests that mimicry embedding works through lowering the data set size requirement for highcapacity architectures, perhaps through including the representations learned from the standardized data set.
In addition, we show that mimicry embedding works on both 2D and, in conjunction with ZedMate, 3D images. We noted that for LeNet and ResNet, mimicry embedding proved more useful for multiclass, as opposed to binary, classification problems. The possible explanation for this is that, compared to binary classification problems, multiclass classification problems may elevate the requirements for the data set size even for shallower networks, like LeNet.
As such, mimicry embedding is a valuable addition to the data science arsenal. When used together with CapsNet (16), LeNet, and ResNet, mimicry embedding proved to be a promising method for detection of complex pathogen phenotypes in vitro. We show that transforming real-world images such that they resemble landmark data sets ensures compatibility with, and seamless switching between, cutting-edge architectures. Embedding data in such a way allows one to maintain full compatibility with weights of the first layers, thereby improving transfer. Using in vivo pathogen data, we further demonstrate that mimicry embedding can yield a model with higher accuracy than one obtained through cutting-edge neural architecture search. Collectively, our results suggest that ZedMate and mimicry embedding, although employed here for the analysis of host-pathogen interaction, can be useful for AI analysis of small unbalanced 2D and 3D fluorescent data sets.