Defining host–pathogen interactions employing an artificial intelligence workflow

For image-based infection biology, accurate unbiased quantification of host–pathogen interactions is essential, yet often performed manually or using limited enumeration employing simple image analysis algorithms based on image segmentation. Host protein recruitment to pathogens is often refractory to accurate automated assessment due to its heterogeneous nature. An intuitive intelligent image analysis program to assess host protein recruitment within general cellular pathogen defense is lacking. We present HRMAn (Host Response to Microbe Analysis), an open-source image analysis platform based on machine learning algorithms and deep learning. We show that HRMAn has the capacity to learn phenotypes from the data, without relying on researcher-based assumptions. Using Toxoplasma gondii and Salmonella enterica Typhimurium we demonstrate HRMAn’s capacity to recognize, classify and quantify pathogen killing, replication and cellular defense responses. HRMAn thus presents the only intelligent solution operating at human capacity suitable for both single image and high content image analysis. Editorial note: This article has been through an editorial process in which the authors decide how to respond to the issues raised during peer review. The Reviewing Editor's assessment is that all the issues have been addressed (see decision letter).


Introduction
High content imaging (HCI) has revolutionized the field of host-pathogen interaction by allowing researchers to perform image-based large-scale compound and host genome-wide depletion screens in a high-throughput fashion (Brodin and Christophe, 2011;Mattiazzi Usaj et al., 2016). The majority of these screens assess host-pathogen interactions using bulk colorimetric or automated enumeration of pathogen growth at the population level (Ang and Pethe, 2016;Radke et al., 2018). Additionally, quantification of host-pathogen interaction (e.g. analysis of host protein recruitment to the pathogen) in general is often performed manually. However, to meaningfully dissect cell-mediated pathogen control, it is imperative to quantify the host response and pathogen fate at the single-cell level. Many open-source, e.g. CellProfiler (Carpenter et al., 2006), and proprietary, e.g. Perkin Elmer Harmony, analysis software packages have been developed and successfully employed for this purpose (Eliceiri et al., 2012;Stö ter et al., 2013;Smith et al., 2018). To advance the state of the art in image analysis of host-pathogen interaction, incorporation of cuttingedge machine intelligence algorithms (Simonyan and Zisserman, 2014;LeCun et al., 2015) to stratify the image content without the requirement to program complex integrations is needed. HRMAn relies on the same well-established image segmentation strategies as many other programs but offers an intuitive integration of deep learning for more complex image analysis. Solutions existing to date can be split into two major categories: user-friendly turn-key GUI (TK-GUI)-based solutions and scripts ensembles (SE) solutions. Due to the large support burden of the TK-GUI, these programs lack the implementations of the latest engineering advances. At the same time SE solutions are easier to update but are far from user-friendly and are difficult to migrate between installations.
Deep neural network-based machine intelligence methods have brought about a revolutionary advance in the field of computer vision, by allowing for learning of complex morphologies in a highly generalized fashion (Krizhevsky et al., 2012;LeCun et al., 2015). To date, these methods have not been adapted for the field of host-pathogen interaction. Typically, HCI based fluorescent imaging data from a host-pathogen interaction experiment is analyzed by classical image segmentation (Osaka et al., 2012;Schmutz et al., 2013;Kühbacher et al., 2015;Ovalle-Bracho et al., 2015). Occasionally segmentation combined with machine learning based on calculated features has been employed (Kreibich et al., 2015). Most of these analysis pipelines make use of open-source programs tailored with additional coding by the user to suit their specific needs and are not published in their final form as a universal open-source solution. A major short-coming of these classical image segmentation and machine learning analysis methods is that they fail at the level of quantifying host protein recruitment to the pathogen. This is largely due to the fact that traditional algorithms quantify phenotypes in a rule-based manner, using bulk statistical properties of microscopy images or their segments. Conversely, deep neural networks make use of complex patterns (e.g. shapes) within the dataset to learn phenotypes and their diversity. The neural network derives these patterns in an automatic fashion from expert-labelled data. Thus, using pattern complexity to refine classification (Krizhevsky et al., 2012), deep neural networks improve the biological relevance of the phenotypic readouts.
While some proprietary solutions have been employed to extract host protein recruitment data, these are impractical and insufficient for most researchers as they are tied to single and expensive microscopes and do not operate at human capacity (Polajnar et al., 2017;Touquet et al., 2018). To date, for the analysis of host protein recruitment to pathogens, artificial intelligence-driven automated analysis is neither available as an open-source nor as a commercial package. Thus, there remains a need for an open-source, intuitive, flexible, and trainable host-pathogen interaction analysis software that performs at the level of human analytic capacity (Russakovsky et al., 2015;He et al., 2015;Haberl et al., 2018).
Here we present a high-throughput, high-content, single-cell image analysis pipeline that incorporates machine learning and a deep convolutional neural network (CNN) ensemble for Host Response to Microbe Analysis (HRMAn; https://hrman.org/). To assure its broad applicability to infection biology, HRMAn is based on the data integration environment KNIME Analytics Platform (Berthold et al., 2008). The analysis relies on training of machine learning algorithms and deep neural networks that can be tailored to individual researchers' needs.

Architecture of the high-content image analysis pipeline for analyzing host-pathogen interaction
The HRMAn pipeline (Figure 1), is designed to work with all file types acquired on any HCI platform or fluorescence microscope. Plate maps including experimental layouts, sample groups and replicates can be loaded, enabling HRMAn to automatically cluster results and perform error analysis. Once fed into the HRMAn pipeline, images are automatically pre-processed and clustered based on user-defined parameters (i.e. imaging specifications) and corrected for illumination. The subsequent image analysis proceeds in two stages: in stage 1, HRMAn segments images into pathogen and host cell features for single cell analysis. It then classifies these features using a decision tree learning algorithm previously trained on an annotated dataset. In stage 2, HRMAn analyzes host cell features associated with the pathogens using a CNN HRMAlexNet (derived from AlexNet architecture) trained to distinguish complex phenotypic patterns of host-protein recruitment (Krizhevsky et al., 2012). Robust classification is achieved by passing segmented regions of interest through multiple non-linear convolutional filters to identify characteristic phenotypic details.  Figure 1. Overview of the HRMAn pipeline. Following image acquisition, on a high-content imaging platform or any other fluorescence microscope, the images can be loaded into the HRMAn software. First, the data is pre-processed and clustered based on user-defined parameters and provided plate maps. Images then undergo illumination correction and automated segmentation using Huang algorithm. Segmented images are used by a deep convolution neural network (CNN) and other machine learning based algorithms to analyze infection of cells with intracellular pathogens. Depicted is Finally, data are output as a single spreadsheet file providing the researcher with !15 quantitative descriptions of a pathogen and its interaction with host factors at population and single cell levels (Figure 1; Readouts). Importantly, by separating the analysis, HRMAn offers researchers the flexibility to perform fast, simple quantitative analysis of infection parameters using stage 1, without analyzing host protein recruitment.
Machine learning and a convolutional neural network drive classification of pathogen replication and host defense To train for detection and analysis of host-pathogen interactions, HRMAn was provided an annotated dataset of host cells infected with an eGFP-expressing version of the parasite Toxoplasma gondii (Tg) and stained for various host cell features ( Figure 2A) (Seibenhener et al., 2004;Clough et al., 2016). For stage one pathogen detection and enumeration training a simplistic ML strategy -decision tree performed remarkably well. Over 35,000 Tg-vacuoles were analyzed by decision tree, gradient boosted tree, and random forest machine learning classification algorithms and cross-validated ( Figure 2B). As each performed equally, a simple decision tree with Minimum Description Length (MDL) pruning, to limit overfitting, was employed for speed and accuracy of pathogen detection (>99.5% for Tg). Using these parameters, in addition to the readouts from stage 1 (see Figure 1), HRMAn detected and quantified Tg-containing vacuoles harbouring 1, 2, 4 or >4 fluorescent Tg ( Figure 2C).
For stage 2, host protein recruitment, the CNN was trained for ubiquitin and p62 recruitment using segmented Tg vacuoles defined in Stage 1. Robust classification of host protein recruitment was achieved by passing these regions of interest through multiple non-linear filters to identify and differentiate between no recruitment, recruitment, and analysis artefacts ( Figure 2D). Training over 80 epochs with negative log likelihood as a loss function, the deep CNN achieved 92.1% classification accuracy confirmed by expert-based cross-validation. Precision for 'no recruitment', 'recruitment', and 'artefacts' classes was 0.92, 0.92 and 0.71, while recall was 0.94, 0.89 and 1 respectively, hence achieving the accuracy of a human operator and far exceeding human capacity ( Figure 2E).
To assure that uninvaded Tg parasites do not skew the data, stringent synchronization of infection by centrifugation and washing procedures were employed. In a pilot experiment ( HRMAn allows for accurate high-throughput analysis of the host defense response to Toxoplasma To demonstrate the ability of HRMAn and to expand how researchers define and classify host-pathogen interactions, the impact of IFNg on Tg replication and ubiquitin/p62 recruitment to Tg vacuoles was analyzed (Figure 3).
Previous reports indicate that HeLa cells restrict the growth of Tg through ubiquitination of parasitophorous vacuoles and subsequent non-canonical, p62-dependent autophagy (Selleck et al., 2015;Clough et al., 2016). HeLa cells infected with eGFP Tg ±IFNg were fixed 6 hr post-infection (hpi) and stained with Hoechst (nuclei) and antibodies directed against ubiquitin and p62. A total of Figure 1 continued the CNN diagram representing two-dimensional convolutional filters with respective width, height and depth designated on filters facets. Respective change of stride in the groups of hidden layers is depicted above the diagram, while respective activation functions below the diagram. Finally, the data is written as a single file and will provide the researcher with more than 15 different readouts that describe the interaction between pathogen and host cell during infection. HRMAn is based on the open-source data integration environment KNIME Analytics Platform making it modular and adaptable to a researcher's needs. The analysis is based on training of the machine learning algorithms generating high flexibility, which can be tailored to the needs of the user. 1,350 4-colour images were acquired on an automated microscope and loaded into HRMAn for analysis.
HRMAn automatically detected and analyzed more than 15,000 HeLa cells resulting in 15 quantitative outputs of host-pathogen interaction ( Figure 3). Population level readouts from stage one indicated that IFNg treatment did not impact the percentage of infected cells but decreased the number of vacuoles within host cells as well as the number of parasites per cell ( Figure 3A). As eGFP fluorescence is lost when parasites are killed, a reduction in the ratio between vacuoles and host cells serves as an indirect measurement for parasite killing. At the single cell level, HRMAn found that IFNg treatment resulted in a significant reduction of vacuoles per cell and a minor reduction in mean vacuole size, without impacting vacuole position ( Figure 3B). Concomitant with this reduction in vacuole size, both the percentage of replicating parasites, and the number of parasites per vacuole were significantly reduced by IFNg treatment ( Figure 3C). Thus, IFNgÀmediated control    of Tg in HeLa cells involves both parasite-killing and restriction of Tg replication. Importantly, HRMAn offers a wide range of readouts in stage 1 analysis allowing for more detailed information on the dynamics of infection and clearance than typically seen with manual counting. To allow the user to decide which readouts are best suited to answer their specific research question some redundancy has been purposely built in (e.g. mean vacuole size vs. % Replicating). For example, here we focused on parasites per vacuole and the proportion of infected cells, as opposed to the number of individual vacuoles per host cell.
In stage 2, analysis of the >25,000 vacuoles identified in stage 1, showed that the number of host cells with ubiquitin/p62-positive vacuoles and the percentage of ubiquitin/p62-positive vacuoles per host cell increased with IFNg ( Figure 3D). Distribution analysis indicated that in untreated cells, only 5.92% of vacuoles were decorated with ubiquitin, p62, or both. This number rose to 27.61% in IFNgtreated cells, the majority of which (20.92%) were double-positive for ubiquitin/p62 ( Figure 3D). By quantifying the radial fluorescence intensity distribution of these host factors, HRMAn revealed that ubiquitin was more closely associated with Tg vacuoles than p62 and that recruitment of both was increased by IFNg treatment ( Figure 3E). This is in agreement with the notion that p62 binds a ubiquitinated vacuole substrate through its UBA domain (Seibenhener et al., 2004;Clough et al., 2016). Finally, by analyzing vacuoles that recruit ubiquitin/p62, HRMAn indicated that restriction of Tg replication occurs in vacuoles decorated with these host defense proteins ( Figure 3F). Collectively, this data indicates that in HeLa cells, IFNg drives both parasite killing as well as recruitment of ubiquitin/p62 to Tg vacuoles, which acts to restrict parasite replication ( Figure 3). The results demonstrate the capacity of HRMAn to provide a quantitative, multi-parametric readout of host-pathogen interaction at population and single-cell levels.
As a high-throughput, high-content analysis program, HRMAn removes experimental size constraints imposed by manual quantification. To illustrate this, HRMAn was used to systematically analyze the impact of IFNg treatment on type I and type II Toxoplasma strains in five human cell lines: HeLa (cervical carcinoma epithelial), PMA-differentiated THP-1 (macrophage-like), A549 (lung carcinoma epithelia), HFF (primary fibroblasts), and HUVEC (primary endothelial cells) ( Next, HRMAn was employed on all 5 cell lines infected with either type I and type II Tg ±100 IU/ ml IFNg and immuno-stained for ubiquitin and p62.  . Consistent with this, stage 2 HRMAn showed that all cell types could recruit ubiquitin and/or p62 equally well (Figure 3-figure supplement 6A), while a greater percentage of type II vacuoles per cell were decorated in response to IFNg-priming ( Figure 3-figure supplement 6B). The exception to this were THP-1 cells, which did not mount a strain-specific response (Figure 3-figure supplement 6B). Distribution analysis further indicated that THP-1s display a higher intrinsic capacity to decorate Tg vacuoles than other cell lines, even in the absence of IFNg (Figure 3-figure supplement 6C). While no cell-type dependent differences in ubiquitin or p62 coat distance were observed (Figure 3-figure supplement 7A), THP-1s not only decorate vacuoles with more ubiquitin upon IFNg stimulation, they also appear to recruit p62 in an IFNg-independent fashion (Figure 3-figure supplement 7B). Decorated vacuoles in all host cell types displayed a greater ability to restrict the growth of type II versus type I Tg upon IFNg treatment ( Figure 3-figure supplement 7C-D). These results highlight the ability of HRMAn to provide high-throughput and quantitative single-cell analysis of host-pathogen interactions at a scale not achievable by automated bulk or manual quantification.

HRMAn can be adapted for bacteria-host interaction analysis
To demonstrate its flexibility, HRMAn was trained to recognize the bacterium Salmonella enterica Typhimurium (STm) -a pathogen 16x smaller than Tg (0.5 mm vs. 8 mm) -and then set to analyze the impact of IFNg on bacterial killing, replication, and ubiquitin recruitment. Stage 1 outputs showed that similar to Tg (Figure 3), IFNg treatment in HeLa cells reduced the ratio of STm vacuoles/cell and the bacterial load, without impacting the percent of infected host cells ( Figure 4A). At the single cell level, HRMAn found a significant reduction in the number of STm vacuoles/cell, consistent with a reduction in vacuole size, percent of replicating bacteria, and reduced numbers of STm/vacuole ( Figure 4B-C). These results demonstrate that HeLa cells can control infection with STm through IFNgÀdependent bacterial killing and growth restriction.
For stage 2, we used the Tg recruitment model as input to retrain HRMAn for quantification of ubiquitin recruitment to STm ( Figure 4D). This allowed us to achieve 69.9% classification accuracy, confirmed by expert-based cross-validation, in just 40 epochs using 10-fold less non-augmented image data ( Figure 4D). It's known that HeLa cells restrict STm growth by maintaining vacuole integrity; the small percentage of bacteria that escape vacuoles are decorated with ubiquitin and subsequently cleared by autophagy (Noad et al., 2017;van Wijk et al., 2017). Interestingly, stage 2 HRMAn showed that the percent of host cells which recruit ubiquitin to STm doubles upon IFNg treatment, while the percent of decorated vacuoles/cell increases only slightly ( Figure 4E). As seen with Tg ( Figure 3E, Figure 3-figure supplement 7A), IFNg does not impact the distance of the ubiquitin coat to STm but increases its thickness ( Figure 4F). This indicates that more ubiquitin is recruited to cytosolic STm in the presence of IFNg and growth of decorated bacteria was restricted ( Figure 4G). Consequently, although IFNg treatment increases the number of host cells that recruit ubiquitin to STm and the intensity of that recruitment, at the single-cell level HeLa cells appear to have reached their capacity for detection and autophagy-mediated clearance of cytosolic/ubiquitinated STm independent of IFNg treatment ( Figure 4E-G).

HRMANs versatility allows for rapid adaption to study pathogen biology
To illustrate the versatility of HRMAn and the advantage of the modular architecture combined with the accessible user interface that comes with the KNIME Analytics platform, we performed experiments to stress HRMAn's applicability and adaptability to study pathogen-driven parameters of infection. Using transgenic Tg lines expressing different parasite virulence factors, we were able to reproduce and expand upon published data (Virreira Winter et al., 2011). We confirmed that expression of ROP16 from type I Tg or lack of GRA15 in otherwise isogenic type II parasites (PruA7) reduces the recruitment of murine guanylate binding proteins (Gbps) to the vacuoles. Similarly, expression of type I ROP18 in type III parasites (CEP) also reduced recruitment of murine Gbps 1, 2 and 5 compared to isogenic type III parasites ( Figure 5A). This analysis shows that HRMAn can be used to study effects of pathogen effector proteins on an established host phenotype.
Next, we asked whether HRMAn can accurately measure parasite effector proteins targeted elsewhere within infected host cells. Tg is known to secret multiple effector proteins upon invasion and during replication. Examples include the parasite proteins GRA16, GRA24 and TgIST, that are secreted beyond the boundaries of its PV and subsequently translocated to the host cell nucleus Braun et al., 2013;Gay et al., 2016). Using HRMAn, we were able to visualize accumulation of these three Tg effector proteins (tagged with an HA-tag) in the nucleus of the host cell. HRMAn, further indicated that the levels of nuclear accumulation correlated to the number of parasites contained within the infected cell ( Figure 5B). Thus, HRMAn can be employed to analyze both host and parasite parameters during infection in an unbiased, accurate and high content manner.

Discussion
Recent advances have made deep CNNs a powerful image analysis method (Simonyan and Zisserman, 2014;He et al., 2015;Ioffe and Szegedy, 2015;LeCun et al., 2015;Russakovsky et al., 2015;Haberl et al., 2018). Inspired by abstraction of animal visual cortex architecture, CNNs are able to generalize patterns independent of minor phenotypic differences (Hubel and Wiesel, 1968;Matsugu et al., 2003). Combining automated image segmentation, machine learning and a deep CNN in an ensemble, HRMAn is a powerful open-source, user-friendly software for the analysis of host-pathogen interaction at the single-cell level. We based HRMAn on the KNIME Analytics Platform (Berthold et al., 2008). Being highly modular, GUI-based and user-friendly, HRMAn can rapidly be updated with latest technological advances, yet remains transparent for the average user. Furthermore, the ready-to-use DL4J library modules we employed allow for incorporation of the latest advances in the field of artificial intelligence in a click-through manner with zero coding. To date, HRMAn represents the only open-source CNN-driven host-pathogen analysis solution for fluorescent images. While HRMAlexNet is a rather simple architecture, more complex architectures can be easily implemented through recently introduced KNIME-Keras integration (Chollet, 2015), which may facilitate improvement in the classification accuracy. This is important, as it moves the phenotype of host defense protein recognition of pathogens into the realm of HCI at the level of artificial intelligence and thus human accuracy and capacity. Many automated image analysis programs, some of which incorporate machine learning elements, have been developed and are successfully used for classical image segmentation (Supplementary File 1). However, when presented with the problem of classifying host protein recruitment to a pathogen, inaccurate classical image segmentation could lead to erroneous results. Employing an artificial intelligence algorithm, HRMAn circumvents these problems and delivers user-defined automated and unbiased enumeration of this subset of the host-pathogen interplay.
Using Tg and STm infection models, we demonstrate that HRMAn is capable of detecting and quantifying multiple pathogen and host parameters. Importantly, we show that HRMAn can be adapted easily to two entirely different pathogens, that not only differ in size by a magnitude, but also display distinct growth rates and infection dynamics. The easy adaption of HRMAn for different pathogens and research questions will prove useful for any lab working in image-based infection biology. Designed for biologists, HRMAn requires no coding or specialized computer science knowledge. Its modular architecture and the use of KNIME, which provides a graphical representation of the analysis pipeline, allows users to tailor experimental outputs to their own datasets and questions.
As the models we have generated can be used as primers to lower the training dataset size, computation power and training time requirements, HRMAn can be rapidly applied to similar large-scale, image-based experimental setups. As such, HRMAn will allow a broad range of researchers to extend into the realm of high-throughput, unbiased, quantitative single-cell analysis of host-pathogen interaction.

Code and image availability
All open-source KNIME workflows used in this publication can be found at: https://github.com/ HRMAn-Org/HRMAn and on the homepage https://hrman.org/ under GPLv3 open-source software license to allow for rapid and open dissemination and free availability for the research community. The models and their respective weights obtained through training will be deposited on GitHub and the homepage as well.
All images used to train the neural networks and other machine learning algorithms in this study are available upon request.

Image acquisition
For simple infection analysis (stage 1), 96-well plates (see Microscopy sample generation) were imaged on an ArrayScan VtI Live High Content Imaging Platform (Thermo Scientific) using 20x magnification and depending on the experiment, 15-20 fields of view per well.
For recruitment analysis to Toxoplasma gondii (Tg) vacuoles, glass-bottom 96-well plates were imaged on an ArrayScan VtI Live High Content Imaging Platform but using 40x magnification and depending on the experiment, 50 fields of view per well. In both cases, following image acquisition, the images were exported from HCS Studio Cell Analysis Software as single channel 16-bit tiff files before they were fed into the HRMAn analysis pipeline.
For recruitment analysis to Salmonella enterica Typhimurium (STm) vacuoles, images of coverslips were acquired on a Ti-E Nikon microscope equipped with an LED-illumination and an Orca-Flash4 camera using a 60x magnification. 75 fields of view per coverslip were acquired using multi-position acquisition. Images were exported as single channel 16-bit tiff files with Nikon NIS Elements software before they were fed into the HRMAn analysis pipeline.
Generally, HRMAn can work with any common image file format, but the use of uncompressed, lossless formats like tiff (or png) is recommended. Furthermore, HRMAn can work with images acquired on any type of fluorescence microscope and is truly platform independent.

Image analysis using HRMAn
Following image acquisition, the images were loaded into the HRMAn pipeline. Images can be in any common file format, preferably as single-channel tiff files. The used image reader loads images from all file formats supported by SCIFIO (more information can be found here: http://scif.io). If the images were not acquired on a high-content imaging platform, they can be renamed with HRMAn to mimic the file names and the plate format. This is needed to cluster the output data and perform error calculation. Furthermore, the OME-XML-metadata is loaded and information on the image is extracted (e.g. image size, type and origin). While the images are loaded into KNIME, the user is asked to provide some basic information on the image acquisition and the type of analysis to be performed. This includes the used magnification, type of analysis, channel number and order and providing a plate map to cluster the data.
HRMAn then pre-processes the images, metadata and provided information and lets the user inspect the input images arranged into a grid and sorted by the field of view. Next the input images undergo illumination correction by dividing the background as a mean image of all acquired images in a channel-wise fashion. Following this step, the individual channels are segmented to detect the Nuclei, the pathogens and the cells: 1. Nuclei are detected using Otsu's method thresholding (Otsu, 1979), a watershed and connected component analysis. Fields of view containing insufficient numbers of nuclei (i.e. empty fields) are excluded from the following analysis.
2. The pathogens (or vacuoles) are detected after image normalization and filtering through thresholding using Otsu's method. Incomplete labels are corrected by filling holes and pathogen vacuoles are separated through water-shedding. Labels are created with a connected component analysis. 3. Cell labels are created using Huang thresholding (Huang et al., 2004) and a Voronoi segmentation using the nuclei labels as starting points. Optionally the images can be enhanced using Contrast Limited Adaptive histogram equalization (CLAHE) to improve the segmentation accuracy. All cell labels touching the border of an image are excluded from the analysis.
Furthermore, the created labels are filtered based on their size and the user-defined parameters such as magnification and detector size. The filter values for STm and Tg were empirically determined using thousands of images from different experiments. Based on which pathogen type is chosen by the user, HRMAn will adjust the filters automatically. Using labelling arithmetic, pathogen labels that are not contained within a cell label are removed from the dataset, as they represent extracellular pathogens. The created labels can then be inspected by the user through an interactive label viewer displaying the original image next to the labels.
Using the created labels, the infection readouts for stage 1 are created: First, cell numbers N Cells and vacuole numbers N Vacuoles are determined by counting the numbers of respective labels in all acquired fields per well (=replicate). Using these values, the vacuole to cell ratio is calculated: Next, the dependencies between the cell labels and vacuole labels are used to calculate the proportion of infected cells and the infection levels of the cells. The label dependencies determine which vacuole labels V 1 , V 2 , . . ., V i are contained by which cell label C 1 , C 2 , . . ., C j . If a cell label C contains at least one vacuole label V, the cell is considered as infected cell C inf . This is used to calculate the proportion of infected cells: Also based on the dependencies, the mean euclidean distance d between the centroid of the vacuole labels (with coordinates X V and Y V ) within a cell and its nucleus' centroid (with coordinates X N and Y N ) is determined as the position of the vacuole: Using the vacuole labels, of vacuoles contained within cells, and working on the original images the properties of the vacuoles are measured as mean values for each well. These include mean vacuole size, shape descriptors (Circularity, Perimeter, Convexity, Extent, Diameter) and fluorescence properties (Minimum, Mean and Maximum Fluorescence).
Using the above determined values as attributes, a decision-tree machine learning algorithm determines good classifiers and employs them to classify each vacuole label V i based on how many individual pathogens P Vi it contains. This step requires providing an annotated dataset.
Based on this classification the vacuoles can be divided into individual groups for the number of pathogens they contain (1/ vac = V 1Vac , 2/ vac = V 2Vac , 4/ vac = V 4Vac and four or more/vac = V !4Vac ) and the number of vacuoles in each group is counted (e.g. number of vacuoles that contain just one pathogen = N V1Vac ). To calculate the proportion of replicating pathogens, the number of vacuoles that contain at least two pathogens is divided by the total number of vacuoles: Combining the information on the number of vacuoles and the number of pathogens P Vi each individual vacuole V i contains, the total number of pathogens N Pathogens is calculated: This can be used to determine the Pathogen load by normalization to the cell number: This concludes stage 1 infection analysis performed by HRMAn. In the end of the analysis, the values calculated for each well or replicate is combined with the values for the other wells belonging to the same sample group based on the user-provided plate map and error calculation is performed. If the user decides to perform only stage 1 infection analysis the HRMAn image analysis pipeline will stop here, if host protein recruitment analysis is to be performed the data will be fed into the second stage for which the implemented deep Convolutional Neural Network (CNN) has to be created and trained first.

Deep learning setup and neural network architecture
To classify pathogen recruitment, we employed a deep Convolutional Neural Network (CNN) HRMA-lexNet ( Figure 1) inspired by published AlexNet (Krizhevsky et al., 2012). Our architecture consisted of a total of 5 convolutional layers, where the first two were immediately followed by local response normalization layers and max pooling layers. The last three convolutional layers were followed by one max pooling layer connected to a fully connected layer. All these layers used rectified linear unit (ReLU) as activation function (Nair and Hinton, 2010). To ensure our neural network can be implemented by other researchers with no coding, it was based on the open source DeepLear-ning4J library implementation in KNIME Analytics Platform. Changes to the original AlexNet architecture were partially introduced by the KNIME AG team and partially by us in the process of optimizing the architecture to the fluorescence microscopy data. Here further improvements in architectures may be made through recently introduced KNIME-Keras integration (Chollet, 2015).
While having the same number of convolutional layers as AlexNet, as introduced by the KNIME team, their dimensions are different. HRMAlexNet has two fully connected layers instead of three, which pushes the fitting burden more to the convolutional layers ( Figure 1).
We have changed the input layer dimensions to fit the multichannel fluorescence microscopy data, since our input two channels, rather than the standard RGB dimension of three. Hence, unlike the original AlexNet, HRMAlexNet was suited to take 100 by 100 by two pixels images as input and designed to run on a single graphic processing unit (GPU).
Furthermore, we have modified the SoftMax (Bishop, 2006) output layer of the architecture to fit our dataset and classification problem. While optimizing the network's hyperparameters, we found that learning rate updating algorithm proposed by default in the KNIME architecture, as well as in Krizhevsky et al. -Stochastic Gradient Descent (SGD) with Nesterov momentum -was failing to converge on our fluorescence microscopy data (Krizhevsky et al., 2012). Therefore, we have substituted this algorithm for the more advanced learning rate update algorithm ADAM (Kingma and Ba, 2014) with both mean and Variance decay parameters, which proved to converge well on various data and initial learning rate ranges.
The choice of DeepLearning4J as a deep learning library allowed us to use 16-bit microscopy images directly, preventing information loss upon conversion of scientific imaging data. Our deep learning hardware was based on a single Nvidia 1080 Ti GPU set up in Intel Core i7 4790K system equipped with 32 Gb of RAM and a SSD.

Neural network training and hyperparameters optimization
Our neural networks were trained using SGD based backpropagation (using ADAM updater with ADAM Mean Decay of 0.9 and ADAM Variance Decay of 0.999 (Kingma and Ba, 2014)) on the augmented original data over at least 80 epochs. To fully utilize the GPU capacity, training was performed in mini-batches of 200. We employed the Xavier algorithm for the weight initialization strategy and negative log likelihood (NLL) as our loss function (Kingma and Ba, 2014). We used learning rates between 0.001 and 0.01 adjusted accordingly to ensure the optimal loss curve decay during training. Together with the weights initialization strategy and the updater choice these were the main hyperparameters optimized in multiple iterations to ensure good training process.
To visualize the attention of the trained HRMAlexNet, we have created a class activation map (CAM) based on a test image from 'recruitment' class ( Figure 2). Since HRMAlexNet architecture does not have a Global Average Pooling layer to visualize the CAM we have used the occlusion technique (Zeiler and Fergus, 2013).

Data preparation, Augmentation and Model Validation
Vacuole images used for creation of our deep learning model were segmented from large field of view micrographs obtained from high-content imaging. To ensure the dimension of the images are uniform, we padded all vacuole with zero-value padding to a uniform 100 by 100 pixels size. Next, we manually labelled the segmented vacuoles into recruited, non-recruited and artefactual (in case of erroneous segmentation of the vacuole). This labelled dataset was then split into the training and test datasets.
To ensure our neural network has sufficiently diverse learning data, upon splitting the original labelled dataset into training and test subsets we performed data augmentation using a custom developed macro for ImageJ. During the augmentation processing, the original labelled dataset was concatenated with a modified version of it. The modifications included various rotations, image reflecting, image translation within the field of view. As microscopy data are typically rotation-, translation-or reflection-invariant, such modification allowed us to create a better dataset aiming at a more generalized model.
Model validation was performed using the non-augmented test fraction of the labelled dataset previously unseen by the model. For this, we used the trained model as first input and passed the labelled test data through the classifier in the second input. The classification accuracy was assessed by accuracy score, numbers of true positive, false positive, true negative and false negative, as well as Cohen's kappa values. A direct summary of the accuracy was visualized in a confusion matrix illustrating a mismatch between original label (Ground Truth) and the class assigned by the classifier (Figure 2).

Host protein recruitment analysis in HRMAn
For recruitment analysis, the vacuole labels created in stage 1 of the analysis are dilated over 20 iterations to create non-overlapping regions of interest (ROIs) around them. Simultaneously, the fluorescence images of the pathogen and the respective channel with fluorescence signal of the host protein are merged into a dual channel image. The created ROIs are used to crop the dual channel images, which creates images of the pathogen and its surrounding stained host protein. The images are clipped to 100 by 100 pixel and fed into a feedforward predictor (classification) which uses the provided and trained deep convolution neural network (CNN) to classify the pathogens vacuoles based on their coating.
Once the vacuoles are separated into two groups, they are analyzed with the above described methods of stage one infection analysis but additionally comparing recruited versus non-recruited vacuoles. Thus, in addition to the overall infection parameters from stage one the user is provided with the same parameters but further layered for the cellular response.
In the case of co-recruitment analysis, two images of each vacuole are created with both containing the pathogen signal, but each containing a different second channel, representing the different stainings. After classification with the CNN, the vacuoles can be compared for recruitment or corecruitment and all the above described parameters are calculated for them individually. Using the previously determined label dependencies of vacuoles V i and cells C i and the classification of the vacuoles V i by the CNN, HRMAn can furthermore calculate the proportion of cells that do respond to infection by decorating at least one vacuole and the proportion of vacuoles decorated per cell, if a single cell contains more than one pathogen vacuole.
Furthermore, working only on the decorated vacuoles, we used a custom-made Fiji code to create a pixel-wise radial intensity profile starting from the pathogen centroid. The distance of the maximum fluorescence intensity is then used to define the distance of the coat from the pathogen centre. Moreover, the mean fluorescence intensity of the coat is determined and can be used as a readout for the amount of protein recruited to each pathogen vacuole.
Finally, all mean values and errors for the replicate conditions, as defined by the user's plate map, are calculated and written into a single spreadsheet file. Before this, the user can also define a scaling factor between pixel and actual metric values which will adjust the output values from pixel (px) to mm or to mm 2 respectively.

Interferon stimulation of cells
All five cell lines used in this publication were stimulated for 16 hr in complete medium at 37˚C with addition of 100 IU/mL human IFNg (285-IF, R&D Systems) prior to infection, if not indicated otherwise.

Toxoplasma gondii infection
Parasites were always passaged the day before infection onto new HFFs to obtain parasites with a high viability for infection. Tg were prepared from freshly 25G syringe-lysed HFF cultures in 10% FBS by centrifugation at 50 x g for 3 min and transferring the cleared supernatant into a new tube and subsequent centrifugation at 500 x g for 7 min and re-suspension of the pelleted parasites into fresh complete medium. Then, the parasites were added to the experimental cells at a MOI of 3 for both type I and type II strains. The cell cultures with added Tg were then centrifuged at 500 x g for 5 min to synchronize the infection. Two hours post-infection, the cultures were thoroughly washed two times with warm PBS (806552, Sigma) to remove any uninvaded parasites and fresh complete medium was added prior to culturing at 37˚C, 5% CO 2 for the required time.

Bacterial culture and infection
Salmonella enterica Typhimurium 12023 strain containing the plasmid pFVP25.1, carrying gfpmut3A under the control of the rpsM constitutive promoter (Valdivia and Falkow, 1996) were grown in Luria Bertani (LB) medium supplemented with 50 mg/ml ampicillin (11593027, Gibco). Prior to infection, bacteria were grown to induce SPI-1 T3SS expression: cultures of STm were grown at 37˚C in LB, diluted 1:50 into fresh LB containing 300 mM NaCl (746398, Sigma) the next morning and incubated shaking at 200 rpm until OD600 = 0.9-1.0 was reached. Bacteria were washed in medium without FBS before use. Cells were infected at a MOI of 50 and infections were synchronized by centrifuging bacteria onto the cells at 750 x g for 10 min. 15 min post infection, the cells were thoroughly washed three times with warm PBS to remove extracellular bacteria and medium containing 100 ug/mL Gentamicin (15750060, Gibco) was added for 30 min. Then, Gentamicin concentration was reduced to 10 mg/mL and cells were incubated further at 37˚C, 5% CO 2 for the appropriate amount of time.

Microscopy sample generation Simple infection analysis
For simple infection analysis, 30,000 THP-1s per well were seeded 5 days prior to IFNg treatment and differentiated with 50 ng/mL PMA for three days and then rested for 2 days in complete medium. HFFs were harvested by washing a confluent monolayer with PBS and subsequent lifting of the cells with 0.05% Trypsin-EDTA (Gibco). Cells were centrifuged at 250 x g for five mins, re-suspended in fresh DMEM plus 10% FBS and 20,000 HFFs per well were seeded the day before IFNg treatment. Similarly, HUVECs were harvested and 15,000 cells per well were seeded in complete medium the day before IFNg treatment. A549s and HeLa cells were harvested in the same way and 8,000 cells per well were seeded the morning before IFNg treatment. All cells were seeded on 1% (w/v) porcine gelatin (G1890, Sigma) pre-coated black-wall, clear bottom 96-well plates (Thermo Scientific). In the evening, all cells were treated with 100 IU/mL IFNg or medium and left at 37˚C, 5% CO 2 overnight. The next morning the cells were infected with either Tg or STm as described above. After the appropriate infection duration, the infected cells were again thoroughly washed with warm PBS to remove as many uninvaded pathogens as possible and subsequently fixed with 4% methanolfree paraformaldehyde (28906, Thermo Scientific). Fixed specimens were permeabilized with Perm-Quench buffer (0.2% (w/v) BSA and 0.02% (w/v) saponin in PBS) for 30 min at room temperature. Then PermQuench buffer containing 1 mg/mL Hoechst 33342 (H3570, Invitrogen) and 2 mg/mL Cell-Mask Deep Red plasma membrane stain (C10046, Invitrogen) were added and samples were incubated at room temperature for 1 hr. After staining, the specimens were washed with PBS five times and kept in 200 mL PBS per well for imaging.

Recruitment analysis
For recruitment analysis, the cells were prepared as described above, but they were seeded on 1% (w/v) porcine gelatin pre-coated black-wall, glass bottom 96-well imaging plates CG 1.0 (130-098-264, MACS Miltenyi) to allow higher resolution imaging. After fixation, cells were permeabilized identically and then stained with primary antibody diluted in PermQuench buffer for 1 hr at room temperature. After three washes with PBS, cells were incubated with the appropriated secondary antibody and 1 mg/mL Hoechst 33342 diluted in PermQuench buffer for another hour at room temperature. Then, the specimens were washed with PBS five times and kept in 200 mL PBS per well for imaging.
In the case of recruitment analysis to STm vacuoles, the cells were seeded on 1% (w/v) porcine gelatin pre-coated 9 mm coverslips in 24-well plates. After fixation and identical staining procedure, the coverslips were mounted using 5 mL ProLong Gold Antifade Mountant (P36930, Invitrogen).

Data handling and statistical measurements
Data was plotted using Prism 7.0e (GraphPad Inc.) and presented with error bars as standard error of the mean (SEM). Significance of results was determined by non-parametric one-way ANOVA or unpaired t-test as indicated in the figure legends. Correction for multiple comparisons was performed by controlling the False Discovery Rate with Two-Stage step-up method after Benjamini, Krieger and Yekutieli. Data availability All data are contained within the manuscript. Due to the size of the dataset, the raw data are available upon request from the corresponding author but a 20Gb subset is available via Dryad (doi:10.5061/dryad.6vq2mp0).

Decision letter and Author response
The following dataset was generated: