Artificial intelligence techniques for neuropathological diagnostics and research

Artificial intelligence (AI) research began in theoretical neurophysiology, and the resulting classical paper on the McCulloch‐Pitts mathematical neuron was written in a psychiatry department almost 80 years ago. However, the application of AI in digital neuropathology is still in its infancy. Rapid progress is now being made, which prompted this article. Human brain diseases represent distinct system states that fall outside the normal spectrum. Many differ not only in functional but also in structural terms, and the morphology of abnormal nervous tissue forms the traditional basis of neuropathological disease classifications. However, only a few countries have the medical specialty of neuropathology, and, given the sheer number of newly developed histological tools that can be applied to the study of brain diseases, a tremendous shortage of qualified hands and eyes at the microscope is obvious. Similarly, in neuroanatomy, human observers no longer have the capacity to process the vast amounts of connectomics data. Therefore, it is reasonable to assume that advances in AI technology and, especially, whole‐slide image (WSI) analysis will greatly aid neuropathological practice. In this paper, we discuss machine learning (ML) techniques that are important for understanding WSI analysis, such as traditional ML and deep learning, introduce a recently developed neuropathological AI termed PathoFusion, and present thoughts on some of the challenges that must be overcome before the full potential of AI in digital neuropathology can be realized.


INTRODUCTION A brief history of artificial intelligence research
Artificial intelligence (AI) is an umbrella term that includes machine learning (ML) (Fig. 1). Deep learning (DL) is a subtype of ML that relies on the use of artificial neural networks. It is in DL where especially rapid progress is currently being made. The history of ML began in 1943, when McCulloch and Pitts developed a mathematical model of a neuron. 1 ML consists of algorithms that parse data, learn from them, and then apply what they have learned to make informed decisions. McCulloch and Pitts used algorithms known as "threshold logic." A few years later, Turing developed the test named after him to determine whether a computer possesses intelligence. 2 In 1952, Samuel created the first computer learning program. 3 The first neural network for computers, the singlelayer perceptron consisting of multiple McCulloch-Pitts neurons, was designed by Frank Rosenblatt in 1958. 4 Extending the design, this model became a multilayer perceptron (MLP), which essentially represents a "plain vanilla" contemporary neural network while lacking the depth, i.e., many hidden layers, of today's networks. A step toward allowing computers to use basic pattern recognition was the application of the nearest-neighbor algorithm. 5 In 1979, Fukushima developed the first convolutional neural network (CNN), which had multiple layers. 6 In 1989, LeCun and colleagues demonstrated the first practical application of backpropagation in combination with CNNs for the recognition of handwritten zip codes. 7 Their original dataset remains in use for validation purposes. 8 Hinton coined the term "deep learning" in 2006 to describe new algorithms that allow computers to discriminate objects and text in images and videos. 9 His group's AlexNet (2012) catapulted CNNs into the spotlight, and their related 2017 paper 10 has earned more than 100 000 citations so far. Rapidly increasing computing power and better software have supported the explosive growth of DL techniques in recent years.

Relevance to neuropathology
The human brain is by far the most structurally complex of all organs. 8 Microscopic inspection of tissue sections is the diagnostic gold standard for most nervous system diseases, although neuroimaging and molecular genetic methods, the latter especially in the field of brain tumor diagnostics, are becoming increasingly relevant. The subjectivity of human diagnostic opinion is often used as an argument in favor of developing ML in medicine, but the lack of time required for the examination of large numbers of tissue sections at high microscopic magnification poses an equal or even greater problem in practice. In addition, the reality of neuropathological service and research is that most countries, for one reason or another (ranging from lack of funding to political issues), do not have enough highly trained specialists, although there are attempts, like the Euro-CNS (European Confederation of Neuropathological Societies) examination, 11 to mitigate the problem. It is our expectation, therefore, that in view of the severe shortage of staff specialists in most countries, neuropathology will be among the medical disciplines and research fields that will benefit most from the new AI technology. With the wide adoption of histological slide scanners, AI analysis of whole-slide images has become practical even for smaller departments. Therefore, the goal of this review is to provide an easy-to-understand and concise introduction to AI techniques that are relevant to clinical and research neuropathologists.

Data preprocessing
Image data preprocessing is an important step that helps ML models to cope with image artifacts 12 that can be introduced at various stages of tissue processing and imaging workflow. Image preprocessing removes such data, allowing for accurate processing of large whole-slide images (WSIs) for downstream modeling. Several techniques exist for removing artefacts. 13,14 One popular method utilizes Otsu thresholding, 15 which divides a grayscale image into foreground and background based on maximizing between-class variance. To check on image quality prior to analysis, thresholding can be combined with a variety of other features such as edge detection filters or contrast measurements. 13,14 Color variation of histological stains has been traditionally corrected using normalization that relies on reference images, [16][17][18] but generative adversarial networks (GANs) are now providing an alternative unsupervised approach. [19][20][21][22] Methods in ML ML methods can be categorized based on the learning technique used (e.g., supervised, unsupervised, reinforcement), the type of input data, common tasks (e.g., classification, segmentation), feature extraction methods, and traditional ML versus DL techniques.

Classification versus segmentation of images
There is a large body of literature on image classification and segmentation tasks. However, there is no consensus across studies that would support the conclusion that one algorithm is clearly superior to another. Yet, in most applications, DL-based classifiers tend to outperform traditional classifiers in terms of accuracy. For example, pixel classification and semantic as well as pixelwise segmentation have been used in a recent neuropathological study. 23 Classification Image classification can be thought of as assigning a label to an input image based on a set of rules. Labels are typically drawn from a predefined set of possible categories. Image classification methods that are commonly used involve Bayesian classifiers, geometric classifiers, clustering, and neural network classifiers. Classifiers can aid in identifying types of organ tissue, specific types of cells, recognizing subcellular structures or cells at various stages of differentiation, and disease entities such as tumors and other pathomorphologies. It is important to note that traditional ML methods depend on prior feature extraction. Different ML methods can be used to classify, such as support vector machines (SVMs), decision trees (DTs), naïve Bayes (NB), k-nearest neighbors (k-NN), multilayer perceptron (MLP), and others.  However, impressive results were achieved more recently using DL-based classifiers in medical image analysis. As the development of DL techniques advances rapidly, neural network classifiers, particularly those based on CNNs, are becoming increasingly popular. DL-based classifiers have shown promise even in the absence of any human intervention. CNN classifiers have undergone extensive development and evolution. Examples include LeNet, 47 the previously mentioned AlexNet, 10 VggNet, 48 GoogLeNet, 49 ResNet, 50 and many advanced CNNs proposed after ResNet. In addition, there are specific DL models for the classification of breast cancer, 51,52 colorectal polyps, 53 and gastric carcinoma, 54 among others. As illustrated in Figure 2, Bao et al. 55 developed the PathoFusion system, which includes a novel bifocal CNN approach, 56 to classify malignant features in scans of adjacent hematoxylin and eosin (HE) and immunostained neuropathological tissue sections. Abas et al. 57 proposed using SVMs and DTs to classify glioma cells in smear preparations; SVM and binary classification trees outperformed the other methods, with F1-scores ranging from 0.92 to 0.94. Furthermore, the SVM algorithm on its own was shown to be capable of classifying all four glioma grades with an accuracy of 90%. 58 Yonekura et al. 59 suggested that CNNs could extract relevant features of glioblastoma (GBM) slides with an accuracy of 98%. A recent study by Ker et al. 60 used a Google Inception v3 CNN system to classify brain histology specimens into normal, low-grade gliomas (LGGs) and high-grade gliomas (HGGs). An accuracy of 100% was achieved in the histological diagnosis of HGGs and 98% accuracy on comparing low-grade samples with normal brain tissue. The method proposed by Ertosun and Rubin 61 classified LGGs and HGGs on the basis of automated segmentation and analysis of cell nuclei and morphological features. Image classification is a frequently investigated topic in pathological image analysis. 53,[62][63][64][65][66][67][68][69] Recent examples from the field of neuropathology include a DL-based model for differentiating tauopathies including Alzheimer's disease 70 and a DT classifier for progressive supranuclear palsy and corticobasal degeneration, which simplifies their neuropathologic differential diagnosis. 71 Furthermore, DL-based image classification was shown to differentiate tufted astrocytes, astrocytic plaques, and neuritic plaques with 99% precision, suggesting that ML can be applied to the differential diagnosis of uncommon neurodegenerative diseases. 72 Segmentation Image segmentation and the localization of histological features and of individual cells represent crucial steps in WSI analysis. The segmentation of intracellular compartments can even provide quantitative information on cell function. Traditional segmentation methods include edge detection, threshold processing, region growing, and the morphology watershed algorithm. [73][74][75][76][77][78][79] More recently, DL has become a popular image segmentation method as DL-based segmentation achieves higher accuracy than traditional methods. Examples of applications include cell counting, 80 cell type detection (e.g., mast cells, 81 breast cancer cells, 82 liver cells, 83 blood cells, 84,85 cervical cells, 86 corneal endothelial cells 87 ), phase contrast analysis of cells in culture, 88 and cell nucleus analysis. 89,90 Mehta et al. 91 designed a new multiresolution encoder-decoder model to semantically segment breast biopsy images into biologically meaningful tissue labels. Alzoubi et al. 92 presented a framework that used PathoFusion 55 to classify WSI patches, then segmented the individual cells from the output patches and visualized the results by means of heatmaps, as shown in Figure 3. Random forest (RF) and SVM were 98% accurate in identifying DNA segmentation patterns in LGG and HGG histopathology images. 93 Reza et al. 94 proposed a cell nucleus segmentation method to classify GBM and LGG based on k-means clustering of nuclear morphological features that included area, perimeter, eccentricity, and major axis length and that made use of a MLP. Even earlier, Mousavi et al. 95 were able to distinguish GBMs from LGGs using image features extracted from regions of interest (ROIs) in WSIs. A hierarchical DT mechanism was used to arrive at decisions. Numerous approaches to the segmentation of pathological images exist. 29   Whatever technique is used, the availability of sufficient training data, also known as the ground truth, is critical.

Supervised ML
In supervised learning, the model to be trained is built on data with known correct labels and numerical values. It is the most basic ML method, and it is distinguished by the ease with which it can produce results that are comparable to human prediction and classification. Annotating or labeling of data is an important step in supervised learning. Labeling tasks, such as those involving neuropathology images, need to be carried out by consultant neuropathologists at least initially (until helpers have been specifically trained) because the quality of an annotation has a significant impact on learning outcomes. Depending on how the WSIs are annotated, there are four different types of labeling approaches, as illustrated in Figure 2 and explained in what follows. Pixelwise labeling is used for semantic segmentation and rarely for the analysis of WSIs because it requires experts to manually trace pathologically altered (e.g., tumor-infiltrated) regions and cellular structures in detail. Given the size and morphological complexity of WSIs, this task can be very time-consuming and inefficient, requiring significant human resources and computing power.
Patch-based annotations are typically derived from pixel-level annotations, which require experts to annotate all relevant pixels. A pathologist, for example, would need to localize and annotate tumor cells in a WSI that contains a tumor. This has been done for the detection of breast cancer metastases by extracting patches from normal and tumor regions based on pixel-level labels created by pathologists. [101][102][103] In addition, this method has been used with other diseases. 104 Tilewise labeling is a more common and efficient strategy than pixelwise annotation because experts only need to assign one or more labels (class labels) to the corresponding image tile, e.g., "malignant," instead of manually marking slides. This less resource-intensive labeling method can be used for both classification and segmentation. However, depending on the size of the selected image tile, some accuracy is sacrificed.
Slidewise labeling is the process of assigning a label to entire WSIs, but this is not common, either, because processing entire WSIs without image resizing and dimensionality reduction is computationally inefficient. Image filter-based preprocessing can be used to reduce the number of patches that must be analyzed. Examples include Otsu thresholding, hysteresis thresholding, and other types of thresholding to automatically identify tissues within WSIs. Other operations, like contrast normalization, dilation and erosion, and a problem-specific patch scoring system, can be used to reduce the number of candidate patches even further and enable automatic patch localization. Recently, several DL-based approaches to labeling entire slides have yielded very promising results. [105][106][107][108][109] The techniques used are frequently a combination of multiple instance learning, unsupervised learning, reinforcement learning, and transfer learning. The goal is to identify patches that can collectively or independently predict the slide label.
Casewise labeling is similar to slidewise labeling, but multiple WSIs can be acquired for each patient to improve diagnostic performance. In this case, the ground truth is not WSI-specific but may be patient-specific. [105][106][107] Unsupervised ML Unsupervised learning, for example, 110 identifies groups of data with similar characteristics that distinguish them from input data without the use of a label. Clustering, for instance, using the k-means algorithm, is a common unsupervised learning task. Clustering finds data with similar characteristics and divides them into various groups automatically. WSIs can be analyzed this way. 98,[111][112][113][114] Since data do not need to be prelabeled, unsupervised learning is easier to perform than supervised learning. However, because it can be impossible to predict what kind of classification criteria a computer will generate, it appears that, although it is possible to find a classification method that humans cannot imagine, classification is not always practical. Sornapudi et al. 115 designed a toolbox that uses a combination of the previously developed Epi-thNet model 116 for epithelium detection and segmentation with the DeepCIN classification model 117 for the diagnostic assessment of cervical intraepithelial neoplasia employing WSIs of cervical tissue samples.

Types of ML
ML-based methods for analyzing WSIs are categorized into two types: traditional ML methods and DL methods, as illustrated in methods have been used to analyze images by extracting handcrafted features like morphological and textural characteristics, followed by the design of classifier algorithms like SVMs, RF, and KNN for downstream analysis tasks. However, traditional ML methods are of limited use in WSI analysis. In contrast, DL technology has proven extremely successful in the field of computer vision, including WSI analysis. 118,119 DL algorithms can extract useful features directly from the data input stream, avoiding complex feature extraction steps. As stated in the classical paper by Krizhevsky et al. 10 : "With enough computation and enough data, learning beats programming for complicated tasks that require the integration of many different, noisy cues." Furthermore, DL algorithms appear far more capable of dealing with WSI heterogeneity.

Traditional ML
The challenge when extracting features from (neuro)pathological WSIs is finding higher-level representations that allow capturing information that is relevant for making a medical diagnosis. In view of the importance of morphometric data like shape and size of individual cells in disease diagnosis, several studies have used them as criteria. 27,120,121 Several studies have also combined feature categories to extract information found in WSIs. 24,30,38,[122][123][124][125][126] Maroof et al. 125 proposed a method of using hybrid space by combining color features with morphological and textural features and then changed the color channel to calculate normalized and cumulative histograms in the wavelet domain for better discrimination of mitotic and nonmitotic nuclei. Barker et al. 37 presented a method for the classification of brain tumor WSIs containing gliomas. They divided WSIs into grids of tiles and localized features of shape, color, and texture from tiled regions, which were then clustered using k-means. A detailed analysis was conducted of a single representative tile from each cluster and a classification decision made for each one using an elastic net model. Their method achieved an accuracy of 93.1%. Recently, the recognition of deep features has gained popularity in a variety of image classification tasks, including of WSIs. 44,127 Given the rich structures found in histological WSIs, the majority of studies now combine different types of morphometric and textural features.

CNN-based DL
CNNs currently are the most widely used artificial neural networks (ANNs). 128 They can be thought of as a feature learning process progressing from a low to a high level. A convolution is a mathematical operation, and convolutional layers are used to learn local features. Convolutional layers are the major building blocks of a CNN. It is here where filters are applied to the original image. The pooling layers are interleaved with convolutional layers to reduce the output from the convolutional layers. The final fully connected layers are used to combine the features learned from the convolutional layers, resulting in a complex and high-level representation. Some groundbreaking CNN architectures include LeNet, 47 AlexNet, 10 VggNet, 48 GoogLeNet, 49 and ResNet. 50 To avoid complex feature extraction steps, CNNs use raw images (or large image patches) as input, which are highly resistant to translation, scaling, inclination, and other forms of deformation. Because WSIs consist of highly complex data, DL algorithms are ideally suited for their analysis. Examples include Sharma et al., 54   from an AlexNet-CNN model 10 that had been pretrained on the ImageNet dataset, followed by the use of a linear SVM 129 to identify and segment brain and colon cancer images. Wang et al. 130 adopted the GoogLeNet architecture for the detection of metastatic breast cancer. Jamaluddin et al. 131 used DL on lymph node metastases, and Sirinukunwattana et al. 132 proposed a spatially constrained CNN (SC-CNN) for the classification of colorectal adenocarcinoma images.
Of special interest to neuropathology, Hou et al. 133

Fully Convolutional Network-based DL
CNNs have gained widespread acceptance for solving medical segmentation problems. Architectures based on fully convolutional networks (FCNs) 135 and U-Nets 136 are the most popular. In a FCN, a classical CNN's fully connected layers are replaced with fully convolutional layers. As a result, instead of classification scores, a FCN model generates a spatial segmentation map. In addition, several models originally developed for medical/biomedical image segmentation were inspired by FCNs and encoderdecoder models, such as the U-Net architecture. A U-Net augments the standard CNN architecture by adding a corresponding expansive path (decoder) with the goal of producing a full-resolution semantic prediction, whereas the CNN architecture consists of a contracting path (encoder) that iteratively builds feature maps using continuously down-sampled data.
B andi et al. 137 introduced a FCN and a U-Net network architecture 136 for WSI segmentation. They found that FCN and U-net models outperform older methods. Cui et al. 138 presented a supervised FCN method for nuclear segmentation of histopathology images. First, color normalization is performed on the image, then a FCN is used to detect nuclei and their boundaries in each patch extracted from the WSI. The procedure was tested using different datasets, which showed that the proposed method is accurate, robust, and fast: one 1000 Â 1000-pixel image can be segmented in under 5 s. Gecer et al. 66 presented a FCN system to identify ROIs in breast WSIs to classify WSIs into five classes. Wang et al. 139 exploited image-level labels to introduce a weekly supervised method using a FCN to select discriminative patches for the classification of lung cancer WSIs. Sheikhzadeh et al. 140 proposed using a FCN as an extension of a CNN to localize and quantify immunohistochemical WSIs in one integrated step. Lin et al. 141 utilized a fast scanning neural network, referred to as ScanNet, for metastatic breast cancer detection in WSIs.
The authors further improved their work by converting the standard FCN layers into an anchor layer that speeds up the network. 142 Graph neural networks CNNs have demonstrated great success in extracting image-level representations. They are ideal for dealing with linear, gridlike (Euclidean) data. However, when dealing with relation-aware representations, they are inefficient because they limit the model's ability to capture global contextual information and comprehensively model tissue composition. In tissue diagnosis, the phenotypic characteristics and topological distribution of cellular and histological features are critical. Consequently, graphical neural networks (GNNs) have attracted considerable interest because they capture intra-and interentity-level interactions. 143 In general, graphs describe natural phenomena better. GNNs are generalized versions of CNNs and capable of working with non-Euclidean data. Traditional CNNs analyze local image areas using fixed connectivity as determined by the convolutional "kernel." Graphs, on the other hand, provide greater flexibility when analyzing unstructured data by encoding the spatial context of individual patches. Thus, graph-based methods in (neuro)pathological analysis can record geometrical and topological properties together, allowing them to model both cell-level information and overall tissue microarchitecture simultaneously. Prior to the advent of DL, many graph-based methods in digital pathology processing relied on traditional ML approaches, 144 which are less accurate for graph classification than graph-based DL. However, the capabilities of graph-based DL for disease diagnosis, which bridges the gap between DL methods and traditional cell graphs, have yet to be thoroughly investigated. GNNs have already been used for ROI classification, 145 detection of malignancy and invasiveness, 146 and survival analysis. 147 We believe that they have great potential for neuropathological analysis.
As illustrated in Figure 5, image analysis in (neuro) pathology using graphs involves the following steps: establishment of graph representation (description of entity, embeddings, and definition of edges), design of GNNbased models for processing graph-structured data, and the use of interpretability methodologies (model-based and post hoc). Pathological images can be translated into entity graphs by means of graph representations where nodes and edges indicate entities and interentity interactions, 148,149 respectively. Individual cells, 146,148,[150][151][152][153][154][155][156][157][158] tissue patches, 147,159-166 different types of tissue, [167][168][169] and hierarchical entities [170][171][172][173] can all be used depending on the task. Following entity-graph construction, the graph is processed using GNN-based models; thus, GNN output can be either a node-or graph-level-based classification. Explainers for graph-structured data 174 are then applied to the entity graphs, yielding intuitive explanations for (neuro)pathologists. Attention mechanisms are an example of explainable methodologies in which the attention weights highlight the nodes and edges in their relative order of importance; as such, it can direct a network to focus on the most relevant parts of the input, suppressing irrelevant features, reducing computational cost, and improving accuracy.

Cell graphs
In these types of graphs, the graph is presented using cells or nuclei as entities with the locations of identified cells serving as graph nodes, and the spatial distances between them 148,149 are represented as lines called "edges." Jaume et al. 148 applied cell-graph analysis to breast cancer subtyping. First, they transformed the ROIs in the histology image into a cell graph, and a GNN model was then used to map the entity graph to a corresponding class label. In the next step, they used a post hoc graph explainer. Finally, a set of metrics was used to assess the generated explanations. A cell-graph explainer was used in breast cancer subtyping 152 to eliminate redundant and uninformative graph nodes and edges while retaining tumor cell nuclei relevant for cancer diagnosis.
Anand et al. 146 proposed a framework named Histograph for classifying breast cancer WSIs. The framework is based on a pretrained model that generates features around the nucleus centroids and models nuclei as graph vertices and internuclear distances as graph edges. Sureka et al. 153 also modeled histopathology images as graphs of nuclei but developed a graph convolutional network (GCN) framework based on an attention mechanism and node occlusion to highlight informative nuclei and internuclear relationships. Nuclei were detected using a U-Net on WSIs, and a graph was constructed by linking pairs of nuclei closer than the distance threshold. The latter method emphasizes the relative contribution of each nucleus and its neighborhood in the WSI and visualizes important nuclei for the final binary classification of the breast cancer biopsy. Zhou et al. 156 developed a GCN for colorectal cancer classification where each node is represented by a nucleus, whereas the edges between these nodes are built according to node similarity. The variety of methods applied again signifies that iterative trial and error and optimization form intrinsic parts of the DL approach.

Patch graphs
In a patch graph, the nodes are the ROIs extracted from the image using color-based, cell density, or attentionbased methods. Aygunes et al. 159 proposed a weakly supervised classification method using GCN as a model for ROI classification in breast histopathology. A pretrained ResNet model was used to extract a feature vector for each node represented by fixed-sized patches of the ROI. Then GCN architecture was used to decide on perpatch representations and to propagate information over the neighboring patches in a progressive manner to incorporate the spatial context and, finally, aggregate the resulting patch representations, leading to the assignment of the whole ROI to a diagnostic class. Ye et al. 160 presented a system for segmentation and classification of breast cancer ROI images based on GCNs. The system is composed of a segmentation module and a GCN module. Li et al. 147 developed a GCN with an attention learningbased survival analysis model of WSIs.

Tissue graphs
A cell graph only encodes cellular morphology, and its topology ignores tissue information. Therefore, it is not capable of capturing the tissue microenvironment, i.e., the distribution of tissue areas such as parenchyma or necrosis, for example, which is necessary for the accurate representation of a histopathological context. A tissue graph composed of multiple tissue areas, on the other hand, is incapable of depicting the cell microenvironment. Tissue graphs are built with nodes representing key tissue regions and edges encoding the relationships between these regions. Anklin et al. 167 proposed a weakly supervised segmentation method for representing a histology image using a "superpixel"-based tissue graph. An input image is first transformed into a tissue graph representation, with graph nodes representing tissue "superpixels." Thus, to learn the intrinsic characteristics of pathological tissue changes, multilevel structural information must be aggregated, with the goal of replicating the tissue diagnostic process used by pathologists when analyzing images at different levels of magnification. A GNN then learns contextualized features of the graph nodes using inexact and incomplete labels. Raju et al. 168  texture from a sample of patches and cluster similar texture features into multiple graphs, with each graph containing nodes that represent different tissues as an instance.

Hierarchical graphs
Histological structures cannot be fully described by abstract cellular or tissue interactions alone, so hierarchical graph representations have been proposed as additional tissue representations. Cell graphs and tissue graphs have been shown to provide valuable complementary information (cellular and tissue interactions) for learning the characteristics of a tissue. Hierarchical graphs aggregate multilevel structural information. Zhang and Li et al. 170 presented a multiscale graph wavelet neural network (MS-GWNN) for the classification of histopathological images of breast cancer. Based on GWNN, 61 MS-GWNN was proposed to perform multiscale analyses using spectral graph wavelets after transforming pathological images into graph representations. Multiple GWNNs with different scaling parameters were used in parallel to integrate the multiscale contextual interactions in graph topology. The final predicted image-level classification was produced by aggregating these features at different scales.
Pati et al. 171 proposed a hierarchical cell-to-tissue-graph (HACT) representation using breast cancer as an example. The almost 50 GB of images used in their study are freely available. This framework, which is also available on GitHub, consists of a low-level cell graph that captures cell morphology and topology, a high-level tissue graph that captures tissue area morphology and the spatial distribution of tissue components, and cell-to-tissue hierarchies that encode the relative spatial distribution of cells in relation to tissue features. Furthermore, a hierarchical graph neural network (HACT-Net) was proposed to map HACT representations.

Other approaches
Recurrent neural networks (RNNs), 175 long short-term memory (LSTM), [176][177][178]  The results demonstrate that integration of image features from a LSTM and a genomic pathway score is more closely correlated to a patient's recurrence of disease than standard clinical markers or image texture features.

The PathoFusion framework
We recently developed the PathoFusion platform, an opensource AI framework that allows for the marking, training, and automated recognition of histopathological diagnostic features in WSIs of human tissue sections 55 (Fig. 2). A new bifocal convolutional neural network (BCNN) 56 uses complementary feature information obtained from both shorter and longer image tiles to classify morphological and immunohistochemical features in scans of adjacent HE and immunostained sections. We subsequently extended the applicability of the PathoFusion framework to the cellular level 92 (Fig. 3). Individual cells can be automatically identified, profiled, and counted by the framework. Our studies demonstrate that PathoFusion is capable of autonomously recognizing histological features and detecting, identifying, and counting immunohistochemically labeled cells in WSIs of diagnostic tissue samples. The data can then be used for whole-slide cross-modality analyses, such as for determining the relationship between the presence of immunohistochemical signals and anaplastic histological features. Our experimental results further suggest that PathoFusion may be used to detect subcellular structures and that it could be used in combination with electron microscopy as well as other methods that generate diagnostic images from specialist marking. The BCNN at the core of the PathoFusion platform has the advantage of high trainability, meaning it keeps the number of required training cases comparatively small (double digits) while facilitating the provision of highquality input from busy specialists such as neuropathology consultants because comparatively little consultant time is required. This enables very high-level neuromorphological training of the AI since any feature of interest can be freely defined through simple (direct, i.e., implicit and fast) consultant marking allowing the aforementioned routine, i.e., talent-dependent feature recognition capability, to be transferred.

Challenges
Despite widespread enthusiasm and impressive results shared so far, 62,184 some constraints still limit the applicability of AI methods in digital neuropathology and pathology. Individual WSIs may contain tens of billions of pixels, and the corresponding large file sizes of up to 3 GB can make direct analysis difficult. Consequently, WSIs are commonly divided into patches, and each patch is analyzed separately, such as for ROI detection. The methodology used for integrating the results of all patches still has room for improvement. However, the most significant problem affecting ML analysis of WSIs is the scarcity of training data since their provision depends on expert labeling (ground truth). Pathologists are capable of labeling more samples in less time when using special user-interface tools 185,186 or active learning. 29,[186][187][188] Labels may also be obtained using a variety of instance learning methods 189 or weakly supervised learning. 29,96 In contrast, semisupervised learning methods make use of both labeled and unlabeled data. 190,191 Moreover, class imbalances of data may occur when the number of samples in a training dataset differs between classes. 192,193 One other current drawback of ANNs is their limited interpretability, i.e., failure to explain why a network makes a specific decision, although researchers have begun to address this problem. 194,195 Lastly, artifacts affecting the tissue structure and color variation of standard stains can be introduced at various stages of the WSI creation process. Because these artifacts may impair interpretation, specific algorithms for detecting artifacts such as blur 196 and tissue folds 197 have been proposed that can be used during WSI preprocessing. 141,198 In recent years, GANs 199 have been developed to deal with the problem. Applications such as color normalization 20,200 and data augmentation 201 have also emerged. Even virtual histological staining has become possible. 202

FUTURE
Rapid progress in AI research with relevance to microscopy can be anticipated in the coming years. However, rather than seeing a threat to morphological diagnostics, we agree that by providing more quantitative evidence and appropriate decision support, ML and DL can improve medical decisions and ultimately patient care. 203 ML has already demonstrated its usefulness not only in cancer diagnostics and research where much of the AI work in pathology and neuropathology 204,205 has been carried out but increasingly also in the field of neurodegenerative diseases. 23,[70][71][72]206,207 A new webbased approach 208 indicates special potential for rare diseases. In addition, rural points of patient care that lack specialist support are likely to benefit from the online availability of AI technology. This is of special relevance to neuropathological diagnostic services since most countries do not have the medical specialty and are unlikely to establish it soon. Morphological talent is not evenly distributed, as every microscopist knows, but, like the changes brought about by molecular genetic instrumentation in brain tumor diagnostics, AI can boost nonexperts to expert-level performance. 208

DATA AVAILABILITY STATEMENT
Data sharing not applicable to this article as no datasets were generated or analysed during the current study.