A review of open-source image analysis tools for mammalian cell culture: algorithms, features and implementations

Rationale

In comparison to the available reviews that only assess algorithms for cell analysis in terms of their capabilities and limits, this review stands out in that it advocates the transition of those algorithms to user-friendly technology where the wider community can utilize them regardless of their qualifications. The transition is to resolve traditional cell analysis problems and automate their procedure, allowing faster, more accurate, and consistent judgement. This article includes a comprehensive review of 49 publications relevant to modern cell analysis tools, including their algorithms, implementations, and unique features for non-experts in computer vision. This review explores the potential of existing tools from various application viewpoints. Although some of the imaging software discussed in this review has only been demonstrated to be effective on specific cell types, they can still be used on other cell types that have comparable traits or attributes, such as cellular structure or event. In summary, this review contributes to introducing the existence of cell analysis tools developed by other researchers, as well as defining their core algorithms and distinctive features that help to overcome the limits of traditional and manual cell analysis. Second, this review highlights the prospects of cell analysis tools for individual applications.

Intended audiences

This review is intended for two main groups of audiences with varying levels of computer vision expertise. The first group consists of non-technical individuals, such as biologists and biomedical researchers, who may not have extensive experience with computer vision techniques. The second group includes technical audiences, such as computational biologists and developers from cell imaging-related industries, who may have more familiarity with these techniques. The first objective is to motivate biologists who have relied heavily on conventional cell analysis and lack an understanding of computer vision algorithms to engage with contemporary cell analysis. Free and open-source are heavily emphasized in this review so that beginners can experiment and adapt without risk. For the non-technical audience to compare the functioning of each tool, the distinctions between them are highlighted depending on the cell morphologies and image modalities the articles have experimented with. Most tools focus solely on prediction tasks, but a few offer unique features that the audience can benefit from, such as measurement and manual error correction tools. This endeavour aims to ensure that every community is using cutting-edge approaches to their maximum potential in a variety of applications and progressing along with the Industrial Revolution (IR 4.0). The second objective is to assist and notify computational biologists or developers of the current cell analysis tools that they may openly take inspiration from and make modifications to, preventing them from having to reinvent the wheel.

The rest of the article is structured as follows. The methodology used to conduct this survey is discussed in ‘Survey Methodology’. The common mammalian cell morphologies and image modalities are summarized in ‘Morphologies and Modalities’. ‘Plugins and Pipeline Development’ describes software or tools that enable multiple plugins and allow the combination of different tools or modules in various ways. The following three sections provide a detailed review of detection, segmentation, and tracking tools, including their unique features and algorithms, with each section divided into several subsections based on their respective approaches. While the previous sections mainly focused on local installations, ‘Cloud-based Tools’ introduces cloud-based tools for cell imaging analysis that do not have specific hardware requirements. In ‘Potential Implementation of the Individual Tools’, a discussion about the potential applications of the prediction tools to the larger scope is presented. ‘Conclusion’ provides the conclusion of the article. The taxonomy of this literature review is illustrated in Fig. 1.

Information source and search process

Google Scholar is used as the primary search engine to initiate the process. It offers related articles and unique materials from different databases and has pertinent advanced search options for this review strategy.

The Google Scholar search term is associated with elements of cell analysis such as cell detection, segmentation, and tracking. Therefore, the first strategy is that the articles are found using the Google Scholar advanced search specifically “with all the words”: (user-friendly AND (software OR tool)) AND “with the exact phrase”: (‘cell detection’ OR ‘cell segmentation’ OR ‘cell tracking’) to quickly filter out irrelevant articles.

To avoid excluding any underlying articles on Google Scholar, all terms are combined under the “with all the words” section before the search process is complete. For example, the search term “with all the words”: (user-friendly cell segmentation tool) or (user-friendly cell detection software) due to the fact that articles with any of the keywords will be listed, regardless of the word structure. This second strategy is also applied in general search engines of other databases because, in contrast to Google Scholar, the advanced search features of the additional databases favour searching based on different word locations rather than the word structure in a phrase.

Inclusion and exclusion criteria

In spite of their publishing years, the tools or software for analyzing cell images that are free, open-source, and have a user-friendly interface are the focus of this review. This is due to the fact that some of the older generations proved to be still in use and progressing. Contrary to the inclusion criteria, cell imaging tools that are not accessible to the general public, charge a fee, require an extensive understanding of the algorithms, and require specialized hardware are excluded from this review.

Study selection

Five main structures of existing cell image analysis tools are summarized in Fig. 2 based on their interfaces. The main structures can also be categorized into two groups. The first group of tools consists of plugin- and pipeline-based tools which specialize in management tasks. The second group consists of tools that are specifically designed for detection, segmentation, and tracking, and focus solely on prediction tasks. The first group allows users to create workflows with multiple prediction tasks and arrange processes, while the second group is limited to the specific processes programmed by the developers and does not allow for the creation of workflows with multiple processes.

However, it has been observed that the structure and capabilities of analysis tools are influenced by technological evolution over time. As a result, at an early stage of cell imaging algorithm development, the number of tools within a specific structure is limited. Over time, as computational technology advanced, more tools were available since there was more capacity for sophisticated computer vision algorithms to operate. Therefore, the number of prediction tools appears to be increasing exponentially for each section until the end of the review article. Another aspect stimulated by technological advancement is the tool platform. According to the review, web-based tools have only recently become available. Prior to that, they were only available on local computer systems.

Generally, prediction tools can be divided into three main sections: detection, segmentation, and tracking. Each main section is further divided into subsections based on the approach used. This organization was chosen because the approach is typically the most prominent differentiator among the tools. In contrast, there are fewer tools that have distinct, unique features, making it less appropriate to divide the sections based on feature distinctions. This approach allows for a more comprehensive and coherent review of the tools available.

Traditional computer vision-based

OpenCFU (Geissmann, 2013), a Python-based tool, primarily segments cells using thresholding based on a score map, where the score map represents the properties of circular objects, such as perimeter, area, height, width, aspect ratio, and solidity, and some of which are defined by users. The image is first filtered with a local median filter to remove background noise, and then with a positive Laplacian of Gaussian to improve the appearance of foreground objects. The analysis concludes with a watershed and distance map approach to components classified as ‘multiple objects’, while ‘individual objects’ are accepted and ‘invalid objects’ are rejected, based on the thresholded score-map.

This paragraph discusses MATLAB-based tools for cell detection and counting. NICE (Clarke et al., 2010) uses a combination of thresholding and extended minima for cell counting by regions, with users able to change the threshold or resolution function in terms of the degree of Gaussian smoothing and expected minima size. AutoCellSeg (Aum et al., 2018) is based on fuzzy or trial-and-error a priori feature extraction using the fast marching method for automatic multi-thresholding aided by feedback-based watershed segmentation mechanisms. The watershed segmentation is used to separate neighbouring cells and it adapts the value of the H-maxima transform for the regional maxima of the input image.

Traditional computer vision-based

Traditional computer vision techniques for segmentation are commonly based on image intensity differences between background and foreground parameters. CellShape (Goñi-Moreno, Kim & Lorenzo, 2016) is a standalone software written in Python with several pre-existing modules for segmenting single bacterial cells and accurately measuring the intensity of fluorescent signals they produce with subpixel contour precision of image colour intensity via linear interpolation for level-grid intersection identification, as well as a bandpass filter comprised of a two-thresholded system for shape recognition.

BioImageXD (Kankaanpää et al., 2012; Kankaanpää et al., 2006) is a general-purpose processing program that leverages Visualization Toolkit (VTK) for multi-dimensional microscopy image processing and 3D rendering and the Insight Segmentation and Registration Toolkit (ITK) for segmentation and other image processing tasks. It offers multiple conventional segmentation methods (Kankaanpää et al., 2008), from region growing to the watershed algorithm.

OpenSegSPIM (Gole et al., 2016) also provides a variety of algorithms for nuclei segmentation on light sheet fluorescent microscopy images including watershed, iterative voting methods, level set approach based on gradient flow, and flexible contour model, where pre-segmentation can be manually edited to act as seeds for another segmentation, automated for time series batch processing, and further analyzed for cell membrane segmentation.

Traditional machine learning-based

Machine learning is a subset of the field of artificial intelligence (AI). Artificial intelligence has made strides in a variety of industries. Pharmaceutical, for example, is one of the industries that benefit from the cell culture process, and (Kulkov, 2021) has discussed the AI effects has on its business. Therefore, AI technology in cell culture imaging will add another key point for the pharmaceutical industry’s advancement. In the context of this review, machine learning refers to the ability of a system to predict cells based on what it has learned from human annotation. It can be fully automatic or semi-automatic. Semi-automatic is a feature that a user-friendly tool should have where it can automatically segment the cells and then allow the user to fix the errors and feed them for new training.

Among the learning algorithms is the classical random forest. The random forest consists of an ensemble of decision trees. Using random forest, Ilastik (Sommer et al., 2011) can segment, classify, track and count your cells. It also provides real-time feedback where users can interactively refine segmentation results by providing new labels. MIB (Belevich et al., 2016) employs random forest through graph-cut and k-means clustering algorithms for segmentation. Apart from segmentation, it also includes 2D measurement tools such as radius, angle, linear and freehand distance, calliper, and many others. SurVos (Luengo et al., 2017) GUI consists of three main workflows in order to segment images: manual segmentation, region-based segmentation, and model-based segmentation. In manual segmentation, hierarchical segmentation is allowed for users directly on the voxels of the volume. In the second segmentation, images are segmented using supervoxels and megavoxels, named as Super-Regions, which are completely unsupervised. In the final stage, using annotations and descriptors from previous stages, a model is trained using extremely random forest (ERF), by default, to segment the images, and the predictions are refined with Markov random field (MRF) formulation to be more spatially consistent. Transmission electron microscopy (TEM) images of Trypanosoma brucei procyclic cells and a mammalian cell line that resembles neurons were employed as test cases for SuRVoS segmentation.

SVM is also commonly used as the primary segmentation algorithm in image analysis software. FARSIGHT Bjornsson et al. (2008) emphasizes three key features: the ability to handle morphological diversity, efficient investigator validation, and modular design. It used a divide-and-conquer segmentation strategy as well as associative image analysis concepts for 3D multi-parameter images where the initial classification task is measured using fuzzy c-means unsupervised classifier and the results can be inspected and validated by users and then, trained by a second supervised classifier, support vector machine (SVM). Al-Kofahi et al. (2010) utilizes the modular design of FARSIGHT by incorporating it with a graph-cuts-based algorithm to analyze multi-parameter histopathology images for cell nuclei. fastER Hilsenbeck et al. (2017) also employs SVM with a Gaussian kernel for single-cell segmentation to estimate the likelihood of a cell being located within candidate regions identified by extremal regions. The algorithms are said to have a fast-training process, so it provides a real-time live preview of segmentation results when training labels are added, changed, or removed by users, providing instant feedback on where to add new labels and when to stop training to reduce segmentation errors and increase the quality, respectively. Another SVM-based segmentation is Orbit Image Analysis (Stritt, Stalder & Vezzali, 2020) for whole slide tissue image analysis in a variety of staining protocols such as H&DAB, FastRed, PAS, and three variations of H&E. It is able to run locally as standalone or connect to the open-source image server OMERO and has demonstrated on three applications: quantification of idiopathic lung fibrosis, nerve fibre density quantification, and glomeruli detection in the kidney.

In comparison to others, Trainable Weka Segmentation (TWS) (Arganda-Carreras et al., 2017) includes several algorithms in the Waikato Environment for Knowledge Analysis (WEKA) (Hall et al., 2009) for trainable-based segmentation that is based on a limited number of manual annotations, as well as clustering-based segmentation that is customizable based on user-defined image features. Aside from segmentation, there is an example where TWS is used to count colonies on agar plates and track cells in microscope images (Deter et al., 2019).

Deep learning-based

A deep neural network (DNN) is an artificial neural network (ANN) with multiple layers between inputs and outputs. DNNs can be basically classified into three types: multi-layer perceptron (MLP), convolutional neural network (CNN), and recurrent neural network (RNN). CNN is the most commonly used in computer vision. It can automatically extract simple features from the inputs using convolution layers to complete the tasks like image classification, object detection and localization, and image segmentation.

ConvPath (Wang et al., 2019) incorporates a CNN to segment and recognize cell types, including tumour cells, stromal cells, and lymphocytes, as well as to detect nuclei centroids of lung adenocarcinoma histology image patches. HistomicsML2 (Lee et al., 2021; Lee et al., 2020b), like QuPath, allows whole slide hematoxylin and eosin (H&E) stained image analysis for nuclei segmentation. However, for superpixel segmentation, instead of hand-crafted selection, the regions are subdivided using Simple Linear Iterative Clustering (SLIC) superpixel algorithm for selection, as shown in Fig. 5, with a pre-trained VGG-16 CNN used for feature extraction on the selected patches and MLP for superpixel classification and interactive learning of histology data. HistomicsML2 GUI is web-based, but it does not require an internet connection because it runs on a local host.

DeepTetrad (Lim et al., 2019) allows for high-throughput measurements, particularly for fluorescent images of pollen tetrad analysis, of crossover frequency and interference in individual plants at various magnifications with a combination of Mask Regional convolutional neural network (Mask R-CNN) and residual neural network (ResNet) as the feature pyramid network (FPN) backbone. Another tool that adapts Mask R-CNN is YeastMate (Bunk et al., 2022), which has been experimented with for multiclass semantic segmentation of S. cerevisiae cells from their mating of budding events into mother and daughter cells. Its model is trained based on brightfield and differential interference contrast (DIC) images. It is designed in a modular way that can be used directly as a Python library and also, two GUI frontends including a standalone GUI desktop and Fiji plugin.

Since 2015, the implementation of U-Net (Ronneberger, Fischer & Brox, 2015) and its variants (Siddique, Member & Paheding, 2021) have been widely used in segmentation tools for biomedical images, such as cell segmentation, due to its state-of-the-art of encoder–decoder architecture for accurate localization of objects with fewer training samples. The successor of MIB, DeepMIB (Belevich & Jokitalo, 2021), is updated to enable users to directly train their datasets using four additional deep learning architecture options for data training: 2D or 3D U-Net, Anisotropic 3D U-Net, and 2D SegNet DeepMIB has successfully segmented datasets from electron and multicolour light microscopy with isotropic and anisotropic voxels. MiSiC (Panigrahi et al., 2021) employs U-Net with a shape index map used as a preprocessing step before training and segmentation rather than using image intensity to prevent error due to modality differences. MiSiC can segment various bacterial cell morphologies in microcolonies or closely clustered together, which is not feasible using only intensity thresholds. During practice, it is ready for phase contrast, fluorescence, and bright-field images as input. A hybrid network of U-Net and Region Proposal Networks (RPN) is used in NuSet (Yang et al., 2020) to segment fluorescent nuclear boundaries. RPN is integrated to achieve robust separation of neighbouring nuclei, with its boundary boxes determining nuclear centroids that serve as seeds for a watershed algorithm. NuSet also includes a graphical user interface (GUI) that allows users to customize the hyperparameters and watershed parameters of NuSet to optimize the separation of crowded nuclei cells for specific datasets.

Residual U-Net, one of the U-Net variants, is used in Cellpose (Stringer et al., 2021) for segmentation. It is implemented in Cellpose to segment nuclei and cytoplasm by tracking the centre of a combined gradient vector field predicted. It does, however, necessitate dependent and accurate cell centre identification. To address this, its successor, Omnipose (Cutler et al., 2022), was introduced with a distance transform within the Cellpose framework to calculate the closest distance between a point within the bounded region and the boundary. A dropout layer is added before the densely connected layer in Cellpose, which is a minor modification for Omnipose. Additional pre-trained models were also added into Omnipose involving bacterial phase contrast and fluorescence. Cellpose 2.0 is another version that includes a collection of diverse pre-trained models as well as a human-in-the-loop pipeline for the rapid prototyping of new custom models (Pachitariu & Stringer, 2022). The segmentation of Cellpose has also been specifically improved for 3D image data (Eschweiler, Smith & Stegmaier, 2021).

Similarly to TWS, a traditional machine learning-based tool, some deep learning-based tools are also capable of providing multiple algorithms for prediction. microbeSEG (Scherr et al., 2022) is equipped with two deep learning models: boundary-based and distance transform-based. Both models are used to segment roundish cells and separate neighbouring cells in 2D phase contrast or fluorescence images. Furthermore, it also contains OMERO data management, allowing data to be easily organized and accessed in the browser, as well as ObiWan-Microbi, an OMERO-based data annotator in the cloud. Instead of providing multiple algorithms for one specific task, InstantDL (Jens et al., 2021) offers multiple algorithms for various tasks where U-Net for semantic segmentation and pixel-wise regression, Mask R-CNN for instance segmentation, and ResNet50 for classification. Transfer learning is used to influence InstantDL performance, with ten pre-trained weight sets provided: four for 2D nuclei segmentation, two for 2D lung segmentation, two for 3D in-silico staining, and one for classification of white blood cells and metastatic cancer. Users must provide their own dataset to train the model using the provided algorithms in relation to the task.

Unlike the others, DeepImageJ (Marañón & Unser, 2021) plugin enables the generic use in ImageJ/Fiji of various pre-trained deep learning (DL) for a user-friendly analysis interface without any DL expertise. For example, a U-Net segmentation plugin was presented by (Falk et al., 2019) for ImageJ/Fiji to better analyze cell images with U-Net advantages and to adapt to new tasks with few annotated samples. As a result, as long as researchers make their algorithms public, non-experts can easily import them for cell analysis and prediction. DeepImageJ also offers several trained models for segmentations like photoreceptor cells, pancreatic, HeLa, glioblastoma, and many more.

Traditional computer vision-based

ChipSeg (De Cesare et al., 2021) is based on the Otsu thresholding method for bacterial and mammalian cell segmentation in microfluidic devices. ChipSeg can also do the tracking by assigning a centroid to each segmented cell and monitoring centroid displacement in two consecutive time-lapse images (De Cesare et al., 2021).

While using thresholding as a basis of segmentation, TrackAssist (Chakravorty et al., 2014) tracks lymphocytes using a combination of the Extended-Hungarian approach and the concept of object existence from the Linear Joint Integrated Probabilistic Data Association (LJIPDA), with its GUI providing a video player and image editor functionality to fix tracking errors frame by frame.

RACE (Stegmaier et al., 2016) focuses on extracting 3D cell shape information from confocal and light-sheet microscopy images of Drosophila (fruit fly), zebrafish (fish), and mouse embryos. It employs effective segmentation through three main steps including slide-based extraction, seed detection, and seed-based fusion for 3D cell shapes, with only three parameters adjusted: the binary thresholding value for the seed detection stage, the H-maxima value from Euclidean distance map for neighbouring cells separation and the starting level of the slice-based watershed segmentation algorithm. RACE cell tracking necessitates the use of a tracking-based Gaussian mixture models (TGMM) framework, replacing RACE seed detection.

An extended fast marching method, known as a matching algorithm, is used in FastTrack (Gallois & Candelier, 2021) to track time-lapse deformable shapes such as cells, where each is thresholded in every frame, and their kinematic parameters of position, direction, area, and perimeter are extracted as inputs of the tracking algorithm. Like TrackAssist, its GUI also allows for manual error correction.

Traditional machine learning-based

CytoCensus (Hailstone et al., 2020) uses a random forest algorithm for cell counting and cell division tracking with a point-and-click mechanism for locating the approximate centres of cells rather than defining cell boundaries. Users can also define a region of interest (ROI) to exclude background interference.

CellCognition (Held et al., 2010) utilizes SVM along with hidden Markov modelling to separate HeLa cells from their offspring in confocal images. (Sommer et al., 2012) demonstrated the use of ilastik and CellCognition to detect mitotic cells in histopathology images.

Deep learning-based

CellTracker (Hu et al., 2021) provides four segmentation algorithms for time-lapse images at single-cell resolution including traditional methods (e.g., Otsu threshold, GrabCut, Watershed) for unsupervised segmentation and deep learning approaches (eg. deep U-Net) for supervised segmentation where annotation tool and model training of customized dataset is provided. CellTracker also provides a manual correction tool to edit segmentation and tracking results.

DeepSea (Zargari et al., 2021) uses Residual U-Net for segmentation, quantification, and tracking of biological features of mouse embryonic stem cells, such as cell mitosis and morphology, in phase-contrast images, but it is scaled down to minimize the parameters for computation while maintaining the performance.

Cheetah (Pedone et al., 2021), a real-time computational toolkit for cybergenetic control, is another U-Net backbone tool that includes four modules: data augmentation for training data generation, semantic segmentation for the foreground (cells) and background (unoccupied space) classification, instance segmentation for cell tracking, and control algorithm for user-defined feedback. Similar to ChipSeg in emphasis, Cheetah is reported to perform better in terms of precisely distinguishing individual cells and preventing misidentified cell debris on empty regions of the chamber.

DetecDiv (Aspert, Hentsch & Charvin, 2022) is proposed to segment and track yeast cells and their division events within cell traps of a microfluidic device in brightfield and fluorescence images. DetecDiv employs an image sequence classification using a combination of CNN and long short-term memory (LTSM) for the cellular budding status and death within a cell trap image and a semantic segmentation of cells using DeeplabV3+. The primary user interface for DetecDiv is MATLAB.

In addition to their approaches, the reviewed tracking tools can also be classified based on their purpose, as either tool for tracking single-cell movement or tools for tracking cell division events. The list of tracking tools, organized by purpose, is presented in Table 1.

Cloud-based segmentation tools

Cellbow (Ren et al., 2020) uses a fully convolutional neural network (FCNN) with two convolutional and deconvolutional layers for fluorescent and bright-field image segmentation. Three examples of cell segmentation are demonstrated which are fission yeast, synthetic, and human cancer cells. Moreover, it also comes with an ImageJ plugin. Mask R-CNN and U-Net are used in nucleAIzer (Hollandi et al., 2020) for nucleus segmentation and boundary correction, respectively, providing pre-trained models of The 2018 Data Science Bowl (DSB), fluorescent and histology cell nuclei data for new images via an online interface or CellProfiler. Therefore, when images are uploaded, the nuclei can be well-segmented if the most similar image model is chosen accordingly.

DeepEM3D (Zeng, Wu & Ji, 2017) architecture, which is designed for anisotropic data such as medical images, is used in the Caffe framework for training by CDeep3M (Haberl et al., 2018) on Amazon Web Services (AWS). The implementation of CDeep3M focuses on microscopy segmentation, such as membranes, vesicles, mitochondria, and nuclei, from multiple image modalities, such as X-ray microscopy (XRM), light microscopy, and electron microscopy. Another online prediction tool that is deployed on AWS is DeepLIIF (Ghahremani et al., 2022). DeepLIIF generates multiple pre-trained models with different modalities where ResNet-9 is used to produce modalities and U-Net to segment the cells. DeepLIIF focuses on segmenting cells in immunohistochemical (IHC) images. One of its key features is that it allows interaction with the segmented results which can be manipulated with the segmentation threshold and performing size gating on the image. If users intend to perform new training, DeepLIIF local installation is made available.

Some tools, such as ZeroCostDL4Mic (von Chamier et al., 2020), which provided two segmentation algorithms which are U-Net and StarDist (Schmidt et al., 2018), are intended to be used on the Google Colab interface as the scripts are provided by the developers. To use ZeroCostDL4Mic, users have to train their own datasets and deploy the models themselves using those algorithms. However, users require no coding expertise. As a result, users must have a Google Drive account in order to store their datasets and run any provided DL-based tasks on the platform. STORM also utilizes Google Colab as a platform for hosting U-Net and Mask R-CNN, which are employed for the segmentation of nuclei in single-molecule localization-based super-resolution fluorescence images.

Cloud-based tracking tools

In contrast to the segmentation tools CDeep3M and DeepLIIF, which only run on AWS, DeepCell (Bannon et al., 2018) can also be run on Google Cloud. In addition, it includes four prediction types: Mesmer (Greenwald et al., 2021), Segmentation, Caliban (Moen et al., 2019), and Polaris. Mesmer (Greenwald et al., 2022) and Segmentation are models for whole-cell segmentation of tissue and cell culture data, respectively. Caliban is responsible for cell tracking, while Polaris detects fluorescent spots.

For cloud-based tracking tools, there are two tools that are hosted on Google Colab which are 3DeeCell-Tracker (Wen et al., 2021) and DeLTA (O’Connor et al., 2022). 3DeeCell-Tracker uses 3D U-Net and feedforward network (FFN) to segment and track cells in 3D time-lapse images of deforming organs or freely moving animals, such as cardiac cells in a zebrafish larva or neurons in a ‘straightened’ freely moving worm, respectively. DeLTA revolves around movie processing for segmenting and tracking rod-shaped bacteria growing, such E. coli cells, in 2D setups using U-Net.

Table 2 presents the tools that only require a web browser to host their GUI regardless of computer specifications and some either provide trained models for specific datasets, pre-trained models for further dataset training, or both.