Artificial Intelligence for Cell Segmentation, Event Detection, and Tracking for Label-Free Microscopy Imaging

Abstract

Background: Time-lapse microscopy imaging is a key approach for an increasing number of biological and biomedical studies to observe the dynamic behavior of cells over time which helps quantify important data, such as the number of cells and their sizes, shapes, and dynamic interactions across time. Label-free imaging is an essential strategy for such studies as it ensures that native cell behavior remains uninfluenced by the recording process. Computer vision and machine/deep learning approaches have made significant progress in this area. Methods: In this review, we present an overview of methods, software, data, and evaluation metrics for the automatic analysis of label-free microscopy imaging. We aim to provide the interested reader with a unique source of information, with links for further detailed information. Results: We review the most recent methods for cell segmentation, event detection, and tracking. Moreover, we provide lists of publicly available software and datasets. Finally, we summarize the metrics most frequently adopted for evaluating the methods under exam. Conclusions: We provide hints on open challenges and future research directions.

1. Introduction

Microscopy is a fundamental research pillar enabling scientists to discover the structures and dynamics of cells and subcellular components. Most of these components are phase objects, which means they are transparent and colorless and cannot be visualized under a light microscope. To overcome this limit, a solution consists in staining the components with dyes, also known as fluorophores or fluorochromes. Such molecules absorb short-wavelength light, generally UV, and emit fluorescence light at a longer wavelength. This mechanism represents the basic premise of fluorescence microscopy techniques, and the fluorescence images show the specimen bright on a dark background. The staining techniques use different dyes to point out a specific cellular component. Then, the same specimen can appear with a red, blue, or green color and appearance, as depicted in Figure 1a,b.

As the molecular genetic methodologies and tools advance, the fluorescence techniques become more specific and can be applied to a larger set of different model organisms [1,2]. However, to ensure high-quality images, they require laborious and expensive sample preparation. Furthermore, the dyes reduce their luminance over time due to a light-induced degradation process called photobleaching [3]. Finally, fluorescent techniques are invasive and interfere with the biological processes causing phototoxicity.

On the contrary, Label-free Imaging (LI) techniques can visualize many cellular structures simultaneously with minimal sample preparation, phototoxicity, and no photobleaching, making them particularly suitable for live-cell imaging. Thus, LI provides measurements complementary to fluorescence imaging for several biological studies. Based on optical principles, these techniques measure the light phase change (i.e., the refractive index) passing through the specimen and converting it into intensity modulations producing qualitative phase contrast images, as exemplified in Figure 1c (the phase change information contained in LI images is non-linearly coupled with its luminance intensity and cannot be retrieved quantitatively. An image produced by LI techniques is a map of path-length shifts associated with the specimen, containing information about both the thickness and refractive index of its structure; for further details and explanations, see [4]). Among the traditional techniques to phase change imaging, there are Phase Contrast (PhC) [5] and Differential Image Contrast (DIC) [6] based on the phase gradient method and differential interference contrast, respectively, to measure the refractive index. Another LI technique very similar to DIC is the Hoffman Modulation Contrast (HMC) [7]. Due to intrinsic limitations of the numerical conversion methods, phase contrast images contain artifacts [8] (i.e., bright halo surrounding cell contours) and the “shade-off effect”, which produces low contrast inside the cells with an intensity very similar to the background. Although several methods have been developed to overcome these artifacts, automatic image processing of label-free images is still challenging, especially the segmentation task for separating cells from the background.

Figure 1. Example of differences in appearance of fluorescence and phase contrast microscopy images (culture of human lymphocyte cells [9]): (a) fluorescence image of nuclear envelope; (b) fluorescence image of interior nuclei (DNA); (c) phase contrast image of whole cells.

Figure 1. Example of differences in appearance of fluorescence and phase contrast microscopy images (culture of human lymphocyte cells [9]): (a) fluorescence image of nuclear envelope; (b) fluorescence image of interior nuclei (DNA); (c) phase contrast image of whole cells.

Unlike the previous techniques, quantitative LI provides higher contrast and reduces artifacts. Among them, Quantitative Phase Imaging (QPI) techniques refer to the microscopes showing phase quantitative information. Details concerning QPI techniques can be found in [10].

A further alternative to noninvasive techniques is Bright-Field (BF) microscopy. It represents the most straightforward configuration for the light microscope, which is not only cheaper but also does not require sample preparation [11]. BF images provide information about the cellular organization, and they are preferred to visualize specimens with low contrast from the background (unlike in fluorescence) or with low resolution and magnification visualization of thin cellular components (unlike in phase contrast). Several studies describe the use of the BF channel in cell detection and automated image analysis of cell populations [12,13].

With the aim of observing the dynamic behavior of living cells over time, LI microscopes are also equipped with a real-time imaging tool named time-lapse. Broadly speaking, time-lapse is a speed-up technique to observe events changing over time. This is usually realized by taking images at regular time intervals and merging them into a video. One of the most significant applications of time-lapse microscopy is cell population monitoring to study single-cell behavior in response to physiological or external stimuli and understand the underlying mechanisms. For example, in drug discovery and cancer research [14], time-lapse microscopy is used to look at cell response to anti-mitotic drugs in terms of cell division and cell death. To achieve this goal, quantitative information on cell behavior needs to be obtained and analyzed [15]. Cell proliferation, lineage, and fate are of primary importance among various cellular events. Image analysis of these biological processes is usually performed manually with suitable protocols [16]. As manual analysis of a large volume of light microscopy images is slow, tedious, time-consuming, and subject to observer subjectivity, biological studies see an increased demand for reliable automatic imaging tools. As in many scientific disciplines, Artificial Intelligence (AI) has been changing how imaging data are processed and analyzed and how experiments are carried out. AI refers to artificial systems aiming to adapt previous knowledge to new situations and recognize meaning in data patterns. Machine Learning (ML) [17] is a subset of AI methods that extracts valuable features from large data sets to make predictions or decisions on unseen data. An ML algorithm is not designed to solve a specific problem but rather to train a computer to solve problems. The training is data-driven. Deep Learning (DL) is a set of ML algorithms using a multi-layer “neural networks” to progressively extract higher-level features from data (the number of layers is the depth of the model, hence the terminology “deep learning”; for a quick overview on DL key concepts for microscopy image data, the reader can refer to [18]).

This review mainly focuses on AI methods for the most used traditional LI microscopy techniques, i.e., PhC, DIC, and BF, to investigate fundamental biological events. A comprehensive description of the state-of-the-art methods using QPI data and AI approaches, which are out of our focus, can be found in [19] and references therein. Few recent surveys are available in the literature covering methods for these types of microscopy images and videos. The study presented by Vicar et al. [20] performs a comprehensive comparison of cell image segmentation methods for the most common label-free microscopy techniques, including PhC, DIC, HMC, and QPI. The review covers traditional methods, providing only hints on DL-based segmentation methods. The authors identify an effective image segmentation pipeline composed of four main steps: image reconstruction, foreground-background segmentation, seed-point extraction, and cell segmentation. They discuss and assess the most effective combination of the above steps for the specific microscopy techniques based on software and tools available in the state-of-the-art literature. Furthermore, they compare the accuracy and efficiency of tools containing all the above four steps (named “all-in-one” tools) as well as software implementing only one of them. Software and data were made publicly available (see Section 3 and Section 4). In [21], Emami et al. present a review of methods and tools for cell tracking. Following the traditional object tracking literature, the methods are subdivided into three groups, according to whether tracking is achieved by detection, model evaluation, or filtering, with limited space given to DL approaches. Well-known commercial and open-source cell tracking tools are summarized, and typical challenges are highlighted. Ulman et al. [22] present a comparison of 21 cell-tracking algorithms participating in three editions of the Cell Tracking Challenge (CTC), an initiative promoting the development and objective evaluation of cell segmentation and tracking algorithms. The compared methods are summarized based on common principles, features, and methodologies, as well as pre- and post-processing strategies. They are evaluated for both the segmentation and tracking tasks (see Section 5), and their overall average performance is used to compile the final ranking. Started in 2013, this challenge is still ongoing, and since 2019, it has been articulated into two different challenges, the Cell Tracking Benchmark (CTB) and the Cell Segmentation Benchmark (CSB), sharing the same dataset (see Section 4).

Although the described surveys provide extremely useful insights on specific tasks or specific datasets, the landscape of the scientific research on the subject still appears fragmented. The aim of our review is to provide a broad and up-to-date view of AI methods for the analysis of label-free images and videos acquired by traditional LI microscopy techniques and all the ingredients needed to afford it. Thus, it covers the most recent methods, especially for cellular segmentation, event detection, and tracking over time-lapse videos, available datasets, software, and evaluation metrics.

The review is organized as follows. In Section 2, we introduce the considered microscopy analysis tasks and provide brief descriptions of the reviewed literature methods for each of them. Section 3 and Section 4 provide brief descriptions and links to the publicly available software and data. Section 5 introduces the most frequently used metrics to evaluate the AI algorithms for the considered tasks. Section 6 summarizes the open problems, providing hints on possible future research directions.

2. Literature

2.1. Cell Segmentation

Image segmentation is the main task for producing numerical data from live-cell imaging experiments, thus providing direct insight into the living system from the quantitative cell information [23,24]. Cell segmentation is the process of splitting a microscopy image into “segments”, i.e., Regions Of Interest (ROIs), and produces an image where cells are separated from the image background by cell contours or cell labels. Accurate cell segmentations are crucial for many challenges involved in cellular analysis, including but not limited to cell tracking [25], cellular features quantification, proliferation, morphology, migration, interactions, and counting [26,27,28,29].

Several steps are considered in the literature for achieving cell segmentation. Some authors [8,20,30] consider crucial to initially perform an image reconstruction step, which produces images with higher contrast between foreground and background, increasing the success of the subsequent image processing tasks. The image formation model in phase contrast microscopy can be studied to reduce artefacts and solve the inverse problem through a regularization approach [8]. Some ML-based methods are reviewed by Vicar et al. [20] for PhC and DIC images, while De Haan et al. [30] present an overview of how DL-based frameworks solve these inverse problems in optical microscopy.

Cell detection (or identification) is also frequently adopted previous to segmentation [20,22], with the aim of locating the cells in the image (e.g., via bounding boxes, as exemplified in Figure 2a).

Figure 2. Results of different image processing tasks: (a) cell detection; (b) cell semantic segmentation; (c) cell instance segmentation.

Figure 2. Results of different image processing tasks: (a) cell detection; (b) cell semantic segmentation; (c) cell instance segmentation.

Some specific types of segmentation are frequently considered for cellular images [20,31,32,33]. Semantic segmentation identifies the object (i.e., the cell) category of each pixel for every known object within an image, as exemplified in Figure 2b. Instance segmentation, instead, identifies the object instance (i.e., the cell with specific features) of each pixel for every known object within an image [34]; an example is given in Figure 2c. It should be observed that the problem of cell instance segmentation is sometimes intended as joint cell detection and segmentation (e.g., see [32]), while other times (e.g., see [20]) instant segmentation is used as a synonym for single cell segmentation.

Some of the most recent segmentation algorithms are described in the following. In [23], Van Valen et al. in the software named DeepCell adopt deep convolutional neural networks for cell segmentation for various types of microscopy images (PhC, Fluo, and PhC coupled with images of a fluorescent nuclear marker), also providing hints on design rules in training CNNs for this task (image normalization, data augmentation, hyper-parameter tuning, and segmentation refinement). They also extend the deep convolutional neural networks to perform semantic segmentation (i.e., not only image segmentation but also cell type prediction). The software and the adopted data are publicly available (see Section 3 and Section 4).

Hilsenbeck et al. [25] present fastER, a fast and trainable tool for cell segmentation that extracts texture and shape features from candidate regions, estimates their likelihood to be a cell with a support vector machine (SVM) algorithm, and calculates an optimal set of non-overlapping candidate regions using a divide and conquer approach. Candidate regions are chosen as the so-called extremal regions (regions with maximal size containing only pixels whose intensities are no greater than a specific threshold), similarly to CellDetect [35]. For each candidate region, a feature vector is extracted to train the SVM model, including typical shape and intensity information (size, major/minor axis lengths, eccentricity, average intensity inside and in its neighborhood, average and standard deviation gradient, and average heterogeneity) [36]. Pre-processing of the images consists of denoising with bilateral filtering, while post-processing of the resulting masks includes hole-filling and size filtering. The software made publicly available (see Section 3) is shown to be robust against common cell segmentation challenges but still suffers high cell densities and blurring. Compared with other state-of-the-art methods (e.g., U-Net [37], ilastik [38], CellProfiler [39], and CellDetect), it is shown to be more efficient on various types of data made publicly available (see Section 4) still achieving similarly accurate results.

Yi et al. [32] propose the software ANCIS an Attentive Neural Cell Instance Segmentation method to predict each cell’s bounding box and its segmentation mask simultaneously. The method builds on a joint network that combines the single shot multi-box detector (SSD) one-stage object detector [40] and U-net [37] for cell segmentation. Attention mechanisms are adopted in detection and segmentation modules to focus the model on useful features while suppressing irrelevant information. The software, tested on DIC images of neural cells, is publicly available (see Section 3).

In [24], Lux and Matula use a watershed [41] marker-based approach with two convolutional neural networks (CNN) of hour-glass architecture shape to segment clustered cells in images consisting of five datasets, three of which originate from the Cell Tracking Challenge [22] (DIC-C2DH-HeLa, Fluo-N2DH-SIM+, and PhC-C2DL-PSC). They used normalization by histogram equalization and median scaling as a pre-processing step. Afterwards, they augment the data by randomized rigid geometric transformations and scaling. Then, Lux et al. use one CNN to predict cell marker pixels and the other CNN for image foreground predictions. They utilize these outputs to compute the marker function to obtain the segmentation seeds and the segmentation function to define cell regions that are further used in the marker-controlled watershed segmentation.

Scherr et al. [42] present a method for segmenting touching cells in BF images from the Cell Tracking Challenge [22] by using a novel representation of cell borders, inspired by distance maps. The proposed method uses an adapted U-Net with two decoder paths, one for prediction of cell distance and another for prediction of neighbor distance. These distances are then used for the watershed-based post-processing to obtain segmentations. Results are evaluated using the SEG, DET, and CSB metrics (see Section 5) and show accurate performances, with average SEG and DET scores of 0.726 and 0.975, respectively.

Nishimura et al. [43] propose a weakly supervised cell instance segmentation method that recognizes each cell region by only using weak labels, i.e., point-level (cell centroid positions) rather than pixel-level annotations, as training data. This approach strongly reduces the annotation cost compared with the standard annotation method required for supervised segmentation. They train a cell detection CNN (U-net) and then use it to estimate rough cell positions. The rough cell shapes are extracted from the detection network by backpropagating the activation from output to input, obtaining a relevance map that shows how each pixel in the input image is relevant to the output. The final cell shapes are estimated by graph-cut [44] using the estimated relevance map as a seed. Results on different datasets show that the method works well with different types of microscopy and different contrasts.

Stringer and Pachitariu introduce Cellpose [45] a software library for the instance segmentation of cell images. It implements a CNN using the U-net architecture style. Cellpose provides the probability of a pixel being inside a cell and the flows of pixels in $xy$ coordinates towards the cell center. The flows are then used to construct the cell ROIs. Several results confirm its reliability on a wide range of label-free images without model retraining or parameter adjustment. The authors also propose a 3D extension of the library that does not use 3D-labeled data but works on the 2D model. The recent furthest extension of Cellpose [46] can adapt CNN segmentation models to new microscopy images with very little training data. Code and data are publicly available (see Section 3 and Section 4).

2.2. Event Detection and Classification

Even though segmentation remains the core of the subsequent imaging tasks, automated analysis of microscopy image sequences often bypasses the segmentation task. Usually, the detection of cell events under investigation is performed directly from heuristically generated ROIs. Detecting changes in cellular behavior plays a central role in different studies, where the focus is on identifying the changes in cellular growth, mitosis, and death. Such changes may be related to cell shape, division, and movement. They cannot be detected in a single image but require the analysis of video or time-lapse sequences. The difficulty primarily relies on the wide spatial-temporal variability of such phenomena, which requires suitable methods to handle time-varying phenomena. Nevertheless, the infinite spectrum of possible events faces an inherent shortage of labeled data.

Automatic and robust approaches to detecting the time and location of cell events from image sequences often make use of the classification task. In microscopy, classification refers to identifying and distinguishing different cell types or states. Classification between other cells, types of tumors (benign or malignant), types of cell states (mitosis detection, alive-dead classification), and types of CIC (Cell-In-Cell) structures [47] are some typical applications.

Some recent AI approaches for event cell detection are here presented. Su et al. [48] and Mao and Yin [49] propose a convolutional long short-term memory (CNN-LSTM) network and a Two stream Bidirectional CNN-LSTM network, respectively, on sequences of single-cell image patches and utilize both spatial and temporal information to detect mitosis events. They report an average precision of 0.96 and 0.98, respectively. However, these models need a large amount of manually annotated data to train on, and both papers also report a sharp decrease in accuracy when testing the model on other cell datasets.

A CNN-LSTM model that learns spatial and temporal locations of the cells from a detection map in a semi-supervised manner is proposed by Phan et al. [50] for the detection of mitosis in PhC videos. The method needs only 1050 annotated frames to achieve an F1 score of 0.544–0.822, depending on the video. However, it also shows a decrease in performance with the increase in the input sequence length, which is not ideal for practical situations where time-lapse experiment’s video sequences may contain thousands of frames. The method also will only be able to detect a single event at a time, such as mitosis, whereas these events can randomly occur in multiple places in a single frame.

Nishimura and Bise [51] propose a method for multiple mitosis event detection and localization by estimating a spatial-temporal likelihood map using the 3D CNN architecture V-Net [52]. In the likelihood map, a mitosis position is represented as an intensity peak with a Gaussian distribution, in which multiple mitoses are represented as multiple peaks. The method has an average precision of 0.862 on a private dataset. While the method does take into account the spatial and temporal information, it is only limited to detecting mitosis events and not any other events that may be associated with mitosis. In order to identify other events as well, multiple models based on this method would be needed. Furthermore, the use of this method for other datasets or cell lines requires the generation of laborious manual annotations in the form of Gaussian distributed likelihood maps.

Su et al. [53] present a deep reinforcement learning-based progressive sequence saliency discovery network (PSSD) for mitosis detection in time-lapse PhC images. The discovery of these salient frames is formulated as a Markov Decision Process that progressively adjusts the selection positions of salient frames in the sequence. Then, the pipeline leverages deep reinforcement learning to learn the policy in the salient frame discovery process. The method consists of two parts: (1) the saliency discovery module, which selects the salient frames from the input cell image sequence by progressively adjusting the selection positions of salient frames; (2) the mitosis identification module, which takes a sequence of salient frames and performs temporal information fusion for mitotic sequence classification. The method is evaluated on the C2C12-16 mitosis detection dataset [54] (see Section 4), and is found to outperform the previous state-of-the-art methods, including CNN-LSTM and 3D-CNN among the others.

Theagarajan and Bhanu [55] present DeephESC 2.0, an ML method to detect and classify human embryonic stem cells (hESC) in PhC images. Firstly, they use a mixture of Gaussians to detect the cells [56], where two Gaussian distributions model the intensity distributions of the foreground (cells) and the background (substrate). Then, Generative Multi Adversarial Networks (GMANs) [57] augment data with new synthetic images and improve the performance of the classification step. To classify the images into six different classes, they implement a hierarchical classifier consisting of a CNN and two Triplet CNNs. The software and dataset are publicly available (see Section 3 and Section 4).

La Greca et al. [58] use in the celldeath software some classical DL approaches such as ResNet [59], where they classify cells as dead or alive by using complete frames as input images. On images containing both alive and dead cells, the model can predict the dead ones, which are localized by heat map-like visualizations merging the information provided by the last convolutional layer and the model predictions. These predictions are compared with human performance and are found to largely outperform human ability. The software is publicly available (see Section 3 and Section 4).

2.3. Cell Tracking

Object tracking consists in locating and monitoring one or more objects of interest and their behavior over time [21]. The image sequence containing cells can be acquired at specific time intervals using the time-lapse technique. When discussing cell tracking, it is generally assumed that segmentation or detection and classification have been performed.

Some recent tracking methods are here reviewed. Magnusson et al. [60] propose a global track linking algorithm, which links into tracks cell outlines generated by a segmentation algorithm. It is a batch algorithm that uses the entire image sequence to decide the links. Starting with the hypothesis that there are no cells in the image sequence, it adds one cell track at a time, in a greedy way, choosing the one maximizing a suitable scoring function, using the Viterbi algorithm. The algorithm can handle cell mitosis, apoptosis, and migration in and out of the imaged area and can also deal with false positives, missed detections, and clusters of jointly segmented cells. It has been tested on BF sequences, but in principle, it can be applied to any type of sequences, given a suitable segmentation algorithm to outline the cells. The algorithm has been implemented in several cell trackers, see for example the Baxter Algorithms package (see Section 3).

Grah et al. [61] propose MitosisAnalyser, a framework for detecting, classifying, and tracking mitotic cells in live-cell phase contrast imaging based on mathematical imaging methods. As pre-processing, denoising by Gaussian filter smoothing is applied, followed by rescaling. In the workflow, each mitosis is detected by using the circular Hough transform. The obtained circular contours are used for initializing the tracking algorithm, which is based on variational methods. Backward tracking is used to establish the beginning of mitosis by detecting a change in the cell morphology. This step is followed by forward tracking until the end of mitosis. The output provides the duration of mitosis and information on cell fates (e.g., number of daughter cells, cell death). The Matlab code is publicly available (see Section 3).

In [62], Rea et al. propose a Graphics Processing Unit (GPU)-based algorithm for tracking yeast cells in PhC microscopy images in real-time. The tracking by detection approach determines a minimum cost configuration for each couple of frames, given by the solution of a linear programming (LP) problem. The GPU-parallel software based on the simplex method, a common tool for solving LP problems, is obtained by exploiting parallelization strategies to maximize the overall throughput and minimize memory transfers between host and device, thus exploiting data locality. The software is publicly available (see Section 3).

Tsai et al. [63] introduce Usiigaci, a semi-automated pipeline to segment, track, and visualize cells in PhC sequences. Segmentation is based on a mask regional convolutional neural network (Mask R-CNN) [64], while the tracking module relies on the Trackpy library [65]. A graphical user interface allows the user to verify the results. The software and annotated data are publicly available (see Section 3 and Section 4).

Scherr et al. [42], in the same paper as for segmentation, also propose a graph-based cell tracking algorithm for touching cells in BF microscopy images (BF-C2DL-HSC and BF-C2DL-MuSC datasets) from the Cell Tracking Challenge [22]. The adapted tracking algorithm includes a movement estimation in the cost function to re-link tracks with missing segmentation masks over a short sequence of frames. Their algorithm can track all segmented cells in an image sequence and only a subset, e.g., a selection of manually marked cells. Results for cell tracking are evaluated using the TRA and CTB metrics (see Section 5) and are shown to perform very well, with TRA scores of 0.929 and 0.967 for the BF-C2DL-HSC and BF-C2DL-MuSC images, respectively.

3. Software

As also already discovered in Section 2, it is every day more common for newly proposed methods to make their implementations publicly available, in the light of the recent trend toward open science. In Table 1, we provide links to existing publicly available software, subdivided by task. Moreover, we provide links to software platforms, providing more diverse functionalities for analyzing microscopy images and videos. Besides the software already described in Section 2, here we briefly summarize the remaining ones.

TWS (Trainable Weka Segmentation) is a Fiji plugin that combines ML algorithms with a set of selected image features to produce pixel-based segmentations. Weka (Waikato Environment for Knowledge Analysis) [66] can itself be called from the plugin.

Baxter Algorithms is a software package for tracking and analyzing cells in microscope images, providing an implementation of the global track-linking algorithm in [60]. The software can handle images produced using both 2D transmission microscopy and 2D or 3D fluorescence microscopy.

CellProfiler is a commonly used program designed for biologists with minimal programming knowledge to measure biological phenotypes quantitatively [39]. Algorithms for image analysis are available as individual modules that can be placed in sequential order to create a pipeline. Several commonly used pipelines are available for download and can be used to detect and measure various properties of biological objects.

ilastik [38] is an interactive machine learning tool based on a random forest classifier [67] for image analysis and is widely used by biologists since it no require specific ML knowledge. It provides pipelines for segmentation, classification, tracking, and lineage, performing on multidimensional data (including 3D space, time, and channels). A friendly user interface enables users to interactively implement their image analysis through a supervised machine learning workflow. ilastik classifies pixels and objects by learning from annotations to predict the class of each unannotated pixel and object. It provides an automatic selection of image features based on a first optimization step. Users can introduce sparse annotations or use labeled data or even provide training examples, then correct the classifier precisely at the position where it is wrong. Once a classifier has been trained, new data can be processed in batch mode.

Table 1. Software: name (Name); reference ([Ref]); year of publication (Year); url (Link); programming language or environment (Language). All links were accessed on 28 August 2022.

Name	[Ref]	Year	Link	Language
Cell segmentation
DeepCell	[23]	2016	https://simtk.org/projects/deepcell	Python, C, Ruby
fastER	[25]	2017	https://bsse.ethz.ch/csd/software/faster.html	C++
TWS	[68]	2017	https://imagej.net/plugins/tws/	Java
ANCIS	[32]	2019	https://github.com/yijingru/ANCIS-Pytorch	Python
Vicar et al.	[20]	2019	https://github.com/tomasvicar/Cell-segmentation-methods-comparison	Matlab
Cellpose	[45,46]	2022	https://github.com/MouseLand/cellpose	Python
Cell classification
DeephESC 2.0	[55]	2019	https://www.vislab.ucr.edu/SOFTWARE/software.php	Python
celldeath	[58]	2021	https://github.com/miriukaLab/celldeath	Python
Cell tracking
Baxter Algorithms	[60]	2015	https://github.com/klasma/BaxterAlgorithms	Matlab/C
Rea et al.	[62]	2019	https://dibernardo.tigem.it/software-data	Matlab/C
Software platforms
CellProfiler	[39]	2006	http://cellprofiler.org	Python
MitosisAnalyser	[61]	2017	https://github.com/JoanaGrah/MitosisAnalyser	Matlab
ilastik	[38]	2019	https://www.ilastik.org/index.html	Python
Usiigaci	[63]	2019	https://github.com/ElsevierSoftwareX/SOFTX_2018_158	Python
ZeroCostDL4Mic	[69]	2020	https://github.com/HenriquesLab/ZeroCostDL4Mic	Python
DeepImageJ	[70]	2021	https://deepimagej.github.io/deepimagej	Python
BioImage Model Zoo	[71]	2022	https://bioimage.io	Python
LIM Tracker	[72]	2022	https://github.com/LIMT34/LIM-Tracker	Python/Java
TrackMate 7	[73]	2022	https://imagej.net/plugins/trackmate/trackmate-v7-detectors	Java

Table 1. Software: name (Name); reference ([Ref]); year of publication (Year); url (Link); programming language or environment (Language). All links were accessed on 28 August 2022.

Name	[Ref]	Year	Link	Language
Cell segmentation
DeepCell	[23]	2016	https://simtk.org/projects/deepcell	Python, C, Ruby
fastER	[25]	2017	https://bsse.ethz.ch/csd/software/faster.html	C++
TWS	[68]	2017	https://imagej.net/plugins/tws/	Java
ANCIS	[32]	2019	https://github.com/yijingru/ANCIS-Pytorch	Python
Vicar et al.	[20]	2019	https://github.com/tomasvicar/Cell-segmentation-methods-comparison	Matlab
Cellpose	[45,46]	2022	https://github.com/MouseLand/cellpose	Python
Cell classification
DeephESC 2.0	[55]	2019	https://www.vislab.ucr.edu/SOFTWARE/software.php	Python
celldeath	[58]	2021	https://github.com/miriukaLab/celldeath	Python
Cell tracking
Baxter Algorithms	[60]	2015	https://github.com/klasma/BaxterAlgorithms	Matlab/C
Rea et al.	[62]	2019	https://dibernardo.tigem.it/software-data	Matlab/C
Software platforms
CellProfiler	[39]	2006	http://cellprofiler.org	Python
MitosisAnalyser	[61]	2017	https://github.com/JoanaGrah/MitosisAnalyser	Matlab
ilastik	[38]	2019	https://www.ilastik.org/index.html	Python
Usiigaci	[63]	2019	https://github.com/ElsevierSoftwareX/SOFTX_2018_158	Python
ZeroCostDL4Mic	[69]	2020	https://github.com/HenriquesLab/ZeroCostDL4Mic	Python
DeepImageJ	[70]	2021	https://deepimagej.github.io/deepimagej	Python
BioImage Model Zoo	[71]	2022	https://bioimage.io	Python
LIM Tracker	[72]	2022	https://github.com/LIMT34/LIM-Tracker	Python/Java
TrackMate 7	[73]	2022	https://imagej.net/plugins/trackmate/trackmate-v7-detectors	Java

ZeroCostDL4Mic is a cloud-based platform proposed by von Chamier et al. [69] aiming to simplify the use of DL architectures for various microscopy tasks. It is a collection of Jupyter Notebooks that can efficiently and interactively run Python code, leveraging the free, cloud-based computational resources of Google Colab. Concerning our focus, the tasks covered by ZeroCostDL4Mic include object detection, for which it implements YOLOv2, and cell segmentation, where it implements both the U-net and StarDist [74,75] networks. The outputs generated by StarDist are directly compatible with the TrackMate tracking software, enabling also automated cell tracking.

DeepImageJ [70] is a plugin for ImageJ and Fiji to facilitate the usage of DL models. It aims to offer user-friendly access to pre-trained models designed for various image modalities, including PhC and DIC. Currently, for the two previously mentioned modalities, the DL models are designed for segmentation.

BioImage Model Zoo [71] is an online repository for AI models to facilitate the usage of these pre-trained models by the bioimaging community. They provide a standard and tutorials to upload new models. The users can either download the projects in community partners’ format or in user-friendly Python notebooks that can be used by anyone with the user’s own dataset to perform bioimage analysis tasks. The current community partners are ilastik, ImJoy [76], Fiji [77], deepImageJ, ZeroCostDL4Mic, and HPA [78].

LIM Tracker is a Fiji plugin for cell tracking and analysis expressly aimed at advanced interactivity, usability, and versatility. Three tracking methods are implemented, suitable for fluorescence or PhC microscopy sequences. In the link-type tracking (tracking by detection), cells are first detected based on a Laplacian of Gaussian filter and watershed segmentation. Their ROIs are then linked by the Linear Assignment Problem algorithm [79]. In the sequential search-type tracking method, based on the particle filter framework, a user-specified ROI is tracked by sequentially searching for its corresponding ROI in subsequent frames by pattern matching. The third type of tracking is manual tracking, which allows users to specify the position of ROIs while moving along sequence frames. Several additional functions allow interactive visualization and error correction. A plugin mechanism is provided for using different segmentation modules, including user-defined algorithms or DL algorithms (e.g., StarDist, Cellpose [45,46], YOLACT++ [80], Matterport MaskR-CNN [81], and Detectron2 MaskR-CNN [82]).

TrackMate 7 [73] is an extension of the TrackMate tracking software [83] distributed as a Fiji plugin. It integrates into tracking pipelines (based on five possible particle-linking algorithms) ten segmentation algorithms (including ilastik, Weka, StarDist, and Cellpose), besides any mask or label images computed with any other segmentation algorithm. It can handle fluorescence or label-free microscopy images, both 2D and 3D. The additional TrackMate helper facilitates choosing an optimal combination of segmentation and tracking modules, also allowing a systematic optimization of the tracking parameters for a whole dataset.

For an extended list of commercial and open source tools for tracking, the interested reader can also refer to [21]. A list of publicly available executable versions of 19 algorithms participating in the 2013–2015 CTC challenges is provided in Table 3 of the Supplementary Material of [22]; further links can also be found through the CTC web pages. Open-source DL software for bioimage segmentation is nicely surveyed in [84], where tools in different forms, such as web applications, plug-ins for existing imaging analysis software, and preconfigured interactive notebooks and pipelines are reviewed. Finally, further suggestions can come from the review by Smith et al. [85]. Indeed, even though their survey focuses on phenotypic image analysis, some of the referred software includes cell segmentation and time-lapse analysis tools.

4. Data

Datasets publicly available can be broadly subdivided into those devoted solely to segmentation (see

Table 2. Details of available annotated data for cell segmentation in traditional label-free images: dataset name (Name); reference ([Ref]); type of microscopy data (Content); url (Link); number of annotated images (# imgs), annotated cells (# cells), and cell lines (#cell lines). All links were accessed on 28 August 2022.

Name	[Ref]	Content	Link	# Imgs	# Cells	# Cell Lines
Allen Cell Explorer	[86]	3D Label- Free	https://www.allencell.org/data-downloading.html/#sectionLabelFreeTrainingData	~18,000	~39,000	1
BU-BIL	[87]	PhC	https://www.cs.bu.edu/fac/betke/BiomedicalImageSegmentation/	151	151	3
CTC	[22]	PhC, DIC, BF	http://www.celltrackingchallenge.net	213	1980	5
DeepCell	[23]	PhC	https://doi.org/10.1371/journal.pcbi.1005177.s021, https://doi.org/10.1371/journal.pcbi.1005177.s022, https://doi.org/10.1371/journal.pcbi.1005177.s023	45	~4300	1
EVICAN	[88]	PhC, BF	https://edmond.mpdl.mpg.de/dataset.xhtml?persistentId=doi:10.17617/3.AJBV1S	4640	26,428	30
fastER	[25]	PhC, BF	https://bsse.ethz.ch/csd/software/faster.html	39	1653 (+953) 1	2
LIVEcell	[26]	PhC	https://sartorius-research.github.io/LIVECell/	5239	1,686,352	8
Usiigaci	[63]	PhC	https://github.com/ElsevierSoftwareX/SOFTX_2018_158	37	2641	1
Vicar et al.	[20]	PhC, DIC, HMC	https://zenodo.org/record/1250729	32	4546	1

¹ For other 953 cells, only centroids are provided.

), event detection and classification (and eventually also tracking, see Table 3), or tracking (and eventually also segmentation, see Table 4). Observe that the numbers reported in these tables refer solely to traditional label-free images/image sequences, which is the focus of this review; nonetheless, many of the reported datasets also have data from other microscopy types. The reported numbers specify only images for which annotations exist (there could be other images but without annotations).

Allen Cell Explorer [86] includes a massive collection of light microscopy cell images with manually curated segmentation masks for 12 cellular components, as reported in [88].

BU-BIL (Boston University-Biomedical Image Library) [87] includes six datasets, three of which consisting of PhC images from different cell lines. The main aim of [87] is to evaluate and compare the performance of biomedical image segmentation made by trained experts, non-experts, and automated segmentation algorithms. Therefore, for each image, only one cell is annotated and provided as binary masks obtained in those three different ways. The gold standard annotation is obtained by majority voting of annotations created by the ten trained experts.

CTC (Cell Tracking Challenge) is a time-lapse cell segmentation and tracking benchmark on publicly available data, launched in 2012 to objectively compare and evaluate state-of-the-art whole-cell and nucleus segmentation and tracking methods [22,89]. The datasets consist of 2D and 3D time-lapse video sequences of fluorescent counterstained nuclei or cells moving on top or immersed in a substrate, along with 2D PhC and DIC microscopy videos of cells moving on a flat substrate. The videos cover a wide range of cell types and quality (spatial and temporal resolution, noise levels, etc.). The ground truth consists of manually annotated cell masks (for segmentation) and cell markers interlinked between frames to form cell lineage trees (for tracking).

DeepCell comes from the supporting material of [23]. It consists of a PhC image sequence of HeLa-S3 cells. Annotations for each image are given in terms of cell and nuclei segmentation masks.

EVICAN (Expert VIsual Cell ANnotation) [88] includes partially annotated grayscale images of 30 different cell lines from multiple microscopes, contrast mechanisms, and magnifications. For each image, a subset of cells and nuclei is annotated and provided both as json annotation files and as binary masks. An example is shown in Figure 3a,b. To reduce the influence of unannotated cells on the background class, in their experiments, the authors pre-processed the dataset by blurring (with a Gaussian filter) the images but leaving unchanged the annotated instances. The pre-processed images are also provided with the dataset (see Figure 3c).

fastER [25] includes PhC, BC, and synthetic Fluo images of three different cell lines. For each image, the annotations consist of binary masks that enclose the segmentation of most of the cells and just the centroid for the remaining cells.

LIVEcell [26] is a recently proposed large-scale, manually annotated, and expert-validated dataset of PhC images for benchmarking cell segmentation. It consists of over 5 thousand images, including over 1.6 million cells of seven cell types (human and mouse) having different cell morphologies and culture densities. Annotations are provided as json files.

Usiigaci [63] includes 37 PhC images of T98G cells. Annotations consist of indexed masks, with an index for each cell, followed in time (see Figure 4). Thus, these can be used for both segmentation and tracking. A spreadsheet file is also enclosed, providing information from tracking and various features for each tracked cell.

Figure 4. Example data from the Usiigaci dataset [63]: (a) original image (20180101ef002xy01t01.tif); (b) corresponding indexed mask, where each color indicates a different cell in all sequence images.

Figure 4. Example data from the Usiigaci dataset [63]: (a) original image (20180101ef002xy01t01.tif); (b) corresponding indexed mask, where each color indicates a different cell in all sequence images.

Table 3. Details of annotated data for cellular event detection and classification in traditional label-free sequences: dataset name (Name); reference ([Ref]); application of the data (Task); type of microscopy data (Content); url (Link); number of annotated images (# imgs), and number of annotated events (# events). All links were accessed on 28 August 2022.

Name	[Ref]	Task	Content	Link	# Imgs	# Events
C2C12-16	[54]	Mitosis Detection	DIC	https://www.iti-tju.org/mitosisdetection/download/1	16,208	7159
CTMC	[90]	Mitosis Detection	DIC	https://ivc.ischool.utexas.edu/ctmc/1	80,389	1616
DeephESC	[55]	Classification	PhC	https://www.vislab.ucr.edu/SOFTWARE/software.php2	785	NA

¹ Unavailable at the time of writing. ² Accessed on 30 August 2022.

Table 3. Details of annotated data for cellular event detection and classification in traditional label-free sequences: dataset name (Name); reference ([Ref]); application of the data (Task); type of microscopy data (Content); url (Link); number of annotated images (# imgs), and number of annotated events (# events). All links were accessed on 28 August 2022.

Name	[Ref]	Task	Content	Link	# Imgs	# Events
C2C12-16	[54]	Mitosis Detection	DIC	https://www.iti-tju.org/mitosisdetection/download/1	16,208	7159
CTMC	[90]	Mitosis Detection	DIC	https://ivc.ischool.utexas.edu/ctmc/1	80,389	1616
DeephESC	[55]	Classification	PhC	https://www.vislab.ucr.edu/SOFTWARE/software.php2	785	NA

¹ Unavailable at the time of writing. ² Accessed on 30 August 2022.

C2C12-16 [54] was released as a large-scale time-lapse phase-contrast microscopy image dataset for the mitosis detection task at the first international contest on mitosis detection in phase-contrast microscopy image sequences, held with the workshop on computer vision for microscopy image analysis (CVMI) at CVPR 2019. It is an extension of the Ker et al. dataset [91] with manual annotations of mitosis. The complete dataset contains 16 sequences with 1013 frames per sequence and a total of 7159 mitosis events within the images.

Cell Tracking with Mitosis Detection Challenge (CTMC) is a benchmarked challenge that provides DIC images for 14 cell lines [90]. The data adds up to 86 live-cell imaging videos consisting of 152,584 frames in total. In addition to the images, the challenge grants bounding box-based detection and tracking ground truths for each cell line, in the form of csv files for each video, including, for each frame and each cell, the cell ID and its bounding box coordinates. Recently, the dataset has also been adopted for the CTMC-v1 Challenge at CVPR 2022 (https://motchallenge.net/data/CTMC-v1/, accessed on 30 August 2022).

DeephESC [55] consists of 785 PhC hESC images subdivided according to six classes (cell clusters, debris, unattached cells, attached cells, dynamically blebbing cells, and apoptically blebbing cells).

Table 4. Details of available annotated data for cell tracking in traditional label-free sequences: dataset name (Name); reference ([Ref]); type of microscopy data (Content); url (Link); number of annotated images (# imgs), annotated cells/tracks (# cells/tracks), and cell lines (#cell lines). All links were accessed on 28 August 2022.

Name	[Ref]	Content	Link	# Imgs	# Cells/Tracks	# Cell Lines
CTC	[22]	PhC, DIC, BF	http://www.celltrackingchallenge.net1	213	1980/2944	5
CTMC	[90]	DIC	https://ivc.ischool.utexas.edu/ctmc/2	80,389	1,097,223 3/1616	14
Ker et al.	[91]	PhC	https://osf.io/ysaq2/1	19134	NA 4/2011	1
Usiigaci	[63]	PhC	https://github.com/ElsevierSoftwareX/SOFTX_2018_1581	37	2641/105	1

¹ Accessed on 30 August 2022. ² Unavailable at the time of writing. ³ Only bounding boxes are provided. ⁴ Only centroids are provided.

Table 4. Details of available annotated data for cell tracking in traditional label-free sequences: dataset name (Name); reference ([Ref]); type of microscopy data (Content); url (Link); number of annotated images (# imgs), annotated cells/tracks (# cells/tracks), and cell lines (#cell lines). All links were accessed on 28 August 2022.

Name	[Ref]	Content	Link	# Imgs	# Cells/Tracks	# Cell Lines
CTC	[22]	PhC, DIC, BF	http://www.celltrackingchallenge.net1	213	1980/2944	5
CTMC	[90]	DIC	https://ivc.ischool.utexas.edu/ctmc/2	80,389	1,097,223 3/1616	14
Ker et al.	[91]	PhC	https://osf.io/ysaq2/1	19134	NA 4/2011	1
Usiigaci	[63]	PhC	https://github.com/ElsevierSoftwareX/SOFTX_2018_1581	37	2641/105	1

¹ Accessed on 30 August 2022. ² Unavailable at the time of writing. ³ Only bounding boxes are provided. ⁴ Only centroids are provided.

The dataset by Ker et al. [91] includes 48 PhC image sequences of mouse C2C12 cells under various treatments. Annotations consist of manually tagged centroids and state (e.g., newborn, divided, or mitotic) for 10% of the cells for all the sequences; only for one of the sequences, all the cells are manually annotated. The dataset is also provided with annotations automatically generated for all the cells using in-house software based on segmentation, mitosis detection, and association.

Other annotated microscopy image sets can be downloaded from the Broad Bioimage Benchmark Collection (BBBC) [92]. It is a publicly available collection of microscopy images intended as a resource for testing and validating automated image-analysis algorithms. Being contributed by many different research groups and for various applications, annotations are provided in varying forms (e.g., cell counts, masks, outlines, or bounding boxes).

5. Metrics

Below, we present some metrics commonly adopted to evaluate the results quantitatively. Some of these metrics are directly used from the computer vision and ML/DL domains, while others are more specific to cellular image analysis.

5.1. Metrics for Pixel-Wise Cell Segmentation

Many different metrics are adopted in the literature to evaluate the performance of (cell) segmentation algorithms. The most frequently used is the one adopted for the CTC [22], generally denoted as SEG. Given the ground truth cell segmentation $GT$ and the corresponding segmentation S computed with any segmentation algorithm, the Jaccard similarity index, also known as Intersection over Union (IoU), evaluates the degree of overlap between the true and the computed results and is defined as

$$\mathrm{IoU}(GT,S)=\frac{|GT\cap S|}{|GT\cup S|},$$

(1)

where $|·|$ indicates the cardinality of a set (i.e., the number of pixels) and ∩ and ∪ indicate the set intersection and union, respectively. This metric [63] is sometimes equivalently expressed in terms of the number of true positive pixels $TP$ ($TP=|GT\cap S|$), false negative pixels $FN$ ($FN=GT-S$), and false positive pixels $FP$ ($FP=S-GT$) as

$$\mathrm{IoU}(GT,S)=\frac{TP}{FN+TP+FP}.$$

The SEG metric adopted in the CTC for a particular video is then computed as the mean IuO over all the GT cells of the video. It should be observed that, although many authors refer to this metric as SEG [72,93], others just refer to it as AP [45,46,74,75,88]. Further metrics frequently adopted [23,25,63] include

• the Recall, also known as Sensitivity or True Positive Rate,

  $$\mathrm{Recall}=\frac{TP}{TP+FN},$$

  (2)

  that gives the percentage of detected true positive pixels as compared to the total number of true positive pixels in the ground truth;
• the Precision, also known as Positive Prediction,

  $$\mathrm{Precision}=\frac{TP}{TP+FP},$$

  (3)

  that gives the percentage of detected true positive pixels as compared to the total number of pixels detected by the algorithm, providing an indication on the degree of exactness of the algorithm in identifying only relevant pixels;
• the F-score, also known as F-measure or Figure of Merit,

  $$\mathrm{F}{}_1=\frac{2·\mathrm{Recall}·\mathrm{Precision}}{\mathrm{Recall}+\mathrm{Precision}}=\frac{2·TP}{2·TP+FP+FN},$$

  (4)

  that is the weighted harmonic mean of Precision and Recall.

All the above metrics assume values in [0,1] and higher values indicate better results.

5.2. Metrics for Object-Wise Cell Detection

Generally, in the case of cell detection, the ground truth is given in terms of bounding boxes of the cells contained in the images. Here, the IoU metric of Equation (1) can be adapted to evaluate the degree of overlap between the ground truth bounding boxes ($GT$) and the predicted bounding boxes (S). IoU tresholding can then be used to decide if a detection is correct or not. For a given IoU threshold $\alpha$, a true positive ($TP$), i.e., a correct positive prediction, is a detection for which IoU $(GT,S)\ge \alpha$ and a false positive ($FP$ ), i.e., a wrong positive detection, is a detection for which IoU $(GT,S)<\alpha$. A false negative ($FN$) is an actual instance that is not detected.

Given these adapted concepts, the Recall, Precision, and F-score metrics defined in Equations (2)–(4) can be used to evaluate cell detection algorithms. These are also used to compute the Average Precision at a given IoU threshold $\alpha$, denoted as AP@$\alpha$, defined as the Area Under the Precision-Recall Curve (AUC-PR) evaluated at the IoU threshold $\alpha$, given as

$$\mathrm{AP}@\alpha ={\int }_0^{1}p\left(r\right)dr.$$

According to the Common Objects in Context (COCO) [94] evaluation protocol (https://cocodataset.org/#detection-eval, accessed on 28 August 2022), single values for $\alpha$ can be chosen for thresholding IoU (generally equal to 0.5 or 0.75). Moreover, a set of thresholds can be chosen and the mean Average Precision mAP over these IoU thresholds considered for cell detection evaluation. With the usual choice [26,74,75] of values for $\alpha$ from 0.5 to 0.95 with a step size of 0.05, mAP is thus given by

$$\mathrm{mAP}=\frac{AP@0.5+AP@0.55+\dots +AP@0.95}{10}.$$

In the CTC [22], the detection accuracy of the methods, denoted as DET, is adopted to estimate how accurately each given object has been identified (http://celltrackingchallenge.net/evaluation-methodology/, accessed on 28 August 2022). It is based on the comparison of the nodes of the acyclic oriented graphs representing the objects in both the ground truth and the computed object detection result. Exploiting the Acyclic Oriented Graph Matching measure for detection (AOGM-D) [95], that gives the cost of transforming the set of nodes of the computed objects into the set of ground truth nodes, DET is defined as

$$\mathrm{DET}=1-\frac{min(\mathrm{AOGM}\text{-}\mathrm{D},{\mathrm{AOGM}\text{-}\mathrm{D}}_0)}{{\mathrm{AOGM}\text{-}\mathrm{D}}_0},$$

where AOGM-D₀ is the cost of creating the set of ground truth nodes from scratch. DET always falls in the [0,1] interval, with higher values corresponding to better detection performance. The DET metric is averaged with the SEG metric described in Section 5.1 to provide the overall performance for the CSB

$${\mathrm{OP}}_{CSB}=\frac{1}{2}\left(\mathrm{DET}+\mathrm{SEG}\right).$$

5.3. Metrics for Cell Event Detection

For mitosis detection, Ref. [54] represents each detected mitosis as a triple ($x,y,t$) of spatial and temporal position of the event. The detection is considered a true positive ($TP$) if its distance from the corresponding ground truth triple is below preset spatial and temporal thresholds. Otherwise, it is considered a false positive ($FP$). Undetected ground truth mitotic events are considered as false negative ($FN$). Having so defined $TP,FP$, and $FN$, Ref. [54] adopts Precision, Recall, and F-score metrics defined in Equations (2)–(4), respectively, to evaluate the performance of mitosis detection algorithms. The same metrics are also adopted in [60], where they are extended also to apoptic events.

In [23], DeepCell is also used to perform semantic segmentation, i.e., to both segment individual cells and predict their cell type. For evaluating the obtained results, the authors consider the Cellular Classification Score (CCS_c) for each class c, defined as

$${\mathrm{CCS}}_c=\frac{{\sum }_{i\in Cells}{s}_{i,c}}{{\sum }_{j\in Classes}{\sum }_{i\in Cells}{s}_{i,j}},$$

where $s_{i,j}$ indicates the classification score of pixel i for class j. The authors showed that the closer the CCS_c is to 1, the more likely the prediction is correct.

5.4. Metrics for Cell Tracking

The metrics most frequently adopted for evaluating cell tracking are those introduced by the Multiple Object Tracking (MOT) [96]. The Multiple Object Tracking Accuracy (MOTA) [97] is a MOT tracking metric that represents the object coverage [90], also used for example in [63]. It can be defined as

$$\mathrm{MOTA}=1-\frac{FN+FP+IDSW}{T},$$

where $FN$ is the sum over the entire video of all missed cells (number of ground truth bounding boxes not covered by any computed bounding box), $FP$ is the sum over the entire video of all false positives (number of bounding boxes not covering any ground truth bounding box), $IDSW$ is the number of object identities switched from one frame to the next (number of bounding boxes covering a ground truth bounding box from a track different than in the previous frame), and T is the total number of detections in the ground truth.

Multiple Object Tracking Precision (MOTP) [98] is the average dissimilarity between all correctly assigned detections (true positives) and their ground-truths, defined as

$$\mathrm{MOTP}=\frac{{\sum }_{t,i}{d}_{t,i}}{{\sum }_t{c}_t},$$

where $c_t$ indicates the number of matches in frame t and $d_{t,i}$ is the bonding bob overlap of the detection i with its ground truth. This MOT tracking metric shows the ability of the tracker to estimate precise object positions, independent of its skill at recognizing object configurations, keeping consistent trajectories, and so forth [97].

More recently, the MOT challenge introduced another tracking metric, named IDF1, that quantifies the object’s identity across the frames of a sequence [90] and represents the ratio of the detections that were properly identified over the average number of ground-truth and computed detections [99]. It is an F-score as in Equation (4).

$$\mathrm{IDF}1=\frac{2\ast IDTP}{2\ast IDTP+IDFP+IDFN},$$

where $IDTP,IDFP,$ and $IDFN$ indicate the number of true positive, false positive, and false negative IDs, respectively.

Many other metrics introduced for evaluating MOT challenges could also be applied to the case of cell tracking, such as the Higher Order Tracking Accuracy (HOTA) [100]. Focusing more specifically on cellular microscopy, the metric adopted in CTC [22] for evaluating cell tracking results is the Tracking Accuracy, denoted as TRA, used for example in [72,90,93]. It is a normalized weighted distance between the tracking ground truth and the result of the algorithm, with weights chosen to reflect the effort it takes a human curator to manually carry out the edits needed for matching the two. Tracking results are first represented as acyclic oriented graphs providing the cells lineage. Then the difficulty in transforming a computed tracking graph into the corresponding ground truth graph is estimated as

$$\mathrm{TRA}=1-\frac{min(\mathrm{AOGM},{\mathrm{AOGM}}_0)}{{\mathrm{AOGM}}_0},$$

where AOGM is the Acyclic Oriented Graph Matching (AOGM) measure [95] and AOGM₀ is the AOGM value required for creating the ground truth graph from scratch. TRA assumes values in [0, 1], with higher values corresponding to better tracking performance. The overall performance for the CTB is calculated as the average of the SEG (see Section 5.1) and TRA metrics:

$${\mathrm{OP}}_{CTB}=\frac{1}{2}\left(\mathrm{SEG}+\mathrm{TRA}\right).$$

6. Open Problems and Future Research Directions

Common challenges in microscopy image processing include increasingly high image sizes, image artifacts, and batch effects, especially in the presence of object crowding and overlapping. Nevertheless, insufficient, imbalanced, and inconsistent data annotations [101] prevent the effective usage of data analysis methods. The image size affects computational and storage time, which might prevent the use of modern image processing techniques on standard hardware. Nevertheless, the batch acquisition of images poses problems, as small variations might be present in successive applications of the same technical procedure. That implies that the hypothesis that all images are independently identically distributed statistical units sampled from the same population, which means they all describe the same process independently of its natural variability, might not be true anymore. As a consequence, the training of AI/ML/DL/methods can be biased and produce results that are not reflecting the probability distribution of the original phenomena. Despite the results reviewed here, these problems require further investigation, as the number of images required for training the parameters of a multi-layer architecture is in the order of tens of thousands. The performance of AI systems and their generalization capabilities strongly depend on the quality of annotations from the available datasets. Although many unsupervised algorithms may not require annotated data, these are crucial for understanding and interpreting such systems. As evident from Table 2 and Table 4, which represent available data collections at the publication time, there is a lack of large-scale curated and annotated datasets of light microscopy images. In particular, this is true for adherent cells or suspension-cultured counterparts, where the lack of annotated datasets makes the segmentation difficult for different LI techniques [20]. This difficulty is exacerbated by the natural variability of the observed phenomena, which often show high cell densities, cell-to-cell variability, complex cellular shapes and texture, cell shape varying over time (e.g., due to drug treatments [26,93]), varying image illumination, and low signal-to-noise ratios (SNRs) [24,25]. These problems are also common in cell detection and tracking, making them even more challenging, especially when using AI approaches. An extreme example is neural cell instance segmentation in neuroscience applications [32], which aims at detecting and segmenting every neural cell in a microscopy image. In these experiments, further limiting factors include cell distortion, unclear cell contours, low-contrast cell protrusion structures, and background impurities. However, accurate detection of objects is crucial for the tracking process, as aberrant object detection leads to missing links and the generation of tracks that end prematurely, with multiple short tracks representing the same individual object over time as different entities. Most detection algorithms treat tightly packed objects (e.g., touching and overlapping cells) as a single entity, resulting in breaks in tracks or single tracks linking groups of objects. In addition, cell tracking can produce terabyte-scale movies as experiments often require multi-day monitoring [93]. In such experiments, rapid cell migrations, high cell density, and multiple rounds of mitoses result in multiple neighboring cells being mis-tracked. Extensive 3D data present additional challenges due not only to the size of the image data itself but also to the very high cell densities they show toward the end of the videos [22].

Other key factors that affect the tracking results [21] include noise, occlusions, difficult object motion, complex objects structures, and background subtraction. Further work is also needed for handling scenarios with low SNR or contrast ratio [22]. These challenges can explain why few cell tracking platforms have been developed for label-free microscopy images. The benchmarked ranking of the Cell Tracking Challenge [22] confirms the difficulties in processing these images and highlights further related research, particularly for DIC images [90]. All these current factors limit the possibility of assessing and comparing the capabilities of different methods used to analyze data. Perspectives and future work in LI also include handling big data of continuously growing size, improving the quality and completeness of annotated datasets, continuous modeling of biological processes using regression rather than classification, and interpretability of ML and DL algorithms.

Finally, investigations should be devoted to the integration of multiple microscopy techniques on the same sample to overcome the proper limits of each technique and the lack of training data. We believe that a holistic view of biological processes and functions might be attained by omics imaging [36], which consists in the integration and analysis of next-generation sequencing data with images, in order to provide more insight into available data.