A Novel Approach for the Automatic Estimation of the Ciliated Cell Beating Frequency

Abstract

The qualitative and quantitative evaluation of nasal epithelial cells is interesting in chronic infectious and inflammatory pathologies of the nose and sinuses. Among the cells of the population of the nasal mucosa, ciliated cells are particularly important. In fact, the observation of these cells is essential to investigate primary ciliary dyskinesia, a rare and severe disease associated with other serious diseases such as respiratory diseases, situs inversus, heart disease, and male infertility. Biopsy or brushing of the ciliary mucosa and assessment of ciliary function through measurements of the Ciliary Beating Frequency (CBF) are usually required to facilitate diagnosis. Therefore, low-cost and easy-to-use technologies devoted to measuring the ciliary beating frequency are desirable. We have considered related works in this field and noticed that up to date an actually usable system is not available to measure and monitor CBF. Moreover, performing this operation manually is practically unfeasible or demanding. For this reason, we designed BeatCilia, a low cost and easy-to-use system, based on image processing techniques, with the aim of automatically measuring CBF. This system performs cell Region of Interest (RoI) detection basing on dense optical flow computation of cell body masking, focusing on the cilia movement and taking advantage of the structural characteristics of the ciliated cell and CBF estimation by applying a fast Fourier transform to extract the frequency with the peak amplitude. The experimental results show that it offers a reliable and fast CBF estimation method and can efficiently run on a consumer-grade smartphone. It can support rhinocytologists during cell observation, significantly reducing their efforts.

1. Introduction

Medical treatments and diagnostic methods have been deeply influenced by the technological advancements achieved in recent decades. Thanks to the numerous studies in the field of computer vision applied to the medical and biomedical fields, additional tools are available to specialists to support them in their tasks [1,2,3,4,5,6,7], improving acquisition, transmission, and analysis of digital images and signals [8,9,10]. In many cases, the basic techniques of these tools require segmentation and contour extraction that are challenging tasks in several application fields [11,12,13,14,15,16,17], and in particular in the medical field [18,19]. Many studies have been conducted in the segmentation and classification of cells from digital images, such as in the case of nasal cytology, which is gaining increasing importance in the diagnosis of nasal diseases [20,21,22]. This fields involves studying the nasal mucosa cells, with the aim of identifying cellular variations in the epithelium, which are often exposed to acute or chronic irritation and inflammation caused by viruses, bacteria, or fungi. Recent studies have proposed automatic extraction and classification of the cells of the nasal mucosa based on digital images from a microscopic slide [23,24].

Among the cells of the nasal mucosa population, ciliated cells form 80% of the epithelium in the upper airways and are particularly important. The qualitative evaluation of ciliated cells is also essential in investigating the Primary Ciliary Dyskinesia (PCD) [25]. Dyskinesia refers to the abnormal and uncoordinated movement of cell cilia. In humans, ciliated cells are 15 to 20 nm long and generally beat with a frequency of up to 16 Hz, but the normal frequency can change in response to infections, temperature, age, or generally speaking, in the case of inflammation or infections in the upper airway [26]. Regularly, these cells beat with a constant, smooth pattern, allowing an effective mucociliary clearance. Biopsy or brushing of the ciliary mucosa and the assessment of ciliary function through measurements of the Ciliary Beating Frequency (CBF) are usually required to make a diagnosis. Interesting insights into ciliated cells and the CBF are reported in Appendix A.

The CBF cannot be properly analyzed with the naked eye, due to its high oscillatory motion; therefore, many techniques have been proposed to overcome this limitation in recent decades. The earliest proposed techniques were based on photo-electric effects [27], photodiodes [28], and stroboscopes [29,30]. Successively, CBF analysis became accurate with the use of high-speed video cameras [31,32], however the implemented procedures were semi-automatic and CBF was manually obtained. The first automated CBF measuring technique, called Sisson–Ammons Video Analysis (SAVA) and based on image analysis, was developed in 2003 by Sisson et al. [33]. It automatically analyzes the whole image for each frame. Its main drawback is that SAVA is very time-consuming, because it analyzes many blocks for each video frame. In [34], the authors describe a method devoted to CBF measurement based on the fast Fourier transform (FFT); they acquire the images with an iPhone camera and then perform the analysis on a desktop system. This study was conducted on ciliary cells extracted from a rabbit’s tracheal samples. This technique requires the manual selection of the Regions of Interest (RoIs), and the authors concluded that “locating the RoIs with a single beating cell is not only challenging but also crucial”, hence an improvement of RoI selection strategies is desirable.

Other studies [35,36] have exploited pixel intensity variations in order to determine CBF; however, although effective, their performance depends on the experimental settings for acquiring videos, which are subject to particularly binding conditions. In [37,38], the authors showed that motile areas and the CBF can be determined through the optical flow (OF), which is defined as the “motion of the imaging surface in changing brightness for 2-D images according to the real motion of an object in the 3-D space” [39]. However, it has been shown that the image quality and frequency resolution (sampling frequency or number of frames captured) are key factors that must be taken into consideration to accurately estimate CBF [36,38,39,40].

To the best of our knowledge, tools measuring CBF proposed in the literature show practical limits, as we have highlighted above, or are based on expensive technologies in real cases. As a matter of fact, the domain experts, when not supported by specialized laboratories, can only describe the ciliated epithelium in a qualitative way. For these reasons, it is worthwhile investing in research to design affordable systems capable of measuring CBF in a short time frame, reducing manual interventions. In this regard, this study is focused on achieving a reliable, low-cost, high performing, and fully automated system called BeatCilia. This system can acquire and process images through a smartphone, using visual computing algorithms, thus overcoming the limitations of RoI-based methods. BeatCilia is a fully automated system and its performance is independent of the user experience. A further benefit of systems such as this one is the possibility of sending clinical data directly to an electronic health record (EHR) and sharing them with care providers [41,42,43].

2. Materials and Methods

2.1. Sample Collection and Preparation

The dataset collected in this work included 10 videos acquired by a Nikon Eclipse 600 (1000× magnification) microscope equipped with a camera model MD6iS (Sony IMX236 sensor), with a spatial resolution of 1920 × 1080 pixels and a temporal resolution of 30 frames per second (fps). Videos were recorded with the aim of testing the proposed CBF automatic extraction algorithm in different scenarios, taking into account the following variables:

• Video background noise level, for which we marked three noise levels, as shown in Figure 1;
• The presence of single or multiple cells in a single frame;
• In-place movement of the cell body, which may affect the automatic estimation of the CBF. We labeled this as “weak” if the cell body was almost immotile, “discrete” (resp. “high”, “very high”) if it was slightly (resp. highly, very highly) vibrating in-place.

Table 1. Recap of the dataset with evidence of the different scenarios involved.

Name	Duration (s)	Frame Count	Background Noise Level	Cells Presence	Cell Body Movement
Cilia 1	2.9	87	1	Single	Weak
Cilia 2	4.56	136	1	Single	Discrete
Cilia 3	2.88	86	2	Single	Discrete
Cilia 4	2.86	85	3	Multiple	High
Cilia 5	10.04	301	1	Single	Very High
Cilia 6	4.85	135	2	Multiple	Weak
Cilia 7	2.83	85	1	Multiple	Discrete
Cilia 8	2.43	73	1	Single	Weak
Cilia 9	3.1	93	1	Multiple	Weak
Cilia 10	2.56	77	3	Multiple	Weak

shows the details of the different scenarios considered with the dataset. In particular, cilia 1 and 2 both have clean backgrounds, but only the first one is related to ciliated cells with stable bodies. Cilia 3 has either background noise or an unstable cell body. Cilia 4, which is the worst-case scenario, has a spoiled background and a fast-paced moving body. Videos from Cilia 5 to Cilia 10 contain different scenarios that are a combination of the four previously described ones. Figure 1 shows an example of frames captured from the dataset.

We will show that our system is capable of handling all of these situations, without requiring a full zoomed-in scene of the beating cilia.

2.2. BeatCilia System Description

In this section, a description of BeatCilia will be given in detail, describing the image processing and computer vision techniques used. Figure 2 shows a brief overview of the building blocks of the system, which takes a video as the input and returns the CBF:

• Cell RoI detection, devoted to the detection of regions of interest, i.e., small portions of the image that depict the cilia or the whole cell area;
• Cell body masking, which acts as a fine-tuning step for each of the RoIs detected in the previous step. The aim is to highlight only cilia, explicitly excluding pixels belonging to the cell body;
• CBF estimation, aimed at measuring the beating frequency for each automatically detected ciliated cell.

2.2.1. Cell RoI Detection

This step was essential for excluding user intervention, regardless of the presence of single or multiple cells in the video frames. Hence, the following operations were performed:

• Grayscale conversion and equalization using the Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm [44] to enhance the contrast and sharpness;
• Dense optical flow computation;
• Optical flow magnitude thresholding and morphological filtering;
• RoI detection.

Let I_f be a generic grayscale image in frame index f and N be the number of frames in a video. The detection of the cells is performed using a dense Optical Flow (OF) algorithm [45], applied on two subsequent frames OF(I_t−1, I_t) for t = 1,…, N and then thresholding the optical flow magnitude image with a binary threshold τ. Then, this binary mask is processed by a morphological opening operation, also known as area opening, using a size 3 elliptic structuring element in order to remove small blobs. Finally, cell RoI detection takes place using a connected component analysis. Figure 3 shows an insight into this step, in which it is immediately clear that the detected cell RoI can be smaller or larger, depending on the actual movement of the whole cell.

2.2.2. Cell Body Masking

The results in Figure 3 highlight the need for a fine-tuning of the cell RoI in order to consider only the pixels belonging to the cilia. In fact, the identified cell RoI can comprise both the cilia and cell body, whose contribution in terms of movement can significantly alter the CBF estimation.

The key aspects of the effectively designed cell body masking method—and consequently a fundamental innovative aspect of our approach—were based on two intrinsic characteristics of ciliated cells:

• The darkness of the basal body, a streak where cilia are anchored to, which marks the boundary between cilia and the cell body;
• A strong white glow that surrounds the cytoplasm and stops right below the dark basal body, a visual effect due to the presence of the cell body, which has its own thickness.

Cilia were identified by masking the cell body. To perform this task, cell body contours were estimated by exploiting the white glow around them. Then, the cell contour was found by masking all of the pixels with an intensity lower than the threshold, which were computed using the algorithm in [46]. Finally, the convex hull was computed to identify the approximation of the cell body (i.e., the minimum filling polygon), and consequently to blacken it. This masking process was iteratively applied to all of the RoIs. Figure 4 shows an example of cell body masking.

2.2.3. CBF Estimation

CBF estimation was then performed by applying a fast Fourier transform [47] on the entire time series frames to analyze the signal in the frequency domain. A high-pass filter was applied to remove the fundamental frequency. The CBF was finally estimated by detecting the peak of the amplitude spectrum of the signal. When multiple cells were detected, the average CBF value was returned.

3. Experiments and Results

In order to effectively test and discuss the methodology described in the previous section, three experiments were designed:

• The first experiment was a preliminary test of the proposed method. It was run on ad-hoc video simulations, which reproduced a constant beating pattern;
• The second experiment run on the videos included in our dataset was performed to validate the approach in a real case scenario;
• Finally, we compared the execution times of BeatCilia running on multiple platforms (including a smartphone) and showed our results compared to those achieved by a couple of previous studies (only partially compatible with our system, as they did not use the same facilities).

The software was written in MATLAB and C++ for the desktop platform, whilst Kotlin was used for mobile software.

The first experiment aimed to prove that analyzing the variation of pixel intensities over time. Applying a Fourier transform to them was an effective and efficient way of extracting the beating frequency. To perform this task, simulated signals were used to emulate both single and multiple cell contexts, as well as the presence of noise (

Table 2. Recap of simulated beating patterns for experiment number 1. The simulated beating frequency and the estimated one are identical.

Simulation N.	N. of Beating Objects	Noise	Beating Frequency (Hz)	Estimated Peak Frequency (Hz)
S1	1	No	7.00	7.00
S2	1–3	No	8.00	8.00
S3	1–3	Yes	14.00	14.00

).

The simulated videos had a spatial resolution of 200 × 200 pixels. In particular, S1 shows a squared patch of 50 × 50 pixels, whose grayscale values beat regularly and without any additional noise, as expressed by the following equation:

$$\mathrm{Y}\left(\mathrm{t}\right)=i·\sin \left(2·\pi ·f·t\right)$$

(1)

The parameters can be explained as follows:

• i stands for the required pixel intensity assigned to the square patch;
• f is the square patch frequency, in hertz;
• t represents the time (in seconds), in our experiment ranging from 0 to N.

S2 is similar to S1, with the difference that the number of beating squares is increased. S3 reproduces a more realistic signal, for which both a Gaussian filter and white noise are applied. The equation used to simulate this behavior is the following:

$$\mathrm{Y}\left(\mathrm{t}\right)=i_1·\sin \left(2·\pi ·f_1·t\right)+i_2·\sin \left(2·\pi ·f_2·t\right)$$

(2)

The last column of Table 2 shows that by using simulated data, the proposed methodology is able to extract the beating frequency for both clean and noisy signals. A remark should be given about the practical importance of removing the fundamental frequency from the signal before evaluating the CBF (based on a peak detector), in order to avoid the presence of useless peaks (as shown in Figure 5). Moreover, it is worth noting that the effects of the added noise can be seen as multiple small peaks at different frequencies and reduced amplitudes corresponding to the real peaks.

In order to validate the proposed approach, a second experiment was conducted on real ciliated cell videos taken from the dataset described in Section 2.1. The resulting estimated CBFs were compared with a ground truth value that has been manually determined by an expert. This experiment has three different goals:

• Testing the effectiveness of the cell RoI detection;
• Testing the cell body masking;
• comparing BeatCilia results with the ground truth.

The parameters required to run the tests were experimentally tuned using a subset of the frames taken from the videos as a training set. In particular, dense optical flow parameters were set to:

• The number of pyramid layers = 3;
• The size of each layer = 0.5;
• Number of computing iterations = 3;
• The size of the pixel neighborhood = 5;
• The averaging filter size = 15.

The binary threshold τ was set to 17, while the structuring element used to perform the morphological opening had a size 3 ellipse.

Figure 6 shows the qualitative results of BeatCilia intermediate steps applied to a frame of the “cilia 1” video of our dataset. In particular, an equalized frame is shown in Figure 6a, where the white glow around the ciliated cell and details of the sharpening effects introduced by the equalization are immediately noticeable. Figure 6b shows the binary mask obtained by filtering the optical flow magnitude. Morphological filtering operations to remove small blobs are shown in Figure 6c, leading to the identification of the area containing ciliated cells, i.e., the cell RoI. Finally, the cell body is masked with the convex hull, as reported in Figure 6d, leaving only ciliated cells inside the region of interest (cell body masking operation).

Figure 7 shows the masked ciliated cells extracted automatically under different operative conditions. The regions of interest were captured from cilia 2, cilia 3, and cilia 4, and are shown to demonstrate the capability of the proposed system to correctly mask the cell body and highlight the cilia independently from the cell rotation and shape. In particular, Figure 7a shows a single cell with a modestly vibrating body and an almost clean background. Figure 7b shows a single cell with discrete body movement, along with a spoiled background. Figure 7c shows the worst case scenario: a scene with a noisy background in which multiple cells are recorded, but only one of them is actually beating. Figure 7d–f include some examples of intersections over unions (IoUs). All the cases in our experiments were similar to these. It is worth noting that the cilia remained greatly unmasked in all situations.

Figure 8 shows a graphical representation of BeatCilia outputs in a multiple-cell scenario with a spoiled background (cilia 4). In order to exclude false positives and stationary areas, every detected ROI with a CBF less than 1 Hz was automatically excluded from the results.

Finally,

Table 3. Estimated CBF and absolute error for all the videos in our dataset. The ground truth CBF was measured by an expert by reproducing the videos in slow motion and manually counting how many times the cilia came back to a certain spot. The resulting number was then divided by the length of the video in seconds.

Video	Ground Truth CBF (Hz)	Estimated CBF (Hz)	Absolute Error (Hz)
Cilia 1	2.100	2.069	0.031
Cilia 2	2.000	1.985	0.015
Cilia 3	1.000	1.395	0.395
Cilia 4	3.500	3.529	0.029
Cilia 5	5.450	5.382	0.068
Cilia 6	3.330	3.348	0.018
Cilia 7	4.280	4.235	0.045
Cilia 8	2.500	2.466	0.034
Cilia 9	1.900	1.936	0.036
Cilia 10	1.250	1.169	0.081

reports the CBF estimation for the videos in the dataset. In particular, BeatCilia was able to estimate the CBF, with an average absolute error of 0.075 Hz. In more detail, the absolute error was less than 0.082 Hz in all cases, except for the video of cilia 3, in which only 2.88 s of video recording (86 frames) was available, containing one cell that beat at only 1 Hz. In the other cases, more frames were available to estimate higher CBFs. A scatter plot is also reported in Figure 9 (correlation coefficient = 0.996).

Finally, we measured the execution time of BeatCilia on multiple platforms (including a mid-range smartphone) and compared our results with those achieved in [34,38]. The test consisted of repeating the whole process multiple times for each video in the dataset, keeping track of the execution times for each run. As for the desktop environment, tests were run on a PC equipped with an Intel Core i7-4710HQ CPU (quad-core, 4.5 GHz) with 16 GB of RAM and Windows 10 operating system. As for the mobile system, BeatCilia was tested on a Xiaomi Redmi 7 with a Snapdragon 610 (quad-core, 1.7 GHz) CPU and 4 GB of RAM, running Android 9.

Figure 10 shows the histogram of the elapsed time per frame of MATLAB implementation on the desktop (green bar), C++ implementation on the desktop (yellow bar), and Android implementation on the smartphone (blue bar). It is immediately noticeable that the running time for C++ outperforms MATLAB, being almost 8× faster due to it being optimized. Figure 10 shows that BeatCilia can efficiently run on a consumer-grade smartphone, effectively supporting the specialists by reducing their efforts, as well as giving reliable CBFs.

Table 4. Comparison of BeatCilia with the systems proposed in [34,38].

System	Platform	Frame Size (px)	Elapsed Time per Frame (s)	Wide Microscopic Field	Single/Multiple-Cell CBF	RoI Selection Method
BeatCilia	Mobile	1920 × 1080	0.058	Yes	Multiple	No
BeatCilia	Desktop C++	1920 × 1080	0.023	Yes	Multiple	No
BeatCilia	Desktop MATLAB	1920 × 1080	0.247	Yes	Multiple	No
[34]	Desktop	iPhone 6 (*)	N/A	No	Single	Yes
[38]	Desktop	256 × 192	0.056	No	Single	No

shows a comparison of BeatCilia with the systems proposed in [34] and [38], based on the following characteristics:

• Platform: Shows the platforms on which the system is available;
• Frame size: The size of the processed video frames; (*) authors do not specify the resolution they used;
• Elapsed time per frame: Average time (in seconds) required to process a single frame. For BeatCilia, we chose the worst case, represented by cilia 10. For [38], the value was computed from the total running time declared in the paper. Here, [34] is marked as not available because the authors declare a running time of “minutes”;
• Wide microscopic field: Shows whether the system can process wide microscopic field images or if it requires that the scene is manually zoomed on a single ciliated beating cilia;
• Single- or multiple-cell CBF: Whether the system is capable to estimate CBF for multiple cells in the scene;
• RoI selection method: Whether the system requires manual interaction to select RoIs or to set any parameter.

BeatCilia took up to 0.023 s (0.058 for the mobile implementation) to process each full-HD frame, while the method from [38] took about 0.056 s on 256 × 192 frames. This factor made a noticeable difference to long videos, especially in mobile environments. Moreover, experiments in this paper were carried out on a laptop processor that was slower (much slower in the case of the smartphone we used) than in other studies and with half of the RAM [38], a crucial factor for video processing applications. In [34,38], experiments were not carried out on a mobile platform.

4. Conclusions and Future Works

The proposed system has been proven to be effective in different scenarios, processing videos of single or multiple cells with or without noisy backgrounds. Its effectiveness allows rhinocytologists more flexibility in cell observation and video capturing, allowing lower zoom requirements, and thus providing reliable CBF estimation without a fully zoomed-in scene for the beating cilia. Its capability to automatically detect the cells in confusing scenarios, focusing on cilia through the masking of the cell body, is the key factor that make BeatCilia actually valuable. To our knowledge, this is the first time that two intrinsic characteristics of ciliated cells—namely the darkness of the basal body and the white glow around the cytoplasm—have been exploited in an automated system to identify the cilia. Moreover, the algorithm was designed to be as lightweight as possible, in order to be implemented on a mobile device. A comparison with other systems confirmed that the main goals of BeatCilia, such as efficiency, were achieved. Since it can also be run on a smartphone, specialists could significantly reduce diagnosis and monitoring times, whilst assuring high-quality results. Future directions of research will be aimed at investigating a possible correlation between the elapsed time and background noise level, which were not significant in this case, considering that the three noise classes were qualitatively defined and large noise variability was observed inside each class. Furthermore, the elapsed time can also be influenced by the number of cells identified in the video. Both factors will be taken into consideration. Finally, in-depth tests of BeatCilia on a larger scale, including clinical experimentation, will be performed.