Mobile-Aware Deep Learning Algorithms for Malaria Parasites and White Blood Cells Localization in Thick Blood Smears

Abstract

Effective determination of malaria parasitemia is paramount in aiding clinicians to accurately estimate the severity of malaria and guide the response for quality treatment. Microscopy by thick smear blood films is the conventional method for malaria parasitemia determination. Despite its edge over other existing methods of malaria parasitemia determination, it has been critiqued for being laborious, time consuming and equally requires expert knowledge for an efficient manual quantification of the parasitemia. This pauses a big challenge to most low developing countries as they are not only highly endemic but equally low resourced in terms of technical personnel in medical laboratories This study presents an end-to-end deep learning approach to automate the localization and count of P.falciparum parasites and White Blood Cells (WBCs) for effective parasitemia determination. The method involved building computer vision models on a dataset of annotated thick blood smear images. These computer vision models were built based on pre-trained deep learning models including Faster Regional Convolutional Neural Network (Faster R-CNN) and Single Shot Multibox Detector (SSD) models that help process the obtained digital images. To improve model performance due to a limited dataset, data augmentation was applied. Results from the evaluation of our approach showed that it reliably detected and returned a count of parasites and WBCs with good precision and recall. A strong correlation was observed between our model-generated counts and the manual counts done by microscopy experts (posting a spear man correlation of $\rho$ = 0.998 for parasites and $\rho$ = 0.987 for WBCs). Additionally, our proposed SSD model was quantized and deployed on a mobile smartphone-based inference app to detect malaria parasites and WBCs in situ. Our proposed method can be applied to support malaria diagnostics in settings with few trained Microscopy Experts yet constrained with large volume of patients to diagnose.

1. Introduction

Malaria is still a global health concern with nearly half of the world’s population at risk. Two hundred and twenty-eight million cases were reported in 2018, and the estimated number of malaria deaths stood at 405,000 [1]. These statistics are majorly a reflection of cases in sub Saharan Africa, but other regions, including South-East Asia, Eastern Mediterranean, Western Pacific, and United States of America, are also at risk.

When diagnosing malaria, the quantitative content of malaria parasites in the blood also referred to as malaria parasitemia is key as it plays a pivotal role in decision support for clinicians with respect to the severity of the malaria, choice of treatment, and probable cause of the disease [2]. However, in highly endemic but low resourced regions, like sub-Saharan Africa, there is lack of sufficient skilled human capacity to efficiently determine malaria parasitemia density of patients. The situation is not made any better by the existing conventional methods for parasitemia determination, like microscopy. This de facto gold standard method for malaria detection and parasitemia determination in most endemic countries involves the manual count of parasites (trophozoites) and White Blood Cells (WBCs) (see in Figure 1) by expert Microscopists [3] (as shown in Figure 1). Counting malaria parasites and WBCs by microscopy method is not only laborious and time consuming but also susceptible to subjectivity by Microscopists [4].

In view of the above microscopy challenges, this research sought to develop an efficient computer vision automated system for identification and quantification of malaria parasitemia and couple the solution with a real time mobile smart phone localization app for malaria diagnosis. Malaria diagnosis automation using a mobile smart phone app for parasitemia determination improves reliability and exactness in interpreting malaria disease severity. It also increases throughput especially in highly endemic but low resourced developing countries.

Current advancements in technologies have made it possible to change the way microscopy of malaria is performed in developing countries. New technologies, like deep learning, can be leveraged with a combination of smart phones to improve disease diagnosis. Given that mobile smartphones are widely owned across the developing world, there is a technological opportunity to address malaria microscopy. Mobile smart phones are now equipped with digital cameras and increased computational power which affords them the potential to capture and process microscopy images. This process is reinforced using pre-trained deep learning models based on transfer learning. Since medical image datasets are usually smaller in size often termed inadequate for learning [6,7], its important to use pre-trained deep learning as a way of utilizing small datasets to produce results that would otherwise be produced by a big dataset. Specifically, transfer learning focuses on sharing knowledge gained while solving one task and applying it to a different but related problem [8]. The learned features can serve as initialization for the another model. In transfer learning, representations learned on a large dataset can be transferred to a model with a totally different yet small data domain [9,10]. Transferring such knowledge from already trained models using big datasets makes computational training of small datasets faster and efficient. Since in this study we use a smaller dataset, the would be challenges are hence cured by using the deep learning models where our small dataset is trained through transfer learning by pre-trained models on already existing big datasets, such as Common Objects in Context (COCO) [11].

In this study, therefore, we propose the use of novel pre-trained deep learning models for a multi-class detection task and Mobile smart phone app for automated localization and quantification of malaria density in thick blood smears. Due to limited datasets, we adopted pre-trained models of Faster Regional Convolutional Neural Network (Faster R-CNN) and Single Shot Multibox Detector (SSD) MobileNet. The Faster Regional Convolutional Neural Network (Faster R-CNN) is known for high accuracy and the Single Shot Multibox Detector (SSD) MobileNet is easily deployable on low core devices, like mobile phones, which we actually employ in our model. A combination of the two methods, therefore, is able to grant us high efficiency and easy deployment on mobile smartphone.

The major contributions of this study are threefold:

We demonstrate that the use of pre-trained deep learning models, particularly Faster R-CNN and SSD MobileNet, in a multi-class detection task and localization of both malaria parasites and WBCs in thick blood smears is efficient. Under this contribution, we extend to validate and compare model performance on a held out dataset with expert level detection performance.

Secondly, we extend the image analysis task to a count of parasites and WBCs and provide an end to end approach for determination of malaria parasitemia based on thick blood smears.

Thirdly, we develop a prototype low core device deployment of image analysis on a mobile smartphone for localization of malaria parasites and WBCs. We believe that our proposed approach can relieve the few available skilled Microscopists the burden of manually counting parasites under the conventional method of malaria microscopy a method that has been faulted for being tedious, time-consuming, subjective and error-prone.

The rest of the paper is organized as follows. In Section 2, we present the related work. In Section 3, details of our proposed method are presented. Experimental results, as well as the discussion, are elaborated in Section 4. In Section 5, conclusions of the study and future work are presented.

2. Related Work

Previous studies have attempted to address the problem of parasitemia estimation for example Sio et al. [12] presented a software, MalariaCount, that automatically generates parasitemias from images of thin Giemsa-stained blood smears. In a subsequent effort, Savkare and Narote [13], presented a technique for determining malaria parasitemia based on a count of Red Blood Cells (RBCs) through a classification task of parasitized and non parasitized cells. They both applied the Otsu thresholding technique on gray image and green channel of the blood image for cell segmentation. Here, watershed-transform was used for separation of touching cells and color. Statistical features were then extracted from segmented cells and SVM binary classifier was used fo classification of normal and infected cells. Poostchi et al. [14] proposed an end-to-end automated detection system for identifying and quantifying malaria parasites (P. falciparum) in thin blood smears of both human and mouse. The authors use a combination of color and texture features to characterize segmented RBCs and a linear Support Vector Machine (SVM) model to classify infected and uninfected cells. These studies have mainly focused on thin blood smear images [15]. However, according to WHO, thick blood films are the gold standard for the determination of malaria parasitemia [16] and provide better detection of parasites than thin blood smears [14]. Thick blood smears contain several layers of red cells, whereas thin films contain a single layer of spread red blood cells. Therefore, for a fixed number of microscope fields, thick films allow the microscopist to examine a larger number of red cells for the presence of parasites, and consequently low parasitemia can be readily identified in thick films [17].

Frean [18] used a semi-automated method for parasitemia determination in thick smears by using an open source java based image processing program ImageJ (version 1.41) for image analysis. In essence, the program segments or classifies particles to be counted on the basis of their relative density (darkness) compared with the background via a thresholding process. For this study, conventional statistical evaluations were done using Statistica 8.0 [18]. The image analysis software used was based on morphological feature extraction with no detection specificity reported. The authors enumerated parasites per image manually by counting parasites on the captured images.

Currently, deep learning models come with an added advantage in that they do not require expertise for the hand engineered features [19]. They have, therefore, aided development of systems for medical image analysis [20]. Yang et al. [5] developed a smartphone application based on intensity-based Iterative Global Minimum Screening (IGMS). The researchers use this method for automatic pre-selection of malaria parasites in thick blood smears and a customized Convolutional Neural Network (CNN) classification model of the parasite. In a separate experiment, the authors localized WBCs using segmentation. However, Customized CNN models have been associated with back and forth fine-tuning of the model layers and huge volumes of annotated data for improved accuracy which may not be available for medical image analysis [6,21]. Moreover, the authors implemented a single-class detection task for mobile deployment.

As opposed to semi-automated and customized CNN models for classification of tasks on malaria parasitemia, this study presents an end-to-end automated pre-trained deep learning approach for detection and counting of malaria parasites and WBCs for improved parasitemia determination in thick blood smears. Unlike the previous study that focused on a single class object detection task for malaria parasites in thick blood smear [22], the pre-trained deep learning approach utilized in this study is a multi-class detection task for parasites and WBCs. It is significantly faster, efficient and requires no additional operator input or pre-processing of the images for automated detection and counts. This is well suited for improving malaria parasitemia quantification in resource-constrained settings with few skilled lab technologists to interpret microscopy test results, as well as facilitating training of the models in a low data regime.

3. Materials and Methods

In order to accomplish our proposed task for automated malaria parasitemia task, we designed an end-to end pipeline for achieving automated localization of trophozoites and WBCs in thick blood smear images as shown in Figure 2. We precisely present the data preparation and labeling procedure as elaborated in Section 3.1, we present the deep learning approach and discuss our choice for the approach in achieving our object detection task in Section 3.2, and then we elaborate the training environment of the selected pre-trained deep learning meta-architecture in Section 3.3. In Section 3.4, we provide model evaluation using standard object detection evaluation matrices of mAP (mean Average Precision), precision, recall, and F1 score. The final objects of interest out of our object detection experiments are localizations of trophozoites and WBCs as elucidated in Section 4. SSD model deployment is extended for mobile phone-based inference, as presented in Section 4.1.

3.1. Data Acquisition and Preparation

To accomplish the localization task in this study, 903 images of thick blood smear with field stain were collected from Mulago National referral hospital in Uganda. Before collection of the image samples, proper permissions were obtained from the people providing the samples, as well as the hospital institution for this data collection. All the data collected was treated in line with the ethical considerations and guidelines governing medical data. Images were captured using a smartphone camera attached to an Olympus microscope with objective magnification (1000×). The attachment mechanism of the smartphone to the eyepiece of the microscope was supported by a 3-D printable adapter [23]. For this experiment, images were taken using a Samsung J6 Android smart-phone camera with a resolution of 5 megapixels to producing images with a dimension of 3264 × 2448 pixels. For each blood slide, the microscope was adjusted to obtain different view point images. Each image was obtained with the corresponding setup parameters including the microscope slide number, stage micrometer grid readings (x and y), zoom level of the smartphone used for image collection, objective size of the microscope, and the staining reagent used. Images of thick blood smears captured with the phone and uploaded to the server, were downloaded and manual data annotations were performed by experienced Lab technicians using the annotation tool LabelImg [24]. The Annotations were in form of bounding boxes drawn around the parasites and WBCs in an image of a blood slide. These annotations were then saved in the Pascal VOC (Visual Object Classes) format [25] and they formed the base data input to our selected models. This Dataset is available at (https://drive.google.com/drive/folders/1p45Dt-BJy8hhoI-rYnhcaL6IMl5FsFL-?usp=sharing).

Preparation of our input data for our selected data analysis models involved decoding of the images and their corresponding XML file annotations from the base dataset into TF (Tensorflow) Recordfiles. The TFRecord record is the data input format for a training job optimized for utilizing the TensorFlow Object detection API (Application Programming Interface). To train and evaluate of our models, the dataset was randomly split into a 9:1 train and test set.

The data pre-processing procedure included conversion of malaria trophozoite and WBC labels to TF-Record format [26] that is compatible with the TensorFlow framework.

3.2. Deep Learning Approach to Malaria Trophozoite and WBC Localization

In medical image analysis, Deep learning has particularly become a choice for methodology in different health fields [20]. Unlike conventional machine learning, deep learning does not require expertise for hand crafted features. Deep learning comprises of multiple processing layers to automatically learn good data representations with multiple levels of abstraction. Therefore, deep learning is promising in a wide variety of applications including malaria parasite detection and detection of WBCs for automated parasitemia determination. This can thus compensate for the limited skilled Microscopists in low but highly endemic settings.

In this study, we employed CNN architecture called Faster R-CNN [27] model and SSD model [28] for the task of localization, detection and counting of the malaria trophozoites and WBCs in thick blood smears. Faster R-CNN has displayed good performance in different object detection tasks [29,30,31], including a similar study for detection of malaria parasites [22], while SSD MobileNet provides opportunities for mobile smartphone deployment, a feature that was important for the choice of this model.

Faster R-CNN, is an improvement to the Region based CNN model (R-CNN) whose detection procedure occurs over two stages. The first stage uses Region Proposal Networks (RPNs) to extract features using selective search purposefully to generate proposals (boxes) which constitute feature maps that are later used by the neural network to check the occurrence of objects. The idea is to reduce the search for relevant features related to the objects of interest in the dataset [27]. In the second stage, bounding box proposals obtained in the previous stage are further cropped and processed. This combination helps Faster R-CNN to have a leading performance on accuracy though leads to a two-stage architecture network which reduces the speed of processing of this method [32]. In this study, we thus implement fully convolutional Residual Network(ResNet 101) [33] as the backbone network for Faster R-CNN. Unlike training from scratch, we exploited transfer learning by using the Faster R-CNN ResNet 101 model pre-trained on the COCO (Common Objects in Context) dataset [11] to achieve faster training coupled with high accuracy.

Single Shot Multibox Detector(SSD) [28] was also implemented in our experimentation. SSD relies on a feed-forward convolutional network to produce a fixed-size collection of bounding boxes and scores for object presence in those boxes. Finally, through a non-maximum suppression step, detections are produced. The Single Shot Multibox Detector (SSD) MobileNet architecture model is generated from depth-wise convolution. A single filter is applied to each neural input channel to begin feature extraction, a 1 × 1 point-wise convolution combines the outputs of the depth-wise convolution. Resultant from a dept-wise convolution are two layers generated, a separate layer for filtration and a layer for combination. According to Howard [34], the combination of the two layers minimizes the model size to create room for reducing computational power demands. Furthermore, model efficiency is enhanced through depth-wise convolution by preventing Graphical Processing Unit (GPU) over consumption on less demanding devices such smartphone platforms. However, the low GPU consumption also creates a lack of usage equilibrium, which hinders the training model causing slow progress and intervals. In this study, we implemented SSD MobileNet V2 model because of its efficiency for low core deployment on mobile smartphones.

3.3. Training Approach

All the proposed models in our experiments were trained and tested on Ubuntu system with a 5th Gen Intel Core processor i7 and 16 GB RAM, a GPU-enabled Nvidia GTX 1060 Graphical Processing Unit (GPU) with 6 GB RAM, Python 3.5, with Tensorflow back-end. Training the models was done using the open-source library Tensorflow [26] on a TFrecord generated from a training dataset of 803 images. To train and test model, we employed the implementation of Faster R-CNN ResNet 101 and SSD MobileNet V2 provided by the Google’s open-source Tensorflow object detection API [35]. Following a transfer learning Scheme, pre-trained models were used to initialize the weights of the previously learned model to adapt to a new task [36]. Transfer learning has the benefit of decreasing the training time for a neural network model and can result in lower generalization error. Faster-R-CNN and SSD models used were pre-trained on the Microsoft COCO [11] dataset that has a lot more labeled data than our limited size of dataset. In our experiments, to fine-tune and share the connection weights between a previously learned model on COCO dataset to our dataset, all layers except the fully connected layer were re-trained without freezing and the weights of the pre-trained models were used as initial weights.

To increase the sample size and variation, data augmentation was applied to optimize and obtain better performance. Augmentation of training data was achieved through random vertical and horizontal flipping of images through the TensorFlow object detection API.

Additionally, the TensorFlow Object detection API enables specification of different training options and parameters by editing a configuration file.

Table 1. Training parameters used for each of the models.

Parameter	Faster R-CNN ResNet101	SSD Mobilenet
Learning rate	0.0003	0.0004
Batch size	1	12
Input image size (pixels)	750 × 750	300 × 300
Momentum optimizer	0.9	0.9
IoU threshold	0.5	0.5

shows our hyper-parameter settings. We set a small learning rate (lr) to work well with the other selected hyper-parameters. A base lr was kept very low at 0.0003 and was later reduced to 0.00004 for Faster R-CNN and SSD, respectively.

For training of machine learning models, having a large batch size like one we used for SSD MobileNet (light model) is usually desirable, although, for experiments using Faster R-CNN, a batch size of 1 was used. This was inevitably due to model’s complex architecture and the limited memory resources that were available for us to execute the training job. Subsequently, We chose momentum optimizer over other optimization algorithms because it helps move more quickly towards the minimum, builds speed, and quickens convergence [37]. To avoid fluctuations associated with smaller optimizer beta values (less than 0.5), we used 0.9 beta value for the momentum optimizer to achieve a more smoother curve.

To fine-tune the models to detect the spatial features of the objects of interest (parasites and WBCs), all models were trained for 200,000 time steps though each model had its own level of attained optimal performance(lowest loss and highest mAP) at respective steps (see

Table 2. [email protected], precision, and recall performance for Faster Regional Convolutional Neural Network (R-CNN) model for detection of malaria parasites and WBC.

Algorithm	Steps	[email protected]	Precision	Recall	F1 score
Faster R-CNN (trophozoites_only)	138,700	0.5506	0.6720	0.802	0.7312
Faster R-CNN (WBC_only)	42,800	0.892	0.855	0.959	0.9039
Faster R-CNN (trophozoites + WBC)	98,700	0.6609	parasites: 0.686 WBC: 0.805	parasites: 0.9303 WBC: 0.884	parasites: 0.7897 WBC: 0.8426
SSD MobileNet (trophozoites + WBC)	80,000	0.6292	parasites: 0.760 WBC: 0.89	parasites: 0.501 WBC: 0.806	parasites: 0.6038 WBC: 0.7528

). These are steps at which the model converges with early stopping. It was realized that SSD MobileNet trains for fewer steps in comparison to Faster R-CNN model. SSD is simpler than other networks as it performs all computations in a single network. SSD combines the predictions from numerous feature maps having different resolutions to handle objects of various sizes. It does not involve regional proposal generating or feature re-sampling, like previous networks did. This makes it easy to train and integrate into systems where detection is required [38]. The training processes were monitored on TensorFlow’s visualization toolkit, TensorBoard to understand all variations that occurred during training.

Once training was accomplished, a set of checkpoint files comprising of learned features were generated from the training dataset. The checkpoint files were frozen into a protobuff file (a Tensorflow format for standard representation of machine learning models).

Using the same setup, different experiments were conducted to investigate model performance with respect to a single-class object detection task (that is, only malaria trophozoites detection and only WBCs detection) and multi-class object detection (both malaria trophozoites and WBCs).

3.4. Model Evaluation

The study evaluated the trained models on a validation dataset of 100 thick blood smear images. To clearly understand how well a model performed in accomplishing the goal of counting trophozoites and WBCs on any validation set image, the TensorFlow object detection API was used. It handled the evaluation of a trained model using the standard COCO evaluation metrics [11]. We adopted the standard evaluation metrics for an object detection task. Specifically, the common metrics of mAP(mean Average Precision) [39], precision, recall, and F1 Score were used.

In order to use mAP [39] as a standard evaluation metric for the object detection task, the performance was first evaluated in terms of Intersection-over-Union (IoU) as implemented in the Pascal VOC Challenge [40]. The precision of a single detection can be measured using IoU as defined in Equation (1).

$$IoU(A,B)=|\frac{A\cup B}{A\cap B}|,$$

(1)

where A represents the ground-truth box collected in the annotation, and B represents the predicted result of the transfer learning model. The IoU measures the percentage or ratio of overlap of the predicted bounding box to the ground truth boxes [39]. mAP is calculated by getting the average IoU for all detections made and in our experiments, we considered an IoU threshold of 0.5. The lower bound of the IoU threshold (0.5) reflects the detection accuracy based on bounding box coverage characteristics of each detected object of interest.

For a single class in a single image, Precision is a measure of the ratio of the true positives detected by the model to the total number of detections made as defined in Equation (2):

$$Precision=\frac{TP}{TP+F{P}^{\prime }}$$

(2)

where TP (True Positives) represents the number of true object detections, and FP (False Positives) represents wrongly detected objects. On the other hand, Recall measures how many true detections were identified by the model as defined in Equation (3):

$$Recall=\frac{TP}{TP+F{N}^{\prime }}$$

(3)

where TP (True Positives) represent Number of true object detections, and FN (False Negatives) represent true undetected objects.

In this study, F1 Score was also used to provide a better understanding of the overall model performance for different experiments conducted. Precision and Recall can be combined to generate the F1 Score as defined in Equation (4):

$$F1Score=\frac{2\times (Precision\times Recall)}{Precision+Recall}.$$

(4)

4. Results

Results from our training experiments show that initially, mAP increases fast and reaches an optimum for the number of steps different for each model as indicated in Table 2. Figure 3 and Figure 4 present the graphical representation of the resultant mAP and loss curves of Faster R-CNN and SSD MobileNet.

Figure 3 and Figure 4 show that Faster R-CNN model achieves better accuracy (in terms of mAP and F1 score) than SSD model for localizing both parasites and WBCs as expected [41]. To visualize the results, model performance inference detections on a sample test image are shown in Figure 5. Again, We observed from the depiction that Faster R-CNN (b) performs better than SDD MobileNet model (SSD (c)).

A general representation of our experimental results is summarized in Table 2.

Table 2 shows that a multi-class detection task of both malaria trophozoites (parasites) and WBCs produces better results than a single-class task (trophozoites only), but lower than a single-class of WBCs (mAP = 0.892). This is attributed to the fact that WBCs are bigger objects and as such are easier examples to learn as compared to smaller objects (trophozoites only). The better performance of the multi-class task is attributed to having bigger objects complementing small objects (trophozoites). The multi-class mAP value of 0.6609 (@0.5IOU) was considered to be good enough performance because of the presence of small objects(trophozoites).

We notice that the recall value generated by SSD MobileNet is lower than that of Faster R-CNN to a great extent. This could be attributed to the fact that using more complex (two-stage) object detectors, such as Faster R-CNN, for difficult small images(parasites), provides better optimization than less complex (single-stage) detectors (SSD) [42]. We show that Faster R-CNN ResNet 101 model produces better detection accuracy for our proposed multi-class detection task for trophozoites (F1 score = 0.7897) and WBCs (F1 score = 0.8426) than its variant SSD MobileNet v2 model for detection of trophozoites (F1 score = 0.6038) and WBCs (F1 score = 0.7528).

Furthermore, in Figure 6, Faster R-CNN model-generated parasite counts are higher than both SSD and manual-expert counts, which shows that Faster R-CNN models perform better at identification of parasites and WBCs in comparison to SSD and experts. The performance disparity with respect to the manual expert count could potentially signify the effect of false-positives through low quality annotations, thus a lower precision registered for parasites. We next analyze the magnitude of the disparity between Faster R-CNN and manual expert count. Results generated show that Faster R-CNN model-generated counts correlate well with manual-expert counts on a held-out dataset of 8 thick blood smear films with more than 100 images each (high Spearman’s correlation coefficients for trophozoites ($\rho$ = 0.998) and WBCs ($\rho$ = 0.987)), as presented in Figure 7. In general, the precision, recall, and F1 score of Faster R-CNN multi-class detection task were comparably higher than SSD.

Following from

Table 3. Malaria parasitemia on a unique set of each validation thick blood smear slide with corresponding clinical interpretation.

Slide	Faster R-CNN Parasites/$\mathsf{\mu }$L	SSD Parasites/$\mathsf{\mu }$L	Manual Parasites/$\mathsf{\mu }$L
Slide${}_1$	2326	1028	485
Slide${}_2$	5740	4174	5768
Slide${}_3$	3160	1958	1638
Slide${}_4$	4682	2535	2783
Slide${}_5$	3011	1808	2195
Slide${}_6$	7238	5007	5685
Slide${}_7$	1906	1081	1483
Slide${}_8$	47,982	57,404	30,502

, film-wise parasitemia density based on Parasites/$\mathsf{\mu }$L method per film [43,44] was evaluated through the Equation (5):

$$Parasites/\mathsf{\mu }\mathrm{L}=\frac{\mathrm{No}.\mathrm{of}\mathrm{counted}\mathrm{parasites}\times \mathrm{No}.\mathrm{of}\mathrm{counted}\mathrm{WBC}}{\mathrm{No}.\mathrm{of}\mathrm{assumed}\mathrm{WBC}\left(8000\right).}$$

(5)

Malaria parasitemia clinical interpretations [45] were deduced as shown in Table 3. Slides 1, 2, 3, 4, 5, 6, and 7 represent parasitemia level above which immune patients will exhibit malaria symptoms whereas Slide 8 represents a patient with maximum parasitemia.

4.1. Deployment of SSD MobileNet V2 on a Smartphone

It should be noted that although the mAP, precision, recall, and F1 score values are lower than that of Faster R-CNN model, a quantized SSD MobileNet model can easily be deployed on low cost device operating systems, such as Android. Quantization of the model reduces the size of the model through model compression to allow models run faster on low cost devices. This, however, comes with a trade-off in performance.

The TensorFlow Lite converter (see Figure 8), a Python API was used to convert our pre-trained TensorFlow model (.pb file format) into the TensorFlow Lite format (.tflite). TensorFlow Lite has capability for achieving low latency, optimizing the kernels for mobile deployment and pre-fused activation to allow smaller and faster models [46].

The application was developed in Flutter using the tflite package and a pre-trained quantized SSD-MobileNet model for localizing malaria parasites and WBCs in images and real-time camera streams. Images can be taken from within the app and are then fed to the detection model.

The image-picker was used for picking images from the gallery, tflite for running our model, and Flutter’s camera package for controlling the camera. Once the app is launched, camera descriptors are passed on to all classes that require it. To detect objects in images, we first load the model using the Tflite.loadModel method available in the tflite package. We then retrieve images from the phone gallery or take them from the app camera. On loading the image, we feed it into the model, which returns the detected class, confidence, and points of interest that will help us draw bounding boxes around the objects (in our case trophozoites and WBCs). The basic idea of drawing bounding boxes around the detected object is to use the model’s POIs (Point Of Interests) and the image dimensions (see APK (Android Package Kit) at (https://drive.google.com/file/d/1GmWMyQ_IZnxCZQbmW7g4QUa2OputhLl4/view?usp=sharing)). All mobile smartphone model deployment experiments were performed on a Samsung j6 Android smartphone camera operating at a resolution of 3264 × 2448 pixels (see inference in Figure 9).

5. Conclusions and Future Work

In this study, we have provided an end-to-end machine learning approach for localization of trophozoites and WBC in thick blood smears for the determination of malaria parasite density and integrate the model into a mobile smartphone detection App. We present a low-cost and reliable mobile phone malaria app diagnostic solution that is relevant to highly endemic low resourced settings. We leverage transfer learning and propose a Faster R-CNN ResNet 101 model and SSD MobileNet V2 as the on-device inference approach. All models were pre-trained on COCO dataset for the task of object detection. We show that our proposed models can accomplish parasite and WBC localization and count effectively in a fast, accurate and consistent way for malaria parasitemia determination.

Furthermore, we evaluated and compared our proposed model based on a multi-class task for detection of both malaria trophozoites and WBCs with performance from single-class tasks for detection of trophozoites (only) and WBCs (only) using the same model. The mAP performance of a multi-class task (0.6609) is better than that of a single class of trophozoite detection (0.5506) and thus a higher precision and recall but lower than that of a single-class of WBCs (0.892). It is expected that models perform better with bigger images, like WBCs, compared to smaller objects (trophozoites only). In fact, SSD MobileNet performed better than Faster R-CNN on bigger objects (WBCs). The better performance of the multi-class task is attributed to having bigger objects (WBCs) complementing small objects (trophozoites). Generally, the Faster R-CNN model for a multi-class task achieved a mAP value of 0.6609 (@0.5IOU) with F1 score of (parasites: 0.7897 and WBC: 0.8426) providing better model performance in comparison to SSD MobileNet model with a mAP value of 0.6292 (@0.5IOU) and (parasites: 0.6038 WBC: 0.7528) as expected [32,41]. SSD provided capacity for mobile smartphone deployment. Conclusively, therefore, different models can be chosen according to the actual needs with respect to accuracy versus low-core device model deployment.

In future work, we intend to work towards reducing false positives and false negatives generated by the model. This could be through acquiring a bigger dataset with quality data annotation. In its current form, the mobile smartphone app still outputs false alarms. Model improvement is still needed in respect to reduction of false alarms and accuracy improvements.