Next Article in Journal
Flexible Risk Evidence Combination Rules in Breast Cancer Precision Therapy
Previous Article in Journal
Molecular Mechanisms, Genotype–Phenotype Correlations and Patient-Specific Treatments in Inherited Metabolic Diseases
Previous Article in Special Issue
When to Perform a Colonoscopy in Diverticular Disease and Why: A Personalized Approach
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Semi-Supervised Segmentation Framework for Gastrointestinal Lesion Diagnosis in Endoscopic Images

1
Center for Informational Biology, University of Electronic Science and Technology of China, Chengdu 610054, China
2
School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 610054, China
3
School of Public Health, College of Health Sciences and Medicine, Dilla University, Dilla P.O. Box 419, Ethiopia
4
Department of Computer Engineering, Kwame Nkrumah University of Science and Technology, Kumasi AK-039-5028, Ghana
5
Digestive Endoscopic Center of West China Hospital, Sichuan University, Chengdu 610017, China
*
Author to whom correspondence should be addressed.
J. Pers. Med. 2023, 13(1), 118; https://doi.org/10.3390/jpm13010118
Submission received: 29 September 2022 / Revised: 20 November 2022 / Accepted: 23 November 2022 / Published: 5 January 2023

Abstract

:
Background: Accurate gastrointestinal (GI) lesion segmentation is crucial in diagnosing digestive tract diseases. An automatic lesion segmentation in endoscopic images is vital to relieving physicians’ burden and improving the survival rate of patients. However, pixel-wise annotations are highly intensive, especially in clinical settings, while numerous unlabeled image datasets could be available, although the significant results of deep learning approaches in several tasks heavily depend on large labeled datasets. Limited labeled data also hinder trained models’ generalizability under fully supervised learning for computer-aided diagnosis (CAD) systems. Methods: This work proposes a generative adversarial learning-based semi-supervised segmentation framework for GI lesion diagnosis in endoscopic images to tackle the challenge of limited annotations. The proposed approach leverages limited annotated and large unlabeled datasets in the training networks. We extensively tested the proposed method on 4880 endoscopic images. Results: Compared with current related works, the proposed method validates better results (Dice similarity coefficient = 89.42 ± 3.92, Intersection over union = 80.04 ± 5.75, Precision = 91.72 ± 4.05, Recall = 90.11 ± 5.64, and Hausdorff distance = 23.28 ± 14.36) on the challenging multi-sited datasets, confirming the effectiveness of the proposed framework. Conclusion: We explore a semi-supervised lesion segmentation method to employ the full use of multiple unlabeled endoscopic images to improve lesion segmentation accuracy. Experimental results confirmed the potential of our method and outperformed promising results compared with the current related works. The proposed CAD system can minimize diagnostic errors.

1. Introduction

Gastrointestinal (GI) tract cancers affect the human digestive system, resulting in one of the most critical healthcare problems [1,2]. According to report [3,4], esophageal cancer, colorectal cancer, and stomach cancer are the three most common GI cancers with the highest incidence and mortality rates [5,6]. Endoscopy is the primary method for the examination of the GI tract; gastroscopy examines the upper GI tract, whereas colonoscopy examines the bowel and rectum [4,7]. Endoscopic examinations typically require expensive, highly standardized equipment and expertise.
Nowadays, automatic segmentation of lesions and anatomical structures in biomedical imaging has led to improvements in accurately diagnosing medical conditions. Advances in deep learning (DL), particularly convolutional neural networks (CNNs), have resulted in significant progress in several vision-related tasks [8,9,10], frequently attaining human-level competitiveness in recognizing lesions in tissues and delineating heart structures [11], and segmenting colorectal polyps [10] and esophageal cancer [7] in GI diseases [12,13]. The success of DL usually depends on the availability of massive labeled high-quality datasets [11]. Accordingly, several deep supervised segmentation models are implemented in fully supervised approaches such as U-Net [14], FCN [15], and UNet++ [16]. However, it is difficult and costly to obtain annotated data in the biomedical imaging environment [17]. Furthermore, even where data are available, there is the need for domain experts to manually produce annotations: a process that is often tedious and impractical on large scale.
One way of addressing this limitation is to use a semi-supervised learning (SSL) approach. SSL is a hybrid of supervised and unsupervised learning methods where a model is trained on a large dataset where only a small volume of the data is annotated. Typically, these data will be the targets linked with some samples [18]. Thus, by reducing the need for annotating large volumes of medical images, an efficient SSL framework offers an attractive alternative to supervised DL approaches in automatic segmentation applications in biomedical imaging. To effectively validate the quality of a SSL method, a few studies [19,20,21] have applied a generative adversarial network (GAN) [18]. GANs comprise a generator network and a discriminator network [18,20].
The lack of sufficiently labeled images for classical DL techniques means that there is the need for SSL techniques that (1) require little supervision [20] and (2) can self-evaluate in an unsupervised manner [22,23]. In this work, we propose a robust SSL framework for training deep models with small labeled training samples. Furthermore, we utilize adversarial generative modeling similar to [23] for GI lesion segmentation tasks [21].
Inspired by [20], this work introduces an adversarial-based semi-supervised learning approach into a densely connected CNN for GI automatic lesion segmentation. The key contributions of this work are summarized as follows:
(1)
We propose a GAN-based semi-supervised GI lesion segmentation framework that uses reasonably small labeled endoscopic images.
(2)
We demonstrate a full use of numerous unlabeled GI datasets to improve lesion segmentation accuracy.
(3)
The proposed framework was tested on five multi-sited datasets from different centers and integrated the predicted result to improve the segmentation performance through generative adversarial training.
(4)
The proposed method outperforms baseline supervised segmentation models as well as other related semi-supervised segmentation frameworks.
The next section of this work is organized as follows. In the following section, we describe the materials and methods of our proposed GAN-based semi-supervised framework. Section 3 deals with the experimental setup. In the consecutive section, we describe the results and discussion and conclude in Section 5.

2. Materials and Methods

2.1. GI image Datasets

We collected 4880 GI images in total from a private hospital and publicly available sources to evaluate our method. The details of each dataset are as follows:

2.1.1. West China Hospital Digestive Endoscopy Center Dataset

We employed 2112 GI images of 484 patients collected and verified by gastroenterologists from the Digestive Endoscopy Center of the West China Hospital in Sichuan, China. The images were stored as RGB color channel in JPEG format. Original images were captured at a resolution of 384 × 384 pixels. Lesion areas were marked and labeled as ground truth (GT) by gastroenterologists. The GT images are stored as black and white binary images and also in JPEG format. From 2112 GI images, only 192 images were GT, and eighty percent of the datasets were used for training with the remainder used for testing. Approval from the medical ethical review committee of the University of Electronic Science and Technology of China (UESTC) and West China Hospital and informed patients’ consent were obtained.

2.1.2. Public Datasets

To ensure a robust framework, we used four related public datasets. Firstly, CVC-ClinicDB [24] data were used to train and validate the network. CVC-ClinicDB is a database that contains 612 images of 384 × 288-pixel spatial resolution. Secondly, the ETIS-LaribPolypDB [25] consisting of 196 polyp images from 34 different video sequences with labeled GT images of size of 1225 × 966 pixels was used. The third dataset was collected from the endoscopy artifact detection (EAD2019) [26] challenge for semantic segmentation with seven different classes (specularity, artifact, saturation, contrast, bubbles, blur, and instrument). The images were in different resolutions, including 1920 × 1080, 1349 × 1079, and 295 × 299 in JPG format. From 2622 images, only 960 related images with lesions were used for semi-supervised training. The fourth dataset used in this study was the Kvasir-SEG dataset [27], which has one thousand polyp images and their corresponding GT. The resolution of the images varies from 332 × 487 to 1920 × 1072 pixels which are encoded in JPEG. Samples of raw images are shown in Figure 1.

2.2. Methods

This study discusses an SSL method based on adversarial generative training techniques for GI lesions segmentation. Our framework uses a weighted amalgamation of losses.
The general framework of the semi-supervised GI lesion segmentation is divided into four main consecutive steps, with the output of each step being the input to the subsequent step. The main detailed steps are explained as follows.
Step 1: Preprocessing
In the first stage, unnecessary background and artifacts were removed from each original dataset denoted đ. Moreover, due to the varying nature of training images sizes, all training images were resized into 192 × 192 to fit the input model we used. All the resized RGB images were converted into a PNG file during Step 1.
Step 2: Supervised Training
The preprocessed dataset from Step 1, denoted đ1, was used to train the network in this step in a supervised manner. The outputs of Step 2, i.e., the model weight from supervised learning, denoted đ2, and the unlabeled datasets, denoted đ4,, were forwarded to Step 3 for semi-supervised training.
Step 3: Adversarial Training
As shown in Figure 2, the input to the semi-supervised module consists of the unlabeled dataset, đ4, and the pre-trained model, đ2. The segmentation probability map is generated by adversarial training in Step 3. The output, đ 6, will be reserved in Step 4, for further evaluation. Subsequently, the EN evaluates the input’s segmentation quality, which comprises the GT from the labeled images, and the predicted output from labeled and unlabeled images. The proposed network can accomplish adversarial learning between the SN and EN based on the test scores. Through adversarial training, the predicted maps generated from SN can be close to the GT. Figure 3 shows our semi-supervised network architecture which comprises SN and EN. Initially, the SN is trained with labeled data in a supervised learning fashion.
Step 4: Testing Model
In Figure 3, the EN requires two inputs (i.e., GI images and the corresponding region of interest (ROI) segmentation maps). The ROI segmentation is fed with GT for testing for the labeled images. Finally, the performance results are determined by the EN.

Loss Function

The loss function of the network is defined as in [9]. Both the labeled GI image X ı and the unlabeled GI images X u are of size H   ×   W . The segmentation and the evaluation networks are denoted by S e g (•) and Ev (•), respectively. GT images labeled by the physicians are symbolized as Y ι . Y ι is of size H ×   W and incorporates two channels and Ŷ denotes the predicted maps. The model is implemented by reducing the following loss function as follows:
S = s e g + λ adv adv
where seg denotes the supervised loss and adv adversarial loss. λ adv denotes the weight of the adversarial learning model. The loss function L seg regulates whether the predicted probability generated from the input labeled image is similar to the GT, and it is formulated as:
L s e g ( X ı , Y u ;   θ S ) = L d c e ( S e g ( X ı ) ,   Y u )   + L b c e ( S e g ( X ı ) ,   Y u )
where θ S , L b c e and adv denotes the parameters of the SN, binary cross-entropy loss (bce), and adversarial loss calculated to approximate the performance from either the labeled or unlabeled images. The adv is expressed as:
adv ( X ı , X u ; θ S ) = λ ı   . L b c e ( Ev X ı , S e g X ı , 1 + λ u . L b c e ( Ev ( X u ,   S e g X u , 1
E = ( X ı ,   X u ,   Y ı ;   θ E ) = L b c e ( Ev X ı ,   Y ı , 1 + λ ı   . L b c e Ev X ı ,   S e g X ı , 0 + λ u   . L b c e Ev X u ,   S e g X u , 0
where θ E represents the parameters of the EN. λ ı and λ u correspond to the loss coefficients of the labeled and unlabeled data, respectively.

3. Experimental Setup

3.1. Implementation Details

We implemented the BCE loss during the supervised training with stochastic gradient descent (SGD) [28] optimizer at a learning rate (LR) of 0.001. The semi-supervised loss is to train the semi-supervised models, and SN is trained with SGD with a LR of 0.001 and 0.0001 weight decay. The networks are optimized with Adam optimizer at an initial learning rate of 0.001, batch size 16, and 4000 epochs. The network was implemented in Python 3.6.4 and Tensorflow (https://www.tensorflow.org (accessed on 10 June 2021)) [29]. Experiments were implemented on Ubuntu 16.04.6 LTS (server-based), and the system was equipped with four GPUs of NVIDIA GeForce RTX 2080Ti with 11 GB memory each.

3.2. Evaluation Metrics

The following five pixel-level evaluation metrics are utilized to compare the segmentation performance of the proposed method. Labeled datasets by physicians were used as GT; five different evaluation metrics were utilized to evaluate the performance, namely: Dice similarity coefficient (DSC), intersection over union (IOU), precision (Pre), recall (Rec), and Hausdorff distance (HDist). DSC is the most common metric for comparing the predicted segmentation and GT. IOU is a typical metric to measure a prediction, and it calculates the overlaps between the prediction and its corresponding GT [30]. HDist measures how far the prediction and the GTs are from each other. The lower the HDist is, the better the image difference, and a closer HDist to zero indicates a better image [9]. DSC, IOU, Pre, and Rec were used to validate the overlap between the GT and prediction. HDist was used to evaluate the distance between a model prediction (P) lesion boundary and the GT. The performance indicators implemented are shown below:
P r e = T P T P + F P
P r e = T P T P + F P
D S C   A , B = 2 × A B A + B = 2 × T P 2 × T P + F P + F N
I O U   A , B = A B A B = T P T P + F P + F N
where TP, FP, and FN represent the number of true positive pixels, false-positive pixels, and false-negative pixels in segmentation regions. A   and   B represent the ratio between the overlapped area A B over the total area A B corresponded by the two boxes.
H D i s t   A , B   max a A   { min b B   d a , b }
where a and b are points of A (GT contour) and B (predicted contour), respectively; d(a, b) is a metric between these points. In this case, we take d(a, b) as the Euclidian distance between a and b.

4. Results and Discussion

In this part, we describe performance comparisons for baseline network trained in a supervised fashion and then the semi-supervised technique for adversarial training. Fixed images with GT were utilized for all unlabeled images and trained adversarially for lesion segmentation.

4.1. Comparisons Using the Limited Labeled GI Datasets

Table 1 shows a comparison of our model performance on four GI datasets. To confirm the proposed approach’s effectiveness, we employed a limited GT for each dataset in a supervised learning fashion. Finally, the proposed method had promising results in all datasets’ differences. Different GI datasets prepared by different physicians were confirmed for the next level. Kvasir-SEG dataset, Ref [27] in terms of the mean value and SD (DSC = 84.65 ± 18.09, IOU = 75.14 ± 18.39, Pre = 86.0 ± 18.81, Rec = 85.0 ± 19.30, and HDist = 35.14 ± 20.63) respectively, achieved better results than other datasets. This was a probability of the total number of images used during training. A higher recall (Rec = 91.20 ± 8.45) result was performed on the ETIS dataset [25]. The average value of HDist (HDist = 30.02 ± 19.11) on our dataset showed fewer distance results than other datasets.

4.2. Supervised Learning Comparisons

After preprocessing, we compared the baseline framework with two state-of-art works: U-Net [14] and UNet++ [16]. Each experiment was run with fixed 192 (from our dataset) labeled GI images. Input images were resized in 192 × 192 and the same evaluation metrics were utilized for all models. Three models demonstrated competitive results, as shown in Table 2. The baseline achieved a better mean value on three metrics (DSC = 82.15 ± 10.22, Pre = 84.22 ± 15.94, and HDist = 32.09 ± 26.44). The pre-trained results were utilized for further semi-supervised learning.

4.3. Semi-Supervised Learning Comparisons

We compared our method with two recent related deep adversarial learning-based methods on our dataset to validate the segmentation accuracy. The two comparison methods were DAN [22] and GAN [20]. To ensure the best segmentation results of each method and ensure comparativeness, the input size of the data was set as 192 × 192. The dataset was split into 192, 384, 768, 960, and 1920, respectively. Due to the unavailability of 1920 images from the EAD2019 dataset [26], only the lab dataset was employed for the final training. Using a fixed labeled dataset with GT for all, the rest of the unlabeled dataset was randomly selected for the proposed model. The learning curve of training and validation performance of the proposed method is presented in Figure 4. The proposed model achieves better results (DSC = 89.42 ± 3.92, IOU = 80.04 ± 5.75, Pre = 91.72 ± 4.05, Rec = 90.11 ± 5.64, and HDist = 23.28 ± 14.36). Almost all models achieved lower performance results on the EAD2019 dataset [26].
Moreover, our model can achieve better GI lesion accuracy on all testing datasets. Table 3 indicates the effectiveness of our method compared with other related works. As shown in Figure 5, qualitative results also confirmed the competitive effects of the models. Accurate segmentation of ROI from GI images is essential for CAD procedures. However, due to the variability of GI lesions, it is challenging to advance accuracy with limited annotated data. The proposed GAN–based model can leverage unlabeled data to obtain better accuracy than related models [18,22]. This indicates the proposed method is applicable for lesion segmentation.
Extensive experimental results confirm that the proposed method can produce promising segmentation maps. Usually, GI images’ appearance differs across generating equipment, with an effect on GI lesion [31]. However, the proposed GAN-based approach achieves more promising results than the state-of-the-art methods, indicating the proposed method’s effectiveness on four datasets. Furthermore, the proposed model can aid clinical annotation tasks by reducing medical errors. Thus, accurate segmentation can help physicians and improve the robustness of GI lesion’s automatic diagnosis.

4.4. Comparison of Baseline and Proposed Models

We have performed several experiments with initial validation using two different methods. The first approach is based on a supervised way that includes U-Net [14] convolutional biomedical image segmentation and UNet++ [16], as shown in Table 2. The second method is based on a semi-supervised method utilizing the generator and discriminator network. We compared the proposed method’s performance with DAN [22] and GAN [20] in the second primary method.

5. Conclusions

This paper proposed an adversarial learning-based semi-supervised segmentation framework for GI lesion diagnosis in endoscopic images. Automatic segmentation of lesions from endoscopic images is crucial to assist physicians in GI digestive tract disease diagnosis. We conducted intensive experiments using five different datasets and test data evaluated using five evaluation metrics. We explore a semi-supervised lesion segmentation method to employ the full use of multiple unlabeled images to improve lesion segmentation accuracy. Extensive experimental results demonstrated the potential of our method and outperformed other related works. A proposed CAD system can assist physicians and minimize diagnostic medical errors. Improving the approaches with more robustness and generalizability to other related semi-supervised learning tasks will be the basis of future work.

Author Contributions

Conceptualization, Z.M.L. and N.R.; methodology, Z.M.L., N.R. and P.E.A.; software, D.H.; validation, W.D. and T.G.; formal analysis, Z.M.L.; investigation, C.L.; resources, N.R., T.G. and L.Z.; data curation, T.G., L.Z. and D.H.; writing—original draft preparation, Z.M.L.; writing—review and editing, Z.M.L., P.E.A. and N.R.; visualization, W.D.; supervision, N.R. and C.L.; project administration, N.R. and C.L.; funding acquisition, N.R. All authors have read and agreed to the published version of the manuscript.

Funding

This study was supported by the National Natural Science Foundation of China (Grant No. 62271127, 61872405, and 61720106004), Sichuan Natural Science Foundation (23NSFSC0437) and the Medico-Engineering Cooperation Funds from the University of Electronic Science and Technology of China (No. ZYGX2022YGRH011 and HXDZ22005).

Institutional Review Board Statement

This study was conducted following the ethical standards of the institutional (IRB) and/or research committee and the 1964 Helsinki declaration and its latter amendments or comparable ethical standards.

Informed Consent Statement

All approaches were performed by following the regulations and relevant guidelines.

Data Availability Statement

In this study, we used two primary data sources (hospital and public), available in references. Except for the hospital dataset, which will be available upon request made to a corresponding author due to data privacy restrictions, the rest of the public datasets are available at https://datasets.simula.no/kvasir/ (accessed on 10 March 2022); https://polyp.grand-challenge.org/CVCClinicDB (accessed on 16 February 2021); https://doi.org/10.1007/s11548-013-0926-3 (accessed on 17 February 2021); https://doi.org/10.17632/C7FJBXCGJ9.1 (accessed on 2 April 2022), and https://ead2019.grand-challenge.org (accessed on 2 April 2022).

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

CADcomputer-aided diagnosis
CNNconvolutional neural network
DLdeep learning
DSCdice similarity coefficient
ENevaluation network
FCNfully connected network
GIgastrointestinal
GTground truth
HDisHausdorff distance
GANgenerative adversarial network
LRlearning rate
Preprecision
Recrecall
ROIregion of interests
SNsegmentation network
SSLsemi-supervised learning
SDstandard deviation
SGDstochastic gradient descent

References

  1. Du, W.; Rao, N.; Liu, D.; Jiang, H.; Luo, C.; Li, Z.; Gan, T.; Zeng, B. Review on the Applications of Deep Learning in the Analysis of Gastrointestinal Endoscopy Images. IEEE Access 2019, 7, 142053–142069. [Google Scholar] [CrossRef]
  2. Escobar, J.P.; Gomez, N.; Sanchez, K.; Arguello, H. Transfer Learning with Convolutional Neural Network for Gastrointestinal Diseases Detection Using Endoscopic Images. In Proceedings of the 2020 IEEE Colombian Conference on Applications of Computational Intelligence (IEEE ColCACI 2020), Cali, Colombia, 7–8 August 2020. [Google Scholar] [CrossRef]
  3. Pogorelov, K.; Randel, K.R.; Griwodz, C.; Eskeland, S.L.; De Lange, T.; Johansen, D.; Spampinato, C.; Dang-Nguyen, D.T.; Lux, M.; Schmidt, P.T.; et al. Kvasir: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection. In Proceedings of the 8th ACM Multimedia Systems Conference, MMSys, New York, NY, USA, 20 June 2017; pp. 164–169. [Google Scholar] [CrossRef] [Green Version]
  4. Bray, F.; Ferlay, J.; Soerjomataram, I.; Siegel, R.L.; Torre, L.A.; Jemal, A. Global Cancer Statistics 2018: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J. Clin. 2018, 68, 394–424. [Google Scholar] [CrossRef] [Green Version]
  5. Gondos, A.; Bray, F.; Brewster, D.H.; Coebergh, J.W.W.; Hakulinen, T.; Janssen-Heijnen, M.L.G.; Kurtinaitis, J.; Brenner, H. Recent Trends in Cancer Survival across Europe between 2000 and 2004: A Model-Based Period Analysis from 12 Cancer Registries. Eur. J. Cancer 2008, 44, 1463–1475. [Google Scholar] [CrossRef] [PubMed]
  6. Pasechnikov, V.; Chukov, S.; Fedorov, E.; Kikuste, I.; Leja, M. Gastric Cancer: Prevention, Screening and Early Diagnosis. World J. Gastroenterol. 2014, 20, 13842–13862. [Google Scholar] [CrossRef] [PubMed]
  7. Du, W.; Rao, N.; Dong, C.; Wang, Y.; Hu, D.; Zhu, L.; Zeng, B.; Gan, T. Automatic Classification of Esophageal Disease in Gastroscopic Images Using an Efficient Channel Attention Deep Dense Convolutional Neural Network. Biomed. Opt. Express 2021, 12, 3066. [Google Scholar] [CrossRef] [PubMed]
  8. Li, X.; Yu, L.; Chen, H.; Fu, C.W.; Xing, L.; Heng, P.A. Transformation-Consistent Self-Ensembling Model for Semisupervised Medical Image Segmentation. IEEE Trans. Neural Netw. Learn. Syst. 2021, 32, 523–534. [Google Scholar] [CrossRef] [PubMed]
  9. Rutka, K.; Garkowski, A.; Karaszewska, K.; Łebkowska, U. Imaging in Diagnosis of Systemic Sclerosis. J. Clin. Med. 2021, 10, 248. [Google Scholar] [CrossRef] [PubMed]
  10. Han, L.; Huang, Y.; Dou, H.; Wang, S.; Ahamad, S.; Luo, H.; Liu, Q.; Fan, J.; Zhang, J. Semi-Supervised Segmentation of Lesion from Breast Ultrasound Images with Attentional Generative Adversarial Network. Comput. Methods Programs Biomed. 2020, 189, 105275. [Google Scholar] [CrossRef] [PubMed]
  11. Gong, E.J.; Bang, C.S.; Lee, J.J.; Yang, Y.J.; Baik, G.H. Impact of the Volume and Distribution of Training Datasets in the Development of Deep-Learning Models for the Diagnosis of Colorectal Polyps in Endoscopy Images. J. Pers. Med. 2022, 12, 1361. [Google Scholar] [CrossRef] [PubMed]
  12. Gong, E.J.; Bang, C.S.; Jung, K.; Kim, S.J.; Kim, J.W.; Seo, S.I.; Lee, U.; Maeng, Y.B.; Lee, Y.J.; Lee, J.I. Deep-Learning for the Diagnosis of Esophageal Cancers and Precursor Lesions in Endoscopic Images: A Model Establishment and Nationwide Multicenter Performance Verification Study. J. Pers. Med. 2022, 12, 1052. [Google Scholar] [CrossRef]
  13. Lonseko, Z.M.; Adjei, P.E.; Du, W.; Luo, C.; Hu, D.; Zhu, L.; Gan, T.; Rao, N. Gastrointestinal Disease Classification in Endoscopic Images Using Attention-Guided Convolutional Neural Networks. Appl. Sci. 2021, 11, 11136. [Google Scholar] [CrossRef]
  14. Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Cham, Switzerland, 2015; Volume 9351, pp. 234–241. [Google Scholar] [CrossRef] [Green Version]
  15. Long, J.; Shelhamer, E.; Darrell, T. Fully Convolutional Networks for Semantic Segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 431–440. [Google Scholar] [CrossRef] [Green Version]
  16. Zhou, Z.; Rahman Siddiquee, M.M.; Tajbakhsh, N.; Liang, J. Unet++: A Nested u-Net Architecture for Medical Image Segmentation. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Berlin/Heidelberg, Germany, 2018; Volume 11045 LNCS, pp. 3–11. [Google Scholar] [CrossRef] [Green Version]
  17. Kohli, M.D.; Summers, R.M.; Geis, J.R. Medical Image Data and Datasets in the Era of Machine Learning—Whitepaper from the 2016 C-MIMI Meeting Dataset Session. J. Digit. Imaging 2017, 30, 392–399. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative Adversarial Nets. In Advances in Neural Information Processing Systems; Curran Associates, Inc.: Sydney, Australian, 2014; Volume 3, pp. 2672–2680. [Google Scholar] [CrossRef] [Green Version]
  19. Peng, J.; Wang, Y. Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models. IEEE Access 2021, 9, 36827–36851. [Google Scholar] [CrossRef]
  20. Hung, W.C.; Tsai, Y.H.; Liou, Y.T.; Lin, Y.Y.; Yang, M.H. Adversarial Learning for Semi-Supervised Semantic Segmentation. In Proceedings of the 29th British Machine Vision Conference, BMVC 2018, Newcastle, UK, 3–6 September 2018; pp. 1–17. [Google Scholar]
  21. Adjei, P.E.; Lonseko, Z.M.; Rao, N. GAN-Based Synthetic Gastrointestinal Image Generation. In Proceedings of the 17th International Computer Conference on Wavelet Active Media Technology and Information Processing, ICCWAMTIP 2020, Chengdu, China, 18–20 December 2020; Institute of Electrical and Electronics Engineers Inc.: Piscataway, NJ, USA, 2020; pp. 338–342. [Google Scholar] [CrossRef]
  22. Zhang, Y.; Yang, L.; Chen, J.; Fredericksen, M.; Hughes, D.P.; Chen, D.Z. Deep Adversarial Networks for Biomedical Image Segmentation Utilizing Unannotated Images. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Berlin/Heidelberg, Germany, 2017; Volume 10435 LNCS, pp. 408–416. [Google Scholar] [CrossRef]
  23. Lonseko, Z.M.; Adjei, P.E.; Du, W.; Luo, C.; Wang, Y.; Hu, D.; Gan, T.; Rao, N. Semi-Supervised Gastrointestinal Lesion Segmentation Using Adversarial Learning. In Proceedings of the 2021 IEEE 3rd Eurasia Conference on Biomedical Engineering, Healthcare and Sustainability (ECBIOS), Taiwan, China, 28–30 May 2021; pp. 63–66. [Google Scholar] [CrossRef]
  24. Bernal, J.; Sánchez, F.J.; Fernández-Esparrach, G.; Gil, D.; Rodríguez, C.; Vilariño, F. Polyp—CVC-ClinicDB. In Computerized Medical Imaging and Graphics; Elsevier Ltd.: Amsterdam, Netherlands, 2015; pp. 99–111. [Google Scholar]
  25. Silva, J.; Histace, A.; Romain, O.; Dray, X.; Granado, B. Toward Embedded Detection of Polyps in WCE Images for Early Diagnosis of Colorectal Cancer. Int. J. Comput. Assist. Radiol. Surg. 2014, 9, 283–293. [Google Scholar] [CrossRef] [PubMed]
  26. Ali, S.; Zhou, F.; Daul, C.; Braden, B.; Bailey, A.; Realdon, S.; East, J.; Wagnières, G.; Loschenov, V.; Grisan, E.; et al. Endoscopy Artifact Detection (EAD 2019) Challenge Dataset. arXiv 2019, arXiv:1905.03209. [Google Scholar] [CrossRef]
  27. Jha, D.; Smedsrud, P.H.; Riegler, M.A.; Halvorsen, P.; de Lange, T.; Johansen, D.; Johansen, H.D. Kvasir-SEG: A Segmented Polyp Dataset. In MultiMedia Modeling; Springer: Berlin/Heidelberg, Germany, 2019; pp. 451–462. [Google Scholar]
  28. Bottou, L. Large-Scale Machine Learning with Stochastic Gradient Descent. In Proceedings of the COMPSTAT’2010, Paris, France, 22–27 August 2010; Springer: Berlin/Heidelberg, Germany, 2010; pp. 177–186. [Google Scholar]
  29. Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; et al. TensorFlow: A System for Large-Scale Machine Learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, 2–4 November 2016; USENIX Association: Berkeley, CA, USA, 2016; pp. 265–283. [Google Scholar]
  30. Xue, Y.; Xu, T.; Zhang, H.; Long, L.R.; Huang, X. SegAN: Adversarial Network with Multi-Scale L 1 Loss for Medical Image Segmentation. Neuroinformatics 2018, 16, 383–392. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  31. Liu, D.Y.; Gan, T.; Rao, N.N.; Xu, G.G.; Zeng, B.; Li, H.L. Automatic Detection of Early Gastrointestinal Cancer Lesions Based on Optimal Feature Extraction from Gastroscopic Images. J. Med. Imaging Health Inform. 2015, 5, 296–302. [Google Scholar] [CrossRef]
Figure 1. Sample raw GI images. (a) GI images with the small and large lesion region (lab), (b) CVC-ClinicDB polyp images with small and large lesion region [24], (c) ETIS-LaribPolypDB polyp images [25], (d) EDA2019 challenge images [26], and (e) Kvasir-SEG images dataset with small and large lesions region [27].
Figure 1. Sample raw GI images. (a) GI images with the small and large lesion region (lab), (b) CVC-ClinicDB polyp images with small and large lesion region [24], (c) ETIS-LaribPolypDB polyp images [25], (d) EDA2019 challenge images [26], and (e) Kvasir-SEG images dataset with small and large lesions region [27].
Jpm 13 00118 g001
Figure 2. Semi-supervised GI lesion segmentation framework.
Figure 2. Semi-supervised GI lesion segmentation framework.
Jpm 13 00118 g002
Figure 3. Overview of proposed semi-supervised GI lesion segmentation architecture. Initially, a segmentation network (SN) is applied by using labeled data and corresponding GT data in a fully supervised fashion. Then, an evaluation network (EN) is introduced to give different scores to the segmentation of unlabeled and unlabeled data.
Figure 3. Overview of proposed semi-supervised GI lesion segmentation architecture. Initially, a segmentation network (SN) is applied by using labeled data and corresponding GT data in a fully supervised fashion. Then, an evaluation network (EN) is introduced to give different scores to the segmentation of unlabeled and unlabeled data.
Jpm 13 00118 g003
Figure 4. The learning curve shows accuracy performance over several epochs of the proposed model. At the first epoch, training and validation accuracy was low, with higher losses. After epoch 300, the model was slightly stable, and after epoch 400, it became stable with significant accuracy improvement.
Figure 4. The learning curve shows accuracy performance over several epochs of the proposed model. At the first epoch, training and validation accuracy was low, with higher losses. After epoch 300, the model was slightly stable, and after epoch 400, it became stable with significant accuracy improvement.
Jpm 13 00118 g004
Figure 5. Visual samples of the segmentation performance of semi-supervised learning (right), (middle), and our method (left) on the test dataset. The green, yellow, purple, and red contours denote the GT, proposed, GAN, and DAN segmentation results.
Figure 5. Visual samples of the segmentation performance of semi-supervised learning (right), (middle), and our method (left) on the test dataset. The green, yellow, purple, and red contours denote the GT, proposed, GAN, and DAN segmentation results.
Jpm 13 00118 g005
Table 1. Lesion segmentation comparison using the limited labeled GI datasets.
Table 1. Lesion segmentation comparison using the limited labeled GI datasets.
Data SourceTotalDSC (%) ± SDIoU (%) ± SDPre (%) ± SDRec (%) ± SDHDist (mm) ± SD
Lab19282.70 ± 7.7172.10 ± 10.2087.00 ± 9.4081.01 ± 11.8130.02 ± 19.11
ETIS-LaribPolypDB [25]19677.46 ± 17.3067.01 ± 19.7271.41 ± 22.0291.20 ± 8.4534.92 ± 31.72
CVC-ClinicDB [24] 61284.23 ± 14.0374.56 ± 16.1284.01 ± 15.4687.04 ± 13.5632.48 ± 26.23
Kvasir-SEG [27]100084.65 ± 18.0975.14 ± 18.3986.0 ± 18.8185.0 ± 19.3035.14 ± 20.63
Table 2. Supervised segmentation performance comparisons.
Table 2. Supervised segmentation performance comparisons.
ModelDSC (%) ± SDIOU (%) ± SDPre (%) ± SDRec (%) ± SDHDist (mm) ± SD
U-Net [14]81.04 ± 14.2868.04 ± 16.0185.15 ± 16.5586.62 ± 15.0336.17 ± 28.18
UNet++ [16]81.68 ± 13.6570.56 ± 14.3286.22 ± 15.6185.14 ± 15.2733.49 ± 27.24
Baseline82.15 ± 10.2270.28 ± 12.0484.22 ± 15.9483.78 ± 14.6932.09 ± 26.44
Table 3. Semi-supervised model performance results compared to GI dataset: Lab (A) data and EAD2019 challenge data (B).
Table 3. Semi-supervised model performance results compared to GI dataset: Lab (A) data and EAD2019 challenge data (B).
Labeled/
Unlabeled Data
ModelDSC (%) ± SDIOU (%) ± SDPre (%) ± SDRec (%) ± SDHDis (mm) ±
SD
ABABABABAB
192/192DAN [22]74.56 ± 2069.01 ± 15.4862.45 ± 13.0861.30 ± 19.4283.45 ± 13.2371.12 ± 16.4872.78 ± 14.0572.96 ± 18.1136.11 ± 23.2639.24 ± 34.11
GAN [20]79.62 ± 12.5671.23 ± 14.3468.55 ± 12.7163.02 ± 18.5789.56 ± 12.3272.34 ± 18.0174.12 ± 14.5673.29 ± 19.2337.25 ± 22.3440.11 ± 33.56
Ours83.10 ± 8.4574.20 ± 13.0271.02 ± 9.5866.04 ± 17.2188.24 ± 10.2677.02 ± 16.7881.42 ± 12.0476.26 ± 18.6332.33 ± 20.2637.79 ± 31.26
192/384DAN [22]81.22 ± 9.0875.48 ± 13.0268.45 ± 11.865.89 ± 18.1179.14 ± 17.5675.64 ± 17.687.58 ± 13.4279.43 ± 18.2533.89 ± 21.1338.26 ± 33.24
GAN [20]82.25 ± 8.1176.89 ± 11.3670.58 ± 11.1567.1 ± 16.3879.40 ± 16.2476.7 ± 16.8286.51 ± 11.4280.02 ± 16.1133.8 ± 20.5236.24 ± 32.45
Ours83.45 ± 7.2379.62 ± 11.0172.81 ± 9.5668.45 ± 16.2486.87 ± 9.680.50 ± 16.3282.47 ± 10.981.4 ± 15.5628.9 ± 18.1235.54 ± 29.33
192/768DAN [22]85.86 ± 8.1480.56 ± 10.275.5 ± 11.3773.01 ± 13.1587.25 ± 7.4880.2 ± 15.0179.9 ± 12.0680.56 ± 15.129.4 ± 19.1533.6 ± 26.37
GAN [20]85.39 ± 7.480.25 ± 10.1675.58 ± 10.3973.89 ± 13.789.61 ± 6.5182.62 ± 14.8880.12 ± 11.4581.63 ± 13.530.8 ± 18.933.06 ± 28.4
Ours86.65 ± 5.982.8 ± 9.476.72 ± 8.775.8 ± 10.6992.74 ± 5.8984.56 ± 11.481.85 ± 9.786.64 ± 12.725.4 ± 16.7231.37 ± 22.46
192/960DAN [22]86.8 ± 7.2482.34 ± 9.577.8 ± 10.3674.6 ± 12.6583.5 ± 7.6283.8 ± 12.9691.02 ± 11.385.45 ± 12.3925.6 ± 18.431.2 ± 27.5
GAN [20]86.7 ± 7.3182.81 ± 9.7676.9 ± 9.776.52 ± 13.292.03 ± 7.4684.72 ± 11.3482.36 ± 10.485.41 ± 11.724.7 ± 17.4230.61 ± 22.63
Ours88.6 ± 4.0883.81 ± 8.479.72 ± 6.4277.36 ± 9.4891.69 ± 4.7887.24 ± 10.8586.09 ± 5.9189.28 ± 11.4224.93 ± 15.8330.45 ± 21.05
192/1920DAN [22]87.03 ± 6.15-78.95 ± 9.36-85.62 ± 6.78-89.63 ± 10.13-24.68 ± 16.59-
GAN [20]87.49 ± 5.63-78.41 ± 8.74-92.81 ± 6.29-85.46 ± 9.2-24.05 ± 16.17-
Ours89.42 ± 3.92-80.04 ± 5.75-91.72 ± 4.05-90.11 ± 5.64-23.28 ± 14.36-
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Lonseko, Z.M.; Du, W.; Adjei, P.E.; Luo, C.; Hu, D.; Gan, T.; Zhu, L.; Rao, N. Semi-Supervised Segmentation Framework for Gastrointestinal Lesion Diagnosis in Endoscopic Images. J. Pers. Med. 2023, 13, 118. https://doi.org/10.3390/jpm13010118

AMA Style

Lonseko ZM, Du W, Adjei PE, Luo C, Hu D, Gan T, Zhu L, Rao N. Semi-Supervised Segmentation Framework for Gastrointestinal Lesion Diagnosis in Endoscopic Images. Journal of Personalized Medicine. 2023; 13(1):118. https://doi.org/10.3390/jpm13010118

Chicago/Turabian Style

Lonseko, Zenebe Markos, Wenju Du, Prince Ebenezer Adjei, Chengsi Luo, Dingcan Hu, Tao Gan, Linlin Zhu, and Nini Rao. 2023. "Semi-Supervised Segmentation Framework for Gastrointestinal Lesion Diagnosis in Endoscopic Images" Journal of Personalized Medicine 13, no. 1: 118. https://doi.org/10.3390/jpm13010118

APA Style

Lonseko, Z. M., Du, W., Adjei, P. E., Luo, C., Hu, D., Gan, T., Zhu, L., & Rao, N. (2023). Semi-Supervised Segmentation Framework for Gastrointestinal Lesion Diagnosis in Endoscopic Images. Journal of Personalized Medicine, 13(1), 118. https://doi.org/10.3390/jpm13010118

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop