Semi-Supervised Segmentation Framework for Gastrointestinal Lesion Diagnosis in Endoscopic Images
Abstract
:1. Introduction
- (1)
- We propose a GAN-based semi-supervised GI lesion segmentation framework that uses reasonably small labeled endoscopic images.
- (2)
- We demonstrate a full use of numerous unlabeled GI datasets to improve lesion segmentation accuracy.
- (3)
- The proposed framework was tested on five multi-sited datasets from different centers and integrated the predicted result to improve the segmentation performance through generative adversarial training.
- (4)
- The proposed method outperforms baseline supervised segmentation models as well as other related semi-supervised segmentation frameworks.
2. Materials and Methods
2.1. GI image Datasets
2.1.1. West China Hospital Digestive Endoscopy Center Dataset
2.1.2. Public Datasets
2.2. Methods
Loss Function
3. Experimental Setup
3.1. Implementation Details
3.2. Evaluation Metrics
4. Results and Discussion
4.1. Comparisons Using the Limited Labeled GI Datasets
4.2. Supervised Learning Comparisons
4.3. Semi-Supervised Learning Comparisons
4.4. Comparison of Baseline and Proposed Models
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
CAD | computer-aided diagnosis |
CNN | convolutional neural network |
DL | deep learning |
DSC | dice similarity coefficient |
EN | evaluation network |
FCN | fully connected network |
GI | gastrointestinal |
GT | ground truth |
HDis | Hausdorff distance |
GAN | generative adversarial network |
LR | learning rate |
Pre | precision |
Rec | recall |
ROI | region of interests |
SN | segmentation network |
SSL | semi-supervised learning |
SD | standard deviation |
SGD | stochastic gradient descent |
References
- Du, W.; Rao, N.; Liu, D.; Jiang, H.; Luo, C.; Li, Z.; Gan, T.; Zeng, B. Review on the Applications of Deep Learning in the Analysis of Gastrointestinal Endoscopy Images. IEEE Access 2019, 7, 142053–142069. [Google Scholar] [CrossRef]
- Escobar, J.P.; Gomez, N.; Sanchez, K.; Arguello, H. Transfer Learning with Convolutional Neural Network for Gastrointestinal Diseases Detection Using Endoscopic Images. In Proceedings of the 2020 IEEE Colombian Conference on Applications of Computational Intelligence (IEEE ColCACI 2020), Cali, Colombia, 7–8 August 2020. [Google Scholar] [CrossRef]
- Pogorelov, K.; Randel, K.R.; Griwodz, C.; Eskeland, S.L.; De Lange, T.; Johansen, D.; Spampinato, C.; Dang-Nguyen, D.T.; Lux, M.; Schmidt, P.T.; et al. Kvasir: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection. In Proceedings of the 8th ACM Multimedia Systems Conference, MMSys, New York, NY, USA, 20 June 2017; pp. 164–169. [Google Scholar] [CrossRef] [Green Version]
- Bray, F.; Ferlay, J.; Soerjomataram, I.; Siegel, R.L.; Torre, L.A.; Jemal, A. Global Cancer Statistics 2018: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J. Clin. 2018, 68, 394–424. [Google Scholar] [CrossRef] [Green Version]
- Gondos, A.; Bray, F.; Brewster, D.H.; Coebergh, J.W.W.; Hakulinen, T.; Janssen-Heijnen, M.L.G.; Kurtinaitis, J.; Brenner, H. Recent Trends in Cancer Survival across Europe between 2000 and 2004: A Model-Based Period Analysis from 12 Cancer Registries. Eur. J. Cancer 2008, 44, 1463–1475. [Google Scholar] [CrossRef] [PubMed]
- Pasechnikov, V.; Chukov, S.; Fedorov, E.; Kikuste, I.; Leja, M. Gastric Cancer: Prevention, Screening and Early Diagnosis. World J. Gastroenterol. 2014, 20, 13842–13862. [Google Scholar] [CrossRef] [PubMed]
- Du, W.; Rao, N.; Dong, C.; Wang, Y.; Hu, D.; Zhu, L.; Zeng, B.; Gan, T. Automatic Classification of Esophageal Disease in Gastroscopic Images Using an Efficient Channel Attention Deep Dense Convolutional Neural Network. Biomed. Opt. Express 2021, 12, 3066. [Google Scholar] [CrossRef] [PubMed]
- Li, X.; Yu, L.; Chen, H.; Fu, C.W.; Xing, L.; Heng, P.A. Transformation-Consistent Self-Ensembling Model for Semisupervised Medical Image Segmentation. IEEE Trans. Neural Netw. Learn. Syst. 2021, 32, 523–534. [Google Scholar] [CrossRef] [PubMed]
- Rutka, K.; Garkowski, A.; Karaszewska, K.; Łebkowska, U. Imaging in Diagnosis of Systemic Sclerosis. J. Clin. Med. 2021, 10, 248. [Google Scholar] [CrossRef] [PubMed]
- Han, L.; Huang, Y.; Dou, H.; Wang, S.; Ahamad, S.; Luo, H.; Liu, Q.; Fan, J.; Zhang, J. Semi-Supervised Segmentation of Lesion from Breast Ultrasound Images with Attentional Generative Adversarial Network. Comput. Methods Programs Biomed. 2020, 189, 105275. [Google Scholar] [CrossRef] [PubMed]
- Gong, E.J.; Bang, C.S.; Lee, J.J.; Yang, Y.J.; Baik, G.H. Impact of the Volume and Distribution of Training Datasets in the Development of Deep-Learning Models for the Diagnosis of Colorectal Polyps in Endoscopy Images. J. Pers. Med. 2022, 12, 1361. [Google Scholar] [CrossRef] [PubMed]
- Gong, E.J.; Bang, C.S.; Jung, K.; Kim, S.J.; Kim, J.W.; Seo, S.I.; Lee, U.; Maeng, Y.B.; Lee, Y.J.; Lee, J.I. Deep-Learning for the Diagnosis of Esophageal Cancers and Precursor Lesions in Endoscopic Images: A Model Establishment and Nationwide Multicenter Performance Verification Study. J. Pers. Med. 2022, 12, 1052. [Google Scholar] [CrossRef]
- Lonseko, Z.M.; Adjei, P.E.; Du, W.; Luo, C.; Hu, D.; Zhu, L.; Gan, T.; Rao, N. Gastrointestinal Disease Classification in Endoscopic Images Using Attention-Guided Convolutional Neural Networks. Appl. Sci. 2021, 11, 11136. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Cham, Switzerland, 2015; Volume 9351, pp. 234–241. [Google Scholar] [CrossRef] [Green Version]
- Long, J.; Shelhamer, E.; Darrell, T. Fully Convolutional Networks for Semantic Segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 431–440. [Google Scholar] [CrossRef] [Green Version]
- Zhou, Z.; Rahman Siddiquee, M.M.; Tajbakhsh, N.; Liang, J. Unet++: A Nested u-Net Architecture for Medical Image Segmentation. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Berlin/Heidelberg, Germany, 2018; Volume 11045 LNCS, pp. 3–11. [Google Scholar] [CrossRef] [Green Version]
- Kohli, M.D.; Summers, R.M.; Geis, J.R. Medical Image Data and Datasets in the Era of Machine Learning—Whitepaper from the 2016 C-MIMI Meeting Dataset Session. J. Digit. Imaging 2017, 30, 392–399. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative Adversarial Nets. In Advances in Neural Information Processing Systems; Curran Associates, Inc.: Sydney, Australian, 2014; Volume 3, pp. 2672–2680. [Google Scholar] [CrossRef] [Green Version]
- Peng, J.; Wang, Y. Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models. IEEE Access 2021, 9, 36827–36851. [Google Scholar] [CrossRef]
- Hung, W.C.; Tsai, Y.H.; Liou, Y.T.; Lin, Y.Y.; Yang, M.H. Adversarial Learning for Semi-Supervised Semantic Segmentation. In Proceedings of the 29th British Machine Vision Conference, BMVC 2018, Newcastle, UK, 3–6 September 2018; pp. 1–17. [Google Scholar]
- Adjei, P.E.; Lonseko, Z.M.; Rao, N. GAN-Based Synthetic Gastrointestinal Image Generation. In Proceedings of the 17th International Computer Conference on Wavelet Active Media Technology and Information Processing, ICCWAMTIP 2020, Chengdu, China, 18–20 December 2020; Institute of Electrical and Electronics Engineers Inc.: Piscataway, NJ, USA, 2020; pp. 338–342. [Google Scholar] [CrossRef]
- Zhang, Y.; Yang, L.; Chen, J.; Fredericksen, M.; Hughes, D.P.; Chen, D.Z. Deep Adversarial Networks for Biomedical Image Segmentation Utilizing Unannotated Images. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Berlin/Heidelberg, Germany, 2017; Volume 10435 LNCS, pp. 408–416. [Google Scholar] [CrossRef]
- Lonseko, Z.M.; Adjei, P.E.; Du, W.; Luo, C.; Wang, Y.; Hu, D.; Gan, T.; Rao, N. Semi-Supervised Gastrointestinal Lesion Segmentation Using Adversarial Learning. In Proceedings of the 2021 IEEE 3rd Eurasia Conference on Biomedical Engineering, Healthcare and Sustainability (ECBIOS), Taiwan, China, 28–30 May 2021; pp. 63–66. [Google Scholar] [CrossRef]
- Bernal, J.; Sánchez, F.J.; Fernández-Esparrach, G.; Gil, D.; Rodríguez, C.; Vilariño, F. Polyp—CVC-ClinicDB. In Computerized Medical Imaging and Graphics; Elsevier Ltd.: Amsterdam, Netherlands, 2015; pp. 99–111. [Google Scholar]
- Silva, J.; Histace, A.; Romain, O.; Dray, X.; Granado, B. Toward Embedded Detection of Polyps in WCE Images for Early Diagnosis of Colorectal Cancer. Int. J. Comput. Assist. Radiol. Surg. 2014, 9, 283–293. [Google Scholar] [CrossRef] [PubMed]
- Ali, S.; Zhou, F.; Daul, C.; Braden, B.; Bailey, A.; Realdon, S.; East, J.; Wagnières, G.; Loschenov, V.; Grisan, E.; et al. Endoscopy Artifact Detection (EAD 2019) Challenge Dataset. arXiv 2019, arXiv:1905.03209. [Google Scholar] [CrossRef]
- Jha, D.; Smedsrud, P.H.; Riegler, M.A.; Halvorsen, P.; de Lange, T.; Johansen, D.; Johansen, H.D. Kvasir-SEG: A Segmented Polyp Dataset. In MultiMedia Modeling; Springer: Berlin/Heidelberg, Germany, 2019; pp. 451–462. [Google Scholar]
- Bottou, L. Large-Scale Machine Learning with Stochastic Gradient Descent. In Proceedings of the COMPSTAT’2010, Paris, France, 22–27 August 2010; Springer: Berlin/Heidelberg, Germany, 2010; pp. 177–186. [Google Scholar]
- Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; et al. TensorFlow: A System for Large-Scale Machine Learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, 2–4 November 2016; USENIX Association: Berkeley, CA, USA, 2016; pp. 265–283. [Google Scholar]
- Xue, Y.; Xu, T.; Zhang, H.; Long, L.R.; Huang, X. SegAN: Adversarial Network with Multi-Scale L 1 Loss for Medical Image Segmentation. Neuroinformatics 2018, 16, 383–392. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Liu, D.Y.; Gan, T.; Rao, N.N.; Xu, G.G.; Zeng, B.; Li, H.L. Automatic Detection of Early Gastrointestinal Cancer Lesions Based on Optimal Feature Extraction from Gastroscopic Images. J. Med. Imaging Health Inform. 2015, 5, 296–302. [Google Scholar] [CrossRef]
Data Source | Total | DSC (%) ± SD | IoU (%) ± SD | Pre (%) ± SD | Rec (%) ± SD | HDist (mm) ± SD |
---|---|---|---|---|---|---|
Lab | 192 | 82.70 ± 7.71 | 72.10 ± 10.20 | 87.00 ± 9.40 | 81.01 ± 11.81 | 30.02 ± 19.11 |
ETIS-LaribPolypDB [25] | 196 | 77.46 ± 17.30 | 67.01 ± 19.72 | 71.41 ± 22.02 | 91.20 ± 8.45 | 34.92 ± 31.72 |
CVC-ClinicDB [24] | 612 | 84.23 ± 14.03 | 74.56 ± 16.12 | 84.01 ± 15.46 | 87.04 ± 13.56 | 32.48 ± 26.23 |
Kvasir-SEG [27] | 1000 | 84.65 ± 18.09 | 75.14 ± 18.39 | 86.0 ± 18.81 | 85.0 ± 19.30 | 35.14 ± 20.63 |
Model | DSC (%) ± SD | IOU (%) ± SD | Pre (%) ± SD | Rec (%) ± SD | HDist (mm) ± SD |
---|---|---|---|---|---|
U-Net [14] | 81.04 ± 14.28 | 68.04 ± 16.01 | 85.15 ± 16.55 | 86.62 ± 15.03 | 36.17 ± 28.18 |
UNet++ [16] | 81.68 ± 13.65 | 70.56 ± 14.32 | 86.22 ± 15.61 | 85.14 ± 15.27 | 33.49 ± 27.24 |
Baseline | 82.15 ± 10.22 | 70.28 ± 12.04 | 84.22 ± 15.94 | 83.78 ± 14.69 | 32.09 ± 26.44 |
Labeled/ Unlabeled Data | Model | DSC (%) ± SD | IOU (%) ± SD | Pre (%) ± SD | Rec (%) ± SD | HDis (mm) ± SD | |||||
---|---|---|---|---|---|---|---|---|---|---|---|
A | B | A | B | A | B | A | B | A | B | ||
192/192 | DAN [22] | 74.56 ± 20 | 69.01 ± 15.48 | 62.45 ± 13.08 | 61.30 ± 19.42 | 83.45 ± 13.23 | 71.12 ± 16.48 | 72.78 ± 14.05 | 72.96 ± 18.11 | 36.11 ± 23.26 | 39.24 ± 34.11 |
GAN [20] | 79.62 ± 12.56 | 71.23 ± 14.34 | 68.55 ± 12.71 | 63.02 ± 18.57 | 89.56 ± 12.32 | 72.34 ± 18.01 | 74.12 ± 14.56 | 73.29 ± 19.23 | 37.25 ± 22.34 | 40.11 ± 33.56 | |
Ours | 83.10 ± 8.45 | 74.20 ± 13.02 | 71.02 ± 9.58 | 66.04 ± 17.21 | 88.24 ± 10.26 | 77.02 ± 16.78 | 81.42 ± 12.04 | 76.26 ± 18.63 | 32.33 ± 20.26 | 37.79 ± 31.26 | |
192/384 | DAN [22] | 81.22 ± 9.08 | 75.48 ± 13.02 | 68.45 ± 11.8 | 65.89 ± 18.11 | 79.14 ± 17.56 | 75.64 ± 17.6 | 87.58 ± 13.42 | 79.43 ± 18.25 | 33.89 ± 21.13 | 38.26 ± 33.24 |
GAN [20] | 82.25 ± 8.11 | 76.89 ± 11.36 | 70.58 ± 11.15 | 67.1 ± 16.38 | 79.40 ± 16.24 | 76.7 ± 16.82 | 86.51 ± 11.42 | 80.02 ± 16.11 | 33.8 ± 20.52 | 36.24 ± 32.45 | |
Ours | 83.45 ± 7.23 | 79.62 ± 11.01 | 72.81 ± 9.56 | 68.45 ± 16.24 | 86.87 ± 9.6 | 80.50 ± 16.32 | 82.47 ± 10.9 | 81.4 ± 15.56 | 28.9 ± 18.12 | 35.54 ± 29.33 | |
192/768 | DAN [22] | 85.86 ± 8.14 | 80.56 ± 10.2 | 75.5 ± 11.37 | 73.01 ± 13.15 | 87.25 ± 7.48 | 80.2 ± 15.01 | 79.9 ± 12.06 | 80.56 ± 15.1 | 29.4 ± 19.15 | 33.6 ± 26.37 |
GAN [20] | 85.39 ± 7.4 | 80.25 ± 10.16 | 75.58 ± 10.39 | 73.89 ± 13.7 | 89.61 ± 6.51 | 82.62 ± 14.88 | 80.12 ± 11.45 | 81.63 ± 13.5 | 30.8 ± 18.9 | 33.06 ± 28.4 | |
Ours | 86.65 ± 5.9 | 82.8 ± 9.4 | 76.72 ± 8.7 | 75.8 ± 10.69 | 92.74 ± 5.89 | 84.56 ± 11.4 | 81.85 ± 9.7 | 86.64 ± 12.7 | 25.4 ± 16.72 | 31.37 ± 22.46 | |
192/960 | DAN [22] | 86.8 ± 7.24 | 82.34 ± 9.5 | 77.8 ± 10.36 | 74.6 ± 12.65 | 83.5 ± 7.62 | 83.8 ± 12.96 | 91.02 ± 11.3 | 85.45 ± 12.39 | 25.6 ± 18.4 | 31.2 ± 27.5 |
GAN [20] | 86.7 ± 7.31 | 82.81 ± 9.76 | 76.9 ± 9.7 | 76.52 ± 13.2 | 92.03 ± 7.46 | 84.72 ± 11.34 | 82.36 ± 10.4 | 85.41 ± 11.7 | 24.7 ± 17.42 | 30.61 ± 22.63 | |
Ours | 88.6 ± 4.08 | 83.81 ± 8.4 | 79.72 ± 6.42 | 77.36 ± 9.48 | 91.69 ± 4.78 | 87.24 ± 10.85 | 86.09 ± 5.91 | 89.28 ± 11.42 | 24.93 ± 15.83 | 30.45 ± 21.05 | |
192/1920 | DAN [22] | 87.03 ± 6.15 | - | 78.95 ± 9.36 | - | 85.62 ± 6.78 | - | 89.63 ± 10.13 | - | 24.68 ± 16.59 | - |
GAN [20] | 87.49 ± 5.63 | - | 78.41 ± 8.74 | - | 92.81 ± 6.29 | - | 85.46 ± 9.2 | - | 24.05 ± 16.17 | - | |
Ours | 89.42 ± 3.92 | - | 80.04 ± 5.75 | - | 91.72 ± 4.05 | - | 90.11 ± 5.64 | - | 23.28 ± 14.36 | - |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Lonseko, Z.M.; Du, W.; Adjei, P.E.; Luo, C.; Hu, D.; Gan, T.; Zhu, L.; Rao, N. Semi-Supervised Segmentation Framework for Gastrointestinal Lesion Diagnosis in Endoscopic Images. J. Pers. Med. 2023, 13, 118. https://doi.org/10.3390/jpm13010118
Lonseko ZM, Du W, Adjei PE, Luo C, Hu D, Gan T, Zhu L, Rao N. Semi-Supervised Segmentation Framework for Gastrointestinal Lesion Diagnosis in Endoscopic Images. Journal of Personalized Medicine. 2023; 13(1):118. https://doi.org/10.3390/jpm13010118
Chicago/Turabian StyleLonseko, Zenebe Markos, Wenju Du, Prince Ebenezer Adjei, Chengsi Luo, Dingcan Hu, Tao Gan, Linlin Zhu, and Nini Rao. 2023. "Semi-Supervised Segmentation Framework for Gastrointestinal Lesion Diagnosis in Endoscopic Images" Journal of Personalized Medicine 13, no. 1: 118. https://doi.org/10.3390/jpm13010118
APA StyleLonseko, Z. M., Du, W., Adjei, P. E., Luo, C., Hu, D., Gan, T., Zhu, L., & Rao, N. (2023). Semi-Supervised Segmentation Framework for Gastrointestinal Lesion Diagnosis in Endoscopic Images. Journal of Personalized Medicine, 13(1), 118. https://doi.org/10.3390/jpm13010118