Chromosome Cluster Type Identification Using a Swin Transformer
Abstract
:1. Introduction
- We address the limitation of state-of-the-art CNN-based chromosome cluster type identification methods in capturing long-range dependencies. Towards this end, we propose the chromosome cluster transformer (CCT), which successfully captures the global context required for the successful identification of chromosome cluster types.
- To the best of our knowledge, this is the first work in the domain of chromosome cluster type identification that utilizes a transformer model.
- To circumvent the limited availability of training data for cluster type identification, the proposed CCT exploits a transfer learning approach.
- The proposed CCT outperforms the state-of-the-art traditional vision transformer in chromosome cluster type identification.
- Furthermore, the proposed CCT outperforms the existing state-of-the-art chromosome cluster type identification methods.
- Additionally, to provide insights on the improved performance, we visualize the activation maps obtained using Gradient Attention Rollout [18].
2. Related Work
3. Proposed Method
3.1. Problem Formulation
3.2. Chromosome Cluster Transformer (CCT)
3.3. Partitioning of Shifted Windows
3.4. Transfer Learning for Chromosome Cluster Transformer
3.5. Implementation Details
4. Databases and Experimental Protocol
4.1. Database
4.2. Evaluation Metrics
- Accuracy:
- Precision:
- F1 score:
- Sensitivity:
- Specificity:
5. Results and Analysis
5.1. Comparison with State of the Art
5.2. Cross-Validation Performance
5.3. Confusion Matrix
5.4. Effect of Model Architecture
5.5. Comparison with Traditional Vision Transformer
5.6. Visualization of Model Activation
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Rai, A.; Hirakawa, H.; Rai, M.; Shimizu, Y.; Shirasawa, K.; Kikuchi, S.; Seki, H.; Yamazaki, M.; Toyoda, A.; Isobe, S.; et al. Chromosome-scale genome assembly of Glycyrrhiza uralensis revealed metabolic gene cluster centred specialized metabolites biosynthesis. DNA Res. 2022, 29, dsac043. [Google Scholar] [CrossRef] [PubMed]
- Wang, Y.P.; Wu, Q.; Castleman, K.R.; Xiong, Z. Chromosome image enhancement using multiscale differential operators. IEEE Trans. Med. Imaging 2003, 22, 685–693. [Google Scholar] [CrossRef] [PubMed]
- Qin, Y.; Wen, J.; Zheng, H.; Huang, X.; Yang, J.; Song, N.; Zhu, Y.M.; Wu, L.; Yang, G.Z. Varifocal-net: A chromosome classification approach using deep convolutional networks. IEEE Trans. Med. Imaging 2019, 38, 2569–2581. [Google Scholar] [CrossRef] [Green Version]
- Arora, T.; Dhir, R. A review of metaphase chromosome image selection techniques for automatic karyotype generation. Med. Biol. Eng. Comput. 2016, 54, 1147–1157. [Google Scholar] [CrossRef] [PubMed]
- Remani Sathyan, R.; Chandrasekhara Menon, G.; Thampi, R.; Duraisamy, J.H. Traditional and deep-based techniques for end-to-end automated karyotyping: A review. Expert Syst. 2022, 39, e12799. [Google Scholar] [CrossRef]
- Remya, R.; Hariharan, S.; Keerthi, V.; Gopakumar, C. Preprocessing G-banded metaphase: Towards the design of automated karyotyping. SN Appl. Sci. 2019, 1, 1–8. [Google Scholar] [CrossRef] [Green Version]
- Wei, H.; Gao, W.; Nie, H.; Sun, J.; Zhu, M. Classification of Giemsa staining chromosome using input-aware deep convolutional neural network with integrated uncertainty estimates. Biomed. Signal Process. Control 2022, 71, 103120. [Google Scholar] [CrossRef]
- Huang, K.; Lin, C.; Huang, R.; Zhao, G.; Yin, A.; Chen, H.; Guo, L.; Shan, C.; Nie, R.; Li, S. A novel chromosome instance segmentation method based on geometry and deep learning. In Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China, 18–22 July 2021; pp. 1–8. [Google Scholar]
- Menaka, D.; Vaidyanathan, S.G. Chromenet: A CNN architecture with comparison of optimizers for classification of human chromosome images. Multidimens. Syst. Signal Process. 2022, 33, 747–768. [Google Scholar] [CrossRef]
- Lin, C.; Zhao, G.; Yin, A.; Yang, Z.; Guo, L.; Chen, H.; Zhao, L.; Li, S.; Luo, H.; Ma, Z. A novel chromosome cluster types identification method using ResNeXt WSL model. Med. Image Anal. 2021, 69, 101943. [Google Scholar] [CrossRef]
- Lin, C.; Yin, A.; Wu, Q.; Chen, H.; Guo, L.; Zhao, G.; Fan, X.; Luo, H.; Tang, H. Chromosome cluster identification framework based on geometric features and machine learning algorithms. In Proceedings of the 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Seoul, Republic of Korea, 16–19 December 2020; pp. 2357–2363. [Google Scholar]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia, 26–30 April 2020. [Google Scholar]
- Hu, R.; Chen, J.; Zhou, L. A transformer-based deep neural network for arrhythmia detection using continuous ECG signals. Comput. Biol. Med. 2022, 144, 105325. [Google Scholar] [CrossRef]
- Han, K.; Wang, Y.; Chen, H.; Chen, X.; Guo, J.; Liu, Z.; Tang, Y.; Xiao, A.; Xu, C.; Xu, Y.; et al. A survey on vision transformer. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 45, 87–110. [Google Scholar] [CrossRef] [PubMed]
- Khan, S.; Naseer, M.; Hayat, M.; Zamir, S.W.; Khan, F.S.; Shah, M. Transformers in vision: A survey. ACM Comput. Surv. CSUR 2021, 54, 1–41. [Google Scholar] [CrossRef]
- Hatamizadeh, A.; Tang, Y.; Nath, V.; Yang, D.; Myronenko, A.; Landman, B.; Roth, H.R.; Xu, D. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, 4–8 January 2022; pp. 574–584. [Google Scholar]
- Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 10012–10022. [Google Scholar]
- Chefer, H.; Gur, S.; Wolf, L. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 782–791. [Google Scholar]
- Kimori, Y. Morphological image processing for quantitative shape analysis of biomedical structures: Effective contrast enhancement. J. Synchrotron Radiat. 2013, 20, 848–853. [Google Scholar] [CrossRef] [PubMed]
- Ming, D.; Tian, J. Automatic pattern extraction and classification for chromosome images. J. Infrared Millim. Terahertz Waves 2010, 31, 866–877. [Google Scholar] [CrossRef]
- Altinsoy, E.; Yang, J.; Yilmaz, C. Fully automatic raw G-band chromosome image segmentation. IET Image Process. 2020, 14, 1920–1928. [Google Scholar] [CrossRef]
- Liu, X.; Fu, L.; Chun-Wei Lin, J.; Liu, S. SRAS-net: Low-resolution chromosome image classification based on deep learning. IET Syst. Biol. 2022, 16, 85–97. [Google Scholar] [CrossRef]
- Arora, T.; Dhir, R. A variable region scalable fitting energy approach for human Metaspread chromosome image segmentation. Multimed. Tools Appl. 2019, 78, 9383–9404. [Google Scholar] [CrossRef]
- Madian, N.; Jayanthi, K.; Suresh, S. Analysis of human chromosome images: Application towards an automated chromosome classification. Int. J. Imaging Syst. Technol. 2018, 28, 235–245. [Google Scholar] [CrossRef]
- Minaee, S.; Fotouhi, M.; Khalaj, B.H. A geometric approach to fully automatic chromosome segmentation. In Proceedings of the 2014 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), Philadelphia, PA, USA, 13 December 2014; pp. 1–6. [Google Scholar]
- Kubola, K.; Wayalun, P. Automatic determination of the g-band chromosomes number based on geometric features. In Proceedings of the 2018 15th International Joint Conference on Computer Science and Software Engineering (JCSSE), Nakhonpathom, Thailand, 11–13 July 2018; pp. 1–5. [Google Scholar]
- Arora, T.; Dhir, R. A novel approach for segmentation of human metaphase chromosome images using region based active contours. Int. Arab. J. Inf. Technol. 2019, 16, 132–137. [Google Scholar]
- Sharma, M.; Vig, L. Automatic chromosome classification using deep attention based sequence learning of chromosome bands. In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil, 8–13 July 2018; pp. 1–8. [Google Scholar]
- Saleh, H.M.; Saad, N.H.; Isa, N.A.M. Overlapping chromosome segmentation using u-net: Convolutional networks with test time augmentation. Procedia Comput. Sci. 2019, 159, 524–533. [Google Scholar] [CrossRef]
- Altinsoy, E.; Yang, J.; Tu, E. An improved denoising of G-banding chromosome images using cascaded CNN and binary classification network. Vis. Comput. 2021, 38, 2139–2152. [Google Scholar] [CrossRef]
- Zhang, D.; Fu, H.; Han, J.; Borji, A.; Li, X. A review of co-saliency detection algorithms: Fundamentals, applications, and challenges. ACM Trans. Intell. Syst. Technol. TIST 2018, 9, 1–31. [Google Scholar] [CrossRef]
- Huang, R.; Feng, W.; Sun, J. Color feature reinforcement for cosaliency detection without single saliency residuals. IEEE Signal Process. Lett. 2017, 24, 569–573. [Google Scholar] [CrossRef]
- Li, T.; Song, H.; Zhang, K.; Liu, Q.; Lian, W. Low-rank weighted co-saliency detection via efficient manifold ranking. Multimed. Tools Appl. 2019, 78, 21309–21324. [Google Scholar] [CrossRef]
- Tang, L.; Li, B.; Kuang, S.; Song, M.; Ding, S. Re-thinking the relations in co-saliency detection. IEEE Trans. Circuits Syst. Video Technol. 2022, 32, 5453–5466. [Google Scholar] [CrossRef]
- Tan, Z.; Wan, L.; Feng, W.; Pun, C.M. Image co-saliency detection by propagating superpixel affinities. In Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada, 26–31 May 2013; pp. 2114–2118. [Google Scholar]
- Zhang, D.; Meng, D.; Han, J. Co-saliency detection via a self-paced multiple-instance learning framework. IEEE Trans. Pattern Anal. Mach. Intell. 2016, 39, 865–878. [Google Scholar] [CrossRef]
- Han, J.; Cheng, G.; Li, Z.; Zhang, D. A unified metric learning-based framework for co-saliency detection. IEEE Trans. Circuits Syst. Video Technol. 2017, 28, 2473–2483. [Google Scholar] [CrossRef]
- Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 2015, 115, 211–252. [Google Scholar] [CrossRef] [Green Version]
- Luo, C.; Wu, Y.; Zhao, Y. SupCAM: Chromosome cluster types identification using supervised contrastive learning with category-variant augmentation and self-margin loss. Front. Genet. 2023, 14, 1109269. [Google Scholar] [CrossRef]
- Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; Chen, L.C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 4510–4520. [Google Scholar]
- Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4700–4708. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Xie, S.; Girshick, R.; Dollár, P.; Tu, Z.; He, K. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1492–1500. [Google Scholar]
- Yao, T.; Li, Y.; Pan, Y.; Wang, Y.; Zhang, X.P.; Mei, T. Dual vision transformer. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 1–13. [Google Scholar] [CrossRef]
Class Label | Image Count |
---|---|
Chromosome Instance | 1712 |
Overlapping Cluster | 1038 |
Touching Cluster | 3029 |
Touching–Overlapping Cluster | 813 |
Model | Precision | Accuracy | F1 | Sensitivity | Specificity |
---|---|---|---|---|---|
MobileNetV2 [40] | 81.83 | 83.41 | 77.52 | 76.85 | 94.47 |
DenseNet121 [41] | 85.59 | 87.65 | 82.23 | 81.68 | 95.88 |
ResNet-50 [42] | 88.30 | 90.15 | 86.08 | 85.68 | 96.72 |
ResNet-101 [42] | 90.65 | 91.89 | 88.32 | 87.92 | 97.30 |
ResNet-152 [42] | 90.71 | 91.97 | 89.09 | 88.79 | 97.32 |
ResNeXt-101-32×8d [43] | 90.79 | 92.27 | 89.36 | 89.10 | 97.42 |
ResNeXt-WSL [10] | 93.35 | 94.13 | 92.41 | 92.20 | 98.04 |
Dual-ViT [44] | 94.07 | 94.10 | 94.05 | 94.10 | 97.69 |
SupCAM [39] | 93.25 | 94.99 | 92.26 | 92.81 | 98.12 |
CCT (Proposed) | 95.02 | 95.30 | 95.02 | 95.03 | 98.26 |
Method | Fold | Precision | Accuracy | F1 | Sensitivity | Specificity |
---|---|---|---|---|---|---|
1 | 94.28 | 94.77 | 93.58 | 93.43 | 98.26 | |
2 | 91.52 | 92.73 | 90.54 | 90.33 | 97.58 | |
ResNeXt- | 3 | 92.70 | 93.63 | 91.21 | 90.82 | 97.88 |
WSL [10] | 4 | 94.42 | 94.99 | 93.42 | 93.23 | 98.33 |
5 | 93.82 | 94.53 | 93.31 | 93.20 | 98.18 | |
Mean (±std) | 93.35 ± 2.19 | 94.13 ± 1.68 | 92.41 ± 2.55 | 92.20 ± 2.68 | 98.04 ± 0.56 | |
1 | 95.37 | 95.75 | 95.36 | 95.36 | 98.44 | |
2 | 94.52 | 94.84 | 94.55 | 94.59 | 98.09 | |
CCT | 3 | 95.00 | 95.07 | 95.02 | 95.01 | 98.15 |
(Proposed) | 4 | 94.84 | 95.29 | 94.83 | 94.81 | 98.29 |
5 | 95.37 | 95.53 | 95.36 | 95.36 | 98.32 | |
Mean (±std) | 95.02 ± 0.32 | 95.30 ± 0.31 | 95.02 ± 0.31 | 95.03 ± 0.30 | 98.26 ± 0.12 |
Model | Precision | Accuracy | F1 | Sensitivity | Specificity |
---|---|---|---|---|---|
CCT-Tiny | 94.11 ± 0.31 | 94.72 ± 0.33 | 94.00 ± 0.29 | 94.05 ± 0.27 | 97.99 ± 0.34 |
CCT-Small | 94.61 ± 0.20 | 95.16 ± 0.21 | 94.51 ± 0.25 | 94.55 ± 0.19 | 98.29 ± 0.23 |
CCT-Base | 94.99 ± 0.15 | 95.13 ± 0.81 | 94.89 ± 0.11 | 95.01 ± 0.36 | 98.12 ± 0.12 |
CCT-Large | 95.02 ± 0.32 | 95.30 ± 0.31 | 95.02 ± 0.31 | 95.03 ± 0.30 | 98.26 ± 0.12 |
Model | Precision | Accuracy | F1 | Sensitivity | Specificity |
---|---|---|---|---|---|
ViT [12] | 90.17 | 92.75 | 91.06 | 91.97 | 97.34 |
CCT (Proposed) | 95.02 | 95.30 | 95.02 | 95.03 | 98.26 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Joshi, I.; Mondal, A.K.; Navab, N. Chromosome Cluster Type Identification Using a Swin Transformer. Appl. Sci. 2023, 13, 8007. https://doi.org/10.3390/app13148007
Joshi I, Mondal AK, Navab N. Chromosome Cluster Type Identification Using a Swin Transformer. Applied Sciences. 2023; 13(14):8007. https://doi.org/10.3390/app13148007
Chicago/Turabian StyleJoshi, Indu, Arnab Kumar Mondal, and Nassir Navab. 2023. "Chromosome Cluster Type Identification Using a Swin Transformer" Applied Sciences 13, no. 14: 8007. https://doi.org/10.3390/app13148007
APA StyleJoshi, I., Mondal, A. K., & Navab, N. (2023). Chromosome Cluster Type Identification Using a Swin Transformer. Applied Sciences, 13(14), 8007. https://doi.org/10.3390/app13148007