Sample Expansion and Classification Model of Maize Leaf Diseases Based on the Self-Attention CycleGAN
Abstract
:1. Introduction
2. Materials and Methods
2.1. The Principle of CycleGAN
2.2. Improvements to the CycleGAN Model
2.2.1. Introduction of Attention Mechanism
2.2.2. Generator Network Architecture
2.2.3. Discriminator Network Architecture
2.2.4. Designing Feature Recombination Loss Function
2.3. Type Training and Parameters
- (1)
- Pre-train the network parameters using a small custom dataset of maize disease.
- (2)
- Train the model using a substantial amount of custom maize disease dataset. The generator and discriminator’s network parameters are trained in a step-by-step manner. Real-time monitoring of the training process is conducted using the TensorBoard module.
- (3)
- Fix the generator parameters and train the discriminator parameters. The discriminator is updated at a 3:1 ratio compared to the generator.
- (4)
- The training is considered complete when both discriminators cannot determine the source of the maize disease leaf image. This is reflected in an output value of 0.5, indicating the Nash equilibrium.
3. Experimental Process
3.1. Evaluation Metrics
3.2. Experimental Data
4. Experimental Results and Analysis
4.1. The Impact of the Improved Model on Model Performance
4.2. Contrast Based on Objective Parameters of Generated Images
- (1)
- Comparison of generated image FID values.
- (2)
- Gray-level histogram feature maps comparison.
4.3. Comparison of Stability of Improved Models
4.4. Ablation Experiment and Comparison with Other Methods
4.5. Comparison Based on Classification Model Accuracy
5. Discussion
- (1)
- In the experiments, to test the influence of the improved methods on the model performance, both the original CycleGAN model and the improved CycleGAN model were used to perform disease transfer on healthy maize leaves. Overall, compared to the original CycleGAN model, the improved CycleGAN model proposed in this study achieved improvements in the PSNR and SSIM values of the three different disease severity diseases, indicating better image quality of the generated images.
- (2)
- We tested the impacts of different mechanisms on the improved CycleGAN model, the original CycleGAN model, the CycleGAN model with only the attention mechanism, the CycleGAN model with only the feature recombination, and the CycleGAN model incorporating both the attention mechanism and feature recombination; all models were used to perform disease transfer on healthy maize leaves. Objective parameter analysis was conducted by calculating the FID values between the generated maize disease leaf images and the original healthy maize leaf images, and visual comparisons were made using grayscale histograms. It was found that the CycleGAN model with both the attention mechanism and feature recombination generated maize disease leaf images that were closer to real maize leaf images, and the image quality was relatively better.
- (3)
- We validated the impact of the feature recombination loss function on the model’s stability, the generated image quality was compared for maize gray leaf spot images under different background complexity levels. The experiment showed that the generated images from the model with the introduced feature recombination loss function had higher image quality, not limited to the best performance in a single experiment.
- (4)
- Comparative experiments were conducted through ablation experiments with the traditional generation models VAE and Pix2Pix. Maize gray leaf spot disease was taken as the experimental object, and the generated image quality was compared under different background complexity levels. The results showed that compared to the original CycleGAN method, the attention-based CycleGAN method, and the feature recombination-based CycleGAN method, the proposed method in this paper had obvious improvements. Compared with the traditional generation models VAE and Pix2Pix, this proposed method not only enhanced the overall clarity of the images, but also realized the generation of maize disease feature information in specific regions, and the edge information of the generated images was also closer to the original images.
- (5)
- In the experiment to validate the effectiveness of the expanded dataset, we utilized the original CycleGAN method, the improved CycleGAN, and the Pix2Pix method for dataset expansion. The experiment showed that on the expanded maize disease dataset using the improved CycleGAN, the average accuracy of the classification tasks performed by six recognition networks (VGG16, VGG19, ResNet50, DenseNet50, DenseNet121, and GoogLeNet) was the highest. The average accuracy in the GoogLeNet model reached , and the dataset expanded by the generation model achieved an average accuracy exceeding in different classification models.
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Wenxia, B.; Xuefeng, H.; Gensheng, H.; Dong, L. Identification of maize leaf diseases using improved convolutional neural network. Trans. Chin. Soc. Agric. Eng. 2021, 37, 160–167. [Google Scholar]
- Chunshan, W.; Chunjiang, Z.; Wu, H.; Ji, Z.; Jiuxi, L.; Huaji, Z. Recognizing crop diseases using bimodal joint representation learning. Trans. Chin. Soc. Agric. Eng. 2021, 37, 180–188. [Google Scholar]
- Dang, M.; Meng, Q.; Gu, F.; Gu, B.; Hu, Y. Rapid recognition of potato late blight based on machine vision. Trans. Chin. Soc. Agric. Eng. 2020, 36, 193–200. [Google Scholar]
- Kai, S.; Zhikun, L.; Hang, S.; Chunhong, G. A research of maize disease image recognition of corn based on BP networks. In Proceedings of the 2011 third International Conference On Measuring Technology and Mechatronics Automation, Shanghai, China, 6–7 January 2011; Volume 1, pp. 246–249. [Google Scholar]
- Kulkarni, A.H.; Patil, A. Applying image processing technique to detect plant diseases. Int. J. Mod. Eng. Res. 2012, 2, 3661–3664. [Google Scholar]
- MENG, L.; ZHONG, J.p.; LI, N. Generating Algorithm of Medical Image Simulation Data Sets Based on GAN. J. Northeast. Univ. (Nat. Sci.) 2020, 41, 332. [Google Scholar]
- Gong, M.; Chen, S.; Chen, Q.; Zeng, Y.; Zhang, Y. Generative adversarial networks in medical image processing. Curr. Pharm. Des. 2021, 27, 1856–1868. [Google Scholar] [CrossRef] [PubMed]
- Wang, C.; Wu, X.; Zhang, Y.; Wang, W. Recognition and segmentation of maize seedlings in field based on dual attention semantic segmentation network. Trans. Chin. Soc. Agric. Eng 2021, 37, 211–221. [Google Scholar]
- Zhu, L.; Lu, C.; Dong, Z.Y.; Hong, C. Imbalance learning machine-based power system short-term voltage stability assessment. IEEE Trans. Ind. Inform. 2017, 13, 2533–2543. [Google Scholar] [CrossRef]
- Batista, G.E.; Prati, R.C.; Monard, M.C. A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explor. Newsl. 2004, 6, 20–29. [Google Scholar] [CrossRef]
- Elreedy, D.; Atiya, A.F. A comprehensive analysis of synthetic minority oversampling technique (SMOTE) for handling class imbalance. Inf. Sci. 2019, 505, 32–64. [Google Scholar] [CrossRef]
- Chou, H.P.; Chang, S.C.; Pan, J.Y.; Wei, W.; Juan, D.C. Remix: Rebalanced mixup. In Proceedings of the Computer Vision—ECCV 2020 Workshops, Glasgow, UK, 23–28 August 2020; Proceedings, Part VI 16. Springer: Berlin/Heidelberg, Germany, 2020; pp. 95–110. [Google Scholar]
- Pei, A.; Chen, G.; Li, H.; Wang, B. Method for cloud removal of optical remote sensing images using improved CGAN network. Trans. Chin. Soc. Agric. Eng 2020, 36, 194–202. [Google Scholar]
- Jin, L.; Tan, F.; Jiang, S. Generative adversarial network technologies and applications in computer vision. Comput. Intell. Neurosci. 2020, 2020, 1459107. [Google Scholar] [CrossRef] [PubMed]
- Kingma, D.P.; Welling, M. Auto-encoding variational bayes. arXiv Prepr. 2013, arXiv:1312.6114. [Google Scholar]
- Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial networks. Commun. ACM 2020, 63, 139–144. [Google Scholar] [CrossRef]
- Ming-Fei, H.; Xin, Z.; Jian-Wei, L. Survey on deep generative model. Acta Autom. Sin. 2020, 41, 1–34. [Google Scholar]
- Ximing, L.; Jiarun, W.; Shaoqian, W. Gans based privacy amplification against bounded adversaries. J. Front. Comput. Sci. Technol. 2021, 15, 1220. [Google Scholar]
- Xu, Z.; Wu, S.; Jiao, Q.; Wong, H.S. TSEV-GAN: Generative Adversarial Networks with Target-aware Style Encoding and Verification for facial makeup transfer. Knowl.-Based Syst. 2022, 257, 109958. [Google Scholar] [CrossRef]
- Frid-Adar, M.; Diamant, I.; Klang, E.; Amitai, M.; Goldberger, J.; Greenspan, H. GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing 2018, 321, 321–331. [Google Scholar] [CrossRef]
- Liu, B.; Tan, C.; Li, S.; He, J.; Wang, H. A data augmentation method based on generative adversarial networks for grape leaf disease identification. IEEE Access 2020, 8, 102188–102198. [Google Scholar] [CrossRef]
- Jing, Y.; Yang, Y.; Feng, Z.; Ye, J.; Yu, Y.; Song, M. Neural style transfer: A review. IEEE Trans. Vis. Comput. Graph. 2019, 26, 3365–3385. [Google Scholar] [CrossRef]
- Andreini, P.; Bonechi, S.; Bianchini, M.; Mecocci, A.; Scarselli, F. Image generation by GAN and style transfer for agar plate image segmentation. Comput. Methods Programs Biomed. 2020, 184, 105268. [Google Scholar] [CrossRef] [PubMed]
- Ledig, C.; Theis, L.; Huszár, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z.; et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4681–4690. [Google Scholar]
- Bulat, A.; Yang, J.; Tzimiropoulos, G. To learn image super-resolution, use a gan to learn how to do image degradation first. In Proceedings of the European Conference on Computer Vision, (ECCV), Munich, Germany, 8–14 September 2018; pp. 185–200. [Google Scholar]
- Zen, G.; Sangineto, E.; Ricci, E.; Sebe, N. Unsupervised domain adaptation for personalized facial emotion recognition. In Proceedings of the 16th International Conference on Multimodal Interaction, Istanbul, Turkey, 12–16 November 2014; pp. 128–135. [Google Scholar]
- Fanny; Cenggoro, T.W. Deep learning for imbalance data classification using class expert generative adversarial network. Procedia Comput. Sci. 2018, 135, 60–67. [Google Scholar] [CrossRef]
- Liu, W.; Luo, Z.; Li, S. Improving deep ensemble vehicle classification by using selected adversarial samples. Knowl.-Based Syst. 2018, 160, 167–175. [Google Scholar] [CrossRef]
- Brock, A.; Donahue, J.; Simonyan, K. Large scale GAN training for high fidelity.natural image synthesis. arXiv 2018, arXiv:1809.11096. [Google Scholar]
- Chen, F.; Zhu, F.; Wu, Q.; Hao, Y.; Wang, E.; Cui, Y. A survey about image generation with generative adversarial nets. Chin. J. Comput. 2021, 44, 347–369. [Google Scholar]
- Kaneko, T. Generative adversarial networks: Foundations and applications. Acoust. Sci. Technol. 2018, 39, 189–197. [Google Scholar] [CrossRef]
- Isola, P.; Zhu, J.Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1125–1134. [Google Scholar]
- Mirza, M.; Osindero, S. Conditional generative adversarial nets. arXiv Prepr. 2014, arXiv:1411.1784. [Google Scholar]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, 5–9 October 2015; Proceedings, Part III 18. Springer: Berlin/Heidelberg, Germany, 2015; pp. 234–241. [Google Scholar]
- Lin, H.; Ren, S.; Yang, Y.; Zhang, Y. Unsupervised image-to-image translation with self-attention and relativistic discriminator adversarial networks. Zidonghua Xuebao/Acta Autom. Sin. 2021, 47, 2226–2237. [Google Scholar]
- Zhu, J.Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2223–2232. [Google Scholar]
- Zhang, J.; Sun, X.; Chen, Y.; Duan, Y.; Wang, Y. Single-Image Defogging Algorithm Based on Improved Cycle-Consistent Adversarial Network. Electronics 2023, 12, 2186. [Google Scholar] [CrossRef]
- Lu, L.; Liu, W.; Yang, W.; Zhao, M.; Jiang, T. Lightweight corn seed disease identification method based on improved shufflenetv2. Agriculture 2022, 12, 1929. [Google Scholar] [CrossRef]
- Hu, L.; Zhang, Y. Facial image translation in short-wavelength infrared and visible light based on generative adversarial network. Acta Opt. Sin. 2020, 40, 0510001. [Google Scholar]
- Li, Q.; Bai, W.; Niu, J. Underwater image color correction and enhancement based on improved cycle-consistent generative adversarial networks. Acta Autom. Sin. 2020, 46, 1–11. [Google Scholar]
- Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 2921–2929. [Google Scholar]
- Kim, T.; Cha, M.; Kim, H.; Lee, J.K.; Kim, J. Learning to discover cross-domain relations with generative adversarial networks. In Proceedings of the International Conference on Machine Learning, PMLR, Fort Lauderdale, FL, USA, 20–22 April 2017; pp. 1857–1865. [Google Scholar]
- Yi, Z.; Zhang, H.; Tan, P.; Gong, M. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2849–2857. [Google Scholar]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef]
- Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Chen, Y.; Pan, J.; Wu, Q. Apple leaf disease identification via improved CycleGAN and convolutional neural network. Soft Comput. 2023, 27, 9773–9786. [Google Scholar] [CrossRef]
- Xiao, D.; Zeng, R.; Liu, Y.; Huang, Y.; Liu, J.; Feng, J.; Zhang, X. Citrus greening disease recognition algorithm based on classification network using TRL-GAN. Comput. Electron. Agric. 2022, 200, 107206. [Google Scholar] [CrossRef]
- Liu, W.; Zhai, Y.; Xia, Y. Tomato Leaf Disease Identification Method Based on Improved YOLOX. Agronomy 2023, 13, 1455. [Google Scholar] [CrossRef]
Network Module | Area | Input Dimension →Output Dimension | Number and Size of Core | Network Layer Information |
---|---|---|---|---|
Generator | Subsampled | 3*ln →64*m*n | 7*7*64 | Convolutional layer, IN,ReLU,stride 1 |
64*m*n →128*m/2*n/2 | 3*3*128 | Convolutional layer, IN,ReLU,stride 2 | ||
128*m/2*n/2 →256*m/4*n/4 | 3*3*256 | Convolutional layer, IN,ReLU,stride 2 | ||
Residual network | 256*m/4*n/4 →256*m/4*n/4 | 3*3*256 | 3 residual blocks,IN, ReLU,stride 1 | |
Attention | 256*m/4*n/4 →512*m/4*n/4 | 3*3*512 | Max pooling and average pooling,stride 1 | |
512*m/4*n/4 →256*m/4*n/4 | 3*3*256 | Convolutional layer, ReLU,stride 1 | ||
Residual network | 256*m/4*n/4 →256*m/4*n/4 | 3*3*256 | 3 residual blocks, IN,ReLU,stride 1 | |
Upsampling | 256*m/4*n/4 →128*m/2*n/2 | 3*3*128 | Convolutional layer, IN,ReLU,stride 2 | |
128*m/2*n/2 →64*m/2*n/2 | 3*3*64 | Convolutional layer, IN,ReLU,stride 2 | ||
64*m/2*n/2 →3*m*n | 7*7*3 | Convolutional layer, Tanh,stride 2 | ||
Discriminator | Subsampling | 3*m*n →64*m/2*n/2 | 4*4*64 | Convolutional layer, AdaIN,ReLU,stride 2 |
64*m/2*n/2 →128*m/4*n/4 | 4*4*128 | Convolutional layer, AdaIN,ReLU,stride 2 | ||
128*m/4*n/4 →256*m/8*n/8 | 4*4*256 | Convolutional layer, AdaIN,ReLU,stride 2 | ||
256*m/8*n/8 →512*m/16*n/16 | 4*4*512 | Convolutional layer, AdaIN,ReLU,stride 2 | ||
Attention | 512*m/16*n/16 →1024*m/16*n/16 | 4*4*1024 | Max pooling and average pooling,stride 1 | |
1024*m/16*n/16 →512*m/16*n/16 | 4*4*512 | Convolutional layer, ReLU,stride 1 | ||
Classifier | 512*m/16*n/16 →1*m/16*n/16 | 4*4*1 | Convolutional layer, ReLU,stride 1 |
Module | CycleGAN | The Attention-Based CycleGAN Method | The Feature Recombination-Based CycleGAN Method | The Incorporating Attention Mechanism and Feature Recombination CycleGAN Method |
---|---|---|---|---|
gray leaf spot (minor/severe) | 158.26/172.17 | 114.26/128.61 | 124.71/136.19 | 110.84/124.71 |
rust disease (minor/severe) | 167.34/181.59 | 113.71/135.86 | 119.93/141.03 | 109.73/131.16 |
leaf spot disease (minor/severe) | 161.32/175.99 | 112.36/127.41 | 127.53/139.76 | 108.36/124.83 |
Method | PSNR/dB | SSIM | ||
---|---|---|---|---|
Mean | Variance | Mean | Variance | |
minor | 23.13 | 0.0083 | 0.89 | 0.0011 |
severe | 20.89 | 0.0012 | 0.81 | 0.0028 |
Method | PSNR/dB | SSIM | ||
---|---|---|---|---|
Minor | Severe | Minor | Severe | |
This research method | 23.13 | 20.89 | 0.89 | 0.81 |
The original CycleGAN method | 18.47 | 14.11 | 0.81 | 0.73 |
The attention-based CycleGAN method | 21.83 | 18.64 | 0.86 | 0.78 |
The feature recombination-based CycleGAN method | 20.02 | 16.91 | 0.83 | 0.75 |
The VAE (Variational Autoencoder) method | 12.11 | 10.38 | 0.71 | 0.67 |
The Pix2Pix method | 22.76 | 20.17 | 0.88 | 0.79 |
Model | VGG16 | VGG19 | DenseNet50 | DenseNet121 | ResNet50 | GoogLeNet |
---|---|---|---|---|---|---|
A | 85.19% | 86.16% | 84.12% | 87.71% | 87.27% | 89.03% |
B | 88.13% | 89.57% | 87.63% | 89.12% | 89.12% | 91.39% |
C | 91.11% | 91.79% | 92.33% | 93.17% | 92.93% | 93.64% |
D | 89.94% | 90.73% | 90.91% | 92.14% | 91.68% | 92.56% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Guo, H.; Li, M.; Hou, R.; Liu, H.; Zhou, X.; Zhao, C.; Chen, X.; Gao, L. Sample Expansion and Classification Model of Maize Leaf Diseases Based on the Self-Attention CycleGAN. Sustainability 2023, 15, 13420. https://doi.org/10.3390/su151813420
Guo H, Li M, Hou R, Liu H, Zhou X, Zhao C, Chen X, Gao L. Sample Expansion and Classification Model of Maize Leaf Diseases Based on the Self-Attention CycleGAN. Sustainability. 2023; 15(18):13420. https://doi.org/10.3390/su151813420
Chicago/Turabian StyleGuo, Hongliang, Mingyang Li, Ruizheng Hou, Hanbo Liu, Xudan Zhou, Chunli Zhao, Xiao Chen, and Lianxing Gao. 2023. "Sample Expansion and Classification Model of Maize Leaf Diseases Based on the Self-Attention CycleGAN" Sustainability 15, no. 18: 13420. https://doi.org/10.3390/su151813420
APA StyleGuo, H., Li, M., Hou, R., Liu, H., Zhou, X., Zhao, C., Chen, X., & Gao, L. (2023). Sample Expansion and Classification Model of Maize Leaf Diseases Based on the Self-Attention CycleGAN. Sustainability, 15(18), 13420. https://doi.org/10.3390/su151813420