A Deep Learning-Based Crop Disease Diagnosis Method Using Multimodal Mixup Augmentation

Abstract

With the widespread adoption of smart farms and continuous advancements in IoT (Internet of Things) technology, acquiring diverse additional data has become increasingly convenient. Consequently, studies relevant to deep learning models that leverage multimodal data for crop disease diagnosis and associated data augmentation methods are significantly growing. We propose a comprehensive deep learning model that predicts crop type, detects disease presence, and assesses disease severity at the same time. We utilize multimodal data comprising crop images and environmental variables such as temperature, humidity, and dew points. We confirmed that the results of diagnosing crop diseases using multimodal data improved 2.58%p performance compared to using crop images only. We also propose a multimodal-based mixup augmentation method capable of utilizing both image and environmental data. In this study, multimodal data refer to data from multiple sources, and multimodal mixup is a data augmentation technique that combines multimodal data for training. This expands the conventional mixup technique that was originally applied solely to image data. Our multimodal mixup augmentation method showcases a performance improvement of 1.33%p compared to the original mixup method.

1. Introduction

Crop diseases pose significant challenges, as they contribute directly to reduction in both crop yield and quality. Hence, the early detection, treatment, and prevention of crop diseases play a vital role in safeguarding farmers’ interests. There are several methods for diagnosing crop diseases, including diagnostic kits [1,2,3], genetic analysis [4,5], and the visual observation of symptoms [6]. Symptoms manifested on leaves, stems, and fruits serve as visual indicators for identifying diseased crops. However, the direct identification and diagnosis of crop diseases by human eyes face limitations due to the diverse types and varying progression of the diseases observed in different crops. Consequently, numerous automatic crop disease diagnosis technologies employing computer vision and deep learning have been developed.

Deep learning has demonstrated excellent performance compared to other image processing methods in automated disease diagnosis. Ferentinos et al. achieved a 99.53% accuracy rate in classifying 58 crop diseases using VGG networks [7], and Picon et al. employed the Resnet for classifying crop diseases collected via mobile devices [8]. In addition, many studies have been conducted to compare the performance based on model architectures or to improve the performance by modifying the model structures [9,10,11]. M. Arsenovic et al. [12] and J. Ahmad et al. [13] corrected low-quality crop images in real environments and used data augmentation methods that simulate real-world conditions to train deep learning models for crop disease diagnosis that perform well in actual environments.

The increase in adoption of smart farms, combined with recent advancements in IoT technology, has facilitated the collection of a broad spectrum of time-series data including temperature, humidity, and CO₂ concentration. Not all environmental data refer to time-series data, but in this study, we assume that environmental data refer to time-series data. Nalini et al. utilized the KNN (K-Nearest Neighbor) classifier and the ensemble method to predict crop diseases based on measured temperature data [14]. Mishra et al. designed a system equipped with sensors for the real-time measurement of temperature and humidity, employing a decision tree algorithm to predict the suitability of the environment for crop cultivation [15].

Previous studies have focused mostly on utilizing either image data or environmental data alone. Currently, few studies predict crop diseases using multimodal data, which comprise diverse types of information with distinct characteristics. Furthermore, there is a lack of research on data augmentation techniques to compensate for insufficient data in methods utilizing time-series data.

To address these issues, we propose a deep learning model that utilizes both crop images and environmental data, including temperature, humidity, and dew point, to predict crop diseases. The assumption that image data and environmental data for crops can play a complementary role in disease prediction is inherent in this approach. For crop leaves affected by powdery mildew, if the image quality is poor or the size of the powdery mildew spots is too small, a deep learning model might incorrectly classify the crop as healthy. However, if environmental variables such as temperature and humidity are also provided, it could help make a more accurate diagnosis. We will experimentally confirm that this assumption is correct in Section 5.

We conduct a comparative analysis of disease prediction performance for the one utilizing either image data or environmental data alone, and for utilizing multimodal data. We also introduce a multimodal mixup augmentation method, which enables the simultaneous application of crop images and environmental data and experimentally demonstrates its effectiveness in improving prediction performance.

2. Related Works

2.1. Deep Learning Approach to Crop Disease Diagnosis Using Multimodal Data

Multimodal data refer to the input of data from various sensors, including visual, auditory, and temperature sensors, encompassing diverse types of information with distinct characteristics. In deep learning, the utilization of multimodal data for learning purposes is commonly known as multimodal deep learning [16].

The analysis of multimodal data using deep learning has been studied in various fields, including healthcare [17,18], automotive [19], and facial recognition [20] domains. Rastgoo et al. introduced a deep learning-based method for the driver’s stress level classification using ECG (electrocardiogram) signals, vehicle dynamics data, and environmental data [19]. It was found that when using multimodal data, the accuracy increased by up to 14.9% compared to when using unimodal data (ECG signals). J.C. Vásquez-Correa et al. analyzed data from speech, handwriting, and gait using a multimodal deep learning model to diagnose Parkinson’s disease [18]. When multimodal data were used, the accuracy improved by 5.3% compared to using unimodal data (speech).

However, most of the deep learning studies for crop disease diagnosis have relied heavily on unimodality (e.g., mostly images). They have leveraged structural improvements of a CNN (convolutional neural network), utilization of multiple resolutions, and ensembles the maximization performance from a given single datum [21,22,23].

In this study, we would like to verify whether performance improves when environmental data as well as images are used to diagnose diseases of crops. We would like to emphasize that in this study, we do not propose a novel model structure for processing multimodal data. To date, most studies on diagnosing crop diseases using deep learning have focused on data using a single modality. However, in this study, we aim to demonstrate that a deep learning model using multimodal data outperforms a model using unimodal data. To the best of our knowledge, this is the first attempt to perform crop disease diagnosis using multimodal data and confirm performance improvement compared to unimodal data.

2.2. Mixup Augmentation

Conventional deep learning models are trained using the ERM (Empirical Risk Minimization) method, which aims to minimize the occurrence of mispredictions. However, the ERM method often exhibits overconfidence in predicting a specific class, even for data points that are located in the vicinity of the distribution boundaries. To address the limitation of ERM, mixup—a data augmentation technique—is introduced [24]. It involves creating augmented samples by applying linear interpolation to pairs of images and labels. Specifically, two samples with different labels are randomly selected from the training dataset, and their images and labels are linearly interpolated to generate new samples. This process results in the creation of new data points that lie on the line connecting the original data points in the feature space. By doing so, mixup allows the model to leverage the distribution information present in the neighborhood of the training data. As a result, the model becomes more robust and capable of better generalizing better to unseen data, as it learns from a wider range of data points and distributions. Mixup effectively regularizes the model, preventing it from memorizing the training samples and improving its generalization ability. We note that mixup augmentation applications have primarily focused on unimodal data augmentation, such as video or audio data [25].

In this paper, we propose a multimodal mixup augmentation method that extends the unimodal mixup augmentation approach. In multimodal mixup, we can combine images with environmental data, leading to more comprehensive and robust models that can handle diverse data types and improve performance across different domains.

3. Materials and Methods

3.1. Dataset Description

We used the dataset from Korea AI Challenge, the ‘Crop Disease Diagnosis based on Agricultural Environmental Changes’. The data collection for the challenge was conducted by LG AI Research, and the challenge was hosted by Dacon Inc. (Seoul, Republic of Korea). The data can be downloaded from the Dacon website [26]. The dataset containing labels consists of 4675 samples. The provided samples consist of image data and environmental data from the same crop. The infected leaves, vegetables, or fruits with diseases are placed in the center of the image. The environmental data include the information of temperature, humidity, and dew point recorded at 10 min intervals for at least 48 h before capturing the crop disease images. The environmental data are provided in the form of comma-separated value files. One of the examples of the given crop image and environmental data is shown in Figure 1.

The objective of the AI challenge is to predict the crop type, disease type, and severity of the disease using the given crop image and environmental data. In this study, we intend to define the term “Crop Disease Diagnosis” in the same way as in the AI Challenge, meaning the simultaneous classification of crop type, disease type, and disease severity. This approach allows us to use the class labels from the dataset utilized in the AI Challenge. The dataset provides class labels for 6 types of crops (including tomatoes, strawberries, and bell peppers), 9 types of diseases (such as normal, powdery mildew, and various nutrient deficiencies), and different levels of infection severity (early, intermediate, and terminal).

Table 1. Class types and the number of samples utilized for crop disease classification. N, P, K, and Ca denote deficiencies in the macronutrient elements nitrogen, phosphorus, potassium, and calcium, respectively.

Crop	Disease	Severity	The Number of Samples
Strawberry	Normal		810
Tomato	Normal		143
	Powdery Mildew	Intermediate	189
Paprika	Normal		1177
	Powdery Mildew	Early	154
		Intermediate	111
		Terminal	42
	Ca	Early	166
	N		142
	P		156
	K		153
Cucumber	Normal		917
Chili	Normal		69
	Anthracnose	Intermediate	99
	N	Early	148
	P		159
	K		157
Grape	Normal		828
	Anthracnose	Early	40
		Intermediate	12
	Powdery Mildew	Early	13
		Intermediate	29
	Sunscald	Early	18
		Intermediate	14
	Corky Core	Early	21
Total			5767

shows 25 classes of the dataset provided.

3.2. Multimodal Deep Learning

The deep learning model architecture used in this study is illustrated in Figure 2. This architecture includes an LSTM (long short-term memory) module for processing environmental data and a CNN (convolutional neural network) for image data. For the CNN model, we used a pretrained Resnet 50 [27] from the ImageNet dataset [28], which is one of the most commonly used networks for image-based deep learning research. After passing through the CNN, crop images are embedded into low-dimensional features with a dimension of 25. Environmental data include temperature, humidity, and dew point. These data are concatenated and passed through an LSTM module, which are then mapped to a feature with a dimension of 200. Next, they are reduced to a low-dimensional feature with a dimension of 25, similar to the image data, by passing through two fc (fully connected) layers. The low-dimensional features from each modality are then concatenated to create a single dimensional feature, which is passed through a final fc layer to generate scores for 25 classes. To prevent overfitting, dropout layers were added at the output of the CNN and at the final layer of the LSTM.

3.3. Multimodal Mixup Augmentation

Equation (1) describes the newly generated data by unimodal mixup augmentation. $x_1$ and $x_2$ represent randomly sampled image data, and $y_1$ and $y_2$ represent the corresponding labels. λ refers to the weight used when mixing two data, and it is sampled from a beta distribution with parameters α and β. That is, λ∼Beta (α, β) for a positive constant α. For convenience, we used both α and β values of 1.0. This implies that λ is sampled from a uniform distribution over the interval [0, 1]. Then, the augmented data $\overline{x}$ and $\overline{y}$ can be generated by linearly interpolating each raw data and label.

$$\overline{\mathit{x}}=\mathit{\lambda }{\mathit{x}}_1+\left(1-\mathit{\lambda }\right){\mathit{x}}_2 \mathit{a}\mathit{n}\mathit{d} \overline{\mathit{y}}=\mathit{\lambda }{\mathit{y}}_1+\left(1-\mathit{\lambda }\right){\mathit{y}}_2$$

(1)

We can also infer that the change in the value of lambda affects the intensity and impact of data augmentation. As λ approaches 0.5, the augmentation becomes stronger, while approaching 0 or 1 makes the augmentation weaker. Also, as the α value decreases, the mixup augmentation effect decreases.

To apply multimodal mixup augmentation properly, we added the augmented environmental data $\overline{z}$ as Equation (2). $z_1$ and $z_2$ represent environmental data, with the corresponding labels $y_1$ and $y_2$.

$$\overline{\mathit{z}}=\mathit{\lambda }{\mathit{z}}_1+\left(1-\mathit{\lambda }\right){\mathit{z}}_2$$

(2)

If the data are multimodal, it is desirable to generate augmented data by linearly combining all modality data. For example, if mixup is performed only for the image modality as in the conventional method, the synthesized labels may play the role of noisy labels for the sequence model that takes environmental data as input, potentially leading to unfavorable learning outcomes.

To linearly interpolate each sampled data, it is necessary to first align their data sizes. The size of the image data sampled was equal; however, the length of the environmental data varied. In general, this can be achieved through resampling, interpolation, or other methods to match the predetermined size. In this study, a crop or padding strategy was used for the convenience of implementation. When the length of the environmental data was excessively long, we cropped them—fixing the size at 320—whereas when they were too short, we applied zero-padding to ensure it reached a length of 320.

Figure 3 shows an example of a proposed mixup augmentation for multimodal data. The top images represent crop images, while the bottom graphs represent environmental data normalized with min-max normalization. Figure 3a represents Paprika-Deficiency (P)-Early, (b) shows Paprika-Normal, and (c) illustrates the results of mixup augmentation. The multimodal data from (a) and (b) were linearly interpolated with a weight of 0.5 each, resulting in synthesized data (c). The synthesized data have labels of 0.5 × ‘Paprika-Deficiency (P)-Early’ and 0.5 × ‘Paprika-Normal’.

4. Experiments

We experimentally analyzed the crop disease classification performance. We compared the result from both image and environmental data with that from crop image data alone. We also compared the classification performance of the proposed multimodal mixup data augmentation method with that of unimodal mixup data augmentation on image data. We utilized the dataset from the Korean AI Challenge, which is introduced in Section 3.1. We divided the dataset into a ratio of 5:3:2 for training, validation, and testing sets, respectively. Due to the imbalanced distribution of data samples, we oversampled for the classes with insufficient sample size during the training phase. The macro F1 score was employed as the performance metric for crop disease diagnosis prediction. This is commonly used for multi-class classification and was also employed as a performance metric in the AI challenge. The F1 score can be expressed as shown in Equation (3), where ${TP}_i$, ${FP}_i$, and ${FN}_i$ are, respectively, the number of true positive samples (correctly classified positive samples), false positive samples (incorrectly classified positive samples), and false negative samples (incorrectly classified negative samples) for a given $i$-th class, and $n$ is the number of classes.

$$\mathit{F}1=\frac{1}{\mathit{n}}\sum _{\mathit{i}}^{\mathit{n}}\frac{{\mathit{T}\mathit{P}}_{\mathit{i}}}{{\mathit{T}\mathit{P}}_{\mathit{i}}+({\mathit{F}\mathit{P}}_{\mathit{i}}+{\mathit{F}\mathit{N}}_{\mathit{i}})/2}$$

(3)

All experiments were conducted in an environment with an Intel(R) Core(TM) i9-10940x CPU@ 3.30 GHz, DDR 128 GB, and NVIDIA RTX3090. The implementation was done using PyTorch version 1.13.0. In all experiments, we used a minibatch size of 8. Also, we used an Adam optimizer [29] to train the deep learning model over 300 epochs, where the initial learning rate was set to 0.001, and the learning rate scheduler was the cosine-annealed warm restart learning scheduler. Adam optimizer is one of the most commonly used in deep learning. It adaptively adjusts learning rates based on the first and second moments of the gradients, which allows for faster convergence and robustness to sparse gradients.

All image data were resized to 256 × 256 and normalized using the mean and standard deviation values used in ImageNet training [28]. Environmental data were normalized using min-max normalization so that the data values could fall between 0 and 1. Then, zero-padding or cropping was performed so that the length of all data could be 320. Augmentation methods including scaling, rotation, translation, flipping, and coarse dropout were only applied to image data.

5. Results

5.1. Comparative Analysis of Crop Disease Diagnosis Performance

Table 2. Crop disease diagnosis F1 score (%) for the testing dataset by data modality.

CNN Model	Environmental Data	Image	Multimodal
Resnet 50	80.84 ± 0.91 (LSTM)	87.79 ± 0.70	90.37 ± 0.59
DenseNet 121		90.60 ± 0.46	92.02 ± 0.74
Xception		88.07 ± 0.89	90.68 ± 0.33
MobileNet V3		89.27 ± 0.73	90.56 ± 0.46
EfficientNet V2-Small		90.34 ± 0.53	92.15 ± 1.18

presents the classification performance of crop disease diagnosis for the testing dataset with respect to data modality. We conducted the same experiment three times, only varying the initial training conditions (e.g., random seed). Furthermore, we present the average F1 score, along with its estimation under a 95% confidence interval, in Table 2.

We expected that multimodal data would demonstrate a wider range of features compared to unimodal data, such as environmental or image data. As expected, the best classification performance was achieved when using multimodal data (e.g., average F1 score of Resnet 50 is 90.37%). However, we also noticed that using only environmental data had a noticeable impact on the occurrence of crop diseases, as a performance of F1 score 80.84% was observed.

Additionally, we conducted iterative experiments by changing CNN models to show that the proposed method was not limited to a particular model. And, we confirmed once again that the performance was the best when environmental data and image data are used together. The models used were DenseNet 121 [30], Xception [31], MobileNetV3 [32], and EfficientNetV2-Small [33]. To improve image classification performance, these models reference the output of all previous layers (DenseNet 121), change the layer structure to increase accuracy while reducing computational cost (Xception, MobileNetV3), or systematically optimize the model’s structure (EfficientNetV2-Small). Since these models have different architectures, they can extract different features and make different judgments, even from the same images. Our experiments with these different architectures demonstrate that the proposed method is robust across various CNN models.

In Section 1, we assumed that image data and environmental data play a complementary role in predicting crop disease diagnosis. Here, we will confirm this assumption through two case studies of crop disease prediction results based on different data modalities. Figure 4 shows an image of a paprika leaf infected with intermediate-stage powdery mildew along with the associated environmental data. When only environmental data were used, the model accurately predicted the crop type and disease. However, when only images were used, the model correctly identified the crop type but incorrectly classified it as healthy. In fact, since the disease was in its early stage, the powdery mildew spores were barely visible in the images. When both image and environmental data were used, the model correctly identified both the crop type and the disease.

Figure 5 shows an image of chili in the early stage of macronutrient element N (nitrogen) deficiency, along with its environmental data. When only environmental data were used, the deep learning model misclassified the crop type and disease. However, when using only image data, the model accurately identified both the crop type and disease. When using both image and environmental data, the model also accurately identified the crop type and disease.

These two cases show that while a deep learning model using a single modality might misclassify crop diseases, it predicted accurately when using multimodal data. This implies that multimodal data can complement each other and enhance the accuracy of crop disease diagnosis.

Table 3. Crop disease diagnosis F1 score (%) for the testing dataset by data augmentation method.

CNN Model	Without Mixup	Image-Only Mixup	Multimodal Mixup
Resnet 50	90.37 ± 0.59	90.93 ± 0.34	92.26 ± 0.79
DenseNet 121	92.02 ± 0.74	92.08 ± 0.42	92.66 ± 0.39
Xception	90.68 ± 0.33	90.97 ± 0.90	91.95 ± 0.56
MobileNet V3	90.56 ± 0.46	91.17 ± 0.80	91.94 ± 0.64
EfficientNet V2-Small	92.15 ± 1.18	90.85 ± 0.46	93.22 ± 0.61

shows the result from crop disease diagnosis with respect to mixup augmentation. Using multimodal mixup augmentation resulted in the best performance, regardless of CNN model types. This implies that the proposed multimodal mixup augmentation method shows generalized performance not limited to specific model architectures.

In Figure 6 and Figure 7, the confusion matrix illustrates the predictions of a deep learning model trained with different data modality. In the confusion matrix, a square box drawn with bold blue lines means that the labels in the box belong to the same crops but have normal status or different disease severity. For example, in the box located at the top left corner, there are two labels for normal and powdery mildew with intermediate severity for tomato. The intensity of the box color represents the number of data. The darker blue, the more data it represents, and the lighter one, the fewer data it represents.

The F1 score for crop disease classification performance is 89.5% when only image data are used, which is 8.6%p higher than when only environmental data are used. On the other hand, when the type of crop is correctly predicted, there are quite a few cases where the severity of disease is also correctly predicted. From the facts above, we conclude that environmental data alone do not have a significant impact on crop type classification, but they provide sufficient information to classify crop diseases. When only image data are used, most of the types of crops are well predicted.

However, there are quite a few cases where, even though the type of crop is correctly predicted, the type and severity of the disease were incorrectly predicted. Consequently, environmental data and image data seem to be able to play complementary roles in deep learning model learning. In fact, using multimodal data, as shown in Table 2, resulted in more than 2%p performance improvement compared to using unimodal data.

Figure 8 and Figure 9 show the confusion matrix for two mixup augmentation methods, that is, image-only mixup and multimodal mixup. Incorporating both environmental and image data notably enhances performance in both cases compared to using data from a single modality. In particular, when the mixup augmentation method is applied to both image and environmental data, the recall score improved by 2.7%p compared to the image-only mixup method. The increase in the recall score indicates improved disease diagnosis performance when crops are diseased. The LSTM module demonstrates resistance to overfitting, and its generalization capability is enhanced through the multimodal mixup augmentation.

5.2. Computational Cost Comparison

We also investigated the impact of including environmental data on the model’s complexity and inference speed. For the environmental data, we utilized the simplest forms of the LSTM layer and two fully connected layers, resulting in a model with low complexity, as shown in

Table 4. Comparison of model complexity and inference time.

Modality of Data	The Number of Params. (M)	FLOPs (G)	F1 Score (%)
Environmental Data	0.19	0.11	80.84 ± 0.91
Image	23.56	10.76	87.79 ± 0.70
Multimodal	23.75	10.88	90.37 ± 0.59

. However, this model exhibited the lowest F1-score among the three modalities. Interestingly, when combining the model for environmental data with Resnet 50 for image data, we observed a 2.5%p increase in F1-score compared to Resnet 50 alone, despite the slight increase in model complexity and computational volume being negligible in comparison.

6. Discussion

In this study, we proposed a multimodal-based crop disease diagnosis method that uses both image and environmental data. We also extended the mixup augmentation, commonly used in the image domain, to be applicable to multimodal data.

Previous studies on crop disease diagnosis have focused primarily on a single modality, such as images or environmental data. To improve accuracy, they have worked on refining model architectures [34,35,36], collecting new data to simulate real-world conditions [12,13], or compressing models for real-time detection [37,38,39]. Recently, with the advancement of IoT technologies like low-power sensors and remote-sensing, it has become easier to acquire various multimodal data. Given these changes, we explored a crop disease diagnosis approach using multimodal data, rather than focusing on enhancing performance with a single modality data. The underlying assumption of this approach is that multimodal data, combining image and environmental data, can be complementary. Experimental results showed that the crop disease diagnosis performance was as follows: environmental data (80.84%) < image data (87.79%) < multimodal data (90.37%). Additionally, we expanded the mixup augmentation method, widely used in deep learning for images, for multimodal deep learning, and observed a 1.33%p performance improvement compared to the original mixup augmentation [24].

This study has several limitations. The first is having performed validation with different datasets, but it is challenging to obtain paired image and environmental data from crops. Our team is searching for new datasets to validate the proposed study and to explore new follow-up research. Another limitation is the difficulty of applying the method in real-world environments. The data used in this study were carefully curated, but real-world images can have different shooting angles and varying lighting conditions. Finding the best combination of data augmentation methods might be important. Environmental data collected from sensors can be noisy. Sensor errors might require signal processing, and missing data may need imputation. The model for environmental data used a simple LSTM structure with two fully connected layers, but more complex models might be required in real-world scenarios.

7. Conclusions

A multimodal deep learning model architecture has been presented to enhance the prediction performance of crop disease diagnosis models. Applying multimodal data, using both the image and environmental data simultaneously in experiments, shows better performance (by 2.58%p in the test dataset) against applying unimodal data on an image. This enhancement comes with a negligible increase in model complexity. The comparison between the result from the multimodal mixup augmentation approach and that from the unimodal one indicates that the multimodal mixup augmentation approach has a 1.33%p performance improvement in the test dataset F1 score for the Resnet 50 CNN model. Our results demonstrate that the best performance on the test dataset was achieved when the multimodal mixup augmentation was applied to both images and environmental data. This finding underscores the fact that utilizing multimodal data, which includes crop images and environmental data, along with appropriately extended mixup augmentation not only enhances disease classification performance but also improves the model’s generalization abilities.