Automatic Tandem Dual BlendMask Networks for Severity Assessment of Wheat Fusarium Head Blight

Abstract

Fusarium head blight (FHB) disease reduces wheat yield and quality. Breeding wheat varieties with resistance genes is an effective way to reduce the impact of this disease. This requires trained experts to assess the disease resistance of hundreds of wheat lines in the field. Manual evaluation methods are time-consuming and labor-intensive. The evaluation results are greatly affected by human factors. Traditional machine learning methods are only suitable for small-scale datasets. Intelligent and accurate assessment of FHB severity could significantly facilitate rapid screening of resistant lines. In this study, the automatic tandem dual BlendMask deep learning framework was used to simultaneously segment the wheat spikes and diseased areas to enable the rapid detection of the disease severity. The feature pyramid network (FPN), based on the ResNet-50 network, was used as the backbone of BlendMask for feature extraction. The model exhibited positive performance in the segmentation of wheat spikes with precision, recall, and MIoU (mean intersection over union) values of 85.36%, 75.58%, and 56.21%, respectively, and the segmentation of diseased areas with precision, recall, and MIoU values of 78.16%, 79.46%, and 55.34%, respectively. The final recognition accuracies of the model for wheat spikes and diseased areas were 85.56% and 99.32%, respectively. The disease severity was obtained from the ratio of the diseased area to the spike area. The average accuracy for FHB severity classification reached 91.80%, with the average F1-score of 92.22%. This study demonstrated the great advantage of a tandem dual BlendMask network in intelligent screening of resistant wheat lines.

1. Introduction

Wheat is one of the main food sources for humans, contributing about 20% of the total dietary calories and proteins [1]. With the growing world population, the steady increase in wheat production is of great significance. Wheat production has traditionally been threatened by various diseases, pests, and abiotic stresses. According to statistics, the global wheat yield loss caused by fungal diseases is as high as 15% to 20% [2]. Among them, wheat Fusarium head blight (FHB) is one of the most harmful fungal diseases. It is mainly caused by Fusarium graminearum, infecting spike flowers at the flowering stage and expanding along the panicle axis during grain filling and maturation. The production and accumulation of toxins, such as deoxynivalenol (DON), Fusarium nivalenol (NIV), and zearalenol (ZEN) [3], can reduce the yield and quality of wheat, causing great harm to human and animal health [4]. The breeding of FHB-resistant varieties is one of the most important means to mitigate the effects of the disease. In order to develop resistant varieties, hundreds of lines must be assessed each year for FHB severity. Protocols for assessing FHB resistance often rely on manual detection. The severity of FHB in wheat can be accurately scored by counting infected spikelets and calculating its percentage in total spikelets [5]. However, this traditional approach is time-consuming, labor-intensive, and prone to human error. Thus, there is an important need to develop a more effective, non-destructive, and high-throughput approach to assess this disease in the field.

The commonly used methods for FHB detection mainly focus on visual analysis, chromatography, polymerase chain reaction (PCR), and enzyme-linked immunosorbent assay (ELIS). Inspection by experienced experts is prone to subjective interference and human error. Biochemical methods, such as chromatography and ELIS, are very accurate, but they often require complex processing steps, which are not suitable for analyzing a large number of FHB-infected wheat spikes in reality [6]. In recent years, imaging and spectroscopic methods, including near infrared spectroscopy (NIRS) and hyperspectral imaging (HSI), have shown strong potential in agriculture and food, especially in crop disease detection [7,8,9,10,11,12,13,14,15,16]. NIRS is based on differences in the absorption, emission, or transmission of light by substances, whose fingerprint features are related to changes in the apparent color, internal composition, and structure of the sample [17]. Peiris et al. [18] proposed an automated single-kernel NIRS method for classifying healthy and FHB-infected wheat with accuracy as high as 99.9%. However, NIRS can only obtain point features of the kernel and cannot achieve large-scale and rapid classification. As a non-invasive, high-throughput, and remote sensing method for plant phenotyping, HSI can merge the spatial and spectral information into a 3D data matrix, but the data in the matrix is too large to be used in real-time analysis [4,19].

With the development of artificial intelligence, color imaging based on machine learning has made great progress. Conventional machine learning methods, such as random forest (RF), K-nearest neighbor (KNN), linear discriminant analysis (LDA), and partial least squares discriminant analysis, have been widely used for crop plant detection and disease evaluation [4,9]. However, these methods perform poorly on large-scale datasets and complex feature scenarios. As a new field of machine learning, deep learning (DL) has gradually begun to show its advantages in image classification, object detection, and natural language processing [20,21]. Deep learning methods have become a preferred approach for disease identification in agricultural fields [22]. Convolutional neural networks (CNN) have been widely used in agriculture in recent years by extracting key features that use different combinations of layers, the translation invariance of convolutional operators, and spatial relationships between adjacent data. Classical CNNs, such as LeNet-5 [23], AlexNet [24], ResNet [25], and VGG [26] were successfully employed for plant disease detection [27,28]. By combining the machine learning and deep learning methods, Hassan SM et al. [29] proposed two methods including shallow VGG with RF and shallow VGG with Xgboost to identify the diseases in corn, potato, and tomato, with the average accuracy as high as 95.70%. Hassan et al. [30] proposed a novel CNN model based on the inception and residual connection to classify the disease of four different plants. The testing accuracies on plant village, rice, and cassava datasets were 99.39%, 99.66%, and 76.59%, respectively. In practical application, the increase of convolutional layers and convolutional kernels of a CNN can enable the model to extract more abstract and refined features, thereby exhibiting excellent performance [26]. However, this procedure may cause the CNN to lose focus on the features and suffer from a vanishing gradient. Residual modules effectively create shortcuts in a sequential network. It leverages a shortcut connection method to weaken the continuous multiplication effect in gradient backpropagation to solve the vanishing gradient problem [25,31]. Girshick, the first author to apply deep learning for object detection, used the region-based convolutional neural network (R-CNN) model to increase the detection rate from 35.1% to 53.7% on the PASCAL VOC dataset [32]. Subsequently, Girshick launched Fast R-CNN and Faster R-CNN network models based on R-CNN, which greatly improved the accuracy and speed of the algorithm in wheat spike detection [33,34,35]. A pulse-coupled neural network (PCNN) with K-means clustering of the improved artificial bee colony (IABC) was developed by Zhang et al. [36] to segment wheat spikes infected with FHB. However, in that research, only one spike in the picture was taken into consideration, which was not practical for high throughput detection in the field condition. Kumar et al. [37] used a deep convolution neural network (DCNN) to automatically classify four wheat rust diseases, achieving an accuracy of 97.16%.

Mask R-CNN is a deep learning algorithm that achieves instance segmentation [38]. Mask R-CNN extends Faster R-CNN by adding a branch that predicts an object mask, replacing the RoI Pooling layer with RoI Align. ROI Align solves the problem of mis-alignment caused by quantization in RoI Pooling operations. Kumar et al. [39] quantified the severity of loose smut in wheat using Mask R-CNN. However, the degree of disease was not graded. Additionally, the accuracy for disease degree classification was not verified. In addition, the calculation of the proportion of the diseased area of the whole leaf was not clearly indicated. Kumar et al. [40] used the Mask R-CNN to recognize the wheat yellow rust disease. However, this study did not include enough datasets, and the result of background segmentation was not adequate. Yang et al. [41] demonstrated the potential of Mask R-CNN to identify leaves in plant images for rapid phenotyping with an average accuracy of up to 91.5%. Su et al. [42] utilized a dual deep learning framework based on Mask R-CNN to detect the wheat for FHB severity in field trials. Due to the deficiencies of the previous strategies, more advanced models are needed to evaluate the resistance of wheat to FHB. In recent years, Chen et al. [43] proposed a more advanced instance segmentation model of BlendMask compared to Mask R-CNN. BlendMask is a state-of-the-art instance segmentation method based on a fully convolutional one-stage (FCOS) object detection network [43]. Xi et al. [44] used two instance segmentation networks including BlendMask and Mask R-CNN to delineate the ginkgo tree crowns. The results showed that the capability of BlendMask outperformed Mask R-CNN. Compared to Mask R-CNN, the BlendMask network model has the characteristics of less computation, higher mask quality, and stable inference time. Therefore, it is necessary to apply the BlendMask model for the image segmentation of wheat spikes and target recognition of the diseased areas.

The assessment of plant disease severity is another important and challenging task in agriculture. Efficient evaluation methods should be of great help to growers and breeders.

Table 1. Summary of the studies on the evaluation crop disease severities based on deep learning methods.

References	Model	Crop	Severity Levels	Accuracy (%)
Esgario et al. [45]	AlexNet, GoogleNet, VGGNet, ResNet, MobileNet	Coffee	Healthy, low, very low, high, very high	84.13
Pan et al. [46]	Faster R-CNN (VGG16)	Strawberry	Healthy, general, serious	88.3
Joshi et al. [47]	VirLeafNet	Vigna mungo	Healthy, mild, severe	91.5
Zhang et al. [48]	Improved YoLo V5	Wheat	Minor, light, medium, heavy, major	91.0
Ji et al. [49]	DeeplabV3+	Grape	Healthy, mild, medium, severe	97.75
Wu et al. [50]	MultiModel_VGR	Pepper	Healthy, general, serious	95.34
Liu et al. [51]	DeeplabV3+, PSPNet, UNet	Apple	Healthy, early, mild, moderate, severe	96.41

summarizes some studies using advanced deep learning methods to assess crop disease severity. Esgario et al. [45] established five models to successfully classify the severity of coffee disease into five grades. ResNet50 achieved the best accuracy of 84.13%. Pan et al. [46] proposed Faster R-CNN (VGG16) and Siamese networks for strawberry leaf scorch severity estimation. An accuracy of 88.3% was achieved on a new dataset, but the manual labeling method is time-consuming and prone to subjective errors. Joshi et al. [47] used VirLeafNet to classify Vigna mungo disease into three grades and the accuracy reached 91.5%. Although the aforementioned studies performed well in determining the severity of plant diseases, the accuracy was lower than the protocol proposed in the current study. In another study, Zhang et al. [48] used the ratio of the number of diseased wheats to the total number of wheats as a method for evaluating disease severity. However, this method ignored the overlapping wheat spikes in the image. In the studies of Ji et al. [49] and Wu et al. [50], improved YoLo V5 and deeplabV3+ achieved the accuracies of 97.75% and 95.34%, respectively, in evaluating disease severities of grape and pepper, respectively. Different from the single-stage segmentation methods mentioned above, Liu et al. [51] developed a two-stage framework to automatically estimate the severity of apple leaf disease in the field, yielding an accuracy of 96.41%. However, the applicability of the framework used in their study lacked validation on multi-leaf images.

With the advancements in machine learning and deep learning techniques, the methods for plant disease detection have shown promising performance. However, the original data for model training were mostly acquired in a lab, which would limit the performance of the method in real field conditions [52]. The main objective of this study was to investigate the feasibility of automatic tandem dual BlendMask networks for assessments of wheat for FHB severity in field trials. The specific steps of this study were to: (1) capture high-quality images of wheat spikes in the field, (2) annotate wheat spikes and diseased areas in the raw images, (3) train a BlendMask model to detect and segment wheat spikes in full-size images, (4) train a second BlendMask model to predict diseased areas in individual spikes, (5) write a program that combines dual BlendMask networks to simultaneously display the results of wheat spike detection and diseased area segmentation in full-size images, and (6) evaluate the disease grade of wheat FHB based on the ratio of the diseased area to the overall wheat spike. To our knowledge, this is the first study to assess the severity of wheat FHB based on automatic tandem dual BlendMask networks.

2. Materials and Methods

2.1. Data Collection and Annotation

Images of wheat spikes from the late flowering stage to maturity were collected in different weather conditions (sunny, cloudy, rainy, etc.) and different backgrounds (other spikes, sky, clouds, and soil). A total of 690 images were captured using an autofocus single-lens reflex (SLR) camera (Canon EOS Rebel T7i, resolution: 6000 × 4000). Each image consists of several wheat spikes. For spike identifications, 524 images (including 12,591 spikes) were selected for the training set and the other 166 images (including 4749 spikes) were selected for the validation set by random. For diseased area detection, a total of 2832 and 922 sub-images of diseased spikes were selected for the training set and validation set by random, respectively. An image annotation software (Labelme, https://github.com/wkentaro/labelme (accessed on 30 October 2021)) was used to label the ground truth for spikes in full-size images and FHB-diseased areas in sub-images. The first step in image annotation was to label wheat spikes in the raw images (Figure 1a,b). After segmenting the labeled individual spikes into sub-images, the diseased regions in the sub-images were then annotated in the second step (Figure 1c,d).

The FHB severity was calculated as the proportion of the diseased area to the total spike area. When the diseased area does not exceed 20%, the wheat can be considered to have the potential to resist FHB. If the infection ratio is above 50%, it means the wheat is very susceptible to the disease. According to these principles, the FHB severity can be divided into four grades: healthy [0–5%), mild [5–10%), moderate [10–20%), and severe [20–50%]. The result of the Levene variance equivalence test and quality t-test of the means are both zeros, which means there are significant differences between any two classes.

2.2. BlendMask

BlendMask is a one-stage dense instance segmentation method model. The main contribution of BlendMask is to propose the Blender module, which can merge the feature both from top-down and bottom-up methods. Specifically, BlendMask is composed of a detector network and a mask branch. In the detector module, FCOS is used to generate the feature from top-down, including the bounding boxes and attention maps. FCOS is composed of a fully convolutional network, which is similar to the pixel-by-pixel prediction method of semantic segmentation to solve target detection. This detection not only becomes anchor free and proposal free, but also reduces the amount of design parameters significantly. The mask branch consists of three parts, including bottom module, top layer, and blender module. Bottom module is used to process the underlying features, generating a score map called Base. The top layer is concatenated on the box head of the detector, which predicts top-level attentions through a convolution layer on each of the detection towers. The blender module is the most important module of BlendMask, which can merge bounding boxes, attention maps and masks to generate the final prediction of instance segmentation.

The backbone of this study is a ResNet model with 50 layers (ResNet-50) mentioned by He et al. [25]. With the increase of the CNN depth, the problems of vanishing gradients [53] and the degradation [25] have been exposed. ResNet applies the shortcut connection of the residual module, which alleviates the problems of gradient disappearance and network degradation to a certain extent [54]. FPN is a top-down hierarchy with lateral connections for constructing high-level semantic features at various scales [55]. The raw images with annotated wheat spikes and the sub-images with annotated diseased areas were used as the inputs to train two BlendMask models for the detection of wheat spikes and diseased areas, respectively. After training the dual BlendMask models, the images in the validation set can be used for the segmentation of the wheat spikes and FHB-diseased areas. The recognition and segmentation results are displayed on the image below as output (Figure 2). The workflow of training dual BlendMask models is presented in Figure 3.

2.3. Evaluation Metrics

In order to evaluate the segmentation results of BlendMask networks on wheat spikes and FHB-diseased areas, several evaluation metrics were computed. It includes precision, recall, F1-score, and average precision (AP). Precision is defined as the proportion of the number of real positive instances among the total number of instances predicted as belonging to the positive category. Recall is defined as the proportion of the number of real positive instances in the total number of instances actually belonging to the positive category [56]. F1-score denotes the harmonic weighted average of precision and recall values, which takes both false positives and false negatives into account [57]. Precision and recall are used to calculate AP, which represents the area under the precision–recall (P-R) curve. The intersection over union (IoU) is defined as the degree to which the manually annotated ground truth box overlaps the bbox (category score) generated by the model. The IoU threshold was used to determine whether a wheat spike was correctly segmented. If the predicted result exactly matches the ground truth, the IoU reaches 100%. The mean intersection over union (MIoU) is another standard metric for image segmentation, which can be calculated as the number of true positive (TP) over the sum of true positive (TP), false negative (FN), and false positive (FP) [58]. The equations of precision, recall, F1-score, AP, IoU, and MIoU are shown below:

$$precision=\frac{TP}{TP+FP}$$

$$recall=\frac{TP}{TP+FN}$$

$$F1=\frac{2\times Precision\times Recall}{Precision+Recall}$$

$$AP=\frac{1}{10}{\displaystyle \sum }_{IoU=0.5}^{0.95}{{\displaystyle \int }}_0^1{P}_{IoU}\left(r\right)dr$$

$$IoU\left(E,F\right)=\left|\frac{E\cap F}{E\cup F}\right|$$

$$MIoU=\frac{1}{k+1}{\displaystyle \sum }_{i=0}^k\frac{P_{ii}}{{{\displaystyle \sum }}_{j=0}^k{P}_{ij}+{{\displaystyle \sum }}_{j=0}^k{P}_{ji}-P_{ii}}$$

where TP (true positive) is the number of correctly segmented wheat spikes or FHB-diseased areas, FP (false positive) is the number of wrong segmented wheat spikes or FHB-diseased areas, and FN (false negative) is the number of ground truth that were not segmented. $P_{IoU}\left(r\right)$ is the precision–recall curve with different IoU values. E represents the manually labeled ground truth box. F represents the bbox generated based on the BlendMask model. The intersection over union (IoU) computes the overlapping area between the predicted wheat spike and the ground truth wheat spike divided by the area of union between them. In some studies, the IoU threshold value is set to 50%, which means that a wheat spike is considered to be correctly segmented if IoU ≥ 0.5. It is considered to be mis-segmented if IoU < 0.5. $k+1$ is the total number of output classes, including an empty class (the background). $P_{ii}$ denotes TP, while $P_{ij}$ and $P_{ji}$ represent FP and FN, respectively.

2.4. Equipment

A computer with a graphics processing unit (GPU) mode (NVIDIA RTX 3090 24 Gb, Shanghai, China) was used to complete the process for model training and validation. It is important to emphasize that these eight GPUs are leased from a GPU sharing platform, and the personal computer can control these leased GPUs on the GPU rental website.

Table 2. Modeling hyperparameter settings.

Hyperparameter	Values
Backbone	Resnet-50
Batch size	16
Base learning rate	0.01
Attention size	14
Max iteration	1,700,000
Bottom resolution	56
Number of classes	2

shows the BlendMask modeling parameters.

Table 3. The total time for model training and validation.

Application	Training Time	Validation Time
Wheat spike identification	72 h 45 min 28 s	14 min 36 s
FHB disease detection	46 h 25 min 43 s	2 min 30 s

depicts the time for model training and validation.

3. Results

3.1. Model Training

Dual BlendMask networks were trained based on the labeled images of wheat spikes and FHB-diseased areas. Figure 4a shows the changing trend of loss during the model training for wheat spike identification. It can be seen from the figure that the curves of the loss of the bbox and the mask decrease sharply in the initial iteration, but the downtrend slows down after 30,000 iterations. The loss curve then fluctuates until the curve trend stabilizes after 160,000 iterations. It can be seen that the loss value of the mask is always greater than that of the bbox. Finally, the loss value of the bbox and mask reached the lowest levels: 0.0012 and 0.024, respectively. Similarly, Figure 4b describes the variation of the loss in the model training for FHB disease. The overall change trend of the loss curve in the image is similar to the previous model for segmenting spikes. The loss values of the mask and the bbox reduced to 0.032 and 0.0016, eventually.

3.2. Wheat Spike Identification

The segmentation results of the wheat spikes on the BlendMask network are shown in Table 3. As can be seen from the data in

Table 4. Results of BlendMask for wheat spikes and FHB disease detection.

Type	Precision (%)	Recall (%)	F1-score (%)	IoU (%)	Ap of Mask (%)	MIoU (%)
Wheat spike	85.36	75.58	80.17	48.23	59.28	56.21
FHB disease	78.16	79.46	78.89	52.41	66.74	55.34

, it is apparent that the algorithm showed an outstanding performance for wheat spike segmentation, with the AP of the mask and the bbox at 57.16% and 56.69%, respectively. In the 166 test images, the overall values of IoU, F1-score, precision, and recall were 48.23%, 80.17%, 85.36%, and 75.58%, respectively. According to statistics, 4053 were correctly identified from the 4749 wheat spikes, yielding a recognition accuracy of 85.56%. All the above results indicate that the BlendMask network has a strong capability to identify wheat spikes under field conditions.

Figure 5 depicts the prediction of selected wheat spike images from the validation dataset. It was observed that most of the spikes were successfully detected. The network model successfully detected the high-density wheat spikes in the field (Figure 5a). Due to the shooting position of the camera and the natural growth state of the wheat spikes, the wheat spikes in many images are partially occluded. There are mainly three types of occlusions, including occlusion by the leaves of the wheat (Figure 5b), occlusion by the wheat awns or straw (Figure 5c), and occlusion by the other wheat spikes (Figure 5d). The training results of the model showed that BlendMask successfully segmented the occluded spikes of wheat, demonstrating the strong performance of the model. Figure 5e shows that wheat spikes cut at the image borders were successfully identified. It is important to identify local spikes, as this allows full utilization of wheat spikes in the dataset to enhance the robustness of the model.

3.3. FHB Disease Evaluation

The trained BlendMask model was used to detect the diseased areas on wheat spikes. Figure 6 shows the diseased area detection and segmentation results from the individual wheat spikes under different conditions including diseased areas with shadow, diseased areas under strong light, diseased areas under low light, and diseased areas under awn occlusion. The bboxes, masks, and category scores were shown in individual spikes. The MIoU value reached 55.34%. The AP values of the mask and the bbox for FHB disease detection were 66.74% and 65.38%, respectively. As seen in Table 3, the overall values of precision, recall, F1-score, and IoU were 78.16%, 79.46%, 78.8%, and 52.41%, respectively. Due to the complexity of crop growth in natural fields and the interference of light intensity or weeds, the individual wheat spikelet that is segmented from full-sized images generate a certain amount of noise, which is mainly manifested in the partial shadow area of the wheat spike, weak light, strong light, and occlusion by wheat awns or straw. All of these bring challenges to the accurate recognition of diseased areas. Nevertheless, the BlendMask model was effective in identifying the lesion areas of these conditions (Figure 6). Eventually, a total of 916 diseased spikes were recognized from 922 samples with an accuracy of 99.32%. The results prove that BlendMask achieved the accurate identification of FHB-diseased areas. Figure 7 shows the flowchart of the combination of dual BlendMask models for evaluating FHB disease severity. Figure 8 shows the visualization of the recognition accuracy and disease severity of different wheat spikes from a full-sized image based on the automatic tandem dual BlendMask framework. It is worth noting that the dual BlendMask framework runs very fast. The recognition results can be output in about 2 s.

Although the trained model performed very well in diseased area detection, some mistakes occurred during validation. For the case of identification failure, the main reasons were model error and annotation error. As shown in Figure 9a, the part adjacent to the lesion area is identified as the lesion area (red rectangle). This type of error belongs to FP. The same error is also shown in Figure 9d in the blue box. The color of the wheat spikes near the diseased area is similar to the unique color of the diseased area due to natural light. Figure 9b shows that the model incorrectly identifies several adjacent lesion areas as one. Figure 9c shows that some lesions (blue rectangle) are not identified due to model errors, which belongs to FN. Figure 9d shows a failure due to annotation errors (red rectangle).

4. Classification of Wheat FHB Severity Grades

Figure 10a depicts the ground truth (the visual rating of spikes in the acquired images by an expert) of wheat spikes with different disease grades in the training sets. As seen in Figure 10a, 21.2%, 28.2%, 32.5%, and 18.1% of the samples in the training set are categorized as healthy, mild, moderate, and severe, respectively. Figure 10b shows the ground truth and prediction of wheat spikes with different disease grades in the validation sets. It was noticed that 20%, 29.6%, 30.3%, and 20.1% of the samples in the validation set were categorized as healthy, mild, moderate, and severe in the ground truth, which is generally similar to those from the training set. The distribution of the predicted results on the four grades is almost identical to that of the ground truth. The maximum error between the predicted quantity and the actual quantity occurs for the severe grade and the number is 10, while the minimum bias is only 1 when it comes to the mild grade.

Through the statistical analysis of the actual value and predicted value of the disease severity in the four disease grades shown in

Table 5. Statistical results of wheat spike FHB disease severity for four grades.

Grade	Type	Severity (%)
		Means ± SD	Max	Min
Healthy	Ground truth	2.8 ± 1.1	5.0	0.0
	Prediction	2.9 ± 0.8	4.9	0.0
Mild	Ground truth	7.4 ± 2.1	10.0	5.1
	Prediction	6.9 ± 1.4	9.8	5.0
Moderate	Ground truth	15.5 ± 2.3	20.0	10.3
	Prediction	14.4 ± 2.1	18.9	10.1
Severe	Ground truth	38.6 ± 4.6	49.8	21.2
	Prediction	36.4 ± 4.1	48.7	20.8

, it can be seen that the predicted value is always less than or equal to the actual value whether it is the average value or the maximum and minimum value of the disease severity. Therefore, the model may have a tendency to underestimate the severity of the disease in practical applications.

To further verify the accuracy of the classification results obtained from the model, the confusion matrix was applied to analyze the similarities and differences between the predicted results and the ground truth. Precision and sensitivity were first obtained. Based on the precision and sensitivity, the F1-score for each of the four grades was calculated. As shown in

Table 6. Results of precision, sensitivity, and F1-score for four grades.

Grade	Precision (%)	Sensitivity (%)	F1-score (%)
Healthy	93.6	88.9	91.1
Mild	93.7	90.9	91.7
Moderate	92.6	92.8	92.9
Severe	94.8	93.1	93.2

, it is obvious that all the values are around 90%. The lowest F1-score score was 91.1% while the highest score was 93.2%. The average F1-score was 92.22%. Figure 11 depicts a confusion matrix for wheat FHB grades. The average accuracy for FHB severity classification was 91.8%. Accuracies of 90%, 91%, 93%, and 93% were obtained for the four grades, respectively. As shown in Figure 11, mild grade samples are most easily misclassified to the healthy grade. This may be because the healthy and mild grades have similar area ratios. Since the diseased area for these two grades is relatively small, the probability of misidentification is likely to increase. Additionally, moderate grade samples are more likely misclassified to the severe grade. This may be due to the smaller difference in the ratio of diseased areas to spike areas for the moderate and severe grades.

5. Discussion

The research proposed a new approach using tandem dual BlendMask networks for automatic severity estimation of wheat FHB in the field. Three main parts were involved in this study, including wheat spike segmentation, FHB disease segmentation, and disease severity classification. RGB images were used to train the dual BlendMask framework to evaluate the severity of wheat FHB disease. Although positive detection results were obtained, there were still some errors in the prediction of the disease severity. These errors may come from the algorithm model or the image annotation. Data annotation is a bottleneck for segmentation tasks [59]. The annotation process is laborious and time-consuming. The annotation results are greatly affected by human factors. In the future, instance segmentation based on semi- and weakly-supervised methods can be considered. In order to maximize the utilization of the dataset, data augmentation is also an essential step. Fang et al., used data augmentation to solve the issue of insufficient training datasets in instance segmentation [60]. The application of random transformations, such as flipping, cropping, and changing saturation to generate new images, can generate new images that effectively augment the training set.

It is a challenging task to realize the recognition of wheat spikes in the field. In this study, FCOS in the BlendMask deep learning framework was used for wheat FHB detection. The model was capable of detecting wheat spikes within a complex environment. Comparing the neural network method with Laws texture energy [61], the BlendMask model achieved a higher accuracy of 85.56% for identification of high-density wheat spikes. The main reason for the success of the proposed model is the FCOS used in the detector module, which can generate the feature from top-down. Although the SpikeSegNet model achieved an accuracy of 99.91% in the study of Misra et al. [62], the wheat spikes were low-density in collected images.

Two trained BlendMask models are connected in series to directly display the recognition results of wheat spikes and diseased areas in the original images. The BlendMask performed very well, yielding a detection rate as high as 99.32% for FHB detection compared to 98.81% for the study of Su et al. [42]. The main reasons for the success of our model are as follows:(1) compared to Mask R-CNN, the Blender module of BlendMask provides higher-quality masks and (2) the wheat in the dataset has been marked with high precision, which helps to improve the performance and robustness of the BlendMask model. The proportion of diseased areas of the whole wheat spike displayed directly on the original input image, thus high-throughput and real-time analysis can be constructed in the field. Many deformation models of the transformer, such as vision transformer (VIT) [63] and swin transformer [64], perform well in image classification, target recognition, and image segmentation. In the future, more advanced semantic segmentation algorithms based on transformers should be considered.

It was noticed that the automatic tandem dual BlendMask networks successfully segmented individual wheat spikes and FHB-diseased areas simultaneously from images of multiple spikes with complex backgrounds. The proposed method showed great potential for non-destructive and high-throughput evaluation of the severity of wheat FHB disease. Su et al. [42] used the same dataset as this study and the Mask R-CNN model to segment wheat individuals and the disease spots on wheat individuals, and the accuracy of segmentation detection reached 77.76% and 98.81%, respectively. In this study, the segmentation detection accuracy of these two parts has been improved to a certain extent, reaching 85.56% and 99.32%. Mask R-CNN takes 250,000 iterations to achieve the accuracy while BlendMask reaches the accuracy mentioned above after 170,000 iterations. Therefore, the BlendMask model is more concise and efficient. By linking the two models, the average time to identify the severity of a wheat plant was 0.09 s.

The objective of this study is to apply the constructed model to a car taking photos in the field, so the model is biased towards proximal perceptual detection. Nowadays, unmanned aerial vehicles (UAVs) are widely used for efficient detection of crop diseases because they can sense a larger range and consume less manpower, but such detection generally relies on the expensive remote sensing equipment on the UAVs. In future studies, we will try to improve the model and make it adapt to the high density of wheat images or videos taken by UAVs. When the UAVs fly at a low altitude over the same kind of wheat in the field, an ordinary camera with a relatively low cost can be used to take photos or video and evaluate the severity of the wheat FHB disease. This is helpful to screen out the wheat with better disease resistance.

6. Conclusions

This study proposed an integrated method for the evaluation of wheat for FHB severity based on the automatic tandem dual BlendMask networks for simultaneous segmentations of wheat spikes and diseased areas under complex field conditions. The BlendMask model demonstrated outstanding performance in the detection of the wheat spikes occluded by awns or cut at the image borders. The recognition accuracies of the model for wheat spikes and diseased areas were 85.56% and 99.32%, respectively. The inference time was within 2 s on average, which would be helpful for real-time monitoring of high-throughput wheat spikes in the field. By calculating the ratio of the lesion area to the overall area, the disease degree of wheat FHB was divided into four grades (healthy, mild, moderate, and severe) and the accuracy of disease level prediction reached 91.8%. This study demonstrates the feasibility of dual BlendMask networks for severity evaluation of wheat FHB in the field. This study will be of great help to the breeding of FHB resistant lines in breeding nurseries.