Development of a Drone-Based Phenotyping System for European Pear Rust (Gymnosporangium sabinae) in Orchards

Abstract

Computer vision techniques offer promising tools for disease detection in orchards and can enable effective phenotyping for the selection of resistant cultivars in breeding programmes and research. In this study, a digital phenotyping system for disease detection and monitoring was developed using drones, object detection and photogrammetry, focusing on European pear rust (Gymnosporangium sabinae) as a model pathogen. High-resolution RGB images from ten low-altitude drone flights were collected in 2021, 2022 and 2023. A total of 16,251 annotations of leaves with pear rust symptoms were created on 584 images using the Computer Vision Annotation Tool (CVAT). The YOLO algorithm was used for the automatic detection of symptoms. A novel photogrammetric approach using Agisoft’s Metashape Professional software ensured the accurate localisation of symptoms. The geographic information system software QGIS calculated the infestation intensity per tree based on the canopy areas. This drone-based phenotyping system shows promising results and could considerably simplify the tasks involved in fruit breeding research.

1. Introduction

The evaluation of fruit genetic resources with regard to their resistance to pathogens is an important task for selection in fruit breeding. Both genetic analysis and phenotyping of specific traits are crucial tools that provide important data for the evaluation process [1,2,3]. Although phenotyping is the most valuable tool in breeding research, besides genetic analysis, it is still frequently performed manually. However, this manual approach is both labour-intensive and time-consuming. Collecting and analysing data requires significant human effort. Consequently, this can lead to delays in breeding progress and limit the effectiveness of identifying desirable traits for breeding programmes. Additionally, manual phenotyping is susceptible to subjectivity and human error. This can result in inconsistencies in data collection and interpretation [1,2,4,5]. In contrast, digital phenotyping offers objective and standardised assessment across various genetic backgrounds, reducing human error and enhancing data reliability. Developing these methods is crucial for the rapid and effective evaluation of diverse genetic resources. It enables breeders to identify and select plants with desirable resistances more efficiently [1,2,3,6,7].

High-throughput phenotyping techniques using non-destructive imaging and detection systems have evolved significantly in recent years. In particular, unmanned aerial vehicles (UAVs) are used for phenotyping various crops such as winter wheat [8], blueberries [9], lemons [10], olives [11,12] or almonds [13]. By using drone technologies, a large amount of data can be collected in a consistent manner. This offers the opportunity to evaluate the required data from larger areas more cost-effectively. UAV cameras provide high-resolution images that allow for a detailed analysis of plant characteristics [1,6,14]. Therefore, they are increasingly used for phenotyping fruit trees, e.g., in the evaluation of plant height [15,16,17], flowering characteristics [18] or disease identification [11,12,19,20].

Machine learning and image processing methods can be used to create a wide variety of algorithms for the diagnosis of plant diseases. Object detection is a helpful tool for the detection of plant diseases, as it enables the identification and localisation of defined regions in RGB images [21,22]. RGB sensors are cost-effective and user-friendly and require less memory capacity [23,24,25]; furthermore, RGB images can be easily generated using UAVs. For the fast and accurate detection of objects, efficient algorithms such as YOLO (“You only look once”) can be used [26,27,28,29]. In various studies, good detection performance was achieved on UAV-RGB images using different YOLO algorithms, e.g., for the detection of tea leaf blight caused by Colletotrichum camelliae [20], bacterial blight on wild rice caused by Xanthomonas oryzae pv. oryzae [30], Verticillium fungus in olive trees [12] or pine wilt disease caused by the pine wood nematode Bursaphelenchus xylophilus [31].

The aim of this study was to establish drone-based phenotyping of pathogens in orchards, using European pear rust (Gymnosporangium sabinae) as a model pathogen. Our approach utilised two YOLO models, namely the model YOLO version 5 large (YOLOv5lu) [29] and YOLO version 8 large (YOLOv8l) [27]. In order to recognise the different stages of pear rust infection in the complex structure of the canopies, large models had to be used [32]. This was necessary because the images were taken under very different conditions, including varying flight altitudes and different environmental factors. In addition, the models had to be able to recognise symptoms on a variety of pear genotypes. Due to their larger model architecture, these are more suitable for complex object detection tasks and still have a high level of accuracy. However, this requires powerful systems and more computing capacity than when using smaller models. The v5u variant from the YOLOv5 (v7.0) release was used for object detection tasks. YOLOv5lu was further developed based on the YOLOv5 algorithm. The algorithm uses a modified Cross Stage Partial (CSP) darknet as the backbone of a Convolutional Neural Network (CNN) to extract features from the input data, reduce the computational effort and maintain the learning capability. The neck of the network uses a path aggregation network (PANet), which displays the input feature maps at different scales, allowing for small as well as larger structures to be recognised in the image and leading to better predictions. The Spatial Pyramid Pooling Fast (SPPF) module processes the features and summarises them in a fixed-size map. This makes it easier to detect and classify objects of different sizes and shapes. The neck then refines the features extracted from the spine. The head uses these features to predict the location of the bounding boxes and classify the objects within them. In contrast to YOLOv5, YOLOv5u uses an anchor-free and object-free split head, allowing object detection to be better adapted to different situations and datasets to detect and localise objects in images. YOLOv8, like YOLOv5u, uses a CSPDarknet backbone. Instead of a CSP layer, a coarse-to-fine (C2f) module is used, improving recognition accuracy and efficiency. The anchor-free and object-free split head is also used to increase the overall accuracy of the model. YOLOv5u and YOLOv8 can optimise the hyperparameters for custom datasets by applying genetic algorithms and provide the best possible recall [12,27,33,34,35,36,37].

The models were trained with images of infected leaves captured after natural infection. These images were collected through ten UAV flight campaigns conducted under different weather conditions and with varying flight parameters at the Julius Kühn Institute’s (JKI) experimental orchard for fruit crop breeding research in Dresden-Pillnitz, Germany. With an average symptom prediction accuracy of 81.3% (mAP@50), we present a workflow describing the detection of pear rust symptoms using a YOLO model [27], the photogrammetric processing of the UAV images and the visualisation of the detected symptoms using Metashape Professional [38]. Additionally, we describe the process of georeferencing the individual symptoms using QGIS software [39].

2. Materials and Methods

2.1. Data Collection, Data Preparation and Symptom Annotation

For the creation of an image dataset featuring pear rust-infected leaves, UAV-RGB images were captured in three experimental orchards [51°00′01″ N, 13°53′12″ E] of the Julius Kühn Institute in Dresden-Pillnitz, Germany. These orchards included 828 different pear genotypes from 24 cross progenies, 36 trees of 10 different pear cultivars, and 176 trees representing 34 Pyrus species. The pear trees were not treated with fungicides, allowing for natural infection. The drone images were taken in 2021, 2022 and 2023, starting with the appearance of the first pear rust symptoms in May and continuing until the appearance of advanced symptoms in August. By recording images of different pear genotypes and infection stages of pear rust symptoms over several flight days and under various weather conditions, we aimed to ensure the recognition of symptoms across a broad spectrum of morphologically diverse pear leaves. This comprehensive dataset forms the basis for a robust object detection model.

A total of 1394 RGB UAV images were collected during ten flight campaigns (

Table 1. Drone and camera system, as well as the selected flight parameters: relative flight altitude above ground (starting point), wind speed and degree of cloud coverage during the time of the image recordings per date.

Date	Drone/ Camera System	Relative Altitude [m]	Wind Speed 1 [m/s]	Cloud Cover 2 [octa]
5 July 2021	DJI P4P	4.90 to 6.60	1.6	6
20 May 2022	DJI P4P	4.90 to 5.10	0.9–1.2	7–8
29 June 2022	DJI P4P	4.80 to 6.00	2.3–3.2	8
12 July 2022	DJI P4P	5.20 to 5.50	1.9	3–4
13 July 2022	DJI P4P	5.90 to 6.00	0.7–1.8	6
14 July 2022	DJI P4P	7.20 to 7.60	1.2	6
17 August 2022	DJI Matrice 300 RTK 3	6.04 to 12.00	1.3–1.7	4–7
24 August 2023	DJI P4P	9.90 to 10.00	0.9–1.1	4–8
30 August 2023	DJI Matrice 300 RTK 3	11.98 to 12.02	1.2–1.4	8
5 September 2023	DJI P4P	7.70 to 8.00	3.0	3

¹ Hourly mean wind speed during the recording period [41]. ² The cloud cover was recorded by the Dresden-Klotzsche weather station. An octa value of zero indicates a completely clear sky and an octa value of eight indicates a completely cloudy sky. The weather station is located about 9 km north-west air distance of the experimental orchard in Dresden-Pillnitz [42]. ³ Including high-resolution camera Zenmuse P1.

[40]). The image data were captured with two quadcopters, the DJI Phantom 4 Pro V2.0 (DJI P4P, Shenzhen, China) and the DJI Matrice 300 RTK (DJI, Shenzhen, China). The Matrice 300 RTK was equipped with a high-resolution Zenmuse P1 camera model (v03.00.01.04, v07.00.01.10, DJI, Shenzhen, China), which has a 45-megapixel full-frame sensor (8192 × 5460 pixels) and a focal length of 24 mm, which corresponds to a 35 mm full-frame format. The DJI P4P has an RGB camera (model FC6310S, v01.08.1719) with a 1-inch CMOS sensor with 20 megapixels (5472 × 3648 pixels) and a focal length of 8.8 mm, which corresponds to a 24 mm full-frame format.

Both drones operate automatically, with preset flight parameters, from the ground control station DJI Pilot PE (v1.8.0, DJI, Shenzhen, China) and DJI Ground Station GS (v2.0.16, DJI, Shenzhen, China). Flights were performed at a relative altitude of 5 to 12 m. The drone images were taken in at a flight speed of 1 to 2 m/s and at an overlap of around 75 to 90%. The hourly mean wind speed was determined during the recording period [41] and varied between 0.7 and 3.2 m/s. The cloud cover is given in octa values, whereby an octa value of zero describes a completely clear sky and an octa value of eight describes a completely cloudy sky. The cloud cover was recorded by the Dresden-Klotzsche weather station. The weather station is located about 9 km north-west of the experimental orchard [42].

The original images were checked for their quality and the presence of disease symptoms (Figure 1) in order to crop them to a pixel size of 768 × 768 pixels without overlap using the Pillow library (v9.5.0, Secret Labs Inc., Östergötland, Sweden) [43] via Python (v3.11.5, Python Software Foundation, Delaware, USA) [44] in pre-processing and save them in JPG format. The creation of smaller image sections was chosen because they made the annotation work much easier, required less memory, were faster to process and were better suited to the network size and GPU capacity. After a qualitative examination of the image sections for the presence of pear rust infections, these were used for symptom annotation. Symptom annotation of early, clear, advanced and late symptoms (Figure 1) was performed by experts using the Computer Vision Annotation Tool (CVAT, v1.1.0) [45].

2.2. Data Augmentation, Model Selection and Model Training

Following the correction work and the completion of the annotated image dataset, a 5-fold cross-validation was performed for the state-of-the-art (SOTA) YOLO algorithm version 5 large (v5lu) [29] and version 8 large (v8l) [27] in order to be able to make a general statement about the expected model performance based on the existing dataset [46]. The 5-fold cross-validation was performed with the SciKit Learn library for machine learning (v1.3.2) [47]. Using the configuration shuffle = True, which allowed the image data to be shuffled, and random_state = 42, a value for the reproducibility of the experiment, five different datasets were generated. Each dataset contains a total of all image files, split differently between the training and validation datasets.

To enlarge the annotated image dataset for model training, the images of the training datasets were first mirrored horizontally and the corresponding annotation coordinates were adjusted according to the mirroring [20] using the open-source Python library OpenCV (v4.9.0, Intel Corporation, California, USA) [48]. Furthermore, these images were then modified using data augmentation methods from the open-source Python library Albumentations (v1.3.1, Albumentations.ai, California, USA) [49]. The principal component analysis (PCA) colour augmentation method at pixel level, also known as FancyPCA [50], was set with the configuration alpha = 0.72 (always apply = True, p = 1.0) and random change in brightness and contrast with the configurations brightness_limit = [−0.14, 0.14] and contrast_limit = [−0.10, 0.10] (p = 1.0, always_apply = True). The five training datasets were expanded with the additionally generated image and annotation data [51], as shown in

Table 2. Structure of the images with annotations (= instances, abbreviated as A. in the table) and background images (abbreviated as BG. in the table) of the datasets after 5-fold cross-validation and data expansion.

Data-Sets 1	Training Datasets 2				Validation Datasets
	A. Images (n)	BG. Images (n)	Sum Instances (n)	Sum Images (n)	A. Images (n)	BG. Images (n)	Sum Instances (n)	Sum Images (n)
1	926	266	26,006	1192	121	29	3248	150
2	930	264	25,182	1194	119	30	3660	149
3	942	252	26,688	1194	113	36	2907	149
4	950	244	26,104	1194	109	40	3199	149
5	924	270	26,028	1194	122	27	3237	149

¹ The SciKit Learn library for machine learning [47] was used to create the datasets of a 5-fold cross-validation with the configuration shuffle = True and random_state = 42. ² For the data augmentations, horizontal mirroring [48], the colour augmentation FancyPCA [50] and random change in brightness and contrast [51] were applied to the cropped UAV images and the associated annotations/instances.

.

The training, testing, predictions and hyperparameter optimisation of deep learning algorithms require high computing power. For this reason, the integrated cloud computing environment Google Colaboratory [52] with the open-source framework PyTorch (v2.1.1, The Linux Foundation, California, USA) [53] and the GPU type NVIDIA Tesla T4 with 15,102 MB total memory were used. The software Ultralytics (v8.0.0, Ultralytics Inc., Maryland, USA) [27] was used for the training runs of the YOLO models and the subsequent hyperparameter tuning. The open-source image augmentation library Albumentations [49] was also integrated and was used to generalise the datasets. In this study, the ToGray parameters of the Albumentation application were set to zero, as initial tests showed that otherwise, for example, roads were identified as a symptom of pear rust. Each dataset (Table 2) was used to train a YOLOv5lu model and YOLOv8l model for 200 epochs, with a batch size of 8, the optimisation method “AdamW” [54] and seed = 7. The following hyperparameters were used: lr0 = 0.01, lrf = 0.01, momentum = 0.937, weight_decay = 0.0005, warmup_epochs = 3.0 and warmup_momentum = 0.8 (Appendix A,

Table A1. Selected settings of the hyperparameters [27] during the 5-fold cross-validation, hyperparameter tuning and training run from scratch [57].

Params	Term	5-Fold Cross- Validation	Hyperparameter Tuning Hyp0	YOLOv5lu Training from Scratch
model	Selected model file for training	yolov5lu.yaml/yolov8l.yaml	yolov5lu.yaml	yolov5lu.yaml
epochs	Number of training epochs	200	10	500
iterations	Number of generations for which the tuning is carried out	-	100	-
patience	Epochs without improvement, which are waited for before training is completed	100	-	100
batch	Defines the number of samples	8	8	9
imgsz	Image size for training	768	768	768
optimizer	Definition of the optimiser for training	AdamW	AdamW	AdamW
verbose	Detailed output during training	true	true	true
seed	Number that ensures the reproducibility of the results	7	7	7
deterministic	Use of deterministic algorithms	true	true	true
iou	Intersection over union	0.7	0.7	0.7
lr0	Initial learning rate	0.01	0.001	0.0085
lrf	Final learning rate	0.01	0.01	0.01213
momentum	Momentum factor for SGD	0.937	0.937	0.81081
weight_decay	L2 regularisation term	0.0005	0.0005	0.00034
warmup_epochs	Number of epochs to warm up the learning rate	3.0	3.0	2.72199
warmup_momentum	Initial momentum for warmup phase	0.8	0.8	0.72251
warmup_bias_lr	Learning rate for bias parameters during the warmup phase	0.0	0.1	0.1
box	Weight of the box loss component	7.5	7.5	4.69446
cls	Weight of the classification loss	0.5	0.5	0.63944
dfl	Weight of the focal distribution loss	1.5	1.5	1.69455
hsv_h	Image, HSV–Hue augmentation	0.015	0.015	0.01697
hsv_s	Image, HSV–Saturation augmentation	0.7	0.7	0.37337
hsv_v	Image, HSV–Value augmentation	0.4	0.4	0.46164
translate	Image, translation	0.1	0.1	0.05561
scale	Image, scale	0.5	0.5	0.39554
flipud	Image, flip up-down	0.5	0.5	0.41081
fliplr	Image, flip left-right	0.0	0.0	0.0
mosaic	Image, mosaic	1.0	1.0	0.74789

). Based on these results, the model was selected by comparing the model parameters precision (P), recall (R), F1-score (F1) and mean average precision (mAP) at intersection over union (IoU).

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{True} \mathrm{Positives}}{\mathrm{True} \mathrm{Positives}+\mathrm{False} \mathrm{Positives}}}$$

(1)

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{True} \mathrm{Positives}}{\mathrm{True} \mathrm{Positives}+\mathrm{False} \mathrm{Negatives}}}$$

(2)

$$\mathrm{F}1=2 \times {\displaystyle \frac{\mathrm{Precision} \times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(3)

$$\mathrm{IoU}= {\displaystyle \frac{{\mathrm{A}}_{\mathrm{O}}}{{\mathrm{A}}_{\mathrm{U}}}}$$

(4)

Correct classifications are indicated by the precision value (Equation (1)), while the recall value (Equation (2)) indicates the completeness of the detections. The F1 score (Equation (3)) is the harmonic mean of the precision and recall metrics. A high F1 score describes good model performance that is characterised by both high precision and high recall. The IoU threshold (Equation (4)) measures the overlap between two bounding boxes. It is the ratio of the area of overlap (A_O) to the area of union (A_U) of the ground truth and the predicted box. An IoU of one means that the predicted box completely overlaps with the ground truth. An IoU of zero means that there is no overlap. The mAP@50 is the average of the precision values at an IoU threshold of 50%. All metrics described can assume values between zero and one. The mAP@50-95 is the average of the precision values over different IoU thresholds of 50% to 95%.

The dataset whose precision, recall, F1-score, mAP@50 and mAP@50-95 metrics were closest to the mean of all models per metric was selected. The hyperparameters of the YOLO model were optimised using hyperparameter tuning over 10 epochs and 100 iterations [27,33,55]. After tuning, the parameters were used for training from scratch (Appendix A, Table A1). Together with the annotation process, the data pre-processing, the selected dataset and the determined hyperparameters, a deterministic YOLO model [56] was created (Figure 2).

2.3. Photogrammetric Construction of the Experimental Orchard

The drone had GPS (Global Positioning System) to record the position and altitude of each image. To increase the accuracy of the image positions, ground control points (GCPs) were used in this study. Based on the corrected position data, a digital map of the orchard was created as a georeferenced 3D model. Using ExifTool (v12.3.8.0, Phil Harvey, Ontario, Canada) [57], important flight and image parameters such as the relative flight altitude and the camera pitch were determined in order to understand the results. Together with a focal length of 8.8 mm (35 mm equivalent: 24.0 mm) and image width of 5472 pixels, the average ground sampling distance (GSD) per flight could be calculated. The GSD refers to the distance between consecutive pixel centres measured on the ground. A higher GSD value corresponds to a lower spatial resolution of the image, resulting in less observable detail [58]. The GSD (Equation (5)) was determined as follows:

$$\mathrm{GSD} \mathrm{in} \mathrm{centimetre}/\mathrm{pixel}={\displaystyle \frac{(\mathrm{Sensor} \mathrm{width} \mathrm{in} \mathrm{millimetre}\times \mathrm{Relative} \mathrm{altitude} \mathrm{in} \mathrm{metre}\times 100)}{(\mathrm{Focal} \mathrm{lenght} \mathrm{in} \mathrm{millimetre}\times \mathrm{Image} \mathrm{width} \mathrm{in} \mathrm{pixels})}}$$

(5)

On 4 September 2023, 804 high-resolution drone images were taken with the DJI P4P over the experimental field in Dresden-Pillnitz at a relative flight altitude of approx. 17 m above the starting point and with 90° nadir view. The image overlap was 90% front overlap and 90% side overlap. The average GSD of the image dataset was around 0.47 cm/pixel. A second flight of the DJI P4P took place on 5 September 2023 at a relative flight altitude of around 8 m above the starting point. A total of 238 drone images were taken with 70% frontal overlap, 58% side overlap and 90° nadir view. The average GSD of the image dataset was around 0.22 cm/pixel. The drone images were taken at a low flight speed of 1–2 m/s. The coordinates of the flight were saved in the WGS84 (EPSG: 4326) coordinate reference system and later transformed into the ETRS89 UTM Zone 33N (EPSG: 25833) coordinate system. The software Metashape Professional (v2.1.2, Agisoft, St. Petersburg, Russia) [38] on an Apple MacBook Pro with 16-core CPU, 40-core GPU M3 Max processor and 64 GB RAM was used to process the images. The following parameter settings were used for image alignment: camera accuracy: high; key point limit: 50,000; tie point limit: zero; adaptive camera model fitting: yes. The dense point cloud elaboration steps were performed with ‘medium quality’ and ‘moderate filtering’. The dense point cloud was then processed to create the georeferenced digital elevation model (DEM) and a digital orthomosaic, as shown in Figure 3.

2.4. YOLO-Based Detection of Pear Rust Symptoms and Assignment to the Individual Tree

For the detection of pear rust symptoms in the field, the YOLOv5lu model with an IoU of 0.7 and a confidence threshold of 0.347 was applied to the 238 unprocessed images taken by the drone at an altitude of 8 m on 5 September 2023. All predictions were output per image in .txt file format (YOLO1.1) and converted into .xml format (PascalVOC) using a Python script [59]. The coordinates of each predicted bounding box (xmax and xmin and ymax and ymin of the infected leaf) were calculated to form a centre point and associate all predictions with the respective UAV image. In order to import the predictions via the Metashape Professional Python API, another Python script [56] was used to display each infected leaf detected by the algorithm as a marker in Metashape Professional [38]. All centre points and, thus, infected leaves were converted into GPS coordinates. The orthomosaic, the DEM and the GPS coordinates of the individual infected leaves were saved and exported as a shape file in the ETRS89 UTM Zone 33N (EPSG: 25833) coordinate system for further processing (Figure 3).

2.5. GIS-Assisted Counting of Diseased Leaves per Individual Tree

For the calculation of the infestation intensity per tree, the digital orthomosaic, the DEM, the shape file with the tree IDs or locations and the shape file with the GPS coordinates of each infected leaf were imported into the georgraphic information system software QGIS (v3.36.0, QGIS Development Team) [39], as shown in Figure 4.

Markers that were used as ground control points have been deleted from the attribute table. The determination of the canopy area of each tree was carried out based on the DEM using the QGIS tool ‘Raster Calculator’. By applying a threshold of 117.2 m, representing the maximum ground level in our study area, the raster information of the trees was subtracted from the undergrowth. This resulted in a binary raster, where values exceeding 117.2 m indicated the presence of tree canopy, facilitating accurate canopy area extraction. Canopies that could be not clearly separated because of branches growing into each other were separated manually using the QGIS tool ‘Split parts’ [39].

The binary raster layer was converted to a vector layer and all empty values of the digital numbers (DN entries) were deleted from the attribute table. The resulting shape file was now linked to the tree IDs and the respective canopy areas (in m²) were calculated with the ‘Field Calculator’ in QGIS (Figure 4).

The number of leaves infected with pear rust per tree were counted using the vector analysis tool ‘Count Points in Polygon’. Based on the number of symptoms per tree and on canopy area, infestation intensity (Equation (6)) was calculated as follows:

$$\mathrm{Infestation} \mathrm{intensity}= {\displaystyle \frac{\mathrm{Symptoms} \mathrm{per} \mathrm{tree} \mathrm{ID}}{\mathrm{Canopy} \mathrm{area} \mathrm{in} \mathrm{square} \mathrm{metres}}}$$

(6)

3. Results

3.1. GYMNSA Dataset

A dataset of RGB UAV images of pear rust symptoms, annotated by experts, was created. A total of 1394 images of different pear genotypes were recorded, including varieties, wild species and crossbred offspring.

The dataset comprises a total of 584 annotated images with 16,251 annotations (=instances) and 162 background images. The training dataset consists of 465 annotated images with a total of 12,591 instances, as well as 132 background images. The validation dataset comprises 119 annotated images with 3660 instances and 30 background images. The GYMNSA dataset [60] is available as an open source, is described in more detail in the associated data article [40] and was used for data augmentation and training of the machine learning model YOLOv5lu [29]. The training dataset for training from scratch was expanded by the data augmentation, as can be seen in

Table 3. Overview of the GYMNSA dataset after expanding the training dataset with the data augmentations.

Dataset	Annotated Images (n)	Background Images (n)	Image Total (n)	Instances (n)
Training 1	930	264	1194	25,182
Validation	119	30	149	3660
Total	1049	294	1343	28,842

¹ For the data augmentations, horizontal mirroring [48], the augmentation method FancyPCA [50] and random changes in brightness and contrast [49] were applied to the cropped UAV images and the associated annotations/instances.

.

3.2. Model Selection and YOLO Model Training

The 5-fold cross-validation results showed that the YOLOv5lu [29] model achieved a mean precision of 77.05%, a mean recall of 65.68%, a mean F1 score of 70.87% and a mAP@50 of 73.55%. The mAP@50-95 was 50.24% (

Table 4. Results and performance metrics of the 5-fold cross-validation of the YOLOv5lu and YOLOv8l model.

Dataset Composition	YOLO Version	P (%)	R (%)	F1 (%)	mAP@50 (%)	mAP@50-95 (%)
Dataset 1	v5lu	77.24	61.65	68.57	70.07	47.08
Dataset 2	v5lu	78.11	64.14	70.44	72.08	49.89
Dataset 3	v5lu	76.78	66.31	71.16	74.28	50.87
Dataset 4	v5lu	77.47	71.74	74.50	79.83	54.55
Dataset 5	v5lu	75.67	64.58	69.69	71.52	48.80
Mean	v5lu	77.05	65.68	70.87	73.55	50.24
Dataset 1	v8l	73.58	60.79	66.57	67.90	44.74
Dataset 2	v8l	75.54	65.49	70.16	71.34	48.18
Dataset 3	v8l	78.66	60.23	68.22	68.61	47.03
Dataset 4	v8l	75.54	65.49	70.16	71.34	48.18
Dataset 5	v8l	73.25	61.66	66.96	67.70	45.47
Mean	v8l	75.31	62.73	68.41	69.38	46.72

P: precision; R: recall; F1: F1-score; the harmonic mean of the precision and recall metrics; mAP: mean average precision. mAP@50 and mAP@50-95 indicate the AP values at an IoU threshold of 50% and at an IoU threshold of 50–95% of the area of overlap (AO) to the area of union (AU) of the ground truth and the predicted box.

). For the YOLOv8l [27] model, the cross-validation results indicated a mean precision of 75.31%, a mean recall of 62.73%, a mean F1 score of 68.41% and a mAP@50 of 69.38%. The mAP@50-95 was 46.72% (Table 4).

The YOLOv5lu model had a higher average accuracy, which indicates better object detection capabilities. Compared to the YOLOv8l model, it showed a more consistent performance in different test scenarios resulting from the validation dataset and was selected for this reason. The dataset that showed the smallest deviations from the mean value of all datasets in the 5-fold cross-validation was selected for the optimisation of the hyperparameter [27,47]. Dataset 2 of the Yolov5lu model showed the best performance values, which deviated only slightly from the mean value (Table 4). Compared to the mean of all v5lu models, dataset 2 showed a mAP@50 of 72.08% (mean 73.55%), a precision of 78.11% (mean 77.05%), a recall of 64.14% (mean 65.68%), an F1 score of 70.44% (mean 70.87%) and a mAP@50-95 of 49.89% (mean 50.24%) and was used for hyperparameter tuning.

The YOLOv5lu model was further optimised through a hyperparameter tuning process over 10 epochs and 100 iterations (Appendix A, Table A1). Together with the hyperparameter, a deterministic YOLOv5lu model was trained from scratch (Figure 5) [56]. The model was evaluated based on precision, recall, mAP@50 and mAP@50-95. The best result was calculated in epoch 168. The precision was increased to 79.81% (before, the score was 78.11%), the recall to 73.35% (before, 64.14%) and the F1 score to 76.44% (before, 70.44%). The mAP@50 was 81.25% (before, 72.08%) and the mAP@50-95 was 55.74% (before, 49.89%), as shown in Figure 5.

Based on the GYMNSA dataset, a customised YOLOv5lu model was developed. It made accurate predictions with an IoU of 0.7 and a confidence threshold of 0.347. The exact confidence threshold was determined using the F1 score as a balanced value between high precision and high recall.

3.3. Digital Orthomosaic and Digital Elevation Model of the Experimental Orchard

On 4 September 2023, 804 high-resolution drone images were taken with the DJI P4P over the test area in Dresden-Pillnitz (17 m flight altitude, 90° nadir view, 90% front and 90% side overlap, average GSD of 0.47 cm/pixel). A second flight took place on 5 September 2023 (8 m flight altitude, 90° nadir view, 70% front and 58% side overlap, average GSD of 0.22 cm/pixel). The predictions of the adapted YOLOv5lu model of the UAV images from 5 September 2023 were defined in advance with the configurations IoU 0.7, the maximum number of detections per image of 500 and the confidence threshold of 0.347. The prediction files in YOLO1.1 file format per image were converted into VOC file format per image using a Python script [59]. The converted predictions could then be transferred to the Metashape Professional software as markers using another Python script [56]. A total of 1039 of the 1042 images were processed and photogrammetrically aligned, resulting in a dense point cloud with 49,238,144 points and 3,067,049 tie points. A digital elevation model and a digital orthomosaic were created to support the localisation of the disease symptoms and the validation of the model results (Figure 3).

3.4. GIS-Assisted Localisation of Disease Symptoms

Spatial analysis in QGIS based on the digital elevation model enabled symptomatic leaves to be identified and precisely assigned to the tree in concern (Figure 4). By integrating images of two flights, it was possible to create a digital field of the distribution of infestation intensity (Figure 6). The combination of different flight phases made it possible to comprehensively monitor the area and thus accurately detect the symptoms of the disease. The calculated infestation intensity ranged from 0 to 33.15 symptoms/m² of the canopy in 2023 and was transferred to the following scoring scale with an interval size of nine:

• 0 symptoms/m² of canopy;
• 0.1 to 9.0 symptoms/m² of canopy;
• 9.0 to 18.0 symptoms/m² of canopy;
• 18.0 to 27.0 symptoms/m² of canopy;
• 27.0 to 36.0 symptoms/m² of canopy.

A total of 116 trees and 29 Pyrus species were classified according to the model predictions and the canopy area using the scoring scale (Figure 6,

Table 5. Classification of infestation intensity based on predictions within the canopy area in 2023. The frequencies of the respective scoring from 5 September 2023 were counted.

Infestation Intensity	Frequency 1
0 symptoms/m2 of canopy	19
0.1 to 9.0 symptoms/m2 of canopy	60
9.0 to 18.0 symptoms/m2 of canopy	23
18.0 to 27.0 symptoms/m2 of canopy	8
27.0 to 36.0 symptoms/m2 of canopy	6

¹ Based on the predictions of the YOLOv5lu model and the photogrammetric processing of UAV images, the symptoms per tree ID were counted on georeferenced images on 116 trees and divided by the canopy area to calculate the infestation intensity.

). Tree IDs 17, 18 and 19 were excluded from the calculation (Figure 6) because the corresponding three images from 5 September 2023 could not be aligned in the photogrammetric process.

4. Discussion

In this study, we present a workflow for the detection and quantification of pear leaves infected with Gymnosporangium sabinae using drones (Figure 2), photogrammetry (Figure 3) and georeferencing (Figure 4) in the field based on computer vision. UAVs have already been used for phenotyping various fruit crops such as lemons [10], olives [11,12] or almonds [13]. Object detection is a helpful and efficient tool for monitoring plant diseases. It enables the identification and localisation of defined regions in RGB images [21,22,28]. Advances in computer vision enable a user-friendly application of previously complex and computationally intensive algorithms. The well-documented open-source project Ultralytics provides a tool for creating customised object detection models [7]. The photogrammetric processing of the drone images with Agisoft’s Metashape Professional and the implementation of the predictions [56] with the integrated Python API enables the digital visualisation of the orchard and the predictions. The geographic information system software QGIS can be used to determine the assignment of the forecasts and the infestation intensity per tree (Figure 6, Table 5).

4.1. GYMNSA Dataset and Model Training

The GYMNSA dataset used in this study [60] is a novel dataset that captures the visual characteristics of pear rust disease symptoms in UAV RGB images caused by the pathogenic fungus Gymnosporangium sabinae. One notable advantage of this dataset is its diversity, encompassing images collected by 10 flight campaigns conducted with the consumer drone DJI P4P and the industry drone DJI Matrice 300. These campaigns were carried out in 2021, 2022 and 2023 under different environmental conditions and at varying flight altitudes, showing early, clear, advanced and late stages of infection (Figure 1) [40].

The YOLOv5lu [29] model was selected over the YOLOv8l [27] model due to its better performance metrics in object detection tasks, which were determined by a 5-fold cross-validation procedure. Following cross-validation, dataset 2 of the YOLOv5lu model demonstrated the best performance metrics which deviated slightly from the mean (Table 4) and was selected for the subsequent workflow. Overall, the five datasets used in the 5-fold cross-validation process exhibited slight variations in the number of images between the training and validation sets. However, these minor differences do not significantly impact the performance metrics, as the datasets are approximately equal in size [46,47]. Images from the drone flight of 5 September 2023 used for the symptom detection includes also background images without any visible symptoms. After symptom prediction, using the Yolov5lu model, it was observed that some features were incorrectly identified as pear rust symptoms (e.g., Taraxacum flowers) and could potentially affect the accuracy of the resulting predictions on the digital map of the experimental orchard. However, these false-positive predictions are not critical to the further workflow, as only the predictions within the tree crowns and the resulting infestation intensity are of interest. The YOLOv5lu model was further optimised by hyperparameter tuning, which led to an improvement in precision, recall and mAP metrics. The process of hyperparameter tuning could be expanded to further improve the performance of the model. In order to increase the model fitness for a customised YOLO model, at least 30 epochs and 150 iterations are usual, which is very computation- and time-intensive. Because of that, a hyperparameter tuning with 10 epochs and 100 iterations (Appendix A, Table A1) was chosen in this experiment in order to be able to present the workflow on the basis of this (Figure 2) [27,55]. The use of deterministic training helped to ensure consistent and reliable results. The YOLO model showed a high accuracy of 81.3% in the detection of pear rust symptoms. The error rate of over 18% is not sufficient for commercial applications in the field of precision agriculture and should be reduced. However, for the evaluation of genetic resources, this level of accuracy is likely to surpass that of manual assessments, where infestation levels are only visually estimated. To enhance the model’s accuracy and robustness, as well as improve detection performance across various disease stages, the inclusion of additional datasets and data augmentation techniques could be considered in future developments [23,24,31]. The inclusion of multispectral or hyperspectral cameras could also improve the detection success of symptoms, but would require the processing and analysis of larger datasets and additional processing steps as well as technical requirements [10,19,24].

Overall, the model is promising, and future research could use this workflow to type further plant diseases using UAV RGB imaging and focus on expanding the dataset and further optimisations to improve and validate the practicality of the system.

4.2. Photogrammetric Reconstruction of the Experimental Orchard

Drone-based monitoring offers significant advantages in terms of effectiveness, objectivity and non-invasiveness [1,3,6,7]. However, detecting small features such as pear rust symptoms in drone images captured at higher flight altitudes can present a challenge. In our study, tests of various flight parameters indicated that predictions from drone images captured at altitudes greater than 8 m above ground tend to be less reliable. As a result, a low-flight strategy was adopted using the DJI P4P drone. However, lower-altitude flights with sufficient image overlap for photogrammetric processing require longer flight durations and increased battery usage, which, in turn, leads to greater amounts of data for processing. In large-scale orchards, this can become a significant challenge, as longer flight times necessitate additional resources and increase operational costs [1]. One possible solution is the use of industrial drones equipped with high-resolution cameras, which enable higher flight altitudes and at the same time shorten flight times. A comparative flight at an altitude of 12 m with a DJI Matrice 300 RTK equipped with a Zenmuse P1 camera led to significantly better results in both symptom detection and photogrammetric processing.

However, as our workflow was meant to be usable for fruit growers, more cost-effective drones were favoured for image capture. A new flight strategy was established to ensure high quality detection of symptoms while minimising the image capture effort. This approach involved two flights: one at lower altitude with reduced image overlap, specifically for symptom detection and prediction, and a second flight at higher altitude with 90% image overlap to optimise photogrammetric processing. This method provides a balance between accurate detection and efficient data acquisition. According to this approach one flight at 5 September 2023, with a low flight altitude of about 8 m for symptom detection and one flight at 4 September 2023 at a high flight altitude of about 17 m for image processing in Metashape Professional software [38] were performed. Three images from 5 September 2023 could not be aligned during the photogrammetric process; therefore, tree IDs 17, 18 and 19 were excluded as their respective forecasts were not processed. The reason for this was a reduced overlap due to a larger tree height and thus a smaller distance to the camera, so that the three images that could not be aligned differed too much in perspective from the neighbouring images. To avoid this, the overlap rate and flight altitude can be adapted in future to ensure photogrammetric processing. During the whole process, the drone GPS ensured exact overlapping and a precise flight. By using ground control points set on the ground before each flight and measured with RTK accuracy, the detected symptoms could be assigned to each individual tree with an accuracy of 1 to 2 cm.

4.3. Automated Counting of Infected Leaves

The integration of the YOLOv5lu model [29] with photogrammetric and GIS techniques enabled the localisation of pear rust symptoms within the experimental orchard. The digital elevation model [38] formed the basis for estimating the canopy area, which was essential for assigning and quantifying the algorithm’s predictions to the specific tree IDs. However, the flight conditions in this example were not ideal due to the high solar radiation and it can be expected that under more favourable flight conditions the detection accuracy could be improved [19].

By polygonising and delimiting the canopy areas on the basis of the digital elevation model, false-positive pear rust predictions outside the canopies could be excluded. However, variations in tree height presented a challenge, as the canopies of smaller trees were difficult or impossible to map accurately. To address this, QGIS was used to manually create polygons around smaller trees, allowing the identification of leaves with symptoms of Gymnosporangium sabinae. Other methods to reduce false-positive detections could include consistent mowing of the undergrowth. Another challenge encountered was the merging of adjacent canopies, which complicated the assignment of predictions to specific tree IDs. This required manual reworking, adding to the overall labour effort. Implementing consistent pruning management could mitigate this issue in future workflows by preventing canopy overlap in advance.

5. Conclusions

The results of this study demonstrate the potential of this integrated approach for broader applications in fruit breeding research, where accurate and timely monitoring of plant health and phenotypic classification of disease symptoms are manual, time-consuming and labour-intensive processes. The trained YOLOv5lu model enables digital phenotyping of different stages of pear rust symptoms on morphologically different genotypes and under different environmental conditions at the individual tree level. The new flight approach with a lower flight altitude for symptom detection and a higher flight altitude for image processing successfully enables a practicable compromise between detection accuracy and optimal image processing. Based on the orthomosaic and the digital elevation model, the infestation intensity per individual tree can be quantified, enabling the precise identification of resistant or tolerant genotypes.

Future research could focus on expanding the dataset, optimising machine learning models and exploring the feasibility of this approach for other plant diseases. The development of resistant varieties and the analysis of genetic resources should be supported by this approach and open up the possibility of applying high-throughput phenotyping using UAV-RGB images, object detection, photogrammetry and georeferencing. However, while drone and machine learning-powered disease monitoring offer promising solutions for large-scale farms and research projects, their accessibility for smaller farms remains uncertain. The initial costs for drones, photogrammetry, and AI software can be prohibitive. Developing more cost-effective, standardised systems could make these technologies more accessible to smaller farms in a user-friendly manner. Additionally, cloud-based data processing systems could reduce the technical burden on farmers and breeders. Renting drones and services from contractors also presents a viable option, lowering the entry barrier for smaller farms and increasing the availability of these technologies.