Apple Detection in Complex Scene Using the Improved YOLOv4 Model

Abstract

To enable the apple picking robot to quickly and accurately detect apples under the complex background in orchards, we propose an improved You Only Look Once version 4 (YOLOv4) model and data augmentation methods. Firstly, the crawler technology is utilized to collect pertinent apple images from the Internet for labeling. For the problem of insufficient image data caused by the random occlusion between leaves, in addition to traditional data augmentation techniques, a leaf illustration data augmentation method is proposed in this paper to accomplish data augmentation. Secondly, due to the large size and calculation of the YOLOv4 model, the backbone network Cross Stage Partial Darknet53 (CSPDarknet53) of the YOLOv4 model is replaced by EfficientNet, and convolution layer (Conv2D) is added to the three outputs to further adjust and extract the features, which make the model lighter and reduce the computational complexity. Finally, the apple detection experiment is performed on 2670 expanded samples. The test results show that the EfficientNet-B0-YOLOv4 model proposed in this paper has better detection performance than YOLOv3, YOLOv4, and Faster R-CNN with ResNet, which are state-of-the-art apple detection model. The average values of Recall, Precision, and F1 reach 97.43%, 95.52%, and 96.54% respectively, the average detection time per frame of the model is 0.338 s, which proves that the proposed method can be well applied in the vision system of picking robots in the apple industry.

1. Introduction and Related Works

Apple is one of the most popular fruits, and its output is also among the top three in global fruit sales. According to incomplete statistics, there are more than 7500 types of known apples [1] in the world. However, experienced farmers are still the main force of agricultural production. Manual work consumes time and increases production costs, and workers who lack knowledge and experience will make unnecessary mistakes. With the continuous progress of precision agricultural technology, fruit picking robots have been widely used in agriculture. In the picking systems, there are mainly two subsystems: the vision system and the manipulator system [2]. The vision system detects and localizes fruits and guides the manipulator to detach fruits from trees. Therefore, a robust and efficient vision system is the key to the success of the picking robot, but due to the complex background in orchards, there are still many challenges in this research.

For the complex background in orchards, the dense occlusion between leaves is one of the biggest interference factors in apple detection, which will cause false detection or missed detection of apples. Therefore, to make the model learn features better, the training data should contain more comprehensive scenes. However, due to the huge number of apples and complex background, apple labeling is a very time-consuming and energy-consuming task, which leads to the number of most datasets ranges from dozens to thousands of images [3,4,5,6,7], and covers a single scene. The data for occlusion scenes is even scarcer, which is not conducive to enhancing the detection ability of the model. To overcome this deficiency, we propose a leaf illustration data augmentation method to expand the dataset. To further expand the number of the dataset and enrich the complexity of the scene, common data augmentation methods such as mirror, crop, brightness, blur, dropout, rotation, scale, and translation are also utilized in this paper. The experimental results show that the model trained by traditional augmentation techniques and an illustration augmentation technique proposed in this paper can well detect apples under complex scenes in orchards.

In recent years, the research on apple detection under complex scenes in orchards has also made some progress. Tian Y et al. [8] proposed an improved YOLOv3 model to detect apples in different growth periods in orchards, with the F1 score of 0.817. Kang H et al. [9] proposed a new LedNet model and an automatic labeling tool, with the Recall and the accuracy at 0.821 and 0.853, respectively. Mazzia V et al. [10] used the YOLOv3-tiny model to match the embedded device, which achieved the detection speed of 30 fps without affecting the mean Average Precision (mAP) (83.64%). Kuznetsova A et al. [11] proposed pre-processing and post-processing operations to adapt to the YOLOv3 model, the detection result shown that the average detection time was 19 ms, 7.8% of the objects were mistaken and 9.2% of apples were not recognized for apples. Gao F et al. [12] used Faster Regions with Convolutional Neural Networks (Faster R-CNN) to detect apples in dense-foliage fruiting-wall trees, the experimental result was that the mAP was 0.879 and the average detection time was 0.241 s, which effectively detected apples under various occlusion conditions; Liu X et al. [13] proposed an apple detection based on color and shape features method, the detection results were that the value of Recall, Precision, and F1 score reached 89.80%, 95.12%, and 92.38%, respectively. Jia W et al. [14] combined ResNet and DenseNet to improve Mask R-CNN, which reduced the input parameters, with the Precision of 97.31% and the Recall of 95.70%.

For picking robots, the model should have fast and accurate detection performance. The YOLO [15,16,17,18] models unify object classification and object detection into a regression problem. The YOLO models do not use the area proposal process but directly use regression to detect objects. Therefore, the detection process is effectively accelerated. Compared with the YOLOv3 model, the latest version YOLOv4 model owns better accuracy under maintaining the same speed. However, the YOLOv4 model has not been widely used for fruit detection. Due to the large size and computational complexity of the YOLOv4 model, it is a huge burden for low-performance devices. EfficientNet [19] uses a compound coefficient to balance the three dimensions (depth, width, and resolution) of the model on limited resources, which can maximize the accuracy of the model. Therefore, we utilize EfficientNet to replace the backbone network CSPDarrknet53 of the YOLOv4 model, and Conv2D is added to the three outputs to further extract and adjust the features, which can make the improved model lighter and better detection performance. The experimental results show that the improved model can be well applied to the vision system of the picking robot.

The rest of this paper is organized as follows. Section 2 introduces the dataset collection, common data augmentation methods, and the proposed illustration data augmentation method. Section 3 introduces YOLOv4, EfficientNet, and the improved EfficientNet-B0-YOLOv4 model. Section 4 is experimental configuration, experimental results, and discussion. Finally, the conclusions and prospects of this paper are described.

2. Dataset and Data Augmentation

2.1. Dataset

In this paper, we choose Red Fuji apple as the experimental object. Since there are a large number of apple-related images on the Internet, we use the Python language to develop an image crawler to download these images in batches, which reduces the cost of data collection and improves the efficiency of data collection.

The main sources of images are Baidu and Google. The search keywords are Red Fuji Apple, Apple Tree, and Apple, etc. Firstly, to ensure image quality, the width or height of the crawled image is set to be at least greater than 500 pixels. Secondly, after the manual screening, the repetitive, fuzzy, and inconsistent images are mainly removed. Finally, 267 high-quality images are obtained, of which 35 images contain only a single apple, 54 images with multiple apples without overlapping, and 178 images with multiple apples overlapping.

Then, these 267 images are expanded to 2670 images by using data augmentation methods. Randomly divide 1922 images as the training set to train the detection model, 214 images as the validation set to adjust the model parameters, and 534 images as the test set to verify the detection performance. To better compare the performance of different models, images in the training set are converted to PASCAL VOC format. The completed dataset is shown in

Table 1. The number of apple images generated by data augmentation methods.

Original	Mirror	Crop	Brightness	Blur	Dropout	Rotation	Scale	Translation	Illustration	Total
267	267	267	267	267	267	267	267	267	267	2670

.

2.2. Common Data Augmentation

In this paper, we use 8 common data augmentation methods to expand the dataset, respectively mirror, crop, brightness, blur, dropout, rotation, scale, and translation operation. These operations are used to further simulate the complex scenes of apple detection in orchards. Figure 1b–i shows the effects of various common data augmentation after these operations.

2.2.1. Image Mirror

In orchards, the position and direction of the apples are various. Therefore, we use 50% probability horizontal mirroring and 50% probability vertical mirroring to process the original image. Both can be used alone or in combination.

2.2.2. Image Crop

In many apples stacked together, there will be various occlusion problems, and some apples will be blocked a little or more. Therefore, we randomly cut off 20% of the original image edges to simulate this scene.

2.2.3. Image Brightness

When the illumination is strong or weak, it will lead to apple color changes, which cause huge interference for the detection. Therefore, to enhance the robustness of the model, we randomly multiply the image with a brightness factor between 0.5 and 1.5.

2.2.4. Image Blur

Sometimes the image captured by the picking robot may be unclear or blurred, which can also cause interference with the apple detection. Therefore, we use the gaussian blur with a mean value of 2.0 and a standard deviation of 8.0 to augment the dataset.

2.2.5. Image Dropout

Apples often encounter the problem of diseases and insect pests, typically leaving traces of numerous spots. Therefore, we randomly dropout the grid points between 0.01 and 0.1 on the original image, and the grid points are filled with black.

2.2.6. Image Rotation

Similar to the mirror method, rotation is to further increase the image viewing angles. Therefore, we use randomly rotating the original image at an angle between −30° and 30° to augment the dataset, and the space vacated by the rotation is filled with black.

2.2.7. Image Scale

Due to the different positions of the apples in orchards, there will be apples of different sizes when capturing images. Therefore, to simulate this scene, we randomly multiply the original image with a scaling factor between 0.5 and 1.5.

2.2.8. Image Translation

Similar to the crop method, translation is to further solve the occlusion problem of the apple. Therefore, we randomly translate 20% of the edges of the original image, and the space after translation is filled with black.

2.3. Illustration Data Augmentation

To enrich the background and texture of training images, in Mixup [20], CutMix [21], Mosaic [18], several original images can be regularly mixed or superimposed to form a new image. For example, four original images are evenly spread into four-square-grid images, which will not increase the cost of training and inference but enhance the localization ability of the model. For apple detection in orchards, the biggest interference to the apple detection is the dense occlusion between leaves, and the complexity and randomness of the background, which makes feature extraction difficult and leads to false detection or missed detection of apples.

Therefore, to simulate the complexity of the scene and enrich the background of the object, we propose a leaf illustration data augmentation method, which uses some leaf illustrations to randomly insert on the original image. Firstly, collect 5 kinds of apple leaf illustrations, as shown in Figure 2. The format of the illustration is PNG, only contains the object itself, and the background is transparent, which helps protect the original image after insertion and avoid adding the invalid background. Secondly, the illustration size is 1/8 to 1/4 of the average value of all the ground-truth in the current image, and the number of insertions is 5 to 15 times. Finally, the original dataset is expanded in batches by using the illustration data augmentation method. Figure 3 shows the augmentation effects of different leaf illustrations.

3. Methodologies

3.1. YOLOv4

YOLOv4 is the state-of-the-art, real-time detection model, which is further improved based on the YOLOv3 model. As a result, on the MS COCO dataset, without a drop in the frames per second (FPS), the mean Average Precision (mAP) is increased to 44%, and the overall performance is significantly improved. There are three major improvements in the network structure: (1) Using the CSPNet [22] to modify Darknet53 to CSPDarknet53, which further promotes the fusion of low-level information and achieves stronger feature extraction capabilities. As shown in Figure 4b, the original residual module is divided into left and right parts. The right parts maintain the original residual stack, and the left parts use a large residual edge to fuse the low-level information with the high-level information extracted from the residual block; (2) Using the Spatial Pyramid Pooling (SPP) [23] to add 4 different max-pooling operations at the last output to further extract and fuse features, the convolution kernel size is (1 × 1), (5 × 5), (9 × 9), and (13 × 13), as shown in Figure 5; (3) Modifying Feature Pyramid Networks (FPN) [24] structure to Path Aggregation Network (PANet) [25], that is, add a top-down structure to the bottom-up structure of FPN to further extract and merge feature information, as shown in Figure 6b.

3.2. EfficientNet

In recent years, the rapid development of deep learning has spawned various excellent convolutional neural networks. From the initial simple network [26,27,28] to the current complex network [29,30,31,32], the performance of the models is getting better and better in all aspects. EfficientNet combines the advantages of previous networks, which summarizes the improvement of network performance into three dimensions: (1) Deepen the network, that is, use the skip connection to increase the depth of the neural network, and achieve feature extraction through deeper layers; (2) Widen the network, that is, increase the number of convolutional layers to achieve more features and obtain more functions; (3) By increasing the input image resolution, the network can learn and express more things, which is beneficial to improve accuracy. Then, use a compound coefficient ϕ to uniformly scale and balance the depth, width, and resolution of the network, and maximize the network accuracy on limited resources. The calculation of the compound coefficient is shown in Equation (1):

$$\begin{array}{l}\hspace{1em} \mathrm{depth}:\hspace{1em}d={\alpha }^{\varphi }\\ \hspace{1em} \mathrm{width}:\hspace{1em}w={\beta }^{\varphi }\\ \mathrm{resolution}:\hspace{1em}r={\gamma }^{\varphi }\\ \hspace{1em}\hspace{1em}\hspace{1em}s.t. \alpha \cdot {\beta }^2\cdot {\gamma }^2\approx 2\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \alpha \ge 1,\beta \ge 1,\gamma \ge 1\end{array}$$

(1)

where the d, w, and r are the coefficients used to scale the depth, width, and resolution of the network. The α, β, and γ are resource allocation for network depth, width, and resolution. According to the research of Tan M [19] in his paper, the network parameters of EfficientNet-B0 are shown in

Table 2. EfficientNet-B0 network parameter.

Stage $\mathit{i}$	Operator ${\stackrel{\wedge }{\mathit{F}}}_{\mathbf{i}}$	Resolution ${\stackrel{\wedge }{\mathit{H}}}_{\mathbf{i}}\times {\stackrel{\wedge }{\mathit{W}}}_{\mathbf{i}}$	#Channels ${\stackrel{\wedge }{\mathit{C}}}_{\mathbf{i}}$	#Layers ${\stackrel{\wedge }{\mathit{L}}}_{\mathbf{i}}$
1	Conv3 × 3	224 × 224	32	1
2	MBConv1, k3 × 3	112 × 112	16	1
3	MBConv6, k3 × 3	112 × 112	24	2
4	MBConv6, k5 × 5	56 × 56	40	2
5	MBConv6, k3 × 3	28 × 28	80	3
6	MBConv6, k5 × 5	14 × 14	112	3
7	MBConv6, k5 × 5	14 × 14	192	4
8	MBConv6, k3 × 3	7 × 7	320	1
9	Conv1 × 1 & Pooling & FC	7 × 7	1280	1

. The optimal coefficients of the network are: α = 1.2, β = 1.1, γ = 1.15. EfficientNet is mainly made up of Stem, 16 Blocks, Conv2D, GlobalAveragePooling2D, and Dense layers. The design of Blocks is mainly based on the residual structure and attention mechanism, and the other structures are similar to conventional convolutional neural networks. Figure 7 shows the EfficientNet-B0, which is the baseline network of EfficientNet.

3.3. EfficientNet-B0-YOLOv4

There are 8 versions of EfficientNet (B0–B7). With the increase of the version, the performance of the model gradually improves, but the corresponding model size and calculation amount also gradually increases. Although the original YOLOv4 model has excellent performance, its size and calculation amount are large, which is not suitable for the application of some low-performance devices. To further improve the accuracy and efficiency of the YOLOv4 model and consider the size of the model, we replace the backbone network CSPDarknet53 of the YOLOv4 model with EfficientNet-B0, and choose P3, P5, and P7 as three different feature layers. Since the output sizes of the three feature layers of the CSPDarknet53 are (256 × 256), (512 × 512), and (1024 × 1024), respectively, the corresponding P3 is (40 × 40), P5 is (112 × 112), and P7 is (320 × 320). Therefore, to match the size and further extract the features, Conv2D is added to adjust the three output features. Figure 8 shows the network structure of EfficientNet-YOLOv4.

The loss function remains the same as the YOLOv4 model, which consists of three parts: classification loss, regression loss, and confidence loss. Classification loss and confidence loss remain the same as the YOLOv3 model, but Complete Intersection over Union (CIoU) [33] is used to replace mean squared error (MSE) to optimize the regression loss.

The CIoU loss function is as follows:

$$LOS{S}_{CIoU}=1-IoU+\frac{{\rho }^2\left(b,b^{gt}\right)}{c^2}+\alpha \upsilon ,$$

(2)

where ${\rho }^2\left(b,b^{gt}\right)$ represents the Euclidean distance between the center points of the prediction box and the ground truth, c represents the diagonal distance of the smallest closed area that can simultaneously contain the prediction box and the ground truth. Figure 9 shows the structure of CIoU.

The formulas of α and υ are as follows:

$$\alpha =\frac{\upsilon }{1-IoU+\upsilon },$$

(3)

$$\upsilon =\frac{4}{{\pi }^2}{\left(\arctan \frac{w^{gt}}{h^{gt}}-\arctan \frac{w}{h}\right)}^2.$$

(4)

The total loss function of the YOLOv4 model is:

$$\begin{array}{l}LOSS=1-IoU+\frac{{\rho }^2\left(b,b^{gt}\right)}{c^2}+\alpha \upsilon -\\ \hspace{1em}\hspace{1em}\hspace{1em} {\displaystyle \sum _{i=0}^{S^2}{\displaystyle \sum _{j=0}^B{I}_{ij}^{obj}\left[\stackrel{\wedge }{C_i}\log \left(C_i\right)+\left(1-\stackrel{\wedge }{C_i}\right)\log \left(1-\stackrel{}{C_i}\right)\right]}}-\\ \hspace{1em}\hspace{1em}\hspace{1em} {\lambda }_{noobj}{\displaystyle \sum _{i=0}^{S^2}{\displaystyle \sum _{j=0}^B{I}_{ij}^{noobj}}}\left[\stackrel{\wedge }{C_i}\log \left(C_i\right)+\left(1-\stackrel{\wedge }{C_i}\right)\log \left(1-C_i\right)\right]-\\ \hspace{1em}\hspace{1em}\hspace{1em} {\displaystyle \sum _{i=0}^{S^2}{I}_{ij}^{obj}}{\displaystyle \sum _{c\in classes}\left[\stackrel{\wedge }{p_i}\left(c\right)\log \left(p_i\left(c\right)\right)+\left(1-\stackrel{\wedge }{p_i}\left(c\right)\right)\log \left(1-p_i\left(c\right)\right)\right]}\end{array}$$

(5)

where S² represents S × S grids, each grid generates B candidate boxes, and each candidate box gets corresponding bounding boxes through the network, finally, S × S × B bounding boxes are formed. If there is no object (noobj) in the box, only the confidence loss of the box is calculated. The confidence loss function uses cross entropy error and is divided into two parts: there is the object (obj) and noobj. The loss of noobj increases the weight coefficient λ, which is to reduce the contribution weight of the noobj calculation part. The classification loss function also uses cross entropy error. When the j-th anchor box of the i-th grid is responsible for certain ground truth, then the bounding box generated by this anchor box will calculate the classification loss function.

4. Experiments and Discussion

4.1. Experimental Details

4.1.1. Simulation Setup

The experimental environment of this paper is under Ubuntu 18.04 system, GPU is Tesla K80 (12 GB), CPU is Intel XEON, and the models are all written with PyTorch. General settings: the training epoch is 100, the learning rate for the first 50 epochs is 1 × 10⁻³ and the batch size is 16, the learning rate for the next 50 epochs is 1 × 10⁻³ and the batch size is 8. Due to the relatively small RAM, the input image size of the YOLOv4 model is changed from 608 × 608 to 416 × 416, which is the same as the original YOLOv3 model.

Table 3. The basic configuration of the local computer.

Computer Configuration	Specific Parameters
CPU	Intel XEON
GPU	Tesla K80
Operating system	Ubuntu18.04
Random Access Memory	12 GB

shows the basic configuration of the local computer.

4.1.2. Evaluation Index

In the binary classification problem, according to the combination of the sample’s true class and the model’s prediction class, it can be divided into 4 types: TP, FP, TN, and FN. TP means true positive, that is, the actual is positive and the prediction is also positive; FP means false positive, that is, the actual is negative but the prediction is positive; TN is true negative, that is, the actual is negative and the prediction is also negative; FN means false negative, that is, the actual is positive but the prediction is negative. The Precision represents the proportion of samples that are true positive among all samples predicted to be positive. The Recall represents the proportion of samples predicted to be positive among the samples that are true positive. The AP value of each class is the area under the P-R curve formed by the Precision and the Recall. The mAP value is the average of the AP values of all classes. The F1 score is based on the harmonic average of the Precision and the Recall. The formula definition is as follows:

Precision:

$$P=\frac{TP}{TP+FP}.$$

(6)

Recall:

$$R=\frac{TP}{TP+FN}.$$

(7)

Average Precision:

$$AP={\displaystyle {\int }_0^{1}p\left(r\right)dr}.$$

(8)

Mean Average Precision:

$$mAP=\frac{1}{n}{\displaystyle \sum _{i=1}^{n}A{P}_i}.$$

(9)

F1 score:

$$F1=2\times \frac{P\times R}{P+R}.$$

(10)

4.2. Experimental Results

4.2.1. Influence of Data Augmentation Methods

To obtain better detection results, traditional data augmentation techniques and a leaf illustration augmentation technique are adopted to expand the dataset. To evaluate the influence of the augmentation techniques on the EfficientNet-B0-YOLOv4 model, the control variate technique is adopted to get rid of one data augmentation approach every time and get the F1 indicators in the absence of this method, as shown in

Table 4. F1 comparison between different data augmentation methods.

Data Augmentation Methods
Illustration	✓	✓	✓	✓	✓	✓	✓	✓	✓
Translation	✓	✓	✓	✓	✓	✓	✓	✓
Scale	✓	✓	✓	✓	✓	✓	✓
Rotation	✓	✓	✓	✓	✓	✓
Dropout	✓	✓	✓	✓	✓
Blur	✓	✓	✓	✓
Brightness	✓	✓	✓
Crop	✓	✓
Mirror	✓
F1	96.54%	95.62%	95.00%	94.87%	94.43%	94.28%	94.01%	93.52%	92.81%	90.76%

. It can be seen from the experimental results that the removal of the illustration data augmentation method has the greatest impact on the performance of the model, and the F1 drops by 2.05%, indicating that the image data augmented by the illustration data augmentation method has a greater contribution to enriching the diversity of the training set. Compared with common data augmentation methods, illustration data augmentation will generate new background and texture for the image, which is of great help to enhance the robustness of model detection, especially under the interference of dense leaves. Due to the complexity and diversity of leaves, the proposed illustration data augmentation method combined with common data augmentation methods to generate apple images can make up for the lack of training images, greatly reduce the workload of labeling, and achieve better results in model detection.

To better assess the performance of the improved model, we count the detection results of original images and augmented images. The test results are shown in

Table 5. Detection results of original and generated images.

Apple Images	Original	Mirror	Crop	Brightness	Blur	Dropout	Rotation	Scale	Translation	Illustration
Number of detected objects	263	270	266	281	276	274	268	266	265	283
Number of missed objects	3	8	8	6	8	8	9	7	4	7
Number of false objects	32	44	40	53	50	48	43	39	35	56

. It can be seen that the EfficinetNet-B0-YOLOV4 model in this paper can achieve desirable detection results for the apple images augmented by using data augmentation methods. Compared with the original image, the methods of mirror, crop, rotation, scale, and translation are mainly based on the change of image position or angle, which hardly adds new texture information, so the detection results are similar to those of the original images. The methods of brightness, blur, dropout, and illustration bring new texture information to the image. Although it will cause more false detections, keeps the number of missed detections similar to the original images, and has more object detections, which shows that the rich background will enhance the learning ability of the model. Compared with the detection results of traditional augmentation techniques, the proposed illustration augmentation technique will lead to more false detections, but the detection quantity and missed detections are consistent with these methods, which shows the effectiveness and feasibility of our proposed method.

To further verify the influence of the proposed illustration data augmentation method on the improved model, as shown in Figure 10, the detection results of the model trained by the illustration augmented image are compared with the model trained by the real image. From the detection results, it can be seen that the model trained by the illustration augmented images and the model trained by the real images both can accurately detect apples under no occlusion between leaves. However, under the dense occlusion between leaves, the model trained by real images hardly detects apples, while the model trained by illustration augmented images can detect apples well and improve the detection results. It can be seen that the illustration augmentation method enriches the leaf occlusion scene in the training images, provides richer features for the learning of the model, and thus helps to improve the learning ability and detection results of the model.

4.2.2. Comparison of Different Models

To verify the superiority of the proposed EfficientNet-B0-YOLOv4 model in this paper, we compare it with YOLOv3, YOLOv4, and Faster R-CNN with ResNet, which are the state-of-the-art apple detection models.

Table 6. Performance comparison between different models.

Different Models	F1	mAP	Precision	Recall
YOLOv3	91.76%	93.98%	95.81%	88.03%
YOLOv4	96.36%	96.85%	98.22%	94.57%
Faster R-CNN with ResNet	66.96%	82.69%	52.07%	93.76%
EfficientNet-B0-YOLOv4	96.54%	98.15%	95.52%	97.43%

shows the comparison of F1, mAP, Precision and Recall of different models.

Table 7. Weight comparison between different models.

Different Models	Time/s	Size/MB	Parameter	FLOPs
YOLOv3	0.405	235	6.15 × 107	3.28 × 1010
YOLOv4	0.410	244	6.40 × 107	2.99 × 1010
Faster R-CNN with ResNet	6.167	108	2.83 × 107	1.79 × 1011
EfficientNet-B0-YOLOv4	0.338	158	3.78 × 107	1.27 × 1010

shows the comparison of the average detection time per frame, weight size, parameter amount and calculation amount (FLOPs) of different models.

Table 8. Detection results of different models.

Different Models	Ground-Truth	Faster R-CNN with ResNet	YOLOv3	YOLOv4	EfficientNet-B0-YOLOv4
Number of detected objects	2340	4214	3745	3004	2712
Number of missed objects	0	147	112	45	68
Number of wrong objects	0	2021	1517	709	440

shows the detection results of different models in the test set. Figure 11 shows the comparison of the P-R curve of different models. Figure 12 shows the detection results of different models, where the green ring represents the missed detection and the blue ring represents the false detection.

Generally, to make robots pick more real apples in orchards, more attention should be paid to the improvement of the Recall. It can be seen from Table 6 and Table 7 that the Faster R-CNN with ResNet model has a better Recall (93.76%), but the other performance and detection results are the worst. Although the weight (108 MB) and the parameter amount (2.83 × 10⁷) are lower than the YOLO models, the two-stage steps are complex and lead to the calculation amount (1.79 × 10¹¹) and the average detection time per frame (6.167 s) greatly exceed the YOLO models. The YOLOv3 model and YOLOv4 model still maintain better real-time detection results, and other indicators in the weight are close to the same, but the detection performance of the YOLOv4 model is better than the YOLOv3 model, where the F1 is 4.60% higher, mAP is 2.87% higher, Precision is 2.41% higher, and Recall is 6.54% higher especially. The EfficientNet-B0-YOLOv4 model proposed in this paper is slightly better than the YOLOv4 model in detection performance, where the F1 is 0.18% higher, mAP is 1.30% higher, Precision is 2.70% lower, and Recall is 2.86% higher especially. But in terms of weight indicators, the improved model is much better than the YOLOv4 model, where the average detection time per frame is reduced by 0.072 s, the weight size is reduced by 86 MB, the parameter amount is reduced by 2.62 × 10⁷, and the calculation amount is reduced by 1.72 × 10¹⁰.

It can be seen from Figure 11 that the P-R curve area under the EfficientNet-B0-YOLOv4 model proposed in this paper is larger, which shows that it has better performance. It can be seen from Table 8 and Figure 12 that in the case of large objects, each model can accurately detect apples, but in the case of small objects, especially in the case of occlusion, the Faster R-CNN with ResNet model will have more missed detections and false detections, which leads to low Precision (52.07%). At the same time, the YOLO models will have fewer missed detections and false detections, the detection result of the YOLOv3 model is relatively poor, the detection result of the YOLOv4 model and the EfficientNet-B0-YOLOv4 model proposed in this paper are close to the same.

Based on the above analysis, the whole results of the EfficientNet-B0-YOLOv4 model proposed in this paper are better than the current popular apple detection models, which can achieve high-recall and real-time detection performance, and reduce the weight size and computational complexity. The experimental results show that the proposed method in this paper is well applied to the vision system of the picking robot.

5. Conclusions

To simulate the possible complex scenes of apple detection in orchards and improve the apple dataset, an illustration data augmentation method is proposed and 8 common data augmentation methods are utilized to expand the dataset. On the expanded 2670 samples, the F1 of using the illustration data augmentation method has increased the most. Given the large size and computational complexity of the YOLOv4 model, EfficientNet is utilized to replace its backbone network CSPDarknet53. The improved EfficientNet-B0-YOLOv4 model has the F1 of 96.54%, the mAP of 98.15%, the Recall of 97.43%, and the average calculation time per frame of 0.338 s, which are better than the current popular YOLOv3 model, YOLOv4 model, and Faster R-CNN with ResNet model. Comparing the proposed EfficientNet-B0-YOLOv4 model with the original YOLOv4 model, the weight size is reduced by 35.25%, the parameter amount is reduced by 40.94%, and the calculation amount is reduced by 57.53%. In future work, we hope to add more apple classes for detection, and conduct level evaluation for each class after picking. For example, each class is divided into three levels: good, medium, and bad, thus forming a complete set of the apple detection system. Furthermore, we will continue to consolidate the illustration data augmentation method to improve the dataset.