Peaches Detection Using a Deep Learning Technique—A Contribution to Yield Estimation, Resources Management, and Circular Economy

Abstract

Fruit detection is crucial for yield estimation and fruit picking system performance. Many state-of-the-art methods for fruit detection use convolutional neural networks (CNNs). This paper presents the results for peach detection by applying a faster R-CNN framework in images captured from an outdoor orchard. Although this method has been used in other studies to detect fruits, there is no research on peaches. Since the fruit colors, sizes, shapes, tree branches, fruit bunches, and distributions in trees are particular, the development of a fruit detection procedure is specific. The results show great potential in using this method to detect this type of fruit. A detection accuracy of 0.90 using the metric average precision (AP) was achieved for fruit detection. Precision agriculture applications, such as deep neural networks (DNNs), as proposed in this paper, can help to mitigate climate change, due to horticultural activities by accurate product prediction, leading to improved resource management (e.g., irrigation water, nutrients, herbicides, pesticides), and helping to reduce food loss and waste via improved agricultural activity scheduling.

1. Introduction

Sustainable agricultural practices can improve product availability in a supply chain. Some traditional practices can dangerously compromise soil structure and an orchard’s abilities to adapt to extreme weather events, climate change, and the stresses of intensive harvest production. All methods, whether new or traditional, must be examined when performing sustainable agriculture practices [1].

Agriculture directly impacts climate change as its activities contribute to a portion of global anthropogenic greenhouse gas (GHG) emissions. The main GHGs produced by agriculture are nitrous oxide (N₂O) and carbon dioxide (CO₂). N₂O is produced by the microbial transformation of nitrogen in soils during applying synthetic fertilizers to the ground, and CO₂ is produced by changes in above- and below-ground carbon stocks caused by land-use changes [2]. Continuous monitoring can provide an up-to-date status of a field in real-time, based on the selected parameters of the plants and fields. This approach is made possible by a computer vision framework on the fields. Fruit detection via computer vision allows monitoring and control of yield, contributing to higher production. Higher yields indicate the possibility of rising production, with less water and without extending the area under agriculture, which means less deforestation and a reduction of natural resource consumption, and GHG emissions.

The application of precision agriculture techniques can help to mitigate climate change. Deep neural networks emerge as powerful tools in precision agriculture as they can provide accurate product prediction, leading to improved resource management, such as irrigation water, nutrients, herbicides, and pesticides, and help reduce food loss and waste by improved agricultural activity scheduling. Alibabaei et al. [3] investigated the ability of an encoder–decoder long short-term memory model, to model the daily evapotranspiration for threes. Assunção et al. [4] used computer vision, based on convolutional neural networks, to classify peach fruit diseases.

Yield estimation is a branch of precision agriculture. It enables planning for cropping, operations, inventory management, and other ancillary services (e.g., fruit pickup by a robot). There are several published works in the field of fruit yield estimation. For example, Häni et al. [5] presented a methodology for apple production estimation. Dorj et al. [6] developed a system for citrus fruit detection and counting. Bargoti and Underwood [7] presented a work on the detection of mangoes, almonds, and apples in orchards. A fruit yield estimation pipeline begins with detection, followed by tracking and counting. The fruit detection phase is critical to the performance of the yield estimation system. Most state-of-the-art fruit detection systems are based on a convolutional neural network (CNN) [8], such as object detection, segmentation, among others. These methods automatically extract features from the appearance of fruit images (i.e., colors and shapes). In this context, the development of fruit detection is specific (because the fruit colors, sizes, shapes, clusters, and distributions in the trees are particular). Figure 1 illustrates this problem. Considering the lack of work with peaches, this paper presents the results for peach detection using the object detection framework faster region-based convolutional neural network (Faster R-CNN). Another contribution is that the images are from a non-controlled environment (outdoors). That is, they come from a natural orchard.

Bargoti and Underwood [9] presented a methodology to obtain an estimate of red and green apples. For the detection phase, they used the traditional computer vision approach based on hue, saturation, and value (HSV) color space for segmentation. With the development of convolutional neural networks (CNNs), many authors use them for fruit detection based on segmentation and object detection approaches. Puttemans et al. [10] proposed a watershed and trinocular stereo triangulation-based segmentation with a cascade of weak classifiers to perform a strawberry and apple detector model. Bargoti and Underwood [11] trained a CNN, where its output was the probability that each pixel belonged to a fruit. Then, the output result was used to generate a binary mask for segmentation. Häni et al. [12] proposed the CNN segmentation model, named U-NET, for apple segmentation and circular Hough transform for fruit detection. The original purpose for this network is medical image segmentation. The author reports an F1 score of 0.858 for detection.

CNNs have also made massive contributions in improving image classification and object detection. In this regard, the object detection framework R-CNN [13] and its variants fast R-CNN [14], faster R-CNN [15], and mask R-CNN [16] are widely used in the literature. Sa et al. [17] proposed faster R-CNN to detect peppers, apples, avocados, mangoes, strawberries, and oranges. However, only the images of peppers are from the orchard. The rest of the fruit images are from the internet (Google Images). CNNs rely on large amounts of data for training to avoid overfitting, and they have good generalization. To achieve good performance with a relatively small set of training data, a transfer learning technique is often used.

Recently, you only look once (YOLO), another branch of object detection based on CNN, became a framework for fruit detection. Koirala et al. [18] applied this model to detect mangoes in orchard images. They used two metrics for evaluation, F1 and average precision (AP), and reported results of 0.96 and 0.98, respectively. Liu et al. [19] applied the mask R-CNN method to detect cucumber fruits. They chose ResNet-101 as the backbone and proposed a modification in the scales and aspect ratios of the anchor boxes. For evaluation, the authors used an F1 score and obtained a score of 0.894 for the test images.

Other studies were developed to detect fruits. However, fruit detection procedures are specific due to the particular distribution of fruit colors, sizes, shapes, branches, and bundles. This study provides the first application (and, consequently, the annotated dataset) of peach detection that is constrained by (1) peaches with different colors, (2) peaches overlapped, and (3) peaches occluded by leafs. The faster R-CNN computer vision framework is used for fruit detection in peach orchards. Fruit detection can be used to count the number of fruits during the culture, and knowing that value, irrigation and all other cultural practices can be conveniently scheduled, and the correct amount of required inputs for fertilization and plant protection can be acquired. This approach improves the productivity and competitiveness of farmers while ensuring environmental concerns. Thus, it can be considered a precision agriculture application based on a deep neural network to help mitigate climate change, as continuous monitoring provides an up-to-date status of the field culture and allows monitoring and control of yields, which may be used as decision-making support systems that lead to increased production with less consumption of natural resources and GHG emissions.

2. Materials and Methods

2.1. Dataset

The images used to compose the dataset were taken with the Eken H9R camera in a peach orchard in Beira Interior, Portugal. All images that composed the training dataset were from the same orchard and were taken on the same day. A particular feature for this peach dataset is its yellowish color. This feature can be a good way to check the generalization ability of the model when testing images from other orchards with a different peach color. Figure 2 shows a training sample image. The original images have a resolution of 5472 × 3648 (pixels), RGB channels with 24-bit color depths. The relationship between the object (fruit) size and image size affects the detection performance. This means that a small object is difficult to detect in a large image. This is due to the peculiarity of the CNN polling operation and anchor size, which is part of the object detection algorithm. To mitigate this problem, we crop each original image in for quadrants. Each of them with a resolution of 2736 × 1824 (pixels). Figure 3 illustrates this process.

For training, 200 images were used, with 1934 annotated fruits. For testing, 40 images (from the same orchard) with 410 annotated peaches were used.

In addition, a small dataset was created from another orchard (Quinta Nova, also in Beira Interior, Portugal), which has a different color, to check the generalization of the model. Figure 4 shows an image that composes the test dataset. It can be seen that the peaches have a reddish color and the illumination varies a lot compared to the training image. This subset of test images has the same resolution as the training images (2736 × 1824). With seven samples, 91 peaches were annotated.

2.2. Object Detection Framework

Object detection is a branch of computer vision that aims to find and classify objects in an image. Histograms of oriented gradients for human detection (HOG) [20] is one of the most popular and traditional (i.e., hand-crafted ) methods used for object detection. Nowadays, traditional object detections are replaced by modern models based on CNNs (e.g., faster R-CNN [15], single shot multibox detector (SSD) [21], you only look once (YOLO) [22]).

In this paper, the faster R-CNN model was used as the basis. The faster R-CNN is a two-stage model in which a CNN backbone is used for feature extraction in the first stage (e.g., VGG16 [23], ResNet [24], Inception [25], and others). Figure 5 shows a simplified workflow for the faster R-CNN in the context of fruit detection. Here, the outputs of the convolution layers are called feature maps. The last layer is used as an input to the region proposal network (RPN), which generates the regions containing possible objects. These regions of interest (ROIs) are used for object classification and bounding box prediction.

Figure 5 shows the faster R-CNN object detection model in the context of fruit detection.

The TensorFlow model application programming interface (API) was used to perform the experiments [26]. This API implements the object detection frameworks faster R-CNN and SSD. In this work, the faster R-CNN with the Inception v2 backbone was used because it has better detection accuracy compared to the other variants implemented in this API. In addition, transfer learning was performed on the COCO dataset [27]. Training and testing were performed on a desktop with Intel(R) Core(TM) i7-4790 CPU @ 3.60 GHz, 8 GB RAM, and an NVIDIA RTX 2080 graphics card with 8 G memory.

2.3. Evaluation Metric

Intersection over union (IOU), defined in (1), is a metric used to evaluate object detection tasks [28]. It provides a value for bounding box prediction. The value obtained from IOU can be used to indicate whether the prediction is true positive (TP), false positive (FP), or false negative (FN), depending on the threshold used. For example, if the IoU threshold is 0.5 and the IoU value for a (positive class) prediction is 0.7, then the prediction is classified as true positive (TP). On the other hand, if the IoU value is 0.3, then the prediction is classified as false negative (FN).

$$IOU=\frac{A\cap B}{A\cup B},$$

(1)

were A is the ground truth area and B is the detected area. Figure 6 illustrates the IOU metric.

Evaluations on the test images were performed using the PASCAL VOC metric average precision (AP) for an IOU threshold of 0.5.

3. Results and Discussions

3.1. Training Result

The training loss curve of faster R-CNN with Inception v2 is shown in Figure 7. The final loss was about 0.1 and the loss curve started to stabilize after 8000 iterations.

3.2. Test Results in the Same Orchard Where the Training Was Performed

Detection performance was assessed by the metric AP with IOU = 0.5 and achieved a value of 0.90. Figure 8 and Figure 9 show the visual results.

3.3. Test Results in a Different Orchard

The following results and discussion are intended to show the evaluation of the model in a different orchard, with many variations, compared to the orchard in which the model was trained, to evaluate more rigorously the generalization ability of the model. The test was performed on seven images with 91 peach fruits and resulted in an AP of 0.77.

Figure 10 shows the visual result for the first test image. The peculiarity of this image is the illumination, in which the light is too reflected in the leaves. The color of the fruit is also reddish. These features are completely different from the images found in the training dataset. As can be seen in Figure 10a,b, the model predicted non-occluded and clustered fruits well, but failed to detect some occluded fruits. In this image, the model only fails in the most difficult case of detection (very occluded). To improve the accuracy of the model, samples from this orchard need to be included in the training dataset.

Figure 11 shows the result for another test image, where the lighting is not good, as the exposure of the camera did not help to distinguish fruits from leaves. Moreover, there were several small fruits. In Figure 11a,b, it can be seen that the model did not detect many small fruits. This issue is a common problem with object detection models. One way to solve this problem is to take pictures close to the trees.

The distinctive feature of the Figure 12 is the many yellow leaves. This feature might confuse the model, since the color of the fruits in the training images is mostly yellow. Figure 12a,b shows the excellent detection results for this particular image. The model copes well with yellow leaves, which could affect the prediction of the model. There is only one false detection for yellow leaves, one for occluded and two for small fruits.

Figure 13 has two special features: many yellow leaves and reddish fruits. Again, the model shows good generalization. For this image, there was only one miss detection for yellow leaves and one miss detection for occluded fruit. The model did well for clustered and red fruits.

3.4. Final Discussions

The results show great potential in using the faster R-CNN model for peach detection in an uncontrolled environment. The model can deal well with occluded fruits, bunches of fruits, and light changes, which are features of a non-controlled environment. For a more accurate evaluation of the model, a test was conducted in another orchard. The results showed that the model is robust, as it handles image detections that are completely different from the training data (e.g., the color of the fruit, the color of the leaves, and the lighting). One possible way to improve the detection accuracy is to extend this work by adding images to the training data from a sequence of frames (from a video). This approach may help the model to better handle the problem of occlusion. Moreover, one may add images from different orchards and take the images closer to the tree.

4. Conclusions

This article contributes toward climate change mitigation through the application of precision agriculture using deep neural networks by providing the field’s current status, resulting in higher yields and, consequently, less deforestation, and a reduction in natural resource consumption and greenhouse gas emissions. In this work, we applied the deep learning approach faster R-CNN object detection model to evaluate the detection of peaches in images from an orchard. Peaches have a specific color, size, shape, fruit clustering, and distribution in a tree. The results show that the model handles all of these peculiarities well, in terms of peach fruit detection, and achieves an AP of 0.90 for the test split images belonging to the same orchard of the training, and an AP of 0.77 for the test images belonging to a different orchard (that has a feature discussed earlier). This shows great performance when compared to hand-crafted fruit detection models. For future work, we propose performing the detection in a sequence of frames, to detect hidden fruits more easily. Moreover, one may add images from different orchards and take the images closer to the tree.