Multi-Feature Patch-Based Segmentation Technique in the Gray-Centered RGB Color Space for Improved Apple Target Recognition

Abstract

In the vision system of apple-picking robots, the main challenge is to rapidly and accurately identify the apple targets with varying halation and shadows on their surfaces. To solve this problem, this study proposes a novel, multi-feature, patch-based apple image segmentation technique using the gray-centered red-green-blue (RGB) color space. The developed method presents a multi-feature selection process, which eliminates the effect of halation and shadows in apple images. By exploring all the features of the image, including halation and shadows, in the gray-centered RGB color space, the proposed algorithm, which is a generalization of K-means clustering algorithm, provides an efficient target segmentation result. The proposed method is tested on 240 apple images. It offered an average accuracy rate of 98.79%, a recall rate of 99.91%, an F1 measure of 99.35%, a false positive rate of 0.04%, and a false negative rate of 1.18%. Compared with the classical segmentation methods and conventional clustering algorithms, as well as the popular deep-learning segmentation algorithms, the proposed method can perform with high efficiency and accuracy to guide robotic harvesting.

1. Introduction

With agricultural production developing toward large-scale, intensive, and precise processes to realize intelligence, demand for intelligent automation of agricultural equipment has been increasing rapidly [1,2,3]. Apples, one of the major fruits in the world, are still picked manually owing to the complex environment of the orchards. To decrease the cost of labor, agricultural production activities must shift from being labor-intensive to technology-intensive [4]. Agricultural fruit-picking equipment based on artificial technology play an important role in accelerating and promoting agricultural modernization [5,6].

The picking robot includes vision and control systems. The main function of the vision system is to accurately identify the target fruit and provide information for motion control [7,8]. However, under the complex natural environment, orchards experience constantly changing weak and strong illumination conditions [9,10]. Depending on the intensity of illumination, different degrees of shadows are formed on the surface of the apples, because of the occlusion caused by fruit branches and leaves as well as clusters of neighboring fruits. Thus, non-uniform halation and shadows are special kind of noise in the images acquired by the vision system, and they cause the loss of information regarding the location of apples in the images, thereby increasing the difficulty of recognition and segmentation [11]. How to effectively and accurately remove or weaken the effect of halation and shadows is one of the key issues of the vision system of agricultural harvesting robots; this topic has received extensive research attention [12].

In the vision system of existing apple-harvesting robots, the effect of illumination on the target is reduced by using specific transformation or enhancement algorithms after image acquisition. Algorithms for image de-lighting and shadow segmentation have also been proposed. Song et al. [13] proposed the method of mixing the illumination-invariant image with the red component extracted from the original image to eliminate the effect of illumination. Huang and He [14] segmented apple targets and the background using the fuzzy 2-partition entropy algorithm in the Lab color space and used the exhaustive search algorithm to determine the optimal threshold for image segmentation. Song et al. [15] used the illumination-invariant image principle to obtain the illumination-invariant image of the shadowed image of the apple. They then extracted the red component information of the original image, added it to the illumination-invariant image, and performed adaptive threshold segmentation on the resulting image to remove shadows. Lü et al. [16] proposed Red-Green (R-G) color feature method in which the main colors are separated to reconstruct the image and the apple target is obtained through threshold segmentation with the original image. Lv et al. [17] extracted the main colors of the image, then the image was reconstructed with the main colors, the reconstructed and original images were subtracted and denoised to realize the extraction of the highlighted region of fruits, the complete fruit target region was obtained by combining the two extracted regions. This method reduced the effect of illumination on apple recognition to a certain extent. However, for varying intensity of halation and degree of shadows on the apple surface, the accuracy of the recognition results is low.

To improve recognition accuracy, Wu et al. [18] proposed combining color and 3D geometric features for fruit point cloud segmentation. In this method, the local descriptor was applied to obtain the candidate regions, and the global descriptor was used to obtain the final segmented results. Sun et al. [19] proposed an improved visual attention mechanism named the GrabCut model, combined it with the Normalized cut (Ncut) algorithm to identify green apples in the orchard with varying illumination, and achieved good segmentation accuracy. Suh et al. [20] proposed a multi-threshold color space conversion algorithm to detect and remove shadows on ground vegetation by illumination, thus improving the accuracy of target recognition. However, the images were processed in a pixel-wise manner, and the inherent spatial information between pixels was ignored. When there is extremely strong illumination and shadow, the recognition accuracy is low, which presents new challenges for the application of the algorithm under the natural orchard environment.

A superpixel segmentation algorithm fully considers the spatial relationship between adjacent pixels. Liu et al. [21] divided the entire image into several superpixel units, then extracted the color and texture features of the superpixels, and used a support vector machine algorithm to classify the superpixels and segment the target apples. However, the execution speed of this method is low. Xu et al. [22] combined group pixels and edge probability maps to generate an image of the apple with superpixel blocks. They removed the effect of shadows by re-illuminating. This method can effectively remove the shadows from the apple’s surface in the image. However, the divided superpixels will eventually affect the segmentation accuracy. Xie et al. [23] proposed sparse representation and dictionary-learning methods for the classification of hyperspectral images by using the pixel blocks to improve the classification accuracy of the images. However, dictionary learning is used to represent images with complex structures, and the learning process is time-consuming and labor-intensive.

Recently, the deep convolutional neural networks (DCNNs) have dominated many fields of computer vision, such as image recognition [24] and object detection [25]. The DCNNs have also dominated the field of image semantic segmentation, such as fully convolutional network (FCN) [26]. An improved deep neural network DaSNet-v2, which can perform detection and instance segmentation on fruits [27]. However, these methods require large, labelled training data sets and a lot of computing power before a reliable result can be calculated. Therefore, new segmentation methods in the color space based on the apple’s characteristics are needed, so that the apples can be identified in real-time in the natural environment of the orchard.

This paper aims to effectively use the characteristic information of the apple’s color to help the picking robot recognize the target. For this purpose, we propose a multi-feature patch-based segmentation technique to segment the apple image in the gray-centered red, green, blue (RGB) color space. Cluster vectors that are not affected by illumination and shadows in the RGB color space are explored and then quaternions are used to decompose the apple image vertically along these vectors, thus obtaining feature maps.

2. Materials and Methods

2.1. Apple Image Acquisition

The variety of apple tested in this study was ‘Fuji’, which is the most popular variety in China. For image acquisition, a PowerShot G16 camera (Canon, Tokyo, Japan) was used to capture the images of apples. The fruit used for imaging were randomly selected from the apple orchards during cloudy and sunny weather conditions, and the images of apples were obtained under natural daylight conditions (08:00–17:00) during sunny days. A total of 300 images were obtained manually in the Baishui Apple experimental demonstration station of Northwest A & F University (109°16′ E, 35°4′ N). They were saved in JPEG format as 24-bit RGB color images. For the dataset, we considered those images with a shooting distance of 30–50 cm. In addition, the shooting angle with the fruit was adjusted to obtain images under different illumination and background conditions. Thus, three types of images were obtained: (1) 80 images having shadows on the apples at varying degrees. (2) 80 images having illumination at varying degrees (existing with the edge of the apple or the inside of the apple), and (3) 80 images had both shadows and illumination at varying degrees. The image resolution was 4000 × 3000 pixels (approximately 12 Megapixels).

The images were processed and analyzed using a computer with an Intel (R) Core (TM) i9-9880H, 2.70 GHz CPU, and equipped with 8 G random access memory. The proposed algorithms were simulated using MATLAB R2018b (The MathWorks Inc., Natick, MA, USA).

2.2. Vector Decomposition in Gray-Centered RGB Color Space

2.2.1. Gray-Centered RGB Color Space

Herein, we use the gray-centered RGB color space, with the origin of the RGB color space placed at the center of the color cube [28]. For 24-bit color images, the translation is achieved by simply subtracting (127.5, 127.5, 127.5) from each pixel value in the RGB space. As a result, all pixels along the same direction from mid-gray have the same hue [29]. This translation operation effectively moves all the pixels on the apple image by half the distance from their own pixels in the RGB color space, forming a new coordinate system with medium gray as the origin, as shown in Figure 1.

2.2.2. Vector Decomposition

A quaternion algebra is a mathematical tool to realize the reconstruction of a three-dimensional color image signal [30]. The quaternion-based method imitates human perception of the visual environment and processes RGB channel information in parallel [31].

A quaternion has four parts and can be written as $q=a+ib+jc+kd$, where $a,b,c,d\in ℝ$, and $i$, $j$, and $k$ satisfy the condition $i^2=j^2=k^2=-1$. For the apple image, the RGB color triple is represented as a purely imaginary quaternion and can be written as $U=R\cdot i+G\cdot j+B\cdot k$ [32,33,34].

Assuming that two pure quaternions P and Q are multiplied together as shown in Equation (1).

$$\begin{array}{l}PQ=\left(P_{1}i+P_{2}j+P_{3}k\right)\left(Q_{1}i+Q_{2}j+Q_{3}k\right)\\ \hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}=-P_1{Q}_1-P_2{Q}_2-P_3{Q}_3+\left(P_2{Q}_3-P_3{Q}_2\right)i\\ \hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}+\left(-P_1{Q}_3+P_3{Q}_1\right)j+\left(P_1{Q}_2-P_2{Q}_1\right)k\\ \hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}=-P\cdot Q+P\times Q\end{array}$$

(1)

where P and Q are two pure quaternions and $i$, $j$, and $k$ is a set of bases.

If $Q=v$ is a unit pure quaternion, P can be decomposed into parallel and perpendicular components about $v$ as shown in Equation (2)

$$\begin{array}{l}\left|P_v\right|=P\cdot v=\left|P\right|\cos \theta \\ \left|P_v\right|=P\cdot v=\left|P\right|\cos \theta \end{array}$$

(2)

where $\left|P_v\right|$, $\left|P_{v\perp }\right|$ respectively represent the component parallel to $v$ and the component perpendicular to $v$.

Let C denote the chosen Color of Interest (COI), and $C=C/\left|C\right|$ is a unit pure quaternion. Given a pure quaternion $U$ and a unit pure quaternion $C$, $U$ may be decomposed into components that are parallel and perpendicular to C as shown in Equation (3):

$$\begin{array}{l}{U}_C=\frac{1}{2}\left[U-CUC\right],\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{U}_C\perp C\\ U_{C\perp }=\frac{1}{2}\left[U+CUC\right],\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{U}_{C\perp }\perp C\end{array}$$

(3)

where $C$, $U$ is a unit pure quaternion and $\perp$, $\parallel$ imply that pure quaternion are regarded as vectors.

2.3. Multiple Shadow and Halation Feature Extraction and Fusion

2.3.1. Pixel Distribution of Apple Image in the RGB Color Space

The color information is significant and distinct feature of images that includes abundant valuable information [35]. To take advantage of this information, the distribution range of pixels in the image in each of the following four cases of the apple surface appearance is counted: no halation and no shadows; only different degrees of shadows; only different degrees of halation (at the edge or inside of the apple image); and different degrees of simultaneous halation and shadows. These pixel distribution ranges are displayed in the RGB color space, which represents all elements of the entire image (i.e., apple, background, halation, and shading), as shown in Figure 2.

The red pixels representing the important characteristics of the apple are easily visible and are significantly different from the background pixels (i.e., those of foliage, soil, and background), indicating that it is easier to segment (as shown in Figure 2a). However, shadows and halation are part of the apple itself as a special kind of noise, which makes distinguishing the apple from the complex background difficult (Figure 2b–d). To analyze the distribution range of various pixels in the above-mentioned four situations in a unified manner, the comprehensive pixel distribution area of the apple and the shadows and halation on its surface is considered as shown Figure 3. Clearly, the distribution of apple images in the RGB space roughly forms a triangle AHC. The background pixels are mainly distributed around the diagonal of the space body, and the shadow and halation pixels from the two-dimensional plane are distributed at the ends of the sides HC and AC of the triangle, respectively. It is not possible to separate the apple from the shadow and halation through one COI. Therefore, in this study, we choose different COIs according to the particularity of shadows and halation in the RGB space. For example, several COIs need to be selected along HC for shadows and along AC for halation.

2.3.2. COI Selection for Shadows and Halation

From the discussion in the previous section, the pixel distribution area of shadows and halation is known. However, the RGB color space has relatively poor uniformity, making it difficult to express the red pixels of the apple by accurate numerical values [36]. The Hue-Saturation-Value (HSV) color space is a common choice for describing colors where brightness and color can be separated. Because the HSV color space is closer to the human visual system than the RGB color space, in this study, we choose the HSV color space to describe the red pixels of apples when selecting the COI for shadows and halation [37]. The formulas for RGB to HSV conversion are given in Equation (4) [38]:

$$\begin{array}{c}V=\max \left(R,G,B\right)\\ S=1-\min \left(R,G,B\right)/V\\ H=\left\{\begin{array}{l}60\times \left(G-B\right)/\left(V-\min \left(R,G,B\right)\right)\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}if\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}V=R\\ 120+60\times \left(B-R\right)/\left(V-\min \left(R,G,B\right)\right)\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}if\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}V=G\\ 240+60\times \left(R-G\right)/\left(V-\min \left(R,G,B\right)\right)\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}if\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}V=B\end{array}\right.\end{array}$$

(4)

where R, G, and B are the red, green, and blue components, respectively, of the RGB color space.

After the RGB space is converted to the HSV space, the value range describing red can be obtained, but the red of the apple is only part of the red in the HSV space. By selecting the range of red distribution on HSV space, so that this range can be consistent with the red range of apple after converting back to RGB space again, then, the distribution of apple red on RGB can be drawn by the value, as shown in Figure 4.

Figure 4 shows that the pixel distribution of the apple’s red in the RGB space obtained from the HSV space roughly forms an irregular geometric figure ADHC. The pixel distribution for background is mainly around the side HA; for halation, it is inside the triangular area AEC; and for shadows, it is inside the triangular area HDE.

In the HSV color space, when the brightness V is less than 0.15, the pixels in the area HFK lose their color information, so the black that appears is not a shadow on the surface of the apple. In Figure 4, shadow pixels in the area HFK are eliminated; as a result, the real shadow range is FKED and the halation range is AEC.

Figure 4a shows the selected ten COIs divided equally between the shadow range FKED and the halation range AEC. Figure 4b shows the resulting image after rotating the five shadow COIs centered at point H. Figure 4c shows the pixel distribution map of the vertically decomposed the image along the rotated shadow COIs. The apple (including shadows) pixel data are concentrated in the area excluding the area HFK and obviously far away from the background pixel range HA. Figure 4d also shows the vertically decomposed image after rotating five of the COIs in the halation pixel distribution range. The apple (including halation) pixel data are concentrated in the area excluding the background pixel range HA.

The apple image is decomposed in parallel and perpendicularly along the ten COIs (experiment to get the most suitable number) and ten feature maps parallel and perpendicular to the COIs are obtained, respectively. Figure 5a shows an apple image with shadows and light coexisting on the target surface. A feature map is obtained after the decomposition of the image along the COIs of the selected shadow and halation.

Figure 5b,c are the images obtained by parallel and vertical decomposition along the COI of the selected shadow, and Figure 5d,e are those obtained by parallel and vertical decomposition along the COI of the selected light. Figure 5b,d are not a grayscale image, they are still color images (RGB components are equal), reflecting the degree of color dark and light in the apple image. Figure 5c,e reflects the color information of the apple image, where in Figure 5c, the shadow is eliminated to a certain extent, but the illuminated area still exists. Thus, we consider the image obtained by vertical decomposition along the ten COIs as the apple features.

2.4. Apple Image Segmentation

2.4.1. Patch-Based Multi-Feature Segmentation Algorithm

Traditional K-means is an unsupervised learning algorithm that places each pixel into two clusters based on the Euclidean distance function [39]. However, this calculation ignores the spatial relationship between pixels in the image, leading to poor segmentation results for complex images. In particular, the local variations of apple images cannot be effectively described by a pixel-based method. Based on the pixel patches, this study proposes a multi-feature patch-based segmentation model, as shown in Equation (5):

$$\left\{\begin{array}{l}\underset{I_i,C_{ij}}{\min }\left\{{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{x\in \mathsf{\Omega }}{\displaystyle \sum _j\frac{w_j}{m_j}{‖R_{m_j}{f}_j\left(x\right)-C_{ij}‖}^2}}}{I}_i\left(x\right)\right\}\\ s.t.\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}I.\hspace{0.33em}{\displaystyle \sum _{i=1}^N{I}_i\left(x\right)=1,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{I}_i\left(x\right)=0\hspace{0.33em}\hspace{0.33em}or\hspace{0.33em}\hspace{0.33em}1,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}i=1,2,\cdot \cdot \cdot N}\end{array}\right.$$

(5)

where f_j(x) is the jth color feature of the original apple image f(x), $R_{m_j}{f}_j\left(x\right)$ is the $\sqrt{m_j}\times \sqrt{m_j}$ patch vector, C_ij is the clustering center, and $I_i\left(x\right)$ is the label function, whose value can be 0 or 1.

As Equation (5) shows, both color features and local contents of the apple images are considered in our model, making it robust to non-uniform illumination, shadows, and local variations in apple images.

Superscript k is used to represent the kth iteration. The iteration solving process can be divided into two steps:

First, in Equation (5), I_i is fixed and C_ij is updated, and the optimization problem shown in Equation (6) needs to be solved.

$$\underset{C_{ij}}{\min }\left\{{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{x\in \mathsf{\Omega }}{\displaystyle \sum _j\frac{w_j}{m_j}{‖R_{m_j}{f}_j\left(x\right)-C_{ij}‖}^2}}}{I}_i^{\left(k\right)}\left(x\right)\right\}$$

(6)

By differentiating, we obtain

$$\underset{C_{ij}}{\min }\left\{{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{x\in \mathsf{\Omega }}{\displaystyle \sum _j\frac{w_j}{m_j}{‖R_{m_j}{f}_j\left(x\right)-C_{ij}‖}^2}}}{I}_i^{\left(k\right)}\left(x\right)\right\}$$

Next, in the second step, in Equation (5), C_ij is fixed and I_i is updated, and resulting optimization problem is given by Equation (7).

$$\left\{\begin{array}{l}\underset{I_i}{\min }\left\{{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{x\in \mathsf{\Omega }}{\displaystyle \sum _j\frac{w_j}{m_j}{‖R_{m_j}{f}_j\left(x\right)-C_{ij}^{\left(k+1\right)}‖}^2}}}{I}_i\left(x\right)\right\}\\ s.t.\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}I.\hspace{0.33em}{\displaystyle \sum _{i=1}^N{I}_i\left(x\right)=1,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{I}_i\left(x\right)=0\hspace{0.33em}\hspace{0.33em}or\hspace{0.33em}\hspace{0.33em}1,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}i=1,2,\cdot \cdot \cdot N}\end{array}\right.$$

(7)

This results in Equation (8).

$$\begin{array}{l}{I}_i^{\left(k+1\right)}\left(x\right)=\left\{\begin{array}{l}1,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}i=i_{\min }\left(x\right),\\ 0\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}i\ne i_{\min }\left(x\right),\end{array}\right.\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{i}_{\min }\left(x\right)=\underset{i}{\arg \min \left(r_i^{\left(k+1\right)}\right)},\\ r_i^{\left(k+1\right)}={\displaystyle \sum _j\left(w_j/m_j\right)}{‖R_{m_j}{f}_j\left(x\right)-C_{ij}^{\left(k+1\right)}‖}^2,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}i=1:N,\hspace{0.33em}\hspace{0.33em}j=1:3\end{array}$$

(8)

2.4.2. Principal Component Analysis (PCA) Dimensionality Reduction

Each apple image has three channels each of R, G, and B. In this study, the selected feature maps will increase from 10 to 30. If the patch size is m × m, the number of feature maps is ${\displaystyle \sum }_{i=1}^{30}{m}_i^2$. It affects not only the calculation timeliness but also the classification performance.

We use the PCA method as the dimensionality reduction processing method to compress the number of features, eliminate redundant data, and reduce the dimension of data. The PCA can maximize the information of data after the intrinsic dimensionality reduction and can determine the importance of the direction by measuring the size of the data variance in the projection direction [40]. W is defined as a matrix consisting of all feature mapping vectors in the column direction. This matrix can better retain the information in the data. The covariance matrix A can be obtained as follows:

$$A=\frac{1}{m-1}{\displaystyle \sum _{i=1}^m\left(x_i-\overline{x}\right)}{\left(x_i-\overline{x}\right)}^T$$

(9)

where m is the number of data participating in dimensionality reduction, x_i is the specific vector expression of random data i, and $\overline{x}$ is the average vector of all data participating in dimensionality reduction.

The output of PCA is Y = W’X. The optimal W is composed of the eigenvectors corresponding to the first k largest eigenvalues of the data covariance matrix as column vectors, thereby reducing the original dimension of X to k dimensions. In this study, we consider the first 6 principal components after the dimensionality reduction of feature data. At this time, the contribution rate reaches more than 96%. This method not only retains the characteristics of multi-dimensional image data, but also ensures the timeliness of model operation.

2.4.3. Halation and Shadow Image Fusion

When the surface of the apple has only shadow or only halation, the shadow COI or halation COI is used to complete the target segmentation in the apple image. In fact, in the natural environment, the distribution of shadows and halation on the surface of the apple is irregular and different degrees of lighting and shadows exist simultaneously. The result of image segmentation obtained by using only the shadow COI or halation COI is not a complete segmentation of the apple.

Figure 6 shows that there is a significant difference in the RGB value distribution between the illuminated area and the shaded area. Thus, this paper sets a B threshold (B = 93), and divides apples image in the RGB space into two areas: halation and shadow. When the B value in an apple image is greater than the threshold, the segmentation result is obtained by using the halation COI, and when the B value is less than the threshold, the segmentation is performed using the shadow COI. Finally, both results are combined to obtain a complete segmentation of the apple under the simultaneous effects of shadows and light, as shown in Figure 7.

3. Experimental and Analysis

To verify the validity and reliability of the proposed apple image segmentation method, 240 images of red ripe apple targets were selected for testing. To test the performance of the algorithm to the greatest extent, the test set consisted of the apple surface images under three conditions: varying degrees of shadow, light, and both conditions simultaneously. To quantitatively evaluate the effectiveness of segmentation by the proposed algorithm, the test results were evaluated in terms of the recall rate, precision rate, F-measure index, false positive rate (FPR), and false negative rate (FNR) [41]. These metrics are calculated as follows.

$$Recall=\frac{TP}{TP+FN}$$

(10)

$$Precision=\frac{TP}{TP+FP}$$

(11)

$$F-measure=\frac{2\ast Precision\ast Recall}{Precision+Recall}$$

(12)

$$FPR=\frac{FP}{FP+TN}$$

(13)

$$FNR=\frac{FN}{TP+FN}$$

(14)

where Truth Positives (TP) represents the number of pixels that are correctly segmented as belonging to the apple; False Negatives (FN) represents the number of pixels belonging to the apple that are incorrectly segmented as the background; False Positives (FP) represents the number of pixels belonging to the background that are incorrectly classified as those belonging to the apple; and Truth Negatives (TN) represents the number of pixels that are correctly segmented as belonging to the background.

The recall rate and precision can be used to measure the ability of the algorithm to identify the apple correctly. The F-Measure is the weighted harmonic average of precision and recall. The FPR gives the percentage of the pixels that belong to the background but are classified as the target. The FNR gives the percentage of the pixels belonging to the target but incorrectly classified as the background. The fruit area in the test images of the data set was manually marked by Labelme software, and the marked results were recorded as the ground truth [42].

Figure 8 shows the comparison of the proposed algorithm with the apple image segmentation method using Red-Blue (R-B) based K-means clustering (Method 1) [17], the fast and robust fuzzy C-means target area acquisition method (Method 2) [43], and depth Comparison of the results of the Mask Regions with Convolutional Neural Network (Mask R-CNN) image target instance segmentation method (method 3) [25] during learning. Figure 8a shows the original image in the test set (the first two are with strong shadows and weak halation, the middle two are with strong lighting and weak shadows, and the last two are with strong halation and strong shadows). Figure 8b,c show the segmentation results obtained using Methods 1 and 2. Both algorithms are based on image global information clustering and segmentation. Figure 8d shows the segmentation results obtained using Method 3, which includes a branch of prediction segmentation mask on the basis of the faster R-CNN. It completes the pixel-level segmentation of the target while achieving target detection.

The segmentation result of the R-B-based K-means clustering algorithm shows more severe loss of the target, especially in the case of strong shadows. In addition, more branches, leaves, grass and sky remain in the segmented image. The fuzzy C-means clustering based on morphological reconstruction and membership filtering increases the local spatial information and membership degree filtering in the image. The segmentation result of the mask R-CNN algorithm has a better segmentation effect on weak shadows and halation, which reduces the mis-segmentation of the background. However, the algorithm does not provide complete segmentation for strong shadows and halation. Although the mask R-CNN algorithm yields outstanding segmentation results compared with the cluster-based image segmentation methods 1 and 2, it could not retain the apple’s edge and requires a large training data. Figure 8e shows the segmentation results of the proposed algorithm. Because this method is designed according to the characteristics of the apple (including shadows and halation), it is more robust to different degrees of shadows and halation and obtains a complete target segmentation.

The results of the segmentation algorithms are compared with the ground truth pixel by pixel, and the performance is evaluated on the basis of the recall, precision, F-measure, FNR, and FPR calculated for each of these algorithms. The results of the comparison test are given in

Table 1. Performance of different methods in terms of the average values of the metrics.

Method	Method Source	Recall	Precision	F-Measure	FPR	FNR
K-means	K-means based on R-B (Jidong Lv et al., 2019)	74.15%	65.31%	69.45%	21.07%	24.93%
Fuzzy C-means	Fast and robust fuzzy C-means (Tao Lei et al., 2017)	93.25%	96.82%	95.00%	1.51%	6.68%
Deep learning	Mask R-CNN (Kaiming HE et al., 2018)	97.69%	97.92%	97.80%	0.33%	2.25%
Proposed algorithm		98.79%	99.91%	99.35%	0.04%	1.18%

. The average recall, precision, F-measure, FPR, and FNR of the proposed algorithm were, respectively, 98.79%, 99.91%, 99.35%, 0.04%, and 1.18%. Those of the K-means clustering algorithm based on R-B (Method 1) were 74.15%, 65.31%, 69.45%, 21.07%, and 24.93%. Those of the fast and robust fuzzy C-means clustering (Method 2) were 93.25%, 96.82%, 95.00%, 1.51%, and 6.68%. Those of the mask R-CNN instance segmentation algorithm (Method 3) were 97.69%, 97.92%, 97.80%, 0.33%, and 2.25%. Therefore, the average values of the recall, precision, and F-measure of the proposed algorithm improved, respectively, by 24.64%, 34.60%, and 29.89% compared with those of Method 1; by 5.54%, 3.09%, and 4.35% compared with those of Method 2; and by 1.10%, 1.99%, and 1.55% compared with those of Method 3. In addition, the average values of the FPR and FNR decreased, respectively, by 21.03% and 23.75% compared with those of Method 1; by 1.47% and 5.50% compared with those of Method 2; and by 0.29% and 1.07% compared with those of Method 3.

4. Discussion

4.1. Location of Apple Targets

After completing the apple target identification, the apple targets need to be localized. Feng et al. [44] performed segmentation of the apple targets and realized two-dimensional localization of each apple target on the basis of the center of mass. Xiao et al. [45] proposed training of an apple color recognition model on the basis of a back propagation neural network. After recognizing the apple target in the image, the outline of the apple target is extracted using morphological operations, and a circle finally determined by the Hough transform algorithm is used to locate the apple target. Niu et al. [46] used the symmetry axis of the extracted apple target to locate the apple target. In this study, we use color prior information based on gray-centered RGB space, and by considering the local variation of the image, we perform the segmentation of apples using pixel patches. As we extract the contour of the apple target with high accuracy, to achieve the timeliness of the entire model, an algorithm for circle fitting on the contour of the apple target is implemented (the parameters of the circle are determined by summing the absolute values of the distances from the data points to the circle). Further, the center coordinates and radius of the fitted circle are used to locate the apple target. The apple target localization based on circle fitting is shown in Figure 9.

4.2. Further Research Perspectives

In recent years, deep learning has become a state-of-the-art technique for many tasks in computer vision [47]. It is trained on highly configured computers by using a large number of data labels, which are not only fast to test but also perform well and are easy to deploy and apply [48]. However, the relationship between different datasets in deep learning methods and different network architecture designs and network generalization capabilities is still being explored by a large number of researchers to find out what the essence [49,50,51]. For different data objects, such as apple images, the relationships between how much data to use, what type of data to use, and which network architecture to use will have acceptable generalization capabilities are still unclear, and their interpretability needs further research in progress.

Another mainstream approach is the model-based image segmentation method used in this paper [52]. We propose a model-based segmentation algorithm for specific images of ripe apples by studying the color features and local variations of the apple images and constructing features from the essence of the segmentation target; this approach makes the segmentation of apple images easy to explain and understand [53]. Since we have a comprehensive understanding of both the data and the underlying algorithms, tuning hyperparameters and changing the model design become simple, reasonable, and interpretable. In addition, the model-based approach is not computationally expensive as fast iteration is possible and various techniques can be implemented in less time. However, traditional image algorithms can only solve certain scenario-specific, manually definable, designable, and understandable image tasks. In addition, the model-based approach requires recalculation of the algorithm model each time a new recognition task is performed, which would lead to large consumption of computational power and data in case of complex recognition targets.

We believe that one of the important reasons for the effectiveness of deep-learning methods is that they coincide with image priors such as multi-scale, non-local, multiple-anisotropy, and non-linearity of images. The discovery of image priors plays a very important role in the field of image processing, and the effective use of new image prior information can lead to effective image representation and understanding. For many years, researchers have been working on the discovery of different image prior information [54,55]. When the apple color information is significantly different from the background, a concise and effective segmentation can be accomplished by ignoring the shape and texture information, but relying only on color information and considering local variations under the a priori assumptions of the model in this paper.

In this paper, a new class of precise prior information is provided for apple images using the model-based approach and an outstanding implementation result is obtained. Therefore, it is an attractive direction to apply this new image prior information to deep learning and further explore the model-based approach or the combination of the advantages of both approaches in order to realize further improved interpretation and generalization of deep-learning methods. As a future research direction, we will further explore the combination of model-based and deep-learning approaches under the a priori conditions considered in this paper.

5. Conclusions

To address the problem of low segmentation accuracy of apple images because of the non-uniform illumination in the natural environment of unstructured orchards, a new segmentation method was established on the basis of the characteristics of the apple images (including halation and shadows). Using this method, segmented apple images using multiple shadow (strong, weak) and halation (strong, weak) features were extracted and then merged.

The segmentation process involves the following steps: First, the pixel distribution of the apple image (with and without halation and shadows) in the RGB color space is observed. Then, the distribution areas of light and shadows are determined, multiple COIs that can cover the two areas are selected, and the image is transformed to the grayscale. Subsequently, the COIs are decomposed by quaternions in the center RGB color space, and the image obtained after vertical decomposition along the COIs is used as the feature map for segmentation.

This study proposed an efficient multi-feature patch-based segmentation algorithm, which is a generalization of the K-means clustering algorithm. To ensure real-time and effective segmentation, pixel patches of appropriate size area are selected; PCA dimensionality reduction of the selected multiple features is carried out, and the segmented images of the apple in the illuminated and shadow areas are obtained. Finally, both segmentation results are combined to obtain the complete apple area. The geometrical shape of the segmented target was well maintained, and the segmentation error was significantly reduced using the proposed patch-based segmentation algorithm.

To verify the effectiveness of segmentation, the designed segmentation method was quantitatively compared and evaluated with a classic algorithm, a modified clustering algorithm, and a deep learning algorithm. The experimental results showed that the proposed method’s recall, precision, and F-measure were higher and it’s FPR and FNR were lower than those of the other three methods.