Plucking Point and Posture Determination of Tea Buds Based on Deep Learning

Abstract

Tea is a significant cash crop grown widely around the world. Currently, tea plucking predominantly relies on manual work. However, due to the aging population and increasing labor costs, machine plucking has become an important trend in the tea industry. The determination of the plucking position and plucking posture is a critical prerequisite for machine plucking tea leaves. In order to improve the accuracy and efficiency of machine plucking tea leaves, a method is presented in this paper to determine the plucking point and plucking posture based on the instance segmentation deep learning network. In this study, tea images in the dataset were first labeled using the Labelme software (version 4.5.13), and then the LDS-YOLOv8-seg model was proposed to identify the tea bud region and plucking area. The plucking points and the central points of the tea bud’s bounding box were calculated and matched as pairs using the nearest point method (NPM) and the point in range method (PIRM) proposed in this study. Finally, the plucking posture was obtained according to the results of the feature points matching. The matching results on the test dataset show that the PIRM has superior performance, with a matching accuracy of 99.229% and an average matching time of 2.363 milliseconds. In addition, failure cases of feature points matching in the plucking posture determination process were also analyzed in this study. The test results show that the plucking position and posture determination method proposed in this paper is feasible for machine plucking tea.

1. Introduction

Tea occupies a significant place in people’s lives due to its high drinking value and economic value. It is widely cultivated in various countries, including China, Japan, England, India, etc. [1]. The growing demand for tea has led to the higher requirement for tea production and processing. There are four main procedures in tea processing: plucking, green removing, rolling, and drying. Of these, plucking is the first process, and its quality has an important impact on the subsequent processing. Tea plucking is time-consuming and tedious, and is low in mechanization. Problems in tea plucking have become more and more prominent in recent years. On the one hand, tea plucking is a laborious job, but with the development of urbanization, the rural labor force population is shrinking and labor costs are rising rapidly. On the other hand, the main laborers in tea gardens are currently old men or women, which is not sustainable as the population is aging [2]. Moreover, tea plucking, especially for famous tea, is in high demand for experience. Without training, plucking quality is not uniform.

To deal with the aforementioned issues and improve the efficiency of tea plucking, mechanized plucking of tea has been proposed. Japan is the first to study the mechanization of tea plucking and developed different types of tea plucking machines, including a self-propelled tea plucking machine, ride-on tea plucking machine, and others [3,4]. The mechanization of tea plucking has also been studied in England, India, and Australia, respectively [1]. However, these early tea plucking machines mainly cut the top of the tea trees without selection at the same height or angle, which is imprecise and may damage the leaves. It is feasible for ordinary tea, but not good for vintage tea. In recent years, some scholars have initiated research and development of tea bud precision-mechanized plucking equipment to achieve accurate harvesting of vintage tea [5,6,7].

Early tea plucking machines cut the tea leaves without determining the position and posture of the leaves, resulting in low plucking quality. Identification of tea leaves is a crucial step in tea plucking, and many scholars have conducted extensive research on this topic. There are two main stages in the study of tea identification. The color and shape features of the tea were used in the first stage, in which tea images were manipulated with various transformations using image processing techniques to obtain the target features [8,9,10]. Thangavel and Murthi proposed a semi-automated system for tea leaf harvesting which identified the different grades of tea leaves using key frame extraction, rice counting, optical flow, and a combination of Prewitt and auto thresholding [11]. Chen et al. analyzed the color properties of tea images using a luminance and color difference model (YUV model), and then they transformed the gray-level images to binary images using the Otsu method, eliminating the noise from the binary image to obtain the tea region [12]. Wu et al. proposed a tea bud identification algorithm using the K-means clustering method based on the a and b components of the Lab color model, and images taken at three different distances were used to compare the recognition effect and efficiency of the Otsu method and the K-means method [13]. Karunasena et al. presented a tea bud detection method using a cascade classifier based on the method integrated with Histogram of Oriented Gradients (HOG) features and Support Vector Machines (SVM) classification, and four sets of tea leaf sample images were taken to conduct the tea bud identification test [14].

Although the image processing method is simple to operate on image feature recognition, its precision is limited and the requirements of the images are high, making it difficult to meet actual demand. In particular, for tea bud recognition, the color and shape features of the foreground and the background are very close to each other, so the recognition results are barely satisfactory [15]. In recent years, deep learning has demonstrated its transcendent ability to recognize objects under complex background conditions, providing a new method for crop recognition and localization. Many experts have studied the detection and recognition of various fruits, such as kiwifruit [16], apple [17,18], Camellia oleifera fruit [19], tomato [20], and waxberry [21], using deep learning [22,23]. Therefore, the second stage of tea leaf feature recognition research is mainly based on deep learning methods [24,25,26,27,28,29]. Xu et al. proposed a two-level fusion network which combines the fast detection capability of YOLOv3 and the high-accuracy classification capability of DenseNet201 to detect tea buds [30]. They also investigated the influence of the shooting angle of the camera on the detection effect. The results show that detection is better on the side-shot tea buds than on the top-shot tea buds. Yang et al. proposed an improved YOLOv3 deep convolutional neural network algorithm to identify tender shoots for high-quality tea, which achieved end-to-end object detection with an image pyramid and residual network, and the K-means method was used to cluster the dimensions of the target boxes [28].

Plucking position and posture determination are the crucial steps after tea bud identification. However, few scholars have studied the plucking point determination method, and even fewer have studied the plucking posture determination. Li et al. detected the tea shoot regions on RGB images using the YOLO network and obtained the tea shoot point clouds using Euclidean clustering processing and the target point cloud extraction algorithm, and then determined the plucking point by setting a minimum cylinder with the radius enclosing the 3D point cloud and the height determined by the growth characteristics of the tea [7]. Chen et al. presented a plucking point locating method by combining the YOLOv3 algorithm, semantic segmentation algorithm, skeleton extraction, and minimum bounding rectangle. They segmented the main veins from the target regions using the Fast-SCNN and skeleton extraction algorithms, and finally required the final endpoint of the traversal as the plucking point [25]. Yang et al. identified the tea shoot regions with the improved YOLOv3 network, obtained the bounding box of the target. The contact position of the tea stem and the identification box were set as the plucking point. However, this method is highly demanding on the camera angle [28]. Chen et al. trained the Faster R-CNN to detect the one tip with two leaves regions (OTTL regions), and then utilized the FCN to detect the plucking point in the OTTL regions [31].

In fact, unlike large fruits, such as apples and oranges, tea buds are small and delicate. Tea buds are mechanically plucked by shearing the tea stems at a special location. If the plucking point is reached on the wrong direction, failure will be achieved. It is therefore necessary to determine the plucking position and posture accurately according to its growth characteristics.

In this study, we aimed to address the problem of plucking posture and position determination. The main innovations and research ideas are as follows:

• An approach for plucking posture and point determination of tea buds based on a deep learning network and an image processing method were proposed. This technique can enhance the accuracy of mechanized tea plucking implementation, thereby contributing to its successful adoption in practice.
• Two approaches of feature point matching were put forward, enabling accurate alignment of two sets of tea bud feature points and the acquisition of the feature line for tea plucking posture. The accuracy and efficiency of these two methods were analyzed in detail.
• The accurate matching of feature points plays a crucial role in determining the plucking posture of tea buds. Therefore, the failure cases of feature point matching were also analyzed. Additionally, future research prospects were presented.

The rest of this paper is arranged as follows. Section 2 presents the method of detection and segmentation of the tea buds and plucking area, as well as the method used for the determination of the plucking posture and position. The effect evaluation of the algorithm and some discussions are presented in Section 3. Conclusions are drawn in Section 4.

2. Materials and Methods

2.1. Dataset Building

2.1.1. Image Acquisition

The image dataset was built using tea bud pictures taken in Guangzhou National Agricultural Science and Technology Park (113.449166 E, 23.394961 N) at Guangdong, China, in June. Yinghong No. 9, which is one of the vintage tea varieties in South China, was used as the study object. To enhance the effectiveness of the proposed algorithm on improving actual plucking accuracy and efficiency, the pictures were captured 3–5 days before the tea buds were harvested under different weather conditions, including sunny, cloudy, and drizzly weather. In view of higher recognition rate and effective plucking, images were taken with a shooting angle of 60° and a photographic distance of 0.5–1 m, as shown in Figure 1. In total, 4024 images were taken with a resolution of 4032 × 3024 and stored in JPEG format.

2.1.2. Data Processing

After removing invalid images, the resolution of all the remaining pictures were scaled to 1008 × 756 so as to reduce the training time. In addition, to improve the generalization capability of deep learning results, the images were augmented using a horizontal mirror. Finally, there were 6450 images in the dataset. They were divided into three subsets: 80% (5000) of images were in the training set, and 20% (1250) of images were in the validation set; in addition, 200 images were used as test set to evaluate the performance of the results.

Image annotation is crucial for dataset processing in deep learning, which can produce feature information to train the deep learning network. The image annotation software Labelme was used as the annotation tool in this study to generate tea bud and plucking area regions [32]. One bud with one leaf (OBOL) is usually considered to be high-quality tea, so the OBOL regions were annotated as the tea buds in the tea image dataset. Meanwhile, in order to localize the plucking point and posture of the tea bud, the plucking area was also annotated. The plucking area was located on the tea stem between the first leaf and the second leaf, and was annotated by 4 lines. The overall annotation result is shown in Figure 2.

2.2. Instance Segmentation of Tea Bud and Plucking Area Based on Improved YOLOv8-Seg

YOLOv8 is an exceptional deep learning network model released by Ultralytics in 2023, which exhibits remarkable capabilities in classification, detection, segmentation, and pose estimation tasks. Compared to the previous versions, YOLOv8 adopts an anchor-free structure; instead of predicting the offset between the target and a predefined anchor, it directly predicts the target center, thereby enhancing both the efficiency and accuracy of the predictions [33]. The YOLOv8-seg model incorporates a mask coefficient and prototype branches, enabling the realization of instance segmentation. To accommodate diverse tasks, YOLOv8-seg offers different models based on the scale factors of depth and width, namely YOLOv8n-seg, YOLOv8s-seg, YOLOv8m-seg, YOLOv8l-seg, and YOLOv8x-seg.

The YOLOv8-seg network structure consists of input, backbone, neck, and head modules, as illustrated in Figure 3. The backbone is composed of a CBS module, C2f module, and SPPF module. The kernel size of the first convolution layer has been modified from the original 6 × 6 to 3 × 3 based on the YOLOv5 network. Moreover, all C3 modules have been replaced with C2f modules, resulting in further enhancements in computational efficiency. The neck module is utilized for feature fusion on the feature maps generated by the backbone network. The decoupled structure is employed in the head component, where multiple parallel branches are utilized to extract category features, position features, and mask features, respectively.

To enhance the segmentation efficiency and accuracy of the YOLOv8-seg network and accommodate the varying shapes of tea buds and plucking areas in this study, an improved algorithm called LDS-YOLOv8-Seg was proposed in this study. Firstly, to reduce the excessive parameters in the head module of the YOLOv8-seg model, shared convolution was employed to effectively reduce the parameters and enhance segment efficiency. Additionally, in order to compensate the potential accuracy degradation resulting from parameter reduction, Group Normalization (GN) was adopted as a replacement for Batch Normalization (BN) for normalization purposes. Secondly, an attention mechanism was added to the SPPF module to improve the model’s ability of feature extraction, which enabling the backbone to prioritize the most relevant parts of the input. Finally, the Deformable Convolution network version 2(DCNv2) was introduced to effectively adapt to the irregular and variable shape characteristics of the segmentation object, thereby acquiring more intricate spatial information. The structure of the LDS-YOLOv8-Seg network is shown in Figure 4.

2.2.1. The Improved Segmentation Head

The module of the head is a crucial component of the YOLOv8-seg model, which is responsible for the final object detection, classification, and segmentation. It comprises four types of branches, with the first branch dedicated to the prototype prediction, primarily consisting of an up-sampling layer. In addition, there are three branches for different sizes of feature maps: one for bounding box regression prediction, and another for classification prediction and the last branch for mask coefficient prediction. The regression prediction branch and classification prediction branch both consist of two CBS modules and a separate conv2d, contributing to the calculation of regression loss and classification loss, respectively. The CBS module comprises the conv2dlayer, BatchNorm2d(BN) layer, and the SiLU activation function.

The original YOLOv8-seg head structure, as shown in Figure 5, contains a large number of parameters, resulting in high computational cost. In this study, the model was lightweight by introducing shared convolution and 1 × 1 convolution to reduce the amount of calculations. The improved YOLOv8-seg head structure is shown in Figure 6, wherein all the white modules employ shared convolutions, including the CGS modules and Conv2d modules for classification and regression. In the regression and classification branches, the output feature maps of the neck component initially undergo convolutional operations using 1 × 1 shared convolutional blocks (CGS) followed by subsequent processing through two 3 × 3 shared convolutional blocks (CGS) for classification and detection. The bounding box regression results of different scales are scaled using a scale factor after the regression calculation in order to address the issue of inconsistent target scales detected by different regression branches due to the shared convolution. The segmentation task is performed at the pixel level, with a focus on calculating more refined spatial features. In order to ensure the segmentation accuracy, this branch abstains from utilizing shared convolutions. The Group Normalization (GN) module, proposed by Wu et al., has been demonstrated to improve calculation accuracy in detection and segmentation tasks compared to Batch Normalization (BN) [34]. The utilization of shared convolution inevitably leads to a certain degree of accuracy degradation. Therefore, in this study, Group Normalization (GN) was employed as an substitute for Batch Normalization (BN) within the CBS, resulting in the formation of the CGS module to enhance the accuracy of final calculation. Moreover, the computational burden of the model can be further reduced by a 1 × 1 convolution.

2.2.2. SPPF-LSKA Module

The attention mechanism simulates the cognitive process of humans in allocating attention when processing information. By assigning different weights to each element of the input data, it enables the model to focus on the most relevant features, thereby enhancing the expressiveness and generalization ability of the model while reducing unnecessary computations. The large separable kernel attention (LSKA) is a spatial attention mechanism, which is improved based on large kernel attention (LKA). The structures of LKA and LSKA are shown in Figure 7, where the $\otimes$ represents the Hadamard product; the LKSA module is obtained by decomposing the two-dimensional convolution kernels of the depth-wise convolution and deep dilated convolution in the LKA module into two cascaded horizontal and vertical one-dimensional separable convolution kernels, thereby further reducing both memory usage and computational complexity. The LSKA exhibits robust long-range dependence in picture, spatial, and channel adaptabilities, as well as in the exceptional scalability of the extremely large kernel.

The LSKA initially employs convolution to extract the horizontal and vertical features from the input feature map. Subsequently, dilated convolutions with different dilation rates are utilized to further enhance the feature extraction, which enables a broader receptive field coverage and obtains additional feature information without incurring additional computational costs. The obtained features are ultimately combined through a 1 × 1 convolution layer to generate the final attention map, which is subsequently subjected to the Hadamard product with the original input feature map.

The LSKA module was incorporated after the Concat layer of the SPPF module in this study. Moreover, the BN layer was replaced by a GN layer in the CBS module to generate the CGS module. The SPPF-LSKA module was formed through these aforementioned enhancements. The enhanced structure is shown in Figure 8. The feature fusion ability of the SPPF module can be enhanced through these improvements, reducing the influence of background features on the tea bud and plucking area detection and improving the model’s accuracy in complex environments.

2.2.3. C2f-DCNv2 Module

Traditional convolution performs well in handling static or regular targets; however, it may encounter challenges when dealing with irregular and variable shapes of tea buds, as well as significant differences in size. Compared to traditional convolution networks, the deformable convolution network (DCN) exhibits better recognition capabilities for diverse shapes and sizes of tea buds by dynamically adjusting the spatial and structural characteristics of the target. The deformable convolution network version 1 (DCNv1) introduces learnable offsets to accommodate the deformation of objects in the image; however, the new position of the sample point after offsetting may move beyond the ideal offset range. To address this limitation, in DCNv2, each sample not only learns the offset as in DCNv1, but also modulates it using the learned modulation parameters. To be specific, the DCNv2 introduces a modulation scalar $\Delta m_k$ in the convolution process, which represents the feature amplitude change at the kth position. This enables the network to prioritize critical features, reduce irrelevant information interference, and consequently enhance the model’s performance. The structure of the DCNv2 is shown in Figure 9.

The deformation characteristics of the target in the deep layers are more intricate in the deep learning networks. Therefore, in this study, the convolution operation in the final layer, C2f, of the backbone network was replaced by DCNv2 in this study, and the C2f-DCNv2 module was proposed. To be specific, the convolution in the bottleneck structure of C2f was specially replaced by DCNv2, enabling more flexible sampling of the input feature maps and better learning of the target scale and deformations. After the convolution layer, the residual connection was employed to fuse the output feature map with the input feature map. The structure of the improved C2f-DCNv2 is shown in Figure 10, where $\oplus$ represents the residual connection.

2.3. Matching the Tea Bud with Its Plucking Point

After the instance segmentation of the tea images, the tea bud mask and plucking area mask were acquired. But each tea bud mask and plucking area mask received from the instance segmentation was a single mask; they were not related to each other. The plucking point and the tea bud posture were what we wanted in the tea plucking activity. To obtain this information, the plucking area and the tea bud on the same tea should be matched first. The accurate matching of the plucking area and the tea bud is a crucial prerequisite for obtaining the optimal plucking posture. Enhancing the matching accuracy not only improves the overall matching effectiveness, but also contributes to the precision calculations of the tea bud plucking posture significantly. Furthermore, it facilitates the actual high-precision mechanized tea bud harvesting. Two algorithms were proposed to perform the matching, including the nearest point matching method and the point in range matching method.

2.3.1. Plucking Point Localization

To achieve the matching of the plucking area and the tea bud, the plucking point should be calculated first. The masks of the plucking area we obtained from the instance segmentation were a series of mask matrices. Every element of the matrix was the probability that the position of the original image belongs to the plucking area. As for the plucking area, the coordinates of the plucking point could be obtained by centroid calculating the mask matrix of the plucking area. Centroids of the mask matrix can be calculated with Equations (1) and (2).

$$x_c={\displaystyle \frac{{\displaystyle \sum p_i{x}_i}}{{\displaystyle \sum p_i}}}$$

(1)

$$y_c={\displaystyle \frac{{\displaystyle \sum p_i{y}_i}}{{\displaystyle \sum p_i}}}$$

(2)

where $x_c$ and $y_c$ represent the width and height coordinate of the centroid, $x_i$ and $y_i$ represent width and height coordinate of every element in the mask matrix, and $p_i$ represents the probability that the position of the element belongs to the plucking area.

The plucking point could be obtained from the above calculation, and it is definitely located in the plucking area. To match the plucking area with the tea bud, a feature point of the tea bud area should be calculated too. For the sake of convenience, the central point of the tea bud’s bounding box was chosen as the feature point, which can be calculated with Equations (3) and (4).

$$x_t={\displaystyle \frac{x_{left-top}+x_{right-bottom}}{2}}$$

(3)

$$y_t={\displaystyle \frac{y_{left-top}+y_{right-bottom}}{2}}$$

(4)

where $x_{left-top}$ and $y_{left-top}$ represent the width and height coordinates of the left-top point of the tea bud bounding box, $x_{right-bottom}$ and $y_{right-bottom}$ represent the width and height coordinate of the right-bottom point of the tea bud bounding box.

The central point of the tea bud bounding box and the centroid of the plucking area are shown in Figure 11.

2.3.2. Nearest Point Matching Method

The nearest point matching method (NPM) was proposed to realize the matching of the plucking point and the tea bud in this study. Two point sets were obtained through the above calculation. Point set A was a collection of the plucking points of the tea bud. Point set B was a collection of the central points of the tea buds’ bounding boxes. As shown in Figure 12, the blue points belong to the point set A and the red points belong to the point set B. According to the analysis on the tea image dataset, the plucking point of a tea bud is closer to the central point of its bounding box than to the bounding box of another tea bud. Therefore, the core idea of the nearest point matching method is to match every member of point set A to the nearest member of point set B. More detailed procedures for the nearest point matching method can be seen in Algorithm 1.

Algorithm 1: Nearest point matching method
Input: Point set A: {$x_a,y_a|(x_a,y_a)\in$ centroid of plucking area};             Point set B: {$x_b,y_b|(x_b,y_b)\in$ central points of tea bud bounding box};
Output: Result ←List:$\left(\left(x_i,y_i\right)\right.$,$(x_j,y_j)|$$\left(\left(x_i,y_i\right)\right.$ in A matched with the nearest $(x_j,y_j)|$ in B))
1:	Result ← zeros (row ← min (row of A, row of B), column ← 4)
2:	for i = 1, …, n (n ← row of A)
3:	D ← zeros (row = row of A, column = 1)
4:	for j = 1, …, m (m ← row of B)
5:	$d_j$ ← distance of $(x_a[i],y_a[i])$ and $(x_b[j],y_b[j])$
6:	k ← Index of minimum of d
7:	Result [i] ← $((x_a[i],y_a[i]),(x_b[k],y_b[k]))$
8:	B ← B without $(x_b[k],y_b[k])$
9:	Return Result

2.3.3. Point in Range Matching Method

Generally speaking, the centroid of the plucking area is located in the bounding box of the tea bud because the plucking area is one part of the tea bud. Therefore, the point in range matching method (PIRM) can be used to match the plucking point with the central point of the tea bud’s bounding box. As shown in Figure 3, the width coordinate x of the plucking point was located between the left and the right of the tea bud. Similarly, the height coordinate y of the plucking point was located between the top and the bottom of the tea bud. The core idea of the PIRM is to seek which tea bud bounding box satisfies the conditions of Equations (5) and (6).

$$x_b^{left}\le x_a\le x_b^{right}$$

(5)

$$y_b^{bottom}\le y_a\le y_b^{top}$$

(6)

where $x_a$ and $y_a$ denote width and height coordinates of the plucking point, $x_b^{left}$ and $x_b^{right}$ denote the left and right coordinates of the bounding box, and $y_b^{bottom}$ and $y_b^{top}$ denote the bottom and top coordinates of the bounding box. More detailed procedures for the PIRM can be seen in Algorithm 2.

Algorithm 2: Point in range matching method
$\mathrm{Input}: \mathrm{Point} \mathrm{set} \mathrm{A}\leftarrow \{x_a,y_a|(x_a,y_a)\in$ centroid of plucking area};              Point set B              $\leftarrow \{x_b^{left},x_b^{right},y_b^{bottom},y_b^{top}|(x_b^{left},x_b^{right},y_b^{bottom},y_b^{top})\in$ limits of tea bud area bounding box};
Output: Result $\leftarrow \mathrm{List}: ((x_i,y_i),(x_j,y_j)|((x_i,y_i)$ $\mathrm{in} \mathrm{A} \mathrm{matched} \mathrm{with} (x_j,y_j)$ $\mathrm{calculated} \mathrm{from} \mathrm{B}))$
1:	Result ← zeros (row ← min (row of A, row of B), column ← 4)
2:	for i = 1, …, n (n ← row of A)
3:	for j = 1, …, m (m ← row of B)
4:	$\mathrm{if} x_b^{left}[j]<x_a[i]<x_b^{right}[j],y_b^{bottom}[j]<y_a[i]<y_b^{top}[j]$
5:	Result$\left[\mathrm{i}\right]\leftarrow (x_a[i],y_a[i]),((x_{left}[j]+x_{right}[j])/2,(y_{bottom}[j]+y_{top}[j])/2)$
6:	$B\leftarrow B \mathit{without} (x_b^{left}[j],x_b^{right}[j],y_b^{bpttom}[j],y_b^{top}[j])$
7:	break
8:	else
9:	continue
10:	Return Result

2.4. Plucking Posture Determination

Typically, tea buds grow upwards due to the characteristics of apical dominance and heliotropism. In many studies, the mechanized harvesting of tea leaves was accomplished by vertically descending the plucking end-effector to reach the designated plucking position [35,36,37]. In practice, however, the growth direction of the tea buds is not completely straight up, owing to the influences of the planting position on the tea plant, the situation regarding the growth of the stem on which the tea bud is situated, and so on. The tea buds which are located at the edges of the tea bushes, especially, may even grow horizontally. Therefore, if the plucking manipulator simply moves straight down to the plucking point to pick the tea bud, it may cause failure or even damage the tea plant.

In view of the above problems, a method was proposed to determine the plucking direction of the tea bud. The plucking direction is opposite to the direction of the growth of the tea bud. Hence, the plucking manipulator can move along the plucking direction obtained in this study to achieve better plucking, avoiding errors or even failures caused by a direct downward motion.

Given the above studies, once the plucking point is matched to the central point of the tea bud’s bounding box, the plucking direction of the tea bud can be determined.

2.4.1. Two-Dimensional Plucking Posture Determination

The two-dimensional plucking direction, which is the plucking direction on the image of the tea bud, was determined first. As shown in Figure 13, the direction from the plucking point to the central point of the tea bud bounding box is the growth direction. Inversely, the direction from the central point of the tea bud bounding box to the plucking point is the plucking direction. Figure 13b was the partial enlarged detail of the frame in Figure 13a. Equation (7) shows the calculated formula of growth direction.

$$\overrightarrow{BA}=A-B$$

(7)

where A is the coordinate of the central point of the tea bud bounding box and B is the coordinate of the plucking point.

2.4.2. Three-Dimensional Plucking Posture Determination

The two-dimensional plucking direction cannot be used directly in the plucking operation of the tea bud. It is necessary to compute the three-dimensional plucking direction based on the two-dimensional direction obtained from the above study. A depth camera (such as Intel Realsense, Microsoft Kinect, and so on) can be used to obtain the depth information of every point on the tea bud, which can help us to determine the three-dimensional location of every tea bud. The three-dimensional plucking direction can be obtained when the two-dimensional direction line is matched with the spatial coordinates of the corresponding points.

As shown in Figure 14, the central point of the tea bud bounding box is not necessarily located on the tea bud region, and the spatial coordinates of the points on the tea bud regions vary considerably. But the plucking point is located on the plucking area definitely because it is the centroid of this region and the plucking areas are generally approximating rectangles. Meanwhile, the two-dimensional plucking direction line of the tea bud definitely passes through the plucking area. Therefore, the points of the intersection of the two-dimensional plucking direction line and the plucking area are first calculated, and then the fitted straight line of these points can be used as the three-dimensional plucking direction line.

Equation (8) represents the three-dimensional plucking direction line; its transformation equation can be seen as Equations (9) and (10).

$${\displaystyle \frac{x-x_0}{m}}={\displaystyle \frac{y-y_0}{n}}={\displaystyle \frac{z-z_0}{p}}$$

(8)

$$x={\displaystyle \frac{m}{p}}(z-z_0)+x_0=k_{1}z+b_1$$

(9)

$$y={\displaystyle \frac{n}{p}}(z-z_0)+y_0=k_{2}z+b_2$$

(10)

where the parameters k1, b1, k2, and b2 were transformed using Equations (11)–(14).

$$k_1={\displaystyle \frac{m}{p}}$$

(11)

$$b_1=x_0-{\displaystyle \frac{m}{p}}{z}_0$$

(12)

$$k_2={\displaystyle \frac{n}{p}}$$

(13)

$$b_2=y_0-{\displaystyle \frac{n}{p}}{z}_0$$

(14)

Hence, the spatial line can be regarded as the intersection of the two planes in Equations (9) and (10). The least-squares method was utilized to fit the spatial line, and when the residual analysis was first finished, the residual sum of the squares was calculated as in Equations (15) and (16).

$$R_1={\displaystyle \sum _{i=1}^n{(x_i-k_1{z}_i-b_1)}^2}$$

(15)

$$R_2={\displaystyle \sum _{i=1}^n{(y_i-k_2{z}_i-b_2)}^2}$$

(16)

where n represent the number of the points. To obtain the minimum residual, the partial differential of Equations (15) and (16) were calculated as in Equations (17)–(20).

$${\displaystyle \sum _{i=1}^{n}2(x_i-k_1{z}_i-b_1)\times (-z_i)=0}$$

(17)

$${\displaystyle \sum _{i=1}^{n}2(x_i-k_1{z}_i-b_1)\times (-1)=0}$$

(18)

$${\displaystyle \sum _{i=1}^{n}2(y_i-k_2{z}_i-b_2)\times (-z_i)=0}$$

(19)

$${\displaystyle \sum _{i=1}^{n}2(y_i-k_2{z}_i-b_2)\times (-1)=0}$$

(20)

From Equations (17)–(20), the fitted value of k1, b1, k2, and b2 can be calculated as in Equations (21)–(24).

$$k_1={\displaystyle \frac{n{\displaystyle \sum _{i=1}^n{x}_i{z}_i-{\displaystyle \sum _{i=1}^n{x}_i\times {\displaystyle \sum _{i=1}^n{z}_i}}}}{n{\displaystyle \sum _{i=1}^n{z}_i^2-{\displaystyle \sum _{i=1}^n{z}_i\times {\displaystyle \sum _{i=1}^n{z}_i}}}}}$$

(21)

$$b_1={\displaystyle \frac{{\displaystyle \sum _{i=1}^n{x}_i-k_1{\displaystyle \sum _{i=1}^n{z}_i}}}{n}}$$

(22)

$$k_2={\displaystyle \frac{n{\displaystyle \sum _{i=1}^n{y}_i{z}_i-{\displaystyle \sum _{i=1}^n{y}_i\times {\displaystyle \sum _{i=1}^n{z}_i}}}}{n{\displaystyle \sum _{i=1}^n{z}_i^2-{\displaystyle \sum _{i=1}^n{z}_i\times {\displaystyle \sum _{i=1}^n{z}_i}}}}}$$

(23)

$$b_2={\displaystyle \frac{{\displaystyle \sum _{i=1}^n{y}_i-k_1{\displaystyle \sum _{i=1}^n{z}_i}}}{n}}$$

(24)

From the above equations, the fitted line of the three-dimensional plucking direction can be acquired.

Once the three-dimensional plucking direction is obtained, the movement direction of the plucking manipulator can be calculated using a coordinate transformation. The manipulator can be directed to move to the plucking point along the plucking direction to pluck the tea bud.

2.5. General Summary of the Method

To summarize, this study focuses on determining the tea bud plucking point and plucking posture, which are the crucial aspects of mechanized tea plucking implementation. The workflow of the proposed method is shown in Figure 15. Firstly, a dataset of tea bud images was established by capturing photographs in the tea garden and conducting pre-processing procedures. In this stage, the particular focus lies in annotating both the tea bud and the plucking area together. Secondly, the improved YOLOv8-seg network was employed to accurately segment the tea bud and plucking area. However, it is important to note that while achieving the precise recognition and segmentation of the tea bud and plucking area is desirable, it is not the primary focus of this study. Instead, the main objective in this step was to extract feature points and plucking points through a series of image operations. Finally, the unmatched point sets obtained from the previous step were paired using NPM or PIRM, which were proposed in this study. The connected lines of these pairs represent the two-dimensional plucking directions of the tea buds. Subsequently, the fitted line of the intersection points of the two-dimensional plucking direction and plucking area was calculated as the three-dimensional plucking direction.

3. Results and Discussion

3.1. Evaluation Metrics

Two aspects of performance evaluation were included in this study: one is the evaluation of the detection and segmentation of the tea buds and plucking area, and the other is the evaluation of the feature point matching.

The evaluation of the detection and segmentation was mainly performed by evaluating the results of the YOLO network used in this study. Precision, recall, F1 score, and GFLOPs were included in the evaluation metrics. Precision, calculated with Formula (25), is the ratio of actual true samples to all the samples predicted as true, which represents the accuracy of prediction of the true samples. Recall is the ratio of samples predicted as true in the actual true samples, which can be calculated with Formula (26). F1 score, calculated with Formula (27), is a balance indicator of precision and recall.

$$precision={\displaystyle \frac{TP}{TP+FP}}$$

(25)

$$recall={\displaystyle \frac{TP}{TP+FN}}$$

(26)

$$F1 \mathrm{s}core={\displaystyle \frac{2*precision*recall}{precision+recall}}$$

(27)

where TP is the number of actual true samples which were predicted as true samples, FP is the number of false samples which were predicted as true samples, and FN is the number of actual true samples which were predicted as false samples.

GFLOPs represent the total number of floating-point operations executed using a model during inference or training, serving as a metric for the complexity of a model. A higher GFLOP value generally indicates that the model possesses greater computational complexity and requires more additional computational resources for processing.

The accuracy and influence of feature point matching on plucking efficiency was the main factor considered in the evaluation of the feature point matching. Therefore, matching accuracy and matching time were proposed as the evaluation metrics, which were all obtained using the statistical analysis of the matching test data.

3.2. Model Training of LDS-YOLOv8

The LDS-YOLOv8 deep learning network was built based on the pytorch framework, and Win 10 OS with 128 GB RAM, an Intel(R) Xeon(R) E5-2680 v4 @ 2.40 GHz CPU, and an NVIDIA RTX A6000 48 G GPU was used as the training platform in this study. The software used in the training and testing processes included python3.8, CUDA11.4, cuDNN8.2.2, etc. In view of the balance between computer performance and training efficiency, the batch size was set as 24. The initial learning rate and final learning rate were both set to 0.01.

3.3. Evaluation of Tea Bud Recognition and Plucking Area Detection

3.3.1. Detection and Segmentation Results Using the Improved YOLOv8 Model

The YOLOv8 network architecture provides five different sizes of network models. As the model size increases, there is a corresponding improvement in detection and segmentation accuracy. However, this enhancement comes at the cost of increased computation time. For tea plucking, the calculation time of models of different sizes can all meet their actual needs; therefore, the improvements and tests in this study were all conducted based on the YOLOv8x-seg model for the best accuracy. The overall situations of the training process of the original and improved YOLOv8-seg model were analyzed, and Figure 16 shows the trend in the change in the loss value during training, including box loss, classification loss, distribution focal loss, and segmentation loss. It can be seen that both the original and improved models exhibited a rapid decrease in loss value with the increase in the epoch, and the loss value became stable eventually, which indicated that the training progressed well and that the results were convergent. Additionally, the improved model achieved convergence earlier than the original model, which demonstrated the improved efficiency of its training process.

Table 1. Performance of the detection and segmentation of the tea buds.

Network Model	Precision (B)	Recall (B)	mAP (B)	Precision (M)	Recall (M)	mAP (M)
YOLOv8x-seg	0.799	0.875	0.922	0.806	0.870	0.922
LDS-YOLOv8x-seg	0.835	0.859	0.925	0.842	0.852	0.924

,

Table 2. Performance of the detection and segmentation of the plucking area.

Network Model	Precision (B)	Recall (B)	mAP (B)	Precision (M)	Recall (M)	mAP (M)
YOLOv8x-seg	0.794	0.618	0.707	0.645	0.492	0.499
LDS-YOLOv8x-seg	0.8	0.664	0.745	0.635	0.508	0.514

,

Table 3. Performance of the overall detection and segmentation situation.

Network Model	Precision (B)	Recall (B)	mAP (B)	Precision (M)	Recall (M)	mAP (M)
YOLOv8x-seg	0.797	0.747	0.815	0.726	0.681	0.710
LDS-YOLOv8x-seg	0.818	0.761	0.835	0.739	0.68	0.719

present the results of object detection and instance segmentation acquired using the YOLOv8x-seg and LDS-YOLOv8x-seg models for tea buds, plucking area, and overall situation, respectively. As shown in Table 1, the precision and mAP of the LDS-YOLOv8x-seg model outperformed those of the YOLOv8x-seg model in terms of tea bud detection. As observed in Table 2, the precision, recall and mAP of the LDS-YOLOv8x-seg model all outperformed those of the YOLOv8x-seg model in terms of plucking area detection. From a comprehensive perspective, as shown in Table 3, the precision, recall, and mAP of the object detection achieved using the LDS-YOLOv8x-seg model surpassed those of the YOLOv8x-seg model. In respect to the instance segmentation of the tea buds, as shown in Table 1, both the precision and mAP of the LDS-YOLOv8x-seg model outperformed those of the YOLOv8x-seg model, with a 3.6% increase in precision. Regarding the instance segmentation of the plucking area, as observed in Table 2, the precision of the LDS-YOLOv8x-seg model was marginally lower than that of the original YOLOv8x-seg model; however, both the recall and mAP witnessed significant enhancements. Comprehensively, as depicted in Table 3, in terms of instance segmentation, the LDS-YOLOv8x-seg model manifested notable improvements in both precision and mAP while maintaining a substantially unchanged recall compared to the original YOLOv8x-seg. The results presented above demonstrate that the improved model of this paper exhibited superior performance in detection and segmentation on the tea buds and plucking area compared to the original model.

It can also be seen from Table 1 and Table 2 that both the original and improved models performed better on the detection and segmentation of the tea bud than those of the plucking area, with the precision, recall, and mAP of the tea bud detection and segmentation outperformed those of the plucking area detection and segmentation. Analyzing the reason for this, it may be that the shape characteristics of the tea bud were well defined, while the plucking area is smaller than the tea bud and the color and shape are less distinct from its surrounding area. Moreover, it is undeniable that the tea bud and plucking area of every tea have been labeled together during the image labeling, which is beneficial for the detection of the plucking area.

In addition, the precision, recall, and mAP results of tea bud detection and segmentation exhibited smaller variance; however, for the plucking area detection and segmentation, the results were notably different. Analyzing the reason for this, the plucking area is small, which makes it difficult to detect and segment. Meanwhile, even a small error can lead to a large scaling deviation because of its small dimension.

3.3.2. Performance Comparison Between Different Networks

To further illustrate the effectiveness of this study, the improved algorithm in this paper was compared to other algorithms. Compared to the existing research on tea bud target recognition, there is a scarcity of studies focusing on the instance segmentation of both the tea buds and plucking area. This aspect holds significant importance, as it serves as a crucial prerequisite for determining the optimal plucking posture and identifying the precise plucking points.

Comparative analysis of the algorithm in the relevant study and the typical instance segmentation Mask R-CNN were conducted. Chen et al. [31] achieved the detection and segmentation of tea buds using a combination of the Faster-RCNN and FCN-16s models. They employed the two-stage target detection network Faster-RCNN for tea bud recognition, followed by the application of the trained FCN-16s network to accomplish precise tea bud segmentation. The Mask R-CNN network with different backbones were also calculated in this comparison. As presented in

Table 4. Results of different networks.

Method	Precision of Tea Bud Detection	Average Precision of Overall Performance	Average Recall of Overall Performance	F1 Score of Overall Performance
Chen et al. [31]	0.727	0.668	0.777	/
Mask R-CNN with Resnet 50	0.734	0.666	0.724	0.694
Mask R-CNN with Resnet 101	0.693	0.617	0.713	0.662
Mask R-CNN with Resnet 152	0.724	0.662	0.720	0.689
This paper	0.835	0.739	0.680	0.708

, comparing these methods to the method introduced in this paper, it is observed that the precision of tea bud detection achieved using our method notably surpasses theirs.

In terms of the overall recognition and segmentation effect, the average precision of the method proposed in this paper was also notably higher than others. The average recall of this study did not surpass that of others; however, in the actual mechanized tea bud plucking operation, precision holds greater significance than recall due to the fact that the mechanized plucking operations tend to be repeated in round trips.

The method in this paper simultaneously segmented the tea bud and plucking area, while their method only segmented the plucking area in terms of instance segmentation [31]. In addition, the one-stage YOLO algorithm exhibits a notable efficiency advantage when compared to the two-stage instance segmentation algorithm Mask RCNN.

3.3.3. Ablation Experiments

To investigate the effect of each enhancement strategy, including improved head, SPPF-LSKA, and C2f-DCNv2, on improving the model’s object detection and instance segmentation performance, ablation experiments were performed using the validation dataset in this study. The results are presented in

Table 5. Results of the ablation experiments.

Improved Head	SPPF-LSKA	C2f-DCNv2	Precision (B)	Recall (B)	mAP (B)	Precision (M)	Recall (M)	mAP (M)	GFLOPs
-	-	-	0.797	0.747	0.815	0.726	0.681	0.710	344.5
√	-	-	0.81	0.759	0.831	0.735	0.674	0.714	301.2
-	√	-	0.798	0.747	0.817	0.73	0.681	0.715	345.8
-	-	√	0.793	0.759	0.823	0.731	0.688	0.718	342.5
√	√	√	0.818	0.761	0.835	0.739	0.68	0.719	300.5

. It is apparent from the table that, in comparison to the original YOLOv8-seg model, the improved model demonstrates a significantly enhanced performance effect. Analysis of the three individual enhancements indicates that the network incorporating the improved head module exhibited superior performance in both object detection and instance segmentation tasks. Specifically, in object detection, precision improved by 1.3%, recall increased by 1.2%, and mAP improved by 1.6% relative to the original network model. In terms of instance segmentation, both precision and mAP showed significant improvements, while the model’s computational cost was markedly reduced, achieving a 12.54% decrease compared to the original YOLOv8-seg. This further demonstrated the substantial impact of the improved head module on model lightweighting. For the improved model with the SPPF-LSKA module, it can be seen that the module has a significant improvement in detection and instance segmentation performance; all the indicators in the table were increased compared to the original model. The network enhanced with C2f-DCNv2 also exhibited superior performance in both instance segmentation and object detection, with the recall and mAP increasing by 1.2% and 0.8%, respectively, in detection, and all of the indicators improved during instance segmentation.

Finally, the enhanced model incorporating all three improvement modules exhibited a significantly superior enhancement effect compared to the original model and the improved model with the respective module. Among the seven evaluation indicators shown in the table, the improved model incorporating all three improvement modules outperformed the others in six of them. The overall detection and segmentation performance of the improved model for the tea buds and plucking area was effectively enhanced, with the precision of detection and segmentation improving by 2.1% and 1.3%, respectively. Furthermore, the computational cost of the entire model decreased by 12.77% compared to the original model.

The aforementioned findings demonstrate the efficacy of the YOLOv8-seg model’s enhancements, further proving that the LDS-YOLOv8-seg model is better suited for the detection and segmentation of the tea buds and plucking area.

Figure 17 shows the detection and segmentation results of the tea buds and plucking areas achieved using the original and improved YOLOv8-seg model. The figure demonstrates that the improved method exhibited superior recognition and segmentation capabilities.

3.4. Evaluation of the Matching and Plucking Posture Determination

Matching accuracy and matching time are the key factors that affect the success and efficiency of tea bud plucking. The matching of the central points of the tea buds’ bounding boxes and the centroids of plucking areas using NPM and PIRM were carried out, respectively, on 200 images of the test dataset using the LDS-YOLOv8-seg model in this study. The matching accuracy was calculated by counting the number of identified feature points and the number of points matched correctly. The results of matching accuracy and matching time are listed in

Table 6. Accuracy and time of the feature points matching.

Matching Algorithm	Matching Accuracy	Matching Time
		Average Time (Millisecond)	Standard Deviation (Millisecond)
NPM	90.355%	0.175	0.380
PIRM	99.229%	2.363	0.995

. Using statistical analysis, a total of 788 pairs of feature points were matched using the NPM, of which 712 pairs were correctly matched with an accuracy of 90.355%, while a total of 779 pairs of feature points were matched using the PIRM, of which 773 pairs were correctly matched with an accuracy of 99.229%. The NPM achieved more feature points matching pairs; however, there were more false matches. In terms of matching time, the average consumption time and standard deviation for each image with the NPM were 0.175 ms and 0.380 ms, respectively, while the results of the PIRM were 2.363 ms and 0.995 ms.

As can be seen from the above results, the average matching time using the NPM is shorter than when using the PIRM, but its matching accuracy is lower. In fact, regarding the plucking movement time of the tea bud plucking apparatus, the average matching time of both methods all fully meet the requirements for tea plucking. From a comprehensive point of view, the PIRM has a better effect on the matching of the tea buds and plucking area.

Figure 18 shows the matching results of the tea buds and plucking area.

3.5. Analysis of Failure Cases

Failure cases were encountered in the instance segmentation and feature point matching. The difficulty of detecting small targets is the main reason for the failure of the tea bud and plucking area identification. In addition, the recognition accuracy of the network model is also affected by the large difference between the size of the tea buds and the plucking area. According to the statistics of the pixel areas of the tea buds and plucking area in the images of the training dataset used in this study, as shown in

Table 7. Comparison of the pixel area of the tea bud and the plucking area.

Region	Maximum Pixel Area	Minimum Pixel Area	Average Pixel Area
Tea bud	171,662.354	974.700	16,687.086
Plucking area	5065.741	35.070	475.799

, there is a large gap between the two kinds of pixel areas.

Analysis of the causes of failure in the feature point matching process was the main work in this study. In the failure cases of the PIRM, we learned that the main reason was that the centroid of the plucking area and the central point of the tea bud bounding box were located together or separately in the bounding box of another tea bud. As shown in Figure 19, the plucking point of the NO.1 tea bud was located in the bounding box of the NO.2 tea bud and was closer to the central point of the bounding box of the NO.2 tea bud. In the failure cases of the NPM, the main reason was that the plucking point of another tea was indeed closer to the central point of the bounding box of this tea than the bounding box of that tea. It can be seen in Figure 20a that the plucking point of the NO.1 tea bud was closer to the central point of the bounding box of the NO.2 tea bud than to the NO.1 tea bud.

All in all, the reason for the mistake in feature points matching was that the tea buds to be plucked were growing too close together. In addition, after a pair of points was successfully matched in the algorithm, they were removed from its point set, so once a pair of points was incorrectly matched, there was a second matching error, as shown in Figure 20a, or even if three or more pairs of points were matched incorrectly at the same time, as shown in Figure 20b. In addition, the matching sequence also affects the matching result. As shown in Figure 20a, if the matching started from the central point of the No.1 tea bud bounding box, there would be no error; otherwise, if the matching started from the central point of the No.2 tea bud’s bounding box, both matches were wrong. In particular, as shown in Figure 20b, all three matches were incorrect because the matching started from the central point of the NO.3 tea bud’s bounding box. However, due to the random position of the tea buds in one image, a certain matching sequence could not be set in advance.

4. Conclusions

(1) An improved YOLOv8-seg model was proposed for the detection and instance segmentation of tea buds and their plucking area, aiming at the automatic plucking of the tea buds. Incorporating the improved head, SPPF-LSKA, and C2f-DCNv2 modules, the LDS-YOLOv8-seg model demonstrated superior performance in detecting and segmenting tea buds and the plucking area with detection precision, recall, and mAP values of 0.818, 0.761, and 0.835, as well as segmentation precision, recall, and mAP values of 0.739, 0.68, and 0.719. Furthermore, the computational cost decreased by 12.77% compared to the original model.

(2) It is a critical prerequisite to locate the plucking point of tea for automatic plucking. The centroid of the plucking area was computed by mapping the probability of each pixel in the mask image to the coordinates of its corresponding pixel in the original image. The proposed method is more accurate and versatile compared to traditional plucking point localization methods.

(3) In order to carry out automatic tea plucking more accurately, a plucking posture determination method was developed in this study. The plucking point and the central point of the tea bud bounding box were identified first, and then the matching of these points was achieved using either the PIRM or the NPM proposed in this study. Two-dimensional plucking posture can be determined using the matched points. Combined with three-dimensional information, three-dimensional plucking posture can be obtained. The matching effects of the PIRM and the NPM were also compared, and it is shown that the PIRM performed better, with 99.229% matching accuracy and an average consumption time of 2.363 milliseconds.

(4) The failure cases in the feature points matching process were analyzed in this study. The main reason for the mismatch was that the tea buds that needed to be plucked were growing too close together. In addition, the matching order of different pairs of points on the same image also affected the matching accuracy.

As discussed previously, for tea leaves that grow close together, the method proposed in this paper is susceptible to incorrect matching, which consequently hinders the accurate determination of the plucking posture of tea bud. Therefore, our future research will investigate a more refined algorithm that integrates color and shape features for matching tea bud feature points, aiming to address the issue of failures in feature point matching caused by the sequence of matches. Another important research task is related to exploring the integration of the tea plucking posture determination method proposed in this paper into a tea bud plucking apparatus for field experiments.