Apple Shape Detection Based on Geometric and Radiometric Features Using a LiDAR Laser Scanner

Abstract

Yield monitoring systems in fruit production mostly rely on color features, making the discrimination of fruits challenging due to varying light conditions. The implementation of geometric and radiometric features in three-dimensional space (3D) analysis can alleviate such difficulties improving the fruit detection. In this study, a light detection and range (LiDAR) system was used to scan apple trees before (T_L) and after defoliation (T_D) four times during seasonal tree growth. An apple detection method based on calibrated apparent backscattered reflectance intensity (R_ToF) and geometric features, capturing linearity (L) and curvature (C) derived from the LiDAR 3D point cloud, is proposed. The iterative discretion of apple class from leaves and woody parts was obtained at R_ToF > 76.1%, L < 15.5%, and C > 73.2%. The position of fruit centers in T_L and in T_D was compared, showing a root mean square error (RMSE) of 5.7%. The diameter of apples estimated from the foliated trees was related to the reference values based on the perimeter of the fruits, revealing an adjusted coefficient of determination (R²_adj) of 0.95 and RMSE of 9.5% at DAFB₁₂₀. When comparing the results obtained on foliated and defoliated tree’s data, the estimated number of fruit’s on foliated trees at DAFB₄₂, DAFB₇₀, DAFB₁₀₄, and DAFB₁₂₀ 88.6%, 85.4%, 88.5%, and 94.8% of the ground truth values, respectively. The algorithm resulted in maximum values of 88.2% precision, 91.0% recall, and 89.5 F1 score at DAFB₁₂₀. The results point to the high capacity of LiDAR variables [R_ToF, C, L] to localize fruit and estimate its size by means of remote sensing.

1. Introduction

In the quest of decreasing farming costs and increasing sustainability, leaf area and yield monitoring are considered as one of the most important steps when implementing precision agriculture technologies in orchards [1,2]. Monitoring of leaf area index and yield has been widely studied and applied in cereal crops such as rice [3], sorghum [4], wheat [5], maize [6]. Similarly, in orchards the monitoring of tree morphology and growth is requested, since the variation in the leaf area and resulting light interception influences the fruit-bearing capacity of the tree. Furthermore, the shading of fruit affects the fruit color, which is an important parameter of fruit quality [7]. However, the coinciding structures and the varying reflectance properties of the tree organs work as suspending factor for yield mapping systems, particularly making the automated fruit detection challenging.

The localization of fruit and estimation of fruit size have gradually gained an essential driving force by the emerging implementation of robotics in agriculture [8]. Various machine vision systems have been developed based on color, spectral, and thermal imaging. Color, extracted from two-dimensional RGB images, was utilized as an indicative factor of fruit detection in apple [9,10], citrus [11,12], mango [13], and grapes [14]. However, the captured images are sensitive to light variation, while the detection can be biased in cases of equal color of fruit, leaves and branches [15,16]. Thus, the RGB images were frequently fused with thermal images to overcome this issue, e.g., in green apples [17] or green citrus [18]. Due to increased spectral resolution compared to RGB imaging, spectral imaging can potentially be utilized for fruit detection based on the reflectance intensity altered by the fruit pigments [19] by means of either multispectral [20,21] or hyperspectral cameras [22,23]. However, varying lighting conditions remain a perturbing factor.

The recent advancements in three-dimensional (3D) remote sensing technologies received attention in agriculture, enabling potential prospects to overcome the limitations of 2D imaging methods [24]. The 3D point cloud data can restore the shape of fruit trees, providing highly resolved spatial data for fruit localization. Besides the triangulation techniques, which allow the acquisition of 3D information from stereo vision or structured light cameras, 3D point cloud data and radiometric information can be generated by RGB depth (RGB-D) cameras, which operate based on stereo vision or time-of-flight (ToF) measuring principle. The latter RGB-D sensor captures three different modalities: color, depth, and the backscattered reflectance intensity (R_D) at the wavelength of an integrated ToF depth sensor. By means of these data providing geometric (depth) and radiometric (RGB and R_D) information, the discrimination potential of fruit from foliage and woody parts of the plant are assumed to be enhanced. Gene-Mola et al. [25] used range corrected data capturing color, depth, and R_D in the near-infrared wavelength range from an RGB-D camera to detect the apple shape at the tree, resulting in 0.89 F1-score and 94.8% average precision. The authors pointed out that the depth and R_D provided the most robust variables in the calibration model, while RGB was highly influenced by light shading in the canopy. In mango, RGB-D imaging enabled to detect mangoes with 81% precision, and to estimate the length and width of the fruit, achieving a root mean square error (RMSE) of 4.9 mm and 4.3 mm, respectively [26]. Both studies were undertaken after sunset to achieve consistent lighting conditions. It was noticed that the sensor failed to acquire distance information over 3.5 m. Another technique for acquiring 3D information are the light detection and ranging (LiDAR) laser scanners, which are operated based on the ToF principle with a scanning mechanic for obtaining advanced high-density spatial resolution compared to RGB-D sensing. The data acquired are the laser hits based on ToF principle providing geometric information and optionally the intensity of reflected, apparently backscattered, laser hits gaining radiometric information.

Currently, LiDAR intensity data were applied for feature extraction, classification, and object detection and recognition on natural and urban surfaces [27]. In fruit production, working in a 3D space allows the identification of geometric tree features such as the shape and size aiming the fruit localization [28]. The spatial distribution of the geometric features can be illustrated by the decomposition of the covariance matrix from a set of points in 3D. Brodu and Lague [29] proposed that the set of points within the spherical neighborhood can be utilized to define whether the geometric structure of points is closer to a line, plane surface, or a random cluster. Several researchers in forestry applications used the method to include reflectance intensity (R_ToF) and geometric features such as the linearity to discriminate foliage from woody parts [30,31] or to estimate the leaf angle [32]. Whereas in agriculture, Gene-Mola et al. [33] proposed a methodology that combines the backscattered reflectance intensity and a geometric factor for spherical shapes based on the eigenvalues of clusters in apple trees to localize the fruits. The method gained 87.5% localization success, 82.4% identification success, and a F1-score of 0.86 in relation to the total number of fruits. Moreover, eigenvalues of geometric data on curvature and linearity have been computed to distinguish cotton balls from branches, reaching 90% detection accuracy [34]. Similarly, intensity and geometric features have been exploited to train harvesting robots in greenhouse applications [35,36].

The aim of this study is to propose a methodology for segmenting, localizing, and analyzing fruit on the tree, based on the assumption that apples show enhanced R_ToF at 905 nm compared to foliage and woody parts. The main objectives are: (1) the classification of 3D points of apples based on reflectance and geometric features derived from LiDAR laser scanner; (2) the development of a fruit segmentation algorithm that allows measuring the fruit size in defoliated trees; and (3) the evaluation of the proposed technique on foliated apple trees at different growth stages.

2. Materials and Methods

2.1. Site Description

The study was carried out in the experimental station Field Lab for Digital Agriculture by Leibniz Institute of Agricultural Engineering and Bioeconomy (ATB) located in Marquardt, Germany (Latitude: 52.466° N, Longitude: 12.958° E) during the growing season 2019. The trees of Malus × domestica Borkh. ‘Gala’ on M9 rootstock were planted in a 0.37 ha orchard, 0.95 m within-row distance between trees, which have been trained as tall slender spindle (central leader) formation with 3.3 m maximum tree height. The trees were supported by utilizing four horizontally parallel wires (W₁, W₂, W_3, W₄) with 1 m equal distance from each other. Plants were drip irrigated, fertilized, and treated according to management in commercial orchards and the regional plant protection plan. Hand thinning resulted in 1–3 fruits per inflorescence.

2.2. Data Acquisition and Pre-Processing

A metal frame installed on a tractor was used to carry the sensors along the tree rows [37]. A mobile terrestrial LiDAR laser scanner (LMS-511, Sick AG, Waldkirch, Germany) was mounted vertically on the metal frame at 1.6 m above the ground level.

The LiDAR sensor was configured with a 0.1667° angular resolution, 25 Hz scanning frequency and a scanning angle of 190°. The sensor frame was driven along the rows on both sides of the trees with an average speed of 0.13 m s⁻¹. The measured objects, hit by the laser beam, were assumed as perfectly diffuse reflectors (Lambertian), excluding the influence of incidence angle [38,39]. Thus, the apparent signal was considered as an approximation of hemispherical reflectance. Board targets, coated with white barium sulphate (BaSO₄, CAS Number: 7727-43-7, Merck, Germany) for maximum (R_max) and blackened urethane (S black, Avian Technologies, New London, NH, USA) for minimum (R_min) referencing were applied to calibrate the backscattered intensity (R_ToF) of the LiDAR, obtaining the R_ToF [%] at 905 nm for each point in the 3D point cloud (Equation (1)).

$${\mathrm{R}}_{\mathrm{ToF}}=\frac{({\mathrm{R}}_{\mathrm{measured}}-{\mathrm{R}}_{\min })}{({\mathrm{R}}_{\max }-{\mathrm{R}}_{\min })}$$

(1)

The tractor movement was recorded by an inertial measurement unit (IMU) (MTi-G-710, XSENS, Enschede, the Netherlands), allowing the correction of errors due to the uneven ground surface. The root mean square error (RMSE) of orientation noted at 0.25° for roll, pitch and yaw [40]. Furthermore, a RTK-GNSS (AgGPS 542, Trimble, Sunnyvale, CA, USA) was used for georeferencing each individual point of the 3D point cloud. The horizontal and vertical accuracy of the RTK-GNSS was ± 25 mm + 2 ppm and ± 37 mm + 2 ppm, respectively. The IMU was place 0.24 cm aside from the LiDAR sensor, while the receiver antenna of RTK-GNSS was mounted 0.30 cm above the laser scanner (Figure 1). A multi-thread software was developed in Visual Studio (version 16.1, Microsoft, Redmond, WA, USA) to acquire the data [37].

The 3D point cloud data was processed in the Computer Vision Toolbox™ of Matlab (2017b, Mathworks, Natick, MA, USA). A sparse outlier removal was applied to each point cloud pair to reject the points that were above the average distance using CloudCompare (2.10, GPL software, Paris, France). Moreover, the random consensus was performed to filter the points that belonged to the ground [41]. According to Tsoulias et al. [37], the rigid translations and rotations were applied in each point of the point cloud, while the alignment of the pair tree sides was carried out with the iterative closest point algorithm, using a k-dimensional space to speed up the process.

Data were recorded during fruit growth starting when the end of the cell division stage of fruit was reached and the cultivar-typical red blush color [42] appeared and on subsequent three dates, i.e., 42, 72, 104, and 120 days after full bloom (DAFB₄₂, DAFB₇₂, DAFB₁₀₄, DAFB₁₂₀, respectively). On each measuring date two apple trees (n = 8) were initially scanned with leaves (T_L), while the measurement was repeated with the plants defoliated (T_D). After scanning, the total number of apples was counted on the tree. After harvest, fruit’s diameter (D_Manual) was manually measured in the lab. All apples of each tree were categorized according to their growing position on the slender spindle. The wires (W) were used as borders to discriminate the apples between the ground and W₁ (ΔW_1-G); W₁ and W₂ (ΔW_2-1); W₂ and W₃ (ΔW_3-2); W₃ and W₄ (ΔW_4-3).

2.3. Apple Detection Methodology

2.3.1. Extraction of Radiometric and Geometric Features

The information of the 3D local neighbourhood of points ${\mathrm{P}}_{\mathrm{i}}$ = [${\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{i}}$] was determined using the k-nearest neighbours within a radius (r) equal to the mean D_Manual of apples [43]. The total number of the ${\mathrm{P}}_{\mathrm{i}}$ data set (K) was used to estimate the average $\tilde{\mathrm{P}}$ = $\frac{1}{\mathrm{K}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{K}}({\mathrm{P}}_{\mathrm{i}})$ of the nearest neighbours. Thus, the covariance matrix (Cov) was built after mean centering by means of subtraction of $\tilde{\mathrm{P}}$ value from each ${\mathrm{P}}_{\mathrm{i}}$ of the nearest neighbours set (Equation (2)):

$$\mathrm{Cov}(\mathbf{P})=\frac{1}{\mathrm{K}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{K}}({\mathrm{P}}_{\mathrm{i}}-\tilde{\mathrm{P}})\times {({\mathrm{P}}_{\mathrm{i}}-\tilde{\mathrm{P}})}^{\prime }$$

(2)

The Cov was decomposed based on the singular value decomposition, producing the eigenvalues (${\mathsf{\lambda }}_1$, ${\mathsf{\lambda }}_2$, ${\mathsf{\lambda }}_3$), which were sorted in ascending order ${\mathsf{\lambda }}_1$ ≥ ${\mathsf{\lambda }}_2$ ≥ ${\mathsf{\lambda }}_3$ according to the variance captured, and the corresponding eigenvectors. The eigenvalues, which represent the highest orthogonal variance of the matrix, provided the points dispersion within k-nearest neighbours describing the local spatial structure in 3D. The eigenvalues were scaled between 0 and 100, allowing the comparison of different clusters. Thus, the local geometry of each P_i in the point cloud was analyzed by means of eigenvalues to illustrate the spatial structure considering its linearity (L, Equation (3)) and curvature (C, Equation (4)):

$$\mathrm{L}\left({\mathrm{P}}_{\mathrm{i}}\right)=\frac{{\mathsf{\lambda }}_1-{\mathsf{\lambda }}_2}{{\mathsf{\lambda }}_1}$$

(3)

$$\mathrm{C}\left({\mathrm{P}}_{\mathrm{i}}\right)=\frac{{\mathsf{\lambda }}_3}{{\mathsf{\lambda }}_1+{\mathsf{\lambda }}_2+{\mathsf{\lambda }}_3}$$

(4)

where L and C describe the variation of linearity and curvatures for all points along the direction of the corresponding eigenvalues, respectively [43,44]. More specifically, the closer the values of L or C to 100, the higher the likelihood for the shape of points to be linear or curved, respectively. Therefore, L was used to segment the foliage and woody parts of the tree (leaves, branches, stem), while C was applied to distinguish the apple points.

2.3.2. Apple Segmentation

The data set of T_D at DAFB₁₂₀ (n = 2) was used to define the range of R_ToF, C and L for the class of apple (R_A, C_A, L_A) and of woody parts (R_W, C_W, L_W), while the range of leaf class for the same features (R_L, C_L, L_L) was defined in the T_L. The thresholding was performed for each class by employing the exploratory analysis of normal distribution using the probability density function to define the threshold that will distinguish the 3D points of apples from leaves and woody parts [33,45]. The value with the highest likelihood (mode) within the R_A, C_A, and L_A classes was used as a threshold (R_th, C_th, and L_th). Points which fulfilled the criteria of L_A ≤ L_th, C_th ≤ C_A, and R_th ≤ R_A were segmented and categorized as apples. Figure 1 represents a flowchart of the protocol for apple detection and sizing using the defoliated trees (T_D).

Subsequently, a density-based scan algorithm (DBSCAN) [46] was applied to find the point sets, using the mean D_Manual of apples that was found in each ΔW as a neighborhood search radius (ε) and the value 10 as a minimum number of neighbors. The value of 10 was applied resulting from manually run tests showing that less neighboring points result in random appearance of sets. The mean value of apparent reflectance intensity and of curvature in defoliated trees ($\tilde{R}$ and $\tilde{C}$) were calculated to classify the clusters. The maximum distance in x and y axes was considered as the diameter (D_D) of each point set recognized as an apple.

Following, the k-means clustering was applied to count and find the fruit center (M_D) in each cluster in T_D. However, the shape variation of each apple and the clustering of more than one apple produced different C and R_ToF. Consequently, the utilization of C_th and R_th values possibly exclude points at the edge of the fruit. Therefore, a sphere was placed in each individual apple cluster, with the coordinates of M_D as the center and a radius equal with the estimated D_D to include the filtered apple points. The k-means algorithm was reapplied to estimate the M_D of each cluster based on the added 3D points, while the $\tilde{R}$ and $\tilde{C}$ were recalculated.

The fact that more than one apple developed per inflorescence led to close appearance of 2–3 fruits in the cloud, resulting in the creation of augmented clusters with more than one apple. Thus, the clusters of 3D points within the sphere were evaluated and partitioned based on their extracted features, creating sub-clusters and defining the number of M_D. Therefore, the partitioning was performed for each cluster only on the condition that the three variables, D_D, $\tilde{R}$, and $\tilde{C}$, were close to mean values of D_Manual measured at DAFB₁₂₀ for each ΔW and R_th, C_th. The features of each sub-cluster were iteratively extracted and re-evaluated, while the partitioning was continued until the sub-cluster features stopped, while fulfilling the condition. Subsequently, the spheres were placed based on the M_D and the D_D of each sub-cluster. Those spheres, showing a calculated distance between the M_D centers below the minimum value of D_Manual in each ΔW, were merged. Therefore, their M_D was redefined and their extracted features were recalculated. It should be mentioned that the M_D clusters were considered as ground truth labels for the later evaluation of the method in T_L.

Similarly, the L_th, C_th, R_th were applied to the same trees before their defoliation to segment the apples and find their centers (M_L) (Figure 2). The DBSCAN was used to group the segmented points, while the M_L centers were defined by the k-means algorithm. Furthermore, the maximum distance in x and y axes was considered as the diameter (D_L) of each point set. The main partitioning condition remained the same using the generated features from each cluster. Thus, the features of each T_L sub-cluster were iteratively extracted and re-evaluated until fulfilling the conditions. The number of partitions was used in the k-means algorithm to determine the centers of M_L sub-clusters. The only difference between analysing T_D and T_L was the use of the sphere in T_D to evaluate the eligibility of the cluster or the need to search for subclusters according to the estimated manually measured fruit diameter.

However, in T_L, some of the apples were fully or partially covered by the leaves of the tree. In the latter case, the combination of DBSCAN and k-means algorithm overestimated the number of apple classes. Thus, apple classes producing lower or equal Euclidian distance between their M_L centers compared to the minimum D_Manual were combined and considered as one fruit.

2.4. Evaluation

The calibration of the method was carried out considering data from T_D and T_L at DAFB₁₂₀ (n = 2) to define the R_th, C_th and L_th that describe the apple class. Cross-validation, applying thresholds extracted at DAFB₁₂₀, was performed to the T_D and T_L data sets of the three earlier measuring dates (n = 6). The fruit detection methodology was evaluated by calculating the accuracy, the precision, the recall, and the F1 score:

$$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{N}} [\%]$$

(5)

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}} [\%]$$

(6)

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} [\%]$$

(7)

$$\mathrm{F}1=\frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}} [\%]$$

(8)

The M_D were considered as ground truth labels for the evaluation of M_L. The M_L clusters which coincide with the M_D points >50% were considered as true positive (TP). The true negative (TN) refers to the situation when a cluster was correctly categorized as no fruit. Whereas the M_L clusters which matched <50% were counted as false negative (FN). The clusters which reveal a diameter smaller than the minimum value of D_Manual that was found in each ΔW were categorized as false positives (FP). N denotes the sum of TP, TN, FP and FN. The values were given as percentage.

A regression analysis was performed between the D_Manual and the D_D, and RMSE, mean absolute error (MAE), mean bias error (MBE) were calculated for each measuring date. The position of each M_L was evaluated with the M_D using the Euclidean distance, pairing the centers that revealed the minimum distance between them. In parallel, the difference between the fruit diameter in T_D and T_L was estimated, considering the D_D as ground truth for D_L.

3. Results

3.1. Apple Segmentation

The definition of apple class was carried out on the data set of last day’s measurements at DAFB₁₂₀. The calibrated reflectance intensity R_ToF varied in the 3D point cloud of the foliated trees (Figure 3b). The R_ToF values of apples (R_A) and woody parts (R_W) appeared above 70% and below 55%, respectively. The leaves (R_L) ranged between 40 and 80%. The C values of leaves (C_L) ranged between 35.2% and 70% (Figure 3c). The C values of apples (C_A) were observed above 50%, whereas the points of woody parts (C_W) did not exceed 60%. The L of leaves (L_L) ranged between 20% and 80%, while the woody parts (L_W) were scored above 60% (Figure 3d).

However, the occlusions of leaves and woody parts did not allow the acquisition of the full range of R_ToF, C, and L values of apples. Therefore, the trees were defoliated and scanned with the LiDAR system at DAFB₁₂₀ (Figure 4) and data were used to readjust the thresholds. The R_W values ranged between 0 and 60%, while the R_A appeared from 42 to 90% (Figure 4b). The C_A values of apples ranged between 56 and 98%, and the C_W appeared between 0.1 and 55.2% (Figure 4c). The 3D points of the latter category revealed the highest values varying between 48.8 and 98.8%. The L values of apple points (L_A) were more discretely ranging between 0.2 and 38.3%. (Figure 4d). However, the highest mean value was obtained from the R_A class at 71.8%, whereas the mode value (76.1%) was utilized as R_th for the ‘Gala’ trees.

As mentioned above, the R_ToF values of apples could overlap with the leaves or the woody parts, while some of the surfaces may reveal similar C with apples, producing falsely clusters from leaves or woody parts. The R_W showed a mean value of 38.2% with standard deviation (SD) of 18.5%. In contrast, the R_ToF of leaves (R_L) and of apples (R_A) were marginally deviated (Figure 5a), while most of their values appeared from 50 to 70% and from 65 to 71.8%, respectively. More specifically, the mean value of R_L was indicated at 58.8% with 12.1% SD, and 63.2% as the value with the highest likelihood (mode) within the class.

A similar pattern of the probability density of R_ToF was obtained in C (Figure 5b). The most frequent value within the C_A was illustrated at 73.2% (C_th). The values of C of leaves (C_L) were partly coinciding with the C_A, revealing a mean value at 60.7%, and 9.1% SD. The C_W revealed a mean value of 30.1%, with 19.4% SD. The 3D points of L_W depicted the highest values (Figure 5c), reaching a mean value of 60.2%, and 94.1% as mode value (Figure 6).

The R_A, C_A and L_A values were distinct from the residual classes, allowing the discrimination and localization of apples in the 3D point cloud. The apples were described with R_A and C_A values 76.1% and 73.2% (Figure 6). Therefore, the R_th and C_th were combined and used as thresholds to distinguish the apple points from woody parts. Whereas the remaining points of the latter class were removed using the L_W mode value of 15.5% as threshold (L_th) (Figure 6).

When performing the DBSCAN algorithm, the clusters of filtered points of apples appeared in the 3D point cloud (Figure 7). Subsequently, the k-means partition method was applied to the points of each cluster to acquire the M_D. A sphere of variable radius based on the D_D of the cluster, enclosed the points of the cluster when placed at M_D.

After the localization of M_D, the estimated fruit diameter, D_D, was compared to the D_Manual for each ΔW during the growth stages (

Table 1. Results of manually measured mean diameter (D_Manual) [mm] and the mean LiDAR-based measurement (D_D) [mm] in defoliated trees regarding mean absolute error (MAE) [mm], bias (MBE) [mm], root mean square error (RMSE) [%], and adjusted coefficient of determination (R²_adj).

		DManual [mm]	DD [mm]	MBE [mm]	MAE [mm]	RMSE [%]	R2adj
DAFB42	ΔW1-G	32.0	47.6	6.5	7.4	10.8	0.46
	ΔW2-1	36.1	65.1	7.5	8.7	12.0	0.45
	ΔW3-2	39.2	47.9	4.7	5.5	7.2	0.69
	ΔW4-3	39.3	41.2	5.3	6.2	6.6	0.74
DAFB72	ΔW1-G	66.4	48.0	−10.7	12.4	14.5	0.42
	ΔW2-1	62.8	37.9	−9.11	10.5	15.8	0.38
	ΔW3-2	65.6	63.7	−3.2	4.6	5.7	0.81
	ΔW4-3	67.0	63.7	−3.8	4.4	5.9	0.82
DAFB104	ΔW1-G	74.8	70.3	−5.8	8.3	9.6	0.55
	ΔW2-1	71.1	69.5	−4.5	5.1	7.8	0.66
	ΔW3-2	69.6	68.7	−3.3	3.8	4.5	0.90
	ΔW4-3	74.8	70.3	−3.2	3.5	4.1	0.95
DAFB120	ΔW1-G	69.3	67.3	−6.1	7.1	7.7	0.67
	ΔW2-1	70.5	68.4	−5.6	6.5	6.8	0.74
	ΔW3-2	69.7	68.6	−4.3	4.9	5.8	0.81
	ΔW4-3	68.6	66.4	−5.3	6.1	6.6	0.75

). The D_D was related to the D_Manual. In the upper sections, ΔW_3-2 and ΔW_4-3 of the canopies, where the fruit grow more distinctly, the analysis of apple size resulted in an adjacent coefficient of determination (R²_adj) of 0.69 and 0.74 with RMSE = 7.2 and 6.6% at DAFB₄₂, respectively. Generally, high measuring uncertainty was noticed on the first two measuring dates, when the fruit size was smaller, particularly in the lower sections of the trees with overlapping structures. More specifically, no relation was observed on the first measuring date in ΔW_1-G and ΔW_2-1, presenting an overestimation of 6.5 mm and 7.5 mm, respectively. Similarly, at DAFB₇₂ in the same areas, the MBE points to an underestimation of fruit diameter of −10.7 mm and −9.11 mm. In contrast, the highest relation was revealed in ΔW_4-3 (R²_adj = 0.95) and ΔW_3-2 (R²_adj = 0.90), at DAFB₁₀₄. Whereas, in the same stage, a less pronounced R²_adj was observed in ΔW_1-G (R²_adj = 0.55) and ΔW_2-1 (R²_adj = 0.66) compared to the aforementioned upper zones of the trees. The measuring uncertainty was slightly enhanced again at DAFB₁₂₀. The crop load of trees usually varies between the trees, affecting the fruit size. The high crop load of the trees analysed at DAFB₁₂₀ resulted in slightly smaller fruit size compared to trees with lower crop load sampled at DAFB₁₀₄. Consequently, the method showed lowest measuring uncertainty in the biggest fruit at DAFB₁₀₄ considering the defoliated trees.

3.2. Evaluation

In foliated trees, T_L, the same procedure was followed applying the same threshold values gained in the calibration on the data set obtained at DAFB₁₂₀. For evaluating the performance of the segmentation process in foliated trees, T_L, the data set of defoliated trees T_D was employed as ground truth. For this purpose, the M_L clusters found in T_L were compared to the clusters, M_D, found in T_D.

The total number of the apple clusters in defoliated trees (n_D) was 42 in ΔW_1-G, 84 in ΔW_2-1, 91 in ΔW_3-2, and 51 in ΔW_4-3 considering all measuring dates. It can be assumed that the high measuring uncertainties in the lowest part of the canopy may have resulted from the lower number of fruits additionally to the inclusions. When utilizing the proposed methodology, a high percentage of apples were found considering the TP in the foliated trees with 88.6%, 85.4%, 88.5%, and 94.8% of the ground truth at DAFB₄₂, DAFB₇₂, DAFB₁₀₄, and DAFB₁₂₀, respectively (

Table 2. Performance assessment of the localization algorithm on the segmented apple points in foliated trees (T_L) considering the apple in defoliated trees (T_D) as ground truth over four growth stages given in days after full bloom (DAFB) and in four tree height (W) starting from ground (G) till the maximum tree point (4). Manually counted fruit number (n_Manual), total number of detected fruits in T_D (n_D) and true positives (TP) in T_L with descriptive statistics are given.

		nManual	nD	TP	Accuracy [%]	Precision [%]	Recall [%]	F1 [%]
DAFB42	ΔW1-G	10	10	9	76.9	80.0	83.8	81.8
	ΔW2-1	19	18	16	85.2	82.6	86.3	84.4
	ΔW3-2	34	34	30	74.5	75.6	83.7	79.4
	ΔW4-3	17	17	15	76.9	85.0	89.4	87.1
DAFB72,	ΔW1-G	6	6	6	71.4	85.7	80.8	83.1
	ΔW2-1	15	14	12	75.1	82.8	82.3	82.5
	ΔW3-2	14	14	12	73.3	83.3	86.6	84.9
	ΔW4-3	7	7	5	65.8	71.4	83.3	76.9
DAFB104,	ΔW1-G	11	11	9	77.2	81.1	88.8	84.7
	ΔW2-1	18	18	16	84.1	85.7	81.8	83.3
	ΔW3-2	11	11	10	82.5	85.7	87.0	86.3
	ΔW4-3	12	12	11	80.2	86.6	88.5	87.5
DAFB120	ΔW1-G	15	15	14	73.4	78.9	83.3	81.1
	ΔW2-1	32	31	28	76.4	82.2	84.2	83.1
	ΔW3-2	16	16	16	75.5	85.7	88.0	86.8
	ΔW4-3	15	15	15	88.9	88.2	91.0	89.5

).

A less pronounced accuracy was noted when analyzing the foliated trees at DAFB₄₂ and DAFB₇₂. In parallel with the fruit growth, the number of apple points in the 3D point cloud increased, enhancing the performance of the detection algorithm. More specifically, the precision ranged from 81.1 to 86.6% and the F1 between 83.3 and 87.5% at DAFB₁₀₄, when the highest fruit size was captured. However, the highest performance was observed in ΔW_4-3 of DAFB₁₂₀, revealing an 89.5% F1, 88.2% precision, and 91.0% recall, when many large fruits were found and least overlapping between tree’s parts occurred.

The processing time of the algorithm was 13 sec per tree. The whole process carried out in 64-bit operating system with 16 GB RAM, an Intel^® Core i7-8750H processor (2.2 GHz).

The differences of distance between the M_L and M_D (Figure 8) and the fruit diameter considering D_D and D_L were calculated for each measuring date (

Table 3. Differences between the position of the centers of defoliated (M_D) and foliated (M_L) clusters as well as the fruit diameter estimated (D_L) and ground truth (D_D) in T_L and in T_D, respectively, regarding the standard deviation (SD) and root mean square error (RMSE) [%] during the fruit growth given in days after full bloom (DAFB).

	MD–ML [mm]	SD [mm]	RMSE [%]	DD–DL [mm]	SD [mm]	RMSE [%]
DAFB42	0.2	0.22	19.9	22.3	0.3	16.3
DAFB72	0.1	0.1	15.1	12.6	0.2	13.7
DAFB104	0.1	0.1	6.7	11.3	0.2	10.3
DAFB120	0.1	0.1	5.7	8.7	0.1	9.5

). The worst-case scenario was illustrated at DAFB₄₂ with 19.9% RMSE (Figure 9a). Moreover, in the same growth stage, the highest difference between the D_D and D_L was revealed, presenting a mean difference of 22.3 mm with 0.3 mm SD. The measuring uncertainty was decreased at DAFB₇₂. An even less pronounced RMSE was depicted at DAFB₁₀₄, while the lowest difference between ground truth and estimated position of fruit center in foliated trees was found at DAFB₁₂₀ (Table 3).

More specifically for DAFB₁₂₀, the best-case scenario was depicted considering the position of apple center with a mean value of 0.1 mm difference and 5.7% RMSE (Figure 9b). Furthermore, the minimum difference between the D_D and D_L was analysed, reaching 8.7 mm with 0.1 mm SD and 9.5% RMSE.

4. Discussion

4.1. Segmentation

The application of a LiDAR sensor system enabled the acquisition of 3D point cloud of apple trees, allowing the utilization of radiometric (R_ToF) and geometric features (C, L) as classifiers for fruit detection in the proposed algorithm. The apple class (R_A, C_A, L_A) was separated from leaves as well as delimited from the class capturing woody parts (R_W, C_W, L_W). The latter thresholds were defined after tree defoliation, whereas, the leaf class for the same features (R_L, C_L, L_L) was determined in T_L. The discrete ranges (Figure 5) of R_A (42–90%), R_L (50 to 70%) and R_W (0–60%) is assumed to be caused by their different type of surface [47] and water content [48]. Furthermore, the spherical shape of apples resulted in higher C_A values (56–98%) than C_W (0.1–55.2%) due to the linear shape of woody parts, while a reciprocal opposite range was observed in the correspondent classes of linearity, L_A (0.2–38.3%) and L_W (60.2–94.1%). Moreover, the latter classes were partly coinciding with C_L (40–78%) and L_L (20–80%). The apples were segmented from the woody parts and later clustered using the DBSCAN. The application of sphere in each M_D enabled high resolution in the determination of R_th (76.1%), C_th (73.2%) and L_th (15.5%) at DAFB₁₂₀. Similar value of R_th (60%) was suggested by Gene-Mola et al. [25] for the segmentation of apples in ‘Fuji’ trees.

This served as calibration for the segmentation modes, whereas also the sphere radius improved the detection of the number of fruit. As was previously mentioned, the ε of DBSCAN was based on the D_Manual of each ΔW at DAFB₁₂₀. Thus, earlier growth stages would be influenced by the generation of augmented clusters. This was confirmed by the finding that the measuring uncertainty was elevated mainly in the DAFB₄₂ stage, when a lower D of the fruits was measured manually.

Τhis overestimation of fruit size could result in under- and over-counting of the DBSCAN clustering at DAFB₄₂ and subsequent measuring dates, respectively, since the ε remained constant, and the fruit may be bigger or more small fruits are collected in one cluster. In Sun et al. [34], the wide range of size and shape of cotton boll similarily influenced the splitting operation of clusters. Furthermore, the shape of apples is not perfectly spherical producing an error in the estimation of D_D. Wang et al. [26] combined a cascade detection and an ellipse fitting to recognize and estimate the size of mangoes, using RGB-D depth imaging. However, the not perfectly elliptical shape of fruits produced 4.9 mm and 4.3 mm RMSE in the detected height and width of fruit, respectively. Gongal et al. [49] measured the size of apples based on the 3D coordinates from a 3D camera and they obtained 30.9% MAE, while the MAE decreased to 15.2% when the pixel information from an RGB camera was considered. As was mentioned before, the crop load (number of fruit per tree) varied, consequently trees with higher crop load will produce fuits of smaller size as found at DAFB₁₂₀ compared to DAFB₁₀₄. This affected the ratio of the total number of TP over the n_D, decreasing it from 94.8% at DAFB₁₂₀ (calibration date) to 88.4% at DAFB₁₀₄.

Nevertheless, the present results indicate that the data set of L, C and R_ToF can enable feasible counting and sizing of apples from the 3D point cloud, allowing their discrimination even in areas with high apple density. Furthermore, the values of R_ToF, C and L were described through the surface of apples with sphere segmentation, increasing the resolution in the classes, and subsequently the determination of the mode value.

4.2. Evaluation

The apple clusters in T_D were used as ground truth labels. This derived from the assumption that the shape and the size of apples could be fully described due to the reduced occlusions. Mack et al. [50] employed random sample consensus algorithm to fit a 3D sphere model in grapevine berries, reconstructing the shape of berries with 98% precision and 94% recall. However, the sphere size (D_D) was also applied in T_D as an additional step to evaluate the segmented region. In Rakun et al. [51], a reconstructed sphere to evaluate the 3D shape of segmented voxels was employed. Gene-Mola et al. [33] labelled manually the estimated locations of apples in the 3D point cloud, placing 3D rectangular bounding boxes on each apple. The reflectance and the sphericity derived from a LiDAR system were utilized as principal features in fruit detection, achieving 87.5% localization success and 0.85 F1. However, similar results have been reported from the implementation of 3D shape information with RGB vision systems, reaching a precision of 86.4% and recall 88.9% in fruit localization [35,36]. As a major perturbation factor, the woody parts in Δ_W1-G revealed similar R_ToF and C values with the apples, enhancing the FP cases and subsequently the decline of accuracy in this region. The occlusions produced by branches and leaves hide partly or totally the fruit, resulting in undercounting of apples. Mendez et al. [52] used a LiDAR laser scanner to count the number of fruit in pruned orange trees (R² = 0.63), however, similar to the results of the present study, the correlation was significantly reduced (R² = 0.18) when the method was applied in unpruned trees due to occlusion by leaves.

The fruit size of the calibration data set reduced the accuracy of the method, when smaller fruit size occurred. The accuracy of the proposed methodology was reduced at the first two growth stages, resulting in the lowest percentage (65.8%) in ΔW_4-3 at DAFB₇₂. Stajnko et al. [53] monitored the diameter of green apples several times during the vegetation period, combining thermal and RGB camera. The highest relation between the manually measured and detected fruit diameter (R² = 0.70) was observed at harvest stage, when the fruit reached their maximum size. Confirming, a less pronounced relation (R² = 0.68) was observed in apples of smaller size after fruit drop. Furthermore, the varying fruit size, appearing even at one stage, can enhance the visual complexity of the cluster and consequently influence the identified number of fruit [54].

However, evaluating the distance of M_L and M_D centers allowed to quantify the possible deviations (Table 3). Some of the fruits were partly occluded by branches, creating smaller clusters in T_L. This deviation was also visible in the high RMSE of fruit diameter considering D_D and D_L Summarizing, pomising results for in situ yield monitoring using a 2D LiDAR sensor system. Further research should be carried out to test the method in different apple cultivars having less spherical shape and varying surfaces properties in order to identify and tackle possible deviations of geometric and reflectance values.

5. Conclusions

The radiometric and geometric features of apples (R_A, C_A, L_A) showed higher values compared to wood and leaves compared to fruit elements. Concluding that they can be utilized to segment the apple points in the 3D point cloud. The likelihood of thresholds was enhanced by integrating a sphere to segment the apple clusters and determine the center position. This calibration was carried out on two defoliated trees at DAFB₁₂₀, resulting in R_th = 76.1%, C_th = 73.2%, and L_th = 15.5% that define the points of apple class.

The developed sphere methodology in defoliated trees was able to estimate the fruit diameter, D_D, with low measuring uncertainty at different growth stages. The D_D, showed the highest adjacent coefficient of determination (R²_adj = 0.95) with D_Manual in the upper parts of the canopy (ΔW_4-3) at DAFB₁₀₄, when the fruit showed hardly any occlusion with other tree organs and fruit size was at maximum considering all trees measured.

The evaluation of apple clusters in foliated trees over the clusters of defoliated trees depicted that the robustness is affected by the fruit size, but resulted in highest performance of 89.5% F1 with 91.0% recall and 88.2% precision. At DAFB₁₂₀, the minimum difference between ground truth, D_D, and estimated, D_L, fruit diameter was 8.7 mm representing 9.5% RMSE, which encourages the further development of LiDAR-based fruit size analysis.