Techniques for Canopy to Organ Level Plant Feature Extraction via Remote and Proximal Sensing: A Survey and Experiments

Abstract

This paper presents an extensive review of techniques for plant feature extraction and segmentation, addressing the growing need for efficient plant phenotyping, which is increasingly recognized as a critical application for remote sensing in agriculture. As understanding and quantifying plant structures become essential for advancing precision agriculture and crop management, this survey explores a range of methodologies, both traditional and cutting-edge, for extracting features from plant images and point cloud data, as well as segmenting plant organs. The importance of accurate plant phenotyping in remote sensing is underscored, given its role in improving crop monitoring, yield prediction, and stress detection. The review highlights the challenges posed by complex plant morphologies and data noise, evaluating the performance of various techniques and emphasizing their strengths and limitations. The insights from this survey offer valuable guidance for researchers and practitioners in plant phenotyping, advancing the fields of plant science and agriculture. The experimental section focuses on three key tasks: 3D point cloud generation, 2D image-based feature extraction, and 3D shape classification, feature extraction, and segmentation. Comparative results are presented using collected plant data and several publicly available datasets, along with insightful observations and inspiring directions for future research.

1. Introduction

Remote sensing is a cornerstone for plant phenotyping in smart agriculture, enabling high-throughput, non-invasive, and precise monitoring of crop traits across large fields. Technologies such as multispectral and hyperspectral imaging, UAV platforms, and satellite observations allow for real-time data collection on plant health, growth patterns, and stress responses, which are crucial for optimizing agricultural practices and enhancing crop yields. Traditional phenotyping methods, which are often labor-intensive and prone to error, are significantly improved by remote sensing. This technology allows for the rapid assessment of critical traits like plant height, leaf area index, and chlorophyll content, facilitating targeted interventions that enhance resource efficiency and reduce environmental impacts.

1.1. Background and Motivation

Plant phenotyping is a crucial component in modern agriculture and crop management, enabling researchers and agronomists to gain a deeper understanding of plant traits, growth dynamics, and responses to environmental stressors. The primary objective of phenotyping is to quantitatively measure various plant traits, such as the canopy structure, leaf area, plant height, and biomass, which are vital for plant breeding, yield prediction, and resource management. Traditionally, plant phenotyping has relied on manual measurements, which are time-consuming, labor-intensive, and prone to human error. With the rise of digital agriculture, remote and proximal sensing technologies have become essential tools for the high-throughput, non-destructive monitoring of crops at various scales, from the canopy to organ level [1].

Remote sensing platforms, such as satellites, unmanned aerial vehicles (UAVs), and ground-based systems, offer new opportunities for large-scale phenotyping by capturing diverse datasets using multispectral, hyperspectral, and thermal imaging sensors. These technologies provide a wealth of information on plant health, nutrient status, water content, and structural characteristics, enabling more precise and timely interventions in crop management [2,3]. However, the complexity of plant architectures and the variability in environmental conditions present significant challenges for extracting and interpreting phenotypic traits from these datasets. Thus, developing robust feature extraction and segmentation methods that can handle multi-scale data, such as 2D imagery and 3D point clouds, is crucial for advancing plant phenotyping research [4].

Plant feature extraction and segmentation from 2D images is a rapidly growing field in plant phenotyping, enabling the analysis of plant traits and characteristics from digital images [5]. With the advancement of computer vision and machine learning techniques, 2D image-based methods offer a non-invasive, cost-effective, and high-throughput approach to extract valuable information from plant images, such as the leaf shape, size, color, and texture [6]. Accurate segmentation and feature extraction from 2D images facilitate the quantification of plant growth, development, and responses to environmental stimuli, ultimately supporting plant breeding, crop monitoring, and precision agriculture [7]. Various approaches have been proposed for plant feature extraction and segmentation, including texture-based leaf identification [8], shape and texture feature-based classifications [9], and color histogram-based classification [10]. Recently, multi-descriptor approaches have also been developed for leaf classification [11]. Among the many techniques used in plant phenotyping is the 3D point cloud technology in conjunction with deep neural networks.

Figure 1 illustrates the trend in publications on canopy, plant, and organ-level feature extraction methods using remote sensing techniques, and an analysis from 2000 to 2024 of publication trends in feature extraction methods for plant phenotyping demonstrates significant growth across organ-level, plant-level, and canopy-level studies. Plant-level research holds the highest total publications, indicating a predominant focus on this area.

Notably, all levels experienced a peak in publications around 2021, likely due to technological advancements. The surge in canopy-level studies post-2015 reflects the adoption of remote sensing technologies like drones and satellites for large-scale analyses. These trends underscore the importance of integrating feature extraction methods across different organizational levels and leveraging advanced technologies such as hyperspectral imaging, deep learning, and machine learning algorithms.

This paper aims to provide a holistic view of the latest advancements in remote sensing for plant phenotyping, covering the entire spectrum from canopy-level to organ-level sensing. The insights presented here will serve as a valuable resource for researchers and practitioners looking to develop new phenotyping methods or enhance existing ones for precision agriculture and sustainable crop management.

Through empirical evaluation and case studies, these techniques demonstrate their efficacy in enhancing agricultural productivity, sustainability, and resilience to environmental challenges.

Table 1. Techniques used for plant phenotyping in remote sensing: contributions and limitations.

Year	Authors	Contributions	Limitations
2020	[1]	Comprehensive review of remote sensing applications in precision agriculture; highlights the integration of satellite imagery and UAVs for optimizing inputs like water and nutrients.	Focuses more on existing technologies rather than proposing new methodologies; limited discussion on future innovations.
2024	[2]	Overview of advancements in high-throughput phenotyping platforms using remote sensing; emphasizes the integration of diverse sensors like hyperspectral and 3D imaging.	Limited by regional infrastructure availability, particularly in developing regions.
2014	[3]	Reviews the state-of-the-art UAV remote sensing technologies for field-based crop phenotyping, emphasizing the ability to measure a wide range of phenotypic traits.	Discusses existing technologies, lacks an exploration of emerging trends and future applications.
2024	[4]	Explores the transformative impacts of smart sensors, IoT, and AI in modern agricultural practices; discusses the integration of these technologies with remote sensing for enhanced precision.	Primarily theoretical; requires further empirical validation and field tests.
2024	[12]	Compares UAV, satellite, and ground-based remote sensing approaches for field-based crop phenotyping; finds UAVs superior in resolution and efficiency for large-scale breeding programs.	Limited to specific physiological traits like the canopy temperature and NDVI, does not cover broader phenotypic traits.
2024	[13]	The first study using UAV-based remote sensing to model sustainability traits in crops like switchgrass; highlights UAVs’ potential in high-throughput phenotyping.	Focused on a specific crop (switchgrass), which may limit generalizability to other crop types.

shows the techniques used for plant phenotyping in remote sensing.

1.2. Scope and Objectives

The primary focus of this review is to provide a comprehensive overview of the latest techniques for plant phenotyping using remote and proximal sensing across different scales—canopy, plant, and organ levels. This survey covers satellite-based, UAV-based, and ground-based sensing methods, evaluating their effectiveness in extracting critical phenotypic traits for various applications. The scope includes an in-depth analysis of feature extraction methodologies, such as multispectral and hyperspectral imaging, LiDAR (Light Detection and Ranging), synthetic aperture radar (SAR), and 3D point cloud segmentation. Furthermore, this paper aims to bridge the gap between these multi-scale sensing techniques by providing a unified view of how different data acquisition methods and analytical frameworks can be integrated to achieve high-resolution, multi-modal plant phenotyping.

The objectives are threefold: (1) to highlight the advancements in remote sensing technologies for automated phenotyping, (2) to compare the strengths and limitations of each method for specific phenotypic traits and environmental contexts, and (3) to provide a critical assessment of future directions for research and development in this rapidly evolving field. The overarching goal is to support the development of scalable, efficient, and accurate phenotyping methods that can be deployed across different agricultural landscapes.

1.3. Contributions of This Paper

This paper makes several unique contributions to the field of plant phenotyping using remote and proximal sensing techniques. First, it systematically reviews the current state-of-the-art in feature extraction and segmentation methods, covering both 2D image-based and 3D point cloud-based approaches. Second, the survey provides a detailed comparison of various sensing platforms, from satellite-based large-scale monitoring to UAV-based field-level assessments and organ-level analyses using ground-based systems. This multi-scale perspective helps identify the most suitable techniques for specific phenotyping tasks and highlights the potential of integrating different modalities to capture complex plant traits more effectively.

Additionally, this review identifies key research gaps and challenges in current methodologies, such as the need for improved accuracy in organ-level segmentation and the development of robust algorithms that can handle varying spatial resolutions and data noise. By synthesizing findings from recent studies and experiments, this paper offers valuable insights into how emerging technologies like AI, machine learning, and advanced 3D modeling can be leveraged to enhance the precision and scalability of plant phenotyping systems. It not only reviews progress in remote/proximal sensing for plant phenotyping but also demonstrates various levels of plant feature extraction techniques, ranging from remote and proximal data collection to feature extraction and segmentation of plants. The insights of each experiment provided are expected to help researchers in the field and to inform future research and support the design of next-generation phenotyping platforms that combine the strengths of remote and proximal sensing technologies.

1.4. Structure of the Paper

This paper is structured as follows: Section 2, Applications of Plant Phenotyping in Remote Sensing, provides a detailed exploration of various remote sensing technologies and their applications at the canopy, plant, and organ levels. Section 3, Technical Progress of Machine Learning and 3D Point Cloud Classification in Remote Sensing, delves into advancements in 3D point cloud generation and classification, which are essential for capturing detailed plant structures, particularly when integrated with LiDAR and other imaging systems. Section 4, Plant Feature Extraction, discusses the techniques used to extract meaningful features from remote sensing data, emphasizing the use of machine learning and image processing for an accurate phenotypic trait analysis. Section 5, Remote Sensing and Plant Phenotyping: Insights, presents a synthesis of experimental results from recent studies that have validated these technologies in practical applications. Section 6 is the Discussion. Finally, Section 7, Conclusions, summarizes the key findings, highlights the current challenges, and outlines potential future directions for developing remote sensing technologies for plant phenotyping.

Remote sensing technologies for plant phenotyping can be categorized into three main groups: satellite-based, UAV-based, and ground-based (proximal) sensing (Figure 2). Each platform has unique advantages and challenges based on the spatial resolution, data acquisition cost, and the ability to capture specific phenotypic traits.

2. Overview of Plant Phenotyping Using Remote/Proximal Sensing Technologies

Remote sensing technologies have significantly transformed plant phenotyping by providing high-throughput, non-destructive methods for monitoring and analyzing plant traits across different scales, from the canopy down to the organ level [12,13]. These technologies leverage advancements in imaging and data analysis to offer precise and actionable insights into plant health, growth, and productivity. This section explores recent advancements in satellite and UAV-based remote sensing techniques for plant feature extraction, highlighting their applications, strengths, and limitations. In

Table 2. Phenotyping feature extraction in different plant levels.

Criteria	Canopy-Level Phenotyping	Plant-Level Phenotyping	Plant-Organ-Level Phenotyping
Imaging Techniques	Multispectral Hyperspectral LiDAR Thermal SAR Photogrammetry	High-Resolution RGB Multispectral Hyperspectral LiDAR Thermal	3D Imaging Fluorescence Hyperspectral
Data Resolution	Medium to low (depends on the altitude and sensor, satellite and UAV systems typically have lower resolution at higher altitudes)	High due to proximity (due to closer proximity in UAV and controlled environment setups (CESs))	Very nigh (detailed 3D reconstruction, organ-specific data acquisition)
Phenotyping Focus	Whole canopy cover, vegetation health, biomass estimation	Individual plant shapes, growth rates, nutrient stress	Leaf area, stem diameter, fruit/flower detection, photosynthetic activity
Common Sensors Used	RGB cameras, multispectral/hyperspectral sensors, LiDAR, thermal	RGB cameras, LiDAR, thermal, hyperspectral, multispectral	Fluorescence sensors, 3D scanners, high-resolution RGB
Feature Extraction	NDVI, EVI, 3D canopy structure from LiDAR, thermal stress indices	Detailed 3D plant models, growth tracking, nutrient stress detection	Leaf area, stem diameter, fruit/flower detection, photosynthetic activity (fluorescence)
Data Analysis Techniques	Spectral analysis, image segmentation, 3D modeling, time series analysis	Advanced segmentation, 3D reconstruction, thermal and spectral analysis	Deep learning for organ-level segmentation, photosynthetic efficiency analysis
Data Collection Frequency	Periodic (depending on the satellite or UAV revisit time)	Continuous or high frequency in CES and UAV settings	On-demand or continuous for high-resolution monitoring
Advantages	Large area coverage, whole crop monitoring, temporal analysis	Detailed individual plant monitoring, flexible for research needs	Highly precise phenotyping of organs (leaf, stem, and fruit), detailed trait extraction

, the focus of study and techniques at different levels of plant level phenotyping have been reviewed.

2.1. Satellite-Based Plant Feature Extraction

Satellite remote sensing offers a macroscopic view of agricultural landscapes, enabling extensive monitoring of crops and vegetation across large areas. With recent technological advancements, satellites now provide improved spatial, spectral, and temporal resolutions, facilitating detailed extraction of plant phenotypic traits.

2.1.1. Multispectral and Hyperspectral Imaging with Respect to Satellites

Satellites equipped with multispectral and hyperspectral sensors capture data across a range of wavelengths, which is critical for a detailed analysis of plant health and phenotypic traits. Multispectral imaging, which typically captures a limited number of broad spectral bands, is useful for basic vegetation monitoring through indices like the Normalized Difference Vegetation Index (NDVI) and the Enhanced Vegetation Index (EVI). These indices help assess overall plant health, chlorophyll content, and stress levels by analyzing differences in red and near-infrared reflectance, as demonstrated by the Sentinel-2 satellite’s capabilities [14]. Hyperspectral imaging, in contrast, provides a much finer spectral resolution by capturing hundreds of narrow spectral bands, allowing for a detailed analysis of the biochemical properties of plants, such as the nitrogen content, pigment concentrations, and water status. This high spectral resolution makes hyperspectral imaging particularly powerful for the early detection of diseases, nutrient deficiencies, and water stress, providing a comprehensive view of plant physiology [15].

2.1.2. Thermal Imaging with Respect to Satellites

Thermal sensors on satellites measure the thermal radiation emitted by plants, offering insights into the plant water status and stress levels. This method is particularly effective in detecting water stress because stressed plants often close their stomata to conserve water, leading to an increase in leaf temperature. Thermal imaging thus becomes an invaluable tool for irrigation management and drought monitoring. By analyzing thermal data, researchers can create water stress maps to guide irrigation decisions, optimize water use, and prevent crop losses due to drought [16]. Additionally, thermal imaging can detect heat stress and its impacts on photosynthetic efficiency, enabling better management of crops under varying climatic conditions.

2.1.3. Synthetic Aperture Radar (SAR)

SAR technology provides all-weather, day-and-night imaging capabilities, which are critical for consistent monitoring of plant growth and health, regardless of the weather conditions. SAR’s ability to penetrate cloud cover makes it a reliable tool for estimating biomass, monitoring soil moisture levels, and assessing vegetation structure. SAR sensors are particularly useful in regions with frequent cloud cover or during rainy seasons when optical sensors may be less effective. SAR data have been applied successfully to monitor changes in plant structure, detect lodging in crops, and assess the impacts of flooding on agricultural fields [17]. The typical spatial resolution of SAR is in the range of 1–100 m, depending on the system configuration, while satellite optical imaging systems typically achieve spatial resolutions between 10 and 30 m. SAR complements optical and thermal data by providing structural information about plant canopies, offering a more holistic view of crop conditions.

2.1.4. LiDAR (Light Detection and Ranging)

Although less common on satellite platforms due to its weight and power requirements, LiDAR technology provides high-resolution 3D models of plant canopies. These models offer detailed insights into the plant height, volume, and structural characteristics, which are crucial for understanding plant growth dynamics, biomass accumulation, and canopy architecture [18]. In terrestrial or airborne applications, LiDAR is highly effective for creating precise digital elevation models (DEMs) and for applications such as forest inventory, where accurate measurements of tree height and volume are needed. When used in conjunction with other sensors, LiDAR data can enhance the understanding of the vertical canopy structure and contribute to more accurate estimates of crop yield and biomass. Unlike satellite-based or SAR imaging, LiDAR sensors offer the highest possible resolution for 3D structural mapping, with resolutions up to 0.1 m

2.2. UAV-Based Plant Feature Extraction

Unmanned aerial vehicles (UAVs) provide unparalleled flexibility and high-resolution data collection capabilities at lower altitudes, making them particularly well-suited for detailed phenotyping at the canopy, plant, and organ levels. UAVs allow for more frequent and targeted data collection, which are essential for monitoring dynamic changes in plant phenotypes over time.

2.2.1. Multispectral and Hyperspectral Imaging with Respect to UAVs

UAVs equipped with multispectral and hyperspectral cameras offer high-resolution images that can be tailored to specific phenotyping tasks, such as detecting diseases, assessing the nutrient status, and monitoring plant health. Multispectral UAVs can capture several specific bands that are crucial for evaluating vegetation indices, such as NDVI and the Normalized Difference Red Edge (NDRE), which are indicators of plant vigor and stress [19]. Hyperspectral UAVs, which capture continuous spectral data across many bands, provide even more detailed information on plant biochemistry and physiology, allowing for the detection of subtle stress responses, early disease symptoms, and nutrient deficiencies. UAV-based hyperspectral imaging has been particularly valuable in precision agriculture, where detailed spectral information is used to apply inputs (like fertilizers or pesticides) more precisely and efficiently.

2.2.2. Thermal Imaging with Respect to UAVs

Thermal cameras mounted on UAVs are used to monitor plant temperature, a direct indicator of water stress and transpiration rates. UAV-based thermal imaging provides high spatial resolution and temporal flexibility, enabling frequent monitoring of the water status across different parts of the field [20]. By creating thermal maps, farmers can identify areas of the field that are under water stress and adjust irrigation practices accordingly. Additionally, thermal imaging can help detect plant diseases that alter transpiration rates, thereby providing an early warning system for disease management.

2.2.3. RGB Imaging and Structure from Motion (SfM)

High-resolution RGB cameras on UAVs, combined with Structure from Motion (SfM) algorithms, are used to generate detailed 3D models of plant canopies. These models are essential for measuring the plant height, canopy cover, and biomass, which are important phenotypic traits. The SfM technique reconstructs 3D structures from a series of overlapping 2D images taken from different angles, providing a cost-effective alternative to LiDAR for 3D modeling. RGB imaging is also used to detect visual symptoms of stress or disease and to monitor growth stages by analyzing changes in color and texture over time [21].

2.2.4. LiDAR and Depth Sensing

UAV-mounted LiDAR systems provide high-resolution 3D data that are invaluable for precise measurements of plant structure and morphology. Unlike passive optical sensors, LiDAR can penetrate the canopy and capture the 3D structures of plants, allowing for accurate assessments of the plant height, canopy density, and leaf area. This technology is particularly useful for phenotyping traits related to the plant architecture, such as the leaf angle distribution and branching patterns, which are important for understanding light interception and photosynthesis in crops [21]. The highest possible resolution of UAV-mounted LiDAR systems is approximately 0.1–0.3 m.

Fluorescence Imaging:

UAVs equipped with fluorescence sensors can detect specific wavelengths emitted by plants that are indicative of photosynthetic activity and plant health. Chlorophyll fluorescence, in particular, is a direct indicator of photosynthetic efficiency and can reveal early stress responses that are not visible through other imaging methods [22,23]. This technique is used to monitor plant stress, detect diseases, and assess the effectiveness of treatments. Fluorescence imaging can provide real-time feedback on plant health, enabling rapid decision-making in precision agriculture. Fluorescence imaging typically provides a resolution of up to 0.5 m. The use of remote sensing technologies for plant phenotyping provides valuable insights into crop health, growth, and productivity, empowering farmers and researchers to make precise, data-driven decisions. Advancements in satellite and UAV-based remote sensing technologies have enabled the monitoring and analysis of plant traits at various scales, from the canopy to organ level. By leveraging these tools, modern agriculture can improve efficiency, reduce input costs, and enhance sustainability.

2.3. Phenotyping Scope in Remote and Proximal Sensing

Canopy-Level Phenotyping: Crop yield Estimations, Disease Detection, and Canopy Cover.

At the canopy level, remote sensing through UAVs and satellites primarily focuses on whole-plant or vegetation-level analyses. Multispectral and hyperspectral imaging provide essential data for calculating vegetation indices like NDVI and EVI, which are used for monitoring overall plant health, estimating biomass, and assessing the canopy structure. Thermal imaging helps detect water stress at the canopy level, while LiDAR and RGB images enable 3D reconstructions for measuring plant height and canopy density. More features for canopy-level phenotyping often used in remote sensing are discussed in

Table 3. Features for canopy level phenotyping.

Feature Category	Feature Name	Description	Use in Phenotyping and Crop Modeling	Authors
Spectral Features	Normalized Difference Vegetation Index (NDVI)	Measures the photosynthetic capacity and vegetation health by comparing near-infrared (NIR) and red-light reflectance.	Estimates biomass, crop vigor, and health status. Used in yield prediction and growth monitoring.	[24,25]
	Green NDVI (GNDVI)	Similar to NDVI, focuses on green light, and assesses the chlorophyll content and leaf area index.	Assesses the chlorophyll concentration, which is an indicator of the nitrogen content and crop stress.	[26,27]
	Enhanced Vegetation Index (EVI)	Corrects for atmospheric and soil background noise, useful in areas with a high biomass.	Enhances the signal in dense vegetation for better biomass and health estimations.	[28]
	Soil-Adjusted Vegetation Index (SAVI)	Adjusts for soil brightness in areas with sparse vegetation.	Improves the vegetation signal in areas where bare soil is exposed.	[29]
	Red Edge NDVI	Uses the red-edge band to detect subtle changes in the chlorophyll content and high biomass conditions.	Early detection of plant stress and monitoring of senescence stages. Strongly correlated with the plant biomass due to its sensitivity to chlorophyll variations.	[30]
	Photochemical Reflectance Index (PRI)	Measures light use efficiency (LUE) and CO2 uptake using reflectance changes at 531 nm and 570 nm due to xanthophyll cycle activity.	Estimates gross primary productivity (GPP) and photosynthetic performance. Useful for detecting plant stress and quantifying CO2 fixation under varying environmental conditions.	[31]
	Visible Atmospherically Resistant Index (VARI)	Assesses the physiological status of vegetation using the green, red, and blue bands.	Detects plant stress and is significantly correlated with the grain yield, flowering time, plant height, and anthocyanin concentration. Shows strong utility in maize phenotyping.	[32]
	Normalized Difference Water Index (NDWI)	Measures the water content in vegetation.	Detects drought stress and water retention in crops.	[33]
	Normalized Green–Red Difference Index (NGRDI)	Compares green and red bands to highlight vegetation health.	Effective for monitoring crop health and early stress detection.	[34]
	Enhanced NDVI index (ENDVI)	Combines the NIR and green bands to enhance vegetation signals.	Useful for precision agriculture and assessing the nitrogen status.	[35]
	Green Ratio Vegetation Index (GRVI)	Ratio of NIR and green band reflectance to assess crop health.	Indicates chlorophyll activity and the nitrogen status.	[36]
	Chlorophyll Index Red Edge (CIRE)	Uses the red edge band to estimate the chlorophyll concentration.	Measures subtle changes in vegetation health and productivity.	[37]
	Log Red	Logarithmic transformation of red reflectance to reduce saturation effects.	Improves sensitivity to changes in vegetation cover.	[27]
	Red, Green Vegetation Index (RG)	Weighted combination of red and green bands to highlight vegetation features.	Identifies areas of crop stress and biomass estimation.	[27]
Textural Features	Gray-Level Co-occurrence Matrix (GLCM) Features	Captures texture via contrast, correlation, energy, and homogeneity.	Useful for detecting crop type variations and stress levels in vegetation.	[38]
	Entropy	Quantitative measure of randomness in texture. Higher values indicate diverse texture elements.	Differentiates healthy from stressed crops based on the uniformity of the canopy structure.	[39,40]
	Local Binary Patterns (LBPs)	Describes spatial texture patterns in the canopy.	Useful in differentiating crop species and identifying stress patterns.	[41]
Structural Features	Canopy Height Model (CHM)	Derived from DSM and DTM, represents plant height.	Biomass estimation, growth rate monitoring, and yield potential prediction.	[42]
	Canopy Cover	Measures the proportion of ground covered by the canopy using thresholding techniques.	Essential for yield prediction, irrigation management, and monitoring crop density.	[43]
	Canopy Volume	A 3D representation of the canopy structure.	Biomass estimation and modeling crop architecture.	[44]
	Leaf Area Index (LAI)	Measures the leaf area relative to the ground area.	Estimates the photosynthetic capacity, crop growth, and potential yield.	[27]
Temporal Features	Growth Rate	Measures changes in the canopy height, volume, and spectral indices over time.	Useful for assessing crop growth, phenological changes, and stress responses.	[45]
	Phenological Metrics	Key transitions like greening, flowering, and senescence stages. Combines structural and spectral features to estimate the above-ground biomass.	Tracking phenological development and predicting yield. Critical for yield prediction and crop health monitoring.	[46]
Environmental Features	Evapotranspiration (ET)	Measures evaporation and plant transpiration, derived from UAV and spectral data.	Used in irrigation management and water balance models.	[47]
	Soil Moisture Content	Derived from NDWI and thermal imagery.	Estimates the soil water content and crop water stress.	[48]
Machine Learning Features	Deep Learning-Based Features	Extracts complex patterns from high-dimensional datasets. Uses CNNs, graph-based models and transformers to learn patterns and generate high-resolution crop imagery.	Improves feature extraction and phenotyping accuracy, and simulates realistic crop growth patterns.	[49,50]
	GAN-based Features	Trains GANs to generate synthetic images mimicking real-world growth patterns.	Creates realistic synthetic images for simulations or data augmentation.	[51]
	Ordinary Differential Equations (ODEs) and Partial Differential Equation Methods (PDEs)	Models the dynamics and temporal changes of complex systems. Useful for understanding and predicting physiological interactions in crops.	Facilitates the study of dynamic growth patterns, water use efficiency, and stress responses under varying environmental conditions.	[52]
3D Reconstruction	Point Cloud Density and Distribution	Analyzing 3D point clouds for canopy structure and biomass estimation.	Provides detailed structural information for accurate phenotyping and yield estimation.	[53]

.

The integration of satellite and UAV-based remote sensing technologies, along with controlled environment setups (CESs), has revolutionized plant phenotyping at all levels—canopy, plant, and organ. These advancements provide detailed, high-throughput methods for assessing plant traits, contributing to more precise and efficient agricultural practices. With ongoing improvements in sensor technologies and data analysis techniques, the application of remote sensing in phenotyping will continue to enhance crop management and sustainability.

The diverse methodologies for canopy-level phenotyping are essential for advancing agricultural research and enabling precise crop management strategies. Spectral features such as the Normalized Difference Vegetation Index (NDVI), Green NDVI (GNDVI), Enhanced Vegetation Index (EVI), and Soil-Adjusted Vegetation Index (SAVI) are widely utilized for monitoring vegetation health, estimating biomass, and assessing crop stress. These indices leverage various spectral bands (e.g., NIR, red, and green) to capture subtle changes in photosynthetic activity and the chlorophyll content. The NDVI, for example, compares reflectance in the NIR and red bands to assess plant vigor, making it a powerful tool for evaluating crop growth and development [47,54]. Such indices are crucial for yield prediction and growth monitoring under varying environmental conditions.

Additionally, integrating these spectral indices with evapotranspiration (ET) models enhances the understanding of crop water use and helps optimize irrigation strategies. ET, which represents the combined effect of soil evaporation and plant transpiration, is a key indicator of crop water requirements and overall health. UAV-based imaging combined with spectral indices and thermal imagery provides precise ET monitoring for effective water management in precision agriculture. For instance, the two-source energy balance model (TSEB) can partition soil and canopy ET, improving water use efficiency across different crop stages, which is vital for preventing water stress and enhancing crop yields. By integrating a time series analysis of VIs with ET models, a comprehensive assessment of water use and crop development can be achieved, leading to more efficient agricultural practices [47].

Textural features, such as the Gray-Level Co-occurrence Matrix (GLCM), entropy, and local binary patterns (LBPs), provide insights into crop structure and stress conditions by analyzing texture patterns in plant canopies. These methods are particularly useful for distinguishing healthy from stressed crops based on canopy uniformity and texture variations, making them effective for detecting crop type variations and monitoring stress levels. Structural features such as the canopy height, volume, and leaf area index (LAI) are vital for estimating the biomass, potential yield, and canopy architecture, contributing to accurate phenotyping of plant growth and development. Temporal features, including the growth rate and phenological metrics, are essential for tracking key developmental stages such as flowering and senescence, which are crucial for predicting crop yield and identifying stress responses over time. Environmental features like the soil moisture content and evapotranspiration (ET), derived from the NDWI and thermal images, play significant roles in effective irrigation management and water balance modeling, providing detailed insights into the soil water content and crop water status.

Deep Neural Network techniques, including convolutional neural networks (CNNs), long short-term memory (LSTM) networks, and attention mechanisms, have significantly enhanced feature extraction, phenotyping accuracy, and the simulation of crop growth patterns. These approaches can analyze high-dimensional data, generate synthetic images, and predict physiological changes in crops under various environmental conditions. For instance, integrating deep learning models with a time series analysis of vegetation indices allows growers to assess crop conditions over an entire growing season, leading to improved yield prediction models and the early detection of crop stress.

Finally, 3D reconstruction methods using point cloud data enable detailed analyses of the canopy structure and distribution, providing precise structural information for phenotyping and yield estimations. Point cloud data capture spatial details of plant canopies, allowing for accurate modeling of the canopy architecture, biomass estimation, and the detection of subtle changes in the canopy structure over time. Overall, the integration of these diverse feature extraction techniques provides a comprehensive toolkit for effective canopy-level phenotyping, supporting improved crop management, yield prediction, and agricultural decision-making.

2.3.1. Plant-Level Phenotyping: Plant Height, Biomass Estimation, Plant Health Monitoring

At the individual plant level, UAVs equipped with high-resolution cameras and sensors can monitor specific plant traits such as growth patterns, shape, and nutrient deficiencies. The 3D models constructed from RGB and LiDAR data provide detailed insights into plant morphology, while multispectral and hyperspectral imaging reveal biochemical traits such as photosynthetic activity and stress responses. UAV-mounted systems offer flexibility, enabling continuous data collection throughout different growth stages significantly 3D point clouds.

Skeletonization [55] of plants structure from huge 3D point clouds are more advantageous for plant-level phenotyping. The skeletonization process for plant phenotyping involves denoising the point cloud, applying iterative Laplacian contraction to reduce it to a skeletal structure, adaptively sampling key points, and establishing connections to form a cohesive skeleton while avoiding loops. This method enables precise measurements of morphological traits like plant height and leaf angles in under 100 s, making it efficient for automated maize phenotyping. Skeletonization offers significant advantages: it simplifies data processing, supports semantic segmentation of plant organs, improves trait measurement accuracy, enhances visualization, and is compatible with functional-structural plant models. Its robustness to noise and occlusion makes it a powerful tool for scalable, high-throughput phenotyping across diverse environments, although additional refinement is needed for more complex plant architectures and field conditions.

2.3.2. Organ-Level Phenotyping: Leaf Area Index, Stem Thickness, and Flower and Fruit Counting

For phenotyping plant organs, such as leaves, stems, and fruits, high-resolution imaging techniques are critical. In indoor controlled environments (CESs) or UAV settings, fluorescence imaging plays a significant role in monitoring the chlorophyll content and detecting early stress responses at the organ level. Three-dimensional imaging and detailed organ segmentation allow for the measurement of traits like the leaf area, fruit detection, and stem architecture. Advanced deep learning techniques are often employed to automate the segmentation and analysis of individual organs, providing researchers with highly specific insights into plant physiology. More details regarding organ-level phenotyping methods and techniques are listed in

Table 4. Organ-level and plant-level feature extraction methods.

Methods/Techniques	Description	Authors
Hyperspectral Imaging	Imaging technique capturing spectral information across wavelengths for a detailed analysis.	[56,57,58]
NIR (Near-Infrared) Imaging	Uses near-infrared light to penetrate plant tissues, revealing internal structures.	[59]
Fluorescence Imaging	Exploits natural fluorescence from chlorophyll to assess plant health and segmentation.	[60,61]
Terrestrial Laser Scanning/Laser Techniques	Utilizes laser scanning to create detailed 3D models of plants for precise measurements, and includes Terrestrial Laser Scanning (TLS).	[62]
Microscopic Approach	Involves high-magnification imaging for detailed cellular and tissue analyses. Specialized microscopy focusing on stomata for physiological studies.	[63]
Point Cloud	3D representations of plants and their organs are obtained via scanning techniques like structured light, laser scanning, photogrammetry, and multiview stereo. Point cloud data are crucial for capturing precise structural information of plants and their individual organs. Extraction techniques focus on various methods such as shape modeling, statistical feature descriptors, and segmentation algorithms for analyzing morphological traits and growth patterns at the organ and plant levels.	[64,65,66,67,68,69,70,71,72,73]
For Controlled Environments/Chambers	Enclosed environments designed to maintain stable, regulated conditions for consistent phenotyping and experimentation. Used to control variables such as temperature, humidity, light intensity, and irrigation schedules, ensuring precise environmental monitoring and data consistency.	[74,75]
Digital Image Processing	Application of image processing techniques to enhance, segment, and classify digital images for plant feature extraction, disease detection, and growth rate analyses. Techniques often include filtering, noise removal, segmentation, and classification for plant health assessments and monitoring.	[76,77,78,79,80]
Morphological Feature Extraction	Extracts structural features like the shape, size, leaf edges, and structures of plant organs using morphological transformations, edge feature analysis, and key point detection methods. Applied for plant disease detection, species classification, and structural phenotyping. It can also be used to determine the ripeness or maturity level of vegetables.	[81,82,83,84]
Zoning Feature Extraction	Divides images into zones to extract localized features.	[85]
Wavelets	Analyzes images at multiple resolutions using wavelet transformations to extract features like edges, textures, and shapes. Applied for plant disease detection, species classification, and leaf identification by capturing fine details and patterns in plant images.	[86,87,88]
Moment Invariant	Mathematical descriptors that are invariant to image transformations like scaling and rotation.	[89]
Zernike Movement Invariant (ZMI)	Uses Zernike polynomials for rotation-invariant feature extraction.	[90]
Textural Feature Analysis	Analyzes the texture of images to extract features like smoothness or coarseness.	[91,92,93]
GLCM Features	Gray-Level Co-occurrence Matrix (GLCM) for a texture analysis. This is widely used for texture classification and segmentation in medical imaging, agriculture, and other fields requiring fine-grained texture analysis by computing contrast, dissimilarity, homogeneity, energy, and correlation of the matrix.	[94,95]
Histogram-Oriented Gradient Method	Captures edge or gradient structures in images for feature extraction.	[71,96]
Color Feature Extraction	Extracts features based on color information in images.	[97]
Veins/Leaf Venation Density	Analyzes the venation patterns in leaves for species identification or health assessments.	[98,99]
Shape Recognition—HSV and OTSU Methods	Uses the Hue, Saturation, Value (HSV) color space and Otsu’s thresholding for shape-based segmentation.	[100]
CNN (Convolutional Neural Network)	Convolution layers capture local spatial features by applying different filters. These features are then flattened into a feature vector, excluding fully connected layers, and the VGG16 architecture is used as a feature extractor to capture visual patterns from the images.	[101]
	The model first extracts meaningful features from plant images using ResNet50, processes these features through multiple Dense layers, and then sums the resulting values to predict the leaf count, making it a regression-based leaf counter.	[102]
LSTM (Long Short-Term Memory)	LSTM is used to process time series data, making it suitable for capturing dynamic changes over time. LSTM helps detect subtle early stress indicators that may not be evident in a single image, thereby improving early-stage stress detection.	[101]
Attention Mechanism	Attention mechanisms are designed to selectively emphasize the most relevant features of an object while suppressing irrelevant or noisy features. Neural networks that focus on important parts of input data improve performance on tasks like classification.	[103]
Transformers	Advanced deep learning models using self-attention mechanisms, effective in sequence modeling.	[104,105]
Genetic Programming	Evolutionary algorithm that evolves programs or models to perform specific tasks.	[106,107]
Particle Swarm Optimization	The Particle Swarm Optimization (PSO) algorithm enhances the accuracy and efficiency of leaf disease classification by optimizing the segmentation process. PSO aids in finding optimal cluster centers, ensuring that the best characteristics of the features are applied for separation of diseased regions.	[108,109]
Rider Cuckoo Search Algorithm	The Rider Cuckoo Search Algorithm is a hybrid optimization technique integrating the Rider Optimization Algorithm (ROA) and Cuckoo Search (CS) to improve performance. It optimizes the training of Deep Belief Networks (DBNs) by balancing exploration and exploitation, leading to better classification results.	[110]
Clustering and Segmentation	Groups data points based on similarity, used in the general process of partitioning an image into meaningful regions like segmentation and classification.	[111]
Discriminant Analysis	Statistical method used to find a combination of features that separates classes of objects.	[112]
Thresholding and Edge-Based Techniques	These techniques use intensity thresholds and edge detection for segmenting images into meaningful regions. Thresholding converts gray-scale images into binary images based on defined threshold values, while edge detection identifies boundaries between objects using first-order (e.g., Sobel and Canny) and second-order derivatives. These methods are widely used for medicinal plant image segmentation.	[113]
Improved Saliency Method	Enhances the saliency (prominence) of objects for better segmentation by fusing the Sharif saliency-based (SHSB) method with active contour segmentation, improving the clarity of infected regions in cucumber leaves. This step aids in accurate feature extraction.	[114]
Thermal imaging from flower temperature patterns	Captures the floral surface temperature distribution using infrared imaging, revealing contrasting temperature patterns. Analyzes distinct thermal structures within a flower to identify temperature contrasts between petals and reproductive parts.	[115]
Double Line Clustering	A clustering method that segments diseased plant regions by analyzing pairs of lines for more precise image segmentation. It is used to identify and segment diseased leaf areas in crops like tomatoes, grapes, and cucumbers.	[116]
Hough Transform	Identifies geometric shapes (such as lines, circles, and ellipses) within digital images by mapping the image space into a parameter space. Uses a voting mechanism to extract shape details based on defined parameters, making it highly resilient to noise and incomplete boundaries. This method excels at detecting intricate shapes and patterns in noisy datasets and handles partial or broken edges effectively, ensuring reliable extraction of irregular object shapes. Ideal for shape analyses.	[117]
Adaptive Gamma Correction	Adjusts the image brightness adaptively to enhance features.	[114,117]
Multi-Fractal Analysis	Analyzes complex patterns that exhibit fractal properties at multiple scales using fractal dimensions to capture both the local and global characteristics of an object. The key steps include calculating singularities for each image pixel using Hölder exponents and extracting multifractal spectra such as the Hausdorff dimension for global characterization. This method is particularly effective at identifying small objects, such as insect pests, under challenging conditions like variable lighting. It involves box-counting for estimating the fractal dimension and uses techniques such as regional minima and morphological operations to isolate target regions.	[118]
Graph-Based Methods like Pyramidal Histogram of Oriented Gradients (PHOGs)	Graph Cut segmentation is used to segment flowers from complex backgrounds by treating the image as a graph. PHOGs capture the shape of flowers using histograms of edge orientations at multiple pyramid levels, effectively representing local and global shape features for image matching.	[119]
Piecewise Evolutionary Segmentation	The method divides time series data into smaller segments using evolutionary algorithms to find the optimal segmentation pattern, reducing dimensionality and retaining important features. This segmentation adapts dynamically using genetic algorithms to enhance classification and regression models by finding the best segmentation pattern for each problem.	[120]

.

Plant and organ-level feature extraction techniques play crucial roles in precision agriculture, plant phenotyping, and disease detection. Each technique provides unique insights into various aspects of plant morphology, health, and physiological processes. For instance, hyperspectral imaging and NIR imaging capture detailed spectral data and internal structures, making them ideal for identifying subtle changes in plant health and detecting early stress. Terrestrial laser scanning and point cloud data provide accurate 3D structural models, allowing for the quantification of plant organs and growth analysis. Microscopic approaches enable cellular-level studies, focusing on stomata behavior and other tissue characteristics that are essential for understanding physiological responses to environmental factors. In feature extraction, techniques such as morphological feature analysis and textural feature analysis are widely applied to assess disease impacts, leaf shape, and species classification. GLCMs and Wavelets excel in capturing textural and high-resolution details, making them valuable for segmenting complex patterns in leaves and stems. Moreover, advanced methods like graph-based techniques (PHOGs) and multi-fractal analysis effectively represent the complex geometry and multifractal nature of plant organs, enabling accurate identification and segmentation under diverse conditions.

Additionally, deep learning and optimization methods, such as CNNs, LSTM, and attention mechanisms, are integrated to automate feature extraction and improve the detection accuracy. These models can process multi-temporal data and selectively emphasize key features, enhancing the model’s robustness in real-world applications. Genetic algorithms and other optimization techniques (e.g., particle swarm optimization and Rider Cuckoo Search) further refine feature selection and segmentation, resulting in the more precise classification of diseases or growth stages. By utilizing these diverse methodologies, researchers can extract critical plant features, enabling early disease detection, species classification, and optimized phenotyping, and ultimately contributing to sustainable agricultural practices.

3. Technical Progress in 3D Point Cloud Classification in Remote Sensing

In this section, we review the technical advancements in 3D point cloud classification techniques, one of the most critical and popular techniques for plant phenotyping for both remote and proximal sensing, tracing the evolution from conventional methods to machine learning and deep learning approaches. Recognizing PointNet as a significant milestone in the deep learning-based 3D point cloud classification, we present comparative test results for an urban dataset with complex structures, including vegetation, to highlight the differences between conventional methods and PointNet.

3.1. Conventional Methods

In the early days of 3D point cloud processing, researchers developed traditional methods to analyze and understand these complex datasets. In addition to these histogram-based methods, researchers also explored curvature-based and normal-based approaches. The authors of [121] introduced curvature-based methods, which estimate curvature values at each point to describe the local shape. Similarly, normal-based methods estimate surface normals at each point to describe the local orientation. These approaches enable feature description, segmentation, and registration, and have been used in various applications. Finally, voxel-based methods [122] were developed to discretize the 3D space into voxels, allowing for the efficient processing and analysis of point clouds. These methods enable fast and efficient processing but may lose some accuracy due to the discretization process.

In addition to these local shape descriptor or voxel methods, in 2004, Belongie et al. [123] developed the Shape Context method, which uses 2D histograms to encode the spatial distribution of points. This approach has been successful in feature description and object recognition tasks, particularly in the presence of clutter and occlusions. Shape Context creates a 2D histogram of the angles and distances between neighboring points, capturing the underlying shape and structure of the point cloud. One of the pioneering methods in this field was the Point Feature Histogram (PFH) technique, introduced by Rusu et al. in 2008 [124,125]. The PFH encodes the spatial distribution of points in a local neighborhood, allowing for feature description and registration. The method creates a histogram of the angles and distances between neighboring points, capturing the underlying structure of the point cloud. This approach enables robust feature description and registration, even in the presence of noise and missing data. Building on this work, Rusu et al. [126] later developed the Fast Point Feature Histogram (FPFH) method, which improves upon the PFH by reducing computational complexity. The FPFH uses a smaller set of neighboring points, making it faster and more efficient while still maintaining accurate feature description and registration capabilities. Around the same time, Johnson and Hebert introduced the Spin Images method, which uses 2D histograms to encode the spatial distribution of points. This approach enables feature description and object recognition and has been widely used in various applications. Spin Images creates a 2D histogram of the angles and distances between neighboring points, capturing the underlying shape and structure of the point cloud.

Despite these limitations, researchers continued to develop new methods to process and analyze 3D point cloud data. However, these traditional methods were often cumbersome, computationally expensive, and limited in their ability to handle the complexity and variability of point cloud data. These approaches enabled feature description, segmentation, and registration, but were often sensitive to noise and missing data. These traditional methods laid the foundation for 3D point cloud processing, enabling feature description, segmentation, registration, and object recognition (

Table 5. Conventional 3D point cloud techniques.

Technique	Contribution	Limitation	Authors
Curvature-based	Estimates curvature values at each point for shape analysis and feature extraction	Can be sensitive to noise and high curvature variations	[121]
Normal based	Estimates surface normals at each point for shape analysis and feature extraction	Sensitive to noisy data and missing points	[107]
Spin Images	Encodes the spatial distribution of points using 2D histograms for feature description and object recognition	Sensitive to occlusions and requires precise point alignment	[127]
Voxel-based	Discretizes the 3D space into voxels for efficient processing	May lose accuracy due to discretization	[122]
PFH	Encodes the spatial distribution of points in a local neighborhood for feature description and registration	Computationally expensive and sensitive to noise	[124,125]
FPFH	An improvement over the PFH with reduced computational complexity while maintaining accurate feature description	Limited to small neighborhoods for fast computation	[126]
Shape Context	Uses 2D histograms to encode the spatial distribution, successful in cluttered environments	May require more computational resources for complex point clouds	[123]

). While they have been largely surpassed by deep learning techniques, they remain important milestones in the development of this field. Below is a list of conventional point cloud processing methods deemed as milestones of technical development.

3.2. Deep Learning Approaches

Volumetric CNNs, multiview CNNs, spectral CNNs, and feature-based DNNs are all methods of 3D point cloud processing that have improved upon traditional analytic methods. Volumetric CNNs [128] apply 3D convolutional neural networks to voxel shapes, allowing for faster processing times and improved accuracy compared to traditional methods. However, this method is limited by resolution due to data sparsity and computation costs. Multiview CNNs [129] render 3D point clouds into 2D images and then apply 2D convolutional neural networks, making it easier to analyze complex shapes. However, this method is limited to shape classification and retrieval tasks and may lose important 3D information. Spectral CNNs [130] utilize meshes, specifically suited for manifold meshes like organic objects, offering greater efficiency compared to traditional methods. Feature-based DNNs [131] convert 3D data into vectors by extracting traditional shape features, which can be used in fully connected networks for classification. However, this method is limited by the representation power of the extracted features and may not capture complex shapes. Compared to traditional analytic methods, these methods offer improved efficiency and accuracy, but may have limitations in terms of resolution, shape complexity, and feature representation (

Table 6. Conventional deep learning approaches.

Technique	Contributions	Limitations	Authors
Spectral CNNs	Spectral analysis for shape recognition on meshes. Robust to non-isometric deformations.	Restricted to manifold meshes like organic objects. Not easily extendable to non-isometric shapes like furniture.	[130]
Feature-based DNNs	Converts 3D data into vectors for classification. Fast and efficient processing.	Limited by the representation power of extracted features. Requires domain-specific feature engineering.	[131]
Volumetric CNNs	Three-dimensional convolutional neural networks for shape recognition. Real-time object recognition.	Constrained by resolution due to data sparsity and computation costs. Limited to small-scale 3D data.	[128]
Multiview CNNs	Two-dimensional convolutional neural networks for shape classification and retrieval. Efficient processing of large-scale 3D data.	Limited to shape classification and retrieval tasks. Requires multiple views of the 3D object.	[129]

).

Before the introduction of PointNet [132], the disorder and irregularity of point clouds made it impossible for deep learning technology to directly process point clouds. Early point cloud processing used hand-designed rules for feature extraction, which were time-consuming and limited in their ability to capture complex shapes. The introduction of PointNet and PointNet++ [133] allowed for the direct processing of point clouds, which improved the efficiency and accuracy of 3D point cloud processing. These methods offer significant advantages over traditional analytic methods, which relied on hand-designed rules and were limited in their ability to capture complex shapes. However, PointNet and PointNet++ require large amounts of training data and computational resources, which can be limitations. Overall, the choice of method depends on the specific application and the trade-off between efficiency, accuracy, and complexity. While traditional analytic methods are still useful for simple shapes and small datasets, deep learning methods offer improved efficiency and accuracy for complex shapes and large datasets.

3.3. Deep Learning Advances in 3D Point Cloud Processing

The development of segmentation models specifically tailored for 3D point cloud data has revolutionized the field of plant phenotyping, providing researchers with unprecedented capabilities for high-throughput and automated analyses of complex plant traits. Over the past decade, the focus has shifted from traditional machine learning algorithms to deep learning-based architectures, each designed to tackle unique challenges associated with plant structures, such as overlapping organs, variable point densities, and complex morphologies. Early models like PointNet (Qi et al., 2016) [132] and PointNet++ (Qi et al., 2017) [133] set the foundation by introducing point-wise feature extraction and hierarchical grouping strategies, which proved effective in controlled and less cluttered environments. However, these architectures struggled to capture intricate local and global patterns, making them less suitable for dense canopies and highly variable field conditions. To overcome these limitations, subsequent models such as DGCNN (Dynamic Graph CNN) (Yue et al., 2019) [134], PAConv (Xu et al., 2021) [135], and transformer-based architectures emerged, significantly enhancing the robustness and accuracy of segmentation in more challenging scenarios.

PointNet and PointNet++ employ multi-layer perceptron (MLP) networks to individually process each point, with PointNet++ extending this by incorporating local neighborhood information through hierarchical feature extraction. This design, while effective in relatively simple environments, fails to account for complex inter-point dependencies and cannot maintain context in large-scale plant datasets. On the other hand, DGCNN addresses these issues by employing a graph-based structure that dynamically updates neighboring points using EdgeConv, thereby enabling better modeling of local relationships. This makes DGCNN highly effective at capturing complex patterns, such as those found in dense maize or cotton fields. However, DGCNN’s reliance on local neighborhood aggregation makes it computationally intensive and susceptible to noise in low-density regions. Conversely, CurveNet (Xiang et al., 2021) [136] leverages curve-based features to capture connectivity and context in curved plant structures, improving its performance on datasets with prominent geometric features. Yet, its vulnerability to noisy data significantly limits its applicability in real-world agricultural settings.

The introduction of transformer-based models, namely, Point Transformer (PT) (Zhao et al., 2021) [137] and Stratified Transformer (ST) (Lai et al., 2022) [138] marked a substantial leap in segmentation performance by incorporating self-attention mechanisms for long-range feature aggregation. These architectures excel at handling complex inter-point interactions over large distances, making them ideal for segmenting crops with overlapping and intertwined organs, such as tomato or rose plants. By combining self-attention with multi-scale feature integration, the Stratified Transformer adapts to varying point densities and structural complexities, setting new benchmarks for accuracy in plant organ segmentation. However, these models come with the drawback of significantly higher computational costs, which can hinder their deployment in real-time applications and resource-constrained environments.

Recently, PAConv and Mask3D (Hou, et al., 2023) [139] have emerged as strong contenders for specific segmentation tasks. PAConv introduces a dynamic kernel construction method, where convolutional weights are adaptively generated based on the geometry of local point sets, allowing the model to generalize across different plant scales and structures. This adaptability makes PAConv particularly effective for plants with variable organ sizes, such as in cotton or potato datasets. However, the model’s complexity can lead to an increased training times and a need for more computational resources. Meanwhile, Mask3D extends the capabilities of Mask R-CNN to 3D data using sparse convolutions, making it one of the first methods to achieve robust instance segmentation on large-scale field datasets. Its performance on complex scenes, such as those captured using UAVs, highlights its potential for large-scale agricultural monitoring. Yet, Mask3D’s reliance on detailed pre-computed features limits its flexibility and increases preprocessing overhead, reducing its efficiency in dynamic environments.

The novel SP-LSCnet (Mertoğlu et al., 2024) [140] segmentation method, which integrates unsupervised clustering techniques with an adaptive classification network, offers a promising alternative by partitioning the input point cloud into superpoints, thereby reducing computational load and enabling more efficient segmentation of large-scale datasets. By employing attention-based modules to refine receptive fields, SP-LSCnet is able to dynamically adapt to varying organ geometries, making it particularly suited for scenarios with diverse plant species and complex structural variations. However, the reliance on a two-stage pipeline introduces challenges in maintaining superpoint homogeneity and mitigating misclassifications in regions with ambiguous geometry, such as thin petioles or small leaflets, which can degrade its performance in highly cluttered environments.

The development of segmentation models specifically tailored for 3D point cloud data has revolutionized the field of plant phenotyping, providing researchers with unprecedented capabilities for high-throughput and automated analyses of complex plant traits. Over the past decade, the focus has shifted from traditional machine learning algorithms to deep learning-based architectures, each designed to tackle unique challenges associated with plant structures, such as overlapping organs, variable point densities, and complex morphologies. Early models like PointNet [132] and PointNet++ [133] set the foundation by introducing point-wise feature extraction and hierarchical grouping strategies, which proved effective in controlled and less cluttered environments. However, these architectures struggled to capture intricate local and global patterns, making them less suitable for dense canopies and highly variable field conditions. To overcome these limitations, subsequent models such as DGCNN (Dynamic Graph CNN) [134], PAConv [135], and transformer-based architectures emerged, significantly enhancing the robustness and accuracy of segmentation in more challenging scenarios.

PointNet and PointNet++ employ multi-layer perceptron (MLP) networks to individually process each point, with PointNet++ extending this by incorporating local neighborhood information through hierarchical feature extraction. This design, while effective in relatively simple environments, fails to account for complex inter-point dependencies and cannot maintain context in large-scale plant datasets. On the other hand, DGCNN addresses these issues by employing a graph-based structure that dynamically updates neighboring points using EdgeConv, thereby enabling better modeling of local relationships. This makes DGCNN highly effective in capturing complex patterns, such as those found in dense maize or cotton fields. However, DGCNN’s reliance on local neighborhood aggregation makes it computationally intensive and susceptible to noise in low-density regions. Conversely, CurveNet [136] leverages curve-based features to capture connectivity and context in curved plant structures, improving its performance on datasets with prominent geometric features. Yet, its vulnerability to noisy data significantly limits its applicability in real-world agricultural settings.

The introduction of transformer-based models, namely, Point Transformer (PT) [137] and Stratified Transformer (ST) [138], marked a substantial leap in segmentation performance by incorporating self-attention mechanisms for long-range feature aggregation. These architectures excel at handling complex inter-point interactions over large distances, making them ideal for segmenting crops with overlapping and intertwined organs, such as tomato or rose plants. By combining self-attention with multi-scale feature integration, the Stratified Transformer adapts to varying point densities and structural complexities, setting new benchmarks for accuracy in plant organ segmentation. However, these models come with the drawback of significantly higher computational costs, which can hinder their deployment in real-time applications and resource-constrained environments.

Recently, PAConv [135] and Mask3D [139] have emerged as strong contenders for specific segmentation tasks. PAConv introduces a dynamic kernel construction method, where convolutional weights are adaptively generated based on the geometry of local point sets, allowing the model to generalize across different plant scales and structures. This adaptability makes PAConv particularly effective for plants with variable organ sizes, such as in cotton or potato datasets. However, the model’s complexity can lead to increased training times and a need for more computational resources. Meanwhile, Mask3D extends the capabilities of Mask R-CNN to 3D data using sparse convolutions, making it one of the first methods to achieve robust instance segmentation on large-scale field datasets. Its performance on complex scenes, such as those captured using UAVs, highlights its potential for large-scale agricultural monitoring. Yet, Mask3D’s reliance on detailed pre-computed features limits its flexibility and increases preprocessing overhead, reducing its efficiency in dynamic environments.

The novel SP-LSCnet [140] segmentation method, which integrates unsupervised clustering techniques with an adaptive classification network, offers a promising alternative by partitioning the input point cloud into superpoints, thereby reducing the computational load and enabling more efficient segmentation of large-scale datasets. By employing attention-based modules to refine receptive fields, SP-LSCnet is able to dynamically adapt to varying organ geometries, making it particularly suited for scenarios with diverse plant species and complex structural variations. However, the reliance on a two-stage pipeline introduces challenges in maintaining superpoint homogeneity and mitigating misclassifications in regions with ambiguous geometry, such as thin petioles or small leaflets, which can degrade its performance in highly cluttered environments.

The state-of-the-art segmentation models have evolved significantly, with each model offering unique strengths and limitations. While transformer-based architectures provide superior accuracy in complex environments, their high computational costs pose significant challenges. Conversely, models like PAConv and DGCNN offer a balanced approach for simpler plant structures but may falter in more intricate scenarios. The development of novel approaches such as SP-LSCnet demonstrates the potential of integrating unsupervised and attention-based methods to achieve robust segmentation across diverse agricultural contexts, highlighting the need for future research to focus on hybrid models that balance accuracy, computational efficiency, and adaptability to various plant phenotyping tasks. Contributions and limitations of advanced deep learning approaches are highlighted in

Table 7. Contributions and limitations of advanced deep learning approaches.

Technique	Contributions	Limitations	Reference
PointNet	Novel approach for point-wise feature extraction for 3D point cloud segmentation	Lacks local context information, making it less effective for complex plant structures and dense datasets	[132]
PointNet++	Extended PointNet by incorporating local neighborhood features through hierarchical grouping	Struggles with highly complex inter-point dependencies in large-scale datasets	[133]
Point–voxel CNN	Improves execution efficiency by pooling the advantages of voxels and points	May need additional fine-tuning for optimal performance	[141]
PointASNL	Uses nonlocal neural networks for robust point cloud processing	Can be computationally intensive and requires extensive memory	[142]
PAConv	Utilizes a dynamic kernel construction to adapt convolution weights based on the local geometry	High complexity and increased training times	[135]
DGCNN	Uses a graph-based approach with dynamic EdgeConv operations for local neighborhood relationship modeling	Computationally intensive and sensitive to noise in low-density regions	[134]
CurveNet	Leverages curve-based features to capture connectivity and spatial context in curved plant structures	Vulnerable to noise, reducing its effectiveness in real-world settings	[136]
FatNet	Feature-attentive network for 3D point cloud processing	Can be sensitive to overfitting due to its high focus on features	[143]
POEM	Reduces storage and computing costs with a 1-bit fully connected layer (Bi-FC)	Potential loss of detail due to compression	[144]
LatticeNet	Novel approach for 3D semantic segmentation from raw point clouds	May require significant computational resources for segmentation	[145]
Point Transformer	Introduces self-attention for long-range feature aggregation, excelling in complex organ segmentation	The high computational cost limits real-time applications	[137]
G-PCC++	A KNN-based linear interpolation for geometry restoration, a KNN-based Gaussian distance weighted mapping for attribute enhancement.	Computational complexity, which might limit resource-constrained environments	[146]
Stratified Transformer	Extends Mask R-CNN to 3D data using sparse convolutions, enabling robust instance segmentation of large-scale datasets	Relies on pre-computed features, increasing the preprocessing time and reducing flexibility	[138]
Mask3D	Extends Mask R-CNN to 3D data using sparse convolutions, enabling robust instance segmentation of large-scale datasets	Relies on pre-computed features, increasing the preprocessing time and reducing flexibility	[139]
FNeVR	Enhances facial details for image rendering via neural volume rendering	Specific to facial applications and might not be generalizable	[147]
SP-LSCnet	Combines unsupervised clustering and an adaptive network for efficient segmentation using superpoints	Two-stage processing may lead to misclassification in geometrically ambiguous regions	[140]

.

3.4. Progress in Performance (Complexity/Data Size)

The performance of point cloud processing has dramatically improved with the advent of neural network-based methods, marking a significant leap over conventional techniques. Earlier methods, prior to 2010, were marked by several constraints that limited their efficiency and adaptability. These conventional approaches were often slow, relying on hand-crafted features and rule-based algorithms. These properties made it difficult to handle large-scale point clouds or complex scenes efficiently. Additionally, their limited ability to adapt to new data or scenarios made them less flexible, as they often required manual tuning to work effectively. Neural network-based methods, introduced after 2010, represent a major advancement in the field. As mentioned earlier, PointNet, developed in 2017, was a game-changer because it allowed direct processing of point clouds without needing voxelization or rasterization, which conventional methods often required. This new approach achieved state-of-the-art results in tasks like classification, segmentation, and semantic parsing, demonstrating significant improvements in processing speed and efficiency. PointNet could handle large-scale point clouds more efficiently, opening new possibilities for applications like autonomous vehicles and robotics. Following the success of PointNet, PointNet++ was introduced in 2018. It improved upon the original design by incorporating hierarchical feature learning and multi-scale aggregation, leading to better performance in challenging datasets and complex scenes. This approach also proved more efficient in handling large-scale point clouds, making it suitable for a wider range of applications.

Recent neural network-based methods, developed from 2020 onwards, have introduced even more sophisticated techniques. These newer approaches leverage attention mechanisms, graph neural networks, and transformer architectures, further enhancing performance and efficiency. They allow for real-time processing, which is crucial for applications in autonomous vehicles, robotics, and mobile devices. Additionally, these advanced methods are adaptable and can learn from data, allowing them to adjust to new scenarios without the rigid constraints of conventional methods. Neural network-based methods offer several advantages over traditional approaches. They are significantly faster, achieving real-time performance in some cases, and demonstrate much higher accuracy, consistently delivering state-of-the-art results across various point cloud processing tasks. Their ability to scale and adapt to large datasets and complex scenes makes them highly versatile, while their adaptability to new data and scenarios reflects a key shift in the field’s approach to processing point clouds. These advancements represent a new era of efficiency, accuracy, and scalability in point cloud processing.

Table 8. Comparison of the processing time, frame rate, data size, and storage requirements for 3D point cloud methods.

Method	Processing Time (ms)	FrameRate (FPS)	DataSize (MB)	StorageRequirements (MB/min)
Conventional (before 2010)	100–500	1–5	100–500	1000–5000
PointNet (2017)	10–30	30–100	10–50	100–500
PointNet++ (2018)	5–15	60–200	5–20	50–200
Recent Neural Network-Based Methods (2020 and later)	1–5	200–500	1–10	10–50

shows the significant improvements in speed and data size that have been achieved with the introduction of neural network-based methods. The processing time has been reduced by a factor of 10–50, the frame rate has increased by a factor of 5–20, the data size has been reduced by a factor of 5–10, and the storage requirements have decreased by a factor of 10–50.

Note the following:

• Processing Time—the time it takes to process a single point cloud frame.
• Frame Rate—the number of point cloud frames that can be processed per second.
• Data Size—the size of a single point cloud frame.
• Storage Requirements—the amount of storage needed to store one minute of point cloud data.

3.5. Exploring Point Cloud Segmentation via PointNet and SVM

The main difference between PointNet and other neural networks for 3D point cloud processing is its ability to directly handle point clouds without the need for transformation into regular 3D voxel grids or collections of images [133]. Additionally, PointNet is designed to respect the permutation invariance of points in the input and provides a unified architecture for various applications such as object classification, part segmentation, and scene semantic parsing. Permutation invariance of points in the input refers to the fact that the order of the points in a point cloud does not affect the outcome of the neural network. In other words, the network is invariant to the permutation of the input points. Point clouds are typically represented as a set of 3D points, and the order of these points is arbitrary. The same point cloud can be represented in different ways by permuting the order of the points. A neural network that is permutation invariant can process point clouds in any order and produce the same output. PointNet achieves permutation invariance using symmetric functions such as max pooling that aggregate information from all points in the point cloud. This allows the network to capture global information about the point cloud, regardless of the order of the points. Permutation invariance is an important property for point cloud processing, as it allows the network to focus on the geometric structure of the point cloud, rather than the specific order of the points. This makes the network more robust and able to generalize better to new point clouds.

In this section, we compare a conventional method, Support Vector Machine (SVM), and PointNet for 3D point cloud processing capability. This literature review explores the application of 3D deep learning techniques and SVM methods for segmenting point cloud data in urban environments, with a focus on a case study in Voorst, the Netherlands, utilizing the Actueel Hoogtebestand Nederland (AHN) dataset [148,149]. The AHN dataset, with over 32 million point records, provides a comprehensive representation of the terrain, simplified into 16 different class labels, from which five labels are used for analysis. The study adopts a systematic approach outlined in fundamental steps, including data curation, labeling, feature selection, sub-tiling, and normalization.

3.5.1. Data Processing

Normalization and outlier removal are essential preprocessing steps in handling point cloud data to ensure consistency and eliminate noise. Normalization involves scaling the point cloud data to a standardized range, typically between zero and one, to facilitate uniform processing across different datasets. Outlier removal aims to identify and discard data points that deviate significantly from the rest of the dataset, often indicating errors or anomalies in data acquisition. Downsampling is a technique used to reduce the size of point cloud data while preserving its essential features. This process is crucial for managing large datasets and improving computational efficiency in subsequent analysis tasks. One common method of down sampling is voxel downsampling [150], where points are grouped into voxels, and each occupied voxel generates one point by averaging all points within it. This method effectively reduces the number of points while maintaining the overall structure and characteristics of the dataset.

Weighted random sampling is a downsampling technique that prioritizes certain data points based on their importance or relevance to the task at hand. In the context of point cloud data, weighted random sampling involves computing the weight distribution of unique labels present in the dataset. These weights are then used to guide the random downsampling process, ensuring that points from different classes are represented proportionally in the downsampled dataset. This approach helps maintain the class balance and prevent bias in subsequent analysis or modeling tasks. Then, noise and data augmentation techniques are employed to enhance the robustness and generalization capability of models trained on point cloud data. These techniques involve introducing variations such as translation, scaling, and random rotations to the data to simulate real-world conditions and increase the diversity of the training set. By exposing the model to augmented data, it becomes more resilient to noise and variations encountered during inference.

3.5.2. Methodology

The approach taken in this project for the segmentation of outdoor 3D point clouds involves a multi-step process. First, the study begins with meticulous preprocessing of high-quality airborne LIDAR data obtained from Actueel Hoogtebestand Nederland [149]. This preprocessing step is essential for cleaning and organizing the raw point cloud data before segmentation. It involves tasks such as noise removal, outlier detection, and data normalization to ensure the data’s quality and consistency.

First, the study employs a conventional machine learning approach for semantic segmentation, which involves the integration of Fast Point Feature Histograms (FPFHs) (Rusu 2009) and Support Vector Machines (SVMs) (Figure 3). FPFHs are utilized for their precision in extracting local geometric features from the point cloud data. SVMs are chosen for their classification prowess and robustness against overfitting, complementing the FPFH-SVM framework. While FPFHs process data to generate histograms of point cloud data based on neighboring data with various features, SVMs cluster them into different classes, thus segmenting 3D point cloud data.

In contrast to the machine learning approach, this study also explores the advanced PointNet deep learning framework for semantic segmentation. PointNet offers an end-to-end solution for processing raw point cloud data directly, eliminating the need for handcrafted features. Its neural architecture autonomously identifies and learns representative features from the multi-dimensional nature of LIDAR data, making it exceptionally suitable for semantic segmentation tasks. Both the machine learning and PointNet approaches are evaluated based on their accuracy, computational efficiency, and scalability. The Focal Loss function [151], also known as the Modified Categorical Cross Entropy Loss, is a variation of the standard cross-entropy loss commonly used in classification tasks and is employed in PointNet architecture. It is particularly effective in addressing class imbalance problems, where certain classes in the dataset are heavily outnumbered by others. The focal loss function below is derived from the standard categorical cross-entropy loss but introduces a modulating factor to down-weight the loss assigned to well-classified examples

$$CCE(Sn,Yn)=-{\alpha }_{y_n}\log \left({\displaystyle \frac{{\mathrm{e}}^{s_n}}{\sum _{C=1}^m{\mathrm{e}}^{\mathrm{S}n}}}\right)1\left(y_n\right)$$

(1)

where Sn represents the predicted class score, y_n represents the true class label, ${\alpha }_{y_n}$ represents the class weight for the true class, and 1(y_n) is the indicator function. In the context of segmentation tasks, such as segmenting point clouds using PointNet, the Focal Loss function plays a crucial role in addressing the imbalance in class distributions. Point cloud data often contain classes with varying frequencies, such as rare objects or background clutter. By assigning higher weights to underrepresented classes, the Focal Loss function helps the model focus more on learning from these minority classes, improving the overall segmentation performance. By incorporating the Focal Loss function into the training process of models like PointNet, researchers have observed significant improvements in segmentation accuracy, especially for datasets with imbalanced class distributions. The ability to effectively handle a class imbalance ensures that the model learns to differentiate between classes more effectively, leading to more accurate and reliable segmentation results. Focal Loss incorporation leads to improved segmentation accuracy and robustness, making it a valuable enhancement in the field of point cloud analysis and semantic segmentation.

3.5.3. Results

In our experiment, PointNet shows significant discrepancies in its ability to classify different classes (Figure 4). Notably, it performs well in identifying ‘Ground’ and ‘Vegetation’, ‘Buildings’ classes, with a notable confusion between ‘Vegetation’ and other classes (

Table 9. Comparison of deep learning (PointNet) and machine learning (SVM) models’ performance across class labels.

Metric	Buildings	Ground	Vegetation	Water	Unclassified	Overall Accuracy
Precision (PointNet)	Poor	Good	Good	Poor	Poor	78%
Precision (SVM)	Poor	High	High	Poor	Poor	88%
Recall (PointNet)	Very low	High	Moderate	Very low	-	-
Recall (SVM)	Moderate	High	High	-	-	-
F1 score (PointNet)	Weakness	Moderate	Moderate	Weakness	-	-
F1 score (SVM)	Weakness	High	High	-	-	-

). In addition, it struggles with the ‘Unclassified’ and ‘Water’ classes, showing almost no ability to identify them correctly (Figure 5). Whereas SVM with FPFH displays a strong performance on ‘Ground’ and ‘Vegetation’ classes similar to PointNet, it struggles with ‘Unclassified’, ‘Buildings’, and ‘Water’. Moreover, the resolution of the results also deteriorated. The confusion is notably less for ‘Ground’ and ‘Vegetation’, suggesting better discrimination between these and other classes (Figure 5). For a comparative analysis with traditional machine learning models, we have chosen to evaluate it based on the standard metrics shown in Figure 5, such as precision, recall, F1 score, and overall accuracy (see Table 9 for a comparison). This comparative analysis of SVM with FPFH and PointNet for the semantic segmentation of LIDAR point clouds reveals important insights for practical geospatial applications. While SVM with FPFH achieves higher accuracy in recognizing ‘Ground’ and ‘Vegetation’, it struggles with other classes such as ‘Buildings’ and ‘Water’, indicating challenges with a class imbalance. The diagram in Figure 6 shows the training and testing loss and accuracy for the PointNet architecture, which achieves approximately 81% accuracy.

Conversely, PointNet demonstrates broader class recognition but lower overall accuracy, suggesting its potential for applications requiring diverse class identification. However, both models face challenges in generalizing across diverse classes, highlighting the need for further research to enhance their performance. The impact of the class imbalance is evident in the significantly lower performance on minority classes like ‘Buildings’ and ‘Water’, suggesting the necessity of techniques such as SMOTE [152] or adjusting class weights for improved performance. While FPFH features excel in distinguishing ‘Ground’ and ‘Vegetation’, they may be less effective for complex structures like buildings or sparse classes like water. Exploring additional features or alternative feature extraction techniques could enhance the classification capabilities. For deployment, SVM with FPFH appears robust for applications requiring high accuracy in specific classes, while the computational efficiency of PointNet could be advantageous for real-time processing, despite its lower accuracy. One key observation from above experiments is that, the red, green, blue features along with the x, y, x features of the point cloud helped to segment the trees from the cone shaped buildings. Future research directions include adding more features and exploring hybrid approaches combining the strengths of both models and addressing the class imbalance through synthetic data generation or modified training objectives.

4. Plant Feature Extraction via 2D and 3D Data Modalities

In this section, we review 2D image and 3D point cloud-based plant feature extraction techniques, providing an in-depth analysis of recent technical advancements. Unlike industrial objects with standardized shapes, the complexity of plant structures poses significant challenges for extracting the comprehensive features necessary for operations such as harvesting. Therefore, more advanced data processing techniques are required to address these challenges and improve feature extraction for practical applications.

4.1. Two-Dimensional (2D) Image-Based Plants Organ-Level Feature Extraction

Plant image processing aims to enhance image quality, extract meaningful information, or transform images into different formats by performing specific operations on the original plant images. Typically, 2D RGB images are often used as the input, and various processing techniques, such as color conversion to convert them into RGB images or into multiple other formats such as HSL, HSV or image segmentation, and shape detection, are applied for accurate object recognition. This process is crucial in autonomous and precision agriculture, where images taken from agricultural fields must be transformed into useful information. This information can be then integrated with machine-learning-based methods for autonomous feature extraction and segmentation for application-specific requirements.

When harvesting plants, it is crucial to distinguish between ripe and unripe fruits or vegetables before picking them from the plant. Such decision-making tasks are usually based on human perception and judgment. Emulating this behavior is essential for autonomous agricultural robots to accomplish the allotted task successfully. Computer vision and image processing are vital in emulating the above-described behavior. Various image processing techniques have been developed to accomplish these tasks. These techniques include image segmentation, edge detection, blob analysis, color conversion, feature extraction, and various algorithms such as the Euclidean clustering algorithm, the Random Sample Consensus (RANSAC) algorithm, methods based on deep learning, and neural networks. These techniques and their agricultural applications are reviewed below.

4.1.1. Color Space Analysis and Conversion

Color space is essentially a mathematical model for describing the wide range of colors captured by the camera. Color space analysis is application-oriented and is classified into two categories, device-dependent (RGB, HSV, HSL, HIS, YUV, YIQ, CMYK, YCbCr, and YPbPr) and device-independent (XYZ and CIELAB), as described by Velastegui et al. [153].

For example, in the article by Al-Mashhadani et al. [154], OpenCV was adopted for color conversion (RGB to HSV) to detect fruits and distinguish between ripe and unripe ones. The steps involved in fruit detection included preprocessing techniques such as image resizing and bicubic interpolation, and image segmentation achieved using thresholding and histogram analysis. The process also involved color conversion and feature extraction based on color and shape, as well as training and classifying algorithms. This work utilized the HSV color space, with hue values used to establish the thresholding that distinguishes between ripe and transitioning (unripe) fruits. By combining these techniques, a model was developed to differentiate between ripe and unripe tomatoes, as described by Al-Mashhadani et al. [154]. In the study by Rao et al. [155], a color subtraction module (subtraction value of red and blue) was applied to distinguish cotton fruits from the rest of the plants, based on the observation that the RGB values of the cotton fruits differ from that of the other parts of the plants.

Blob (binary large object) analysis exploits color and shape information for object detection, as described in the article by Ganesan et al. [156]. In the research performed by Dewi et al. [157,158], a BLOB analysis was utilized to group pixels with similar brightness and color and detect the fruits.

4.1.2. Shape Analysis and Morphological Features

Shape analysis within an image extracts meaningful information and features by identifying, describing, and quantifying the geometric characteristics of objects in an image. This process is crucial in various applications such as object recognition, image retrieval, and classification. Various techniques utilized in agriculture for the shape analysis of crops are described in this section. For example, edge detection methods such as Sobel, Prewitt, Canny, and the Laplacian edge detection methods are described in the papers by Dewi et al. [157,158], Bulanon et al. [159], and Fernandes et al. [160], which detect the edges of objects or features within an image to define their boundaries. Extracting the contour or boundary of objects requires techniques like the Douglas–Peucker method used by Fernandes et al. [160], the split and merge algorithm used in the study by Zhang et al. [161], or the Probabilistic Hough Transform algorithm applied by Ge et al. [162].

4.1.3. Segmentation Techniques

Image segmentation is the process of performing a set of operations, such as deleting, adding, or enhancing certain parts of an image, with the aim of separating and distinguishing the desired area of interest, as described by Kodagali et al. [163]. In most cases, image segmentation is the very first image processing operation carried out during the process of object recognition. Thresholding and histogram analysis segmentation techniques were utilized to detect targets by Dewi et al. [157]. In the research work conducted by Vasconcelos et al. [164], semi-autonomous annotation of the acquired dataset was performed. Different segmentation networks were utilized to train the acquired dataset, and benchmark results were obtained. Some researchers aimed to develop a synthetic dataset to advance computer vision performance in agricultural robotics. A synthetic dataset was created to generate an exhaustive image dataset, which was difficult to obtain using the manual data acquisition process. Barth et al. [165] used semantic segmentation to detect objects and compare the similarity between synthetic and original images.

In the published study by Jiao et al. [166], the acquired image is transformed into Lab color space, and the K-means segmentation algorithm is utilized. The outlines of apples are extracted using erosion and dilation techniques (morphological processes). The minimum distance of internal points from the outline is calculated by applying a fast algorithm. The maximum distance among the points obtained above represents the center of the apple. Subsequently, radii are calculated, successfully finishing the localization of the overlapped apple quickly.

4.1.4. Deep Learning for 2D Image-Based Feature Extraction

Single-stage deep learning combined with other image-processing techniques have been utilized to distinguish between ripe and unripe fruits in complex environments. In the research by Wang et al. [167], the authors added the ECA attention mechanism to the backbone network of YOLOv8 to enhance its performance. Focal-EIOU loss was applied to the loss function, which aided in balancing easy- and difficult-to-classify samples. A similar method has been applied by Bu et al. [168], employing the YOLOv4 detection algorithm. This paper resulted in an acceptable processing speed and target detection. After observing the results, the authors noted that errors in detection may have been caused by the small bounding boxes. In the future, preprocessing of the images shall be performed by scaling and cropping the acquired images. Also, the prediction results are often based on the number of training images used to train the model. The image preprocessing method based on YOLOv5, involving HSV color space transformation, thresholding, edge detection using the Canny operator, and a combination of image processing and deep learning methods, was performed to determine the exact cutting point on the stem in the article by Miao et al. [169]. Another research team, Yin et al. [170], also utilized YOLOv4 tiny to detect citrus fruits and fruit navel point locations. Onishi et al. [171] utilized a single deep neural network SSD based on CNN for feature extraction due to its superiority in speed and accuracy compared to other single deep neural network methods. The images were captured from below to reduce occlusion by leaves, branches, and other fruits. However, fruits on the edge of the image or at a farther distance from the camera could not be detected using the above method.

Semantic segmentation using DeepLabV3+ was employed to detect strawberries and plant features in the study conducted by Fujinaga et al. [172]. This preprocessing did not aid in distinguishing ripe and unripe fruits. Other plant features were extracted by applying classical image processing, which essentially involves the color (conversion of the RGB color space to the HSV color space) and shape of the plant features. A similar approach was adopted to distinguish litchi from plants by Peng et al. [173]. A convolutional neural network was utilized for image processing. The DeepLabV3+ model with Atrous Spatial Pyramid Pooling (ASPP) and the Xception feature extraction network were adopted for image segmentation. Fujinaga et al. [174] utilized ResNet-50 as the backbone network for DeepLabV3+ semantic segmentation. The study also utilized ASPP, aiding in feature extraction. The study aimed to detect and distinguish between parts of the plant but did not perform any research on distinguishing between ripe and unripe fruits. The results obtained from the semantic segmentation were further processed to reduce noise (remove noise based on the area size) and false positives (morphological operations, skeletonization [55] feature extraction methods popular in 3D point clouds, and dilation—only for trusses). Another study by Giang et al. [175] fused depth data from an RGB-D camera for the fast detection of tomato suckers in real time by training a two-stage neural network.

The study performed by Kalampokas et al. [176] evaluated different CNN models to compare the segmentation accuracy of each model. Another study by Ge et al. [162] utilized deep CNN (Mask R-CNN) to detect both bounding boxes and masks for each instance, for instance, the segmentation of images to detect strawberries. Localization accuracy was improved by implementing density-based point clustering that removed noise points in 3D point clouds. The study conducted by Lin et al. [177] utilized a fine-tuning strategy to train a fully convolutional neural network (FCN) to segment guava and its branches. Further processing was performed to count the number of detected fruits and to estimate the fruit pose. In the paper by Onishi et al. [170], instance segmentation was achieved by employing Mask R-CNN with ResNet-101 as the backbone and the original model of MS COCO. Principal Component Analysis (PCA) was employed to estimate the orientation of fruit and stem masks. Another study by Hussain et al. [178] utilized DeepLabV3 for semantic segmentation to detect litchi and twigs. Later, morphological processes were applied to detect fruit-bearing branches. The picking position of the fruit-bearing branch was obtained using spatial clustering (revised density-based clustering of applications with noise (DBSCAN)) and Principal Component Analysis (PCA) straight-line fitting.

Table 10. Contributions and limitations of 2D image-based feature extraction/segmentation methods.

Technique	Contributions	Limitations	Authors
Image segmentation	Divides the image into background and foreground region of interests		[158]
Edge detection	Sobel edge detection	Slow process and results in a noisy image
	Prewitt edge detection: faster process	Suitable for a high-contrast and low-noise image
	Canny edge detection: based on the Laplacian algorithm and is insensitive to noise	Encompasses a multistage algorithm, and hence results in longer processing time
Blob analysis (binary large object analysis)	The position of the target can be detected.
Semantic Segmentation	Four different deep learning segmentation methods; semi-automatic annotation process	Requires large amount of annotated image datasets; exhaustive manual annotation process	[164]
Color Conversion	HSV color space: thresholding is utilized to determine ripe and turning fruit colors. Hue values for ripe tomatoes are 0–10, the saturation value is 170–256, and the hue value for turning fruit is 11–20 and the saturation value is 150–256.	When ripe and turning fruits are grouped together, the technique can identify the fruit but cannot distinguish between both fruits. Hence, its counts the fruits as a single fruit than a couple of fruits (multiple fruits). Focal bottleneck transformer network (FBoT-Net) for green apple	[154] [178]
Image Segmentation	Segments the image based on HSV values
	Deep learning based 2D image segmentation
Color Conversion	Identifies ripe strawberries based on the hue (HSV format).		[179]
Single-stage deep learning (YOLOv8)	Improves the performance by combining ECA with YOLO. Results in enhanced identification of ripe strawberries in complex environments (forelight, backlight, and occlusion).	Limitation in Context understanding and struggle in detecting small objects compared to latest vrsions.
Color Analysis	The required target has different color values compared to other parts of the plants.	Requires external memory for image processing	[155]
SSD	High processing speed and accuracy	Difficulty detecting fruits at the edge of the frame and farther away from the frame	[171]
K-means Segmentation	Faster fruit detection	Larger images required longer running time. Occlusions lead to inaccurate results.	[166]
RGB-D Image Segmentation	Real-time fruit detection (138 frames per second)	Slightly noisy results. The neural network structure is complicated and the training process is longer.	[177]
	Low-cost sensor	Guava fruits were easier to detect than the branch. Some of the unsuccessful results were due to sunlight.	[176]

summarizes the contributions and limitations of various 2D-based feature extraction techniques.

4.2. Three-Dimensional (3D) Point Cloud-Based Plant Feature Extraction

In this section, we advance to point cloud data processing for plant structures, focusing on the complexities involved in 3D point cloud processing. Major challenges in processing 3D plant data include shape ambiguity and occlusion, which complicate the accurate representation and analysis of plant structures. There are several techniques and data processing pipelines for 3D image acquisition of plants. Briefly, the data acquisition techniques can be classified as active and passive approaches [180]. In an active approach, the sensors emit their own energy towards the object and detect the reflected signals. The characteristic of the object is formed based on the time delay and intensity of the reflected signal. A common example is a LiDAR (Light Detection and Ranging) system that emits laser pulses and detects reflected pulses to calculate the distance to the object. Another example is the work by Um [181], introducing a novel method to calculate the 3D depth and the surface angle of an object at the same time. The enabling technology is the multiple intensity differentiation method. Single camera-based 3D depth technology demonstrated a real-time feature extracting and gesture processing as well. On the other hand, passive approaches do not use specific structured energy but rely on the light that is reflected, emitted, or scattered by objects to form an image [182]. Both approaches and sensing systems have their own advantages and disadvantages. Briefly, active sensors are relatively accurate and more expensive than passive systems. Data captured by both methods need to go through a series of processes to develop intermediate data products such as 3D point clouds, depth maps, and voxels from which 3D structures of the plants or canopy can be generated. Harandi et al. [180] and Paulus [68] conducted a detailed comparison of these 3D imaging processes/techniques with their advantages, disadvantages, and potential uses for field conditions or controlled environments. In summary, 3D laser scanners and imagery are the most commonly used data collection techniques for 3D plant phenotyping. LiDAR operates based on time of flight (ToF), where it determines the distance to an object by measuring the time it takes for a laser pulse to travel to the object and back. The distance is calculated using the speed of light multiplied by half of the time-of-flight duration. Some of the key advantages include faster data collection, light independence, high precision, and higher resolution data.

Panjvani et al. [183] designed a low-cost LiDAR-based framework (LiDARPheno) and developed a 3D model of plants to extract leaf traits such as length, width, and area. Patel et al. [184] utilized 3D point cloud data of sorghum plants obtained from LiDAR to test deep learning models for extracting phenotypic traits such as plant height, plant crown diameter, plant compactness, stem diameter, panicle length, and panicle width. Zhu et al. [185] explored a large-scale phenotyping approach using backpack LiDAR to acquire a 3D point cloud of a field and extract phenotypic features. Paulus et al. [186] tested the potential use of 3D laser scanners to generate point cloud data for grapevines and wheat. They were able to classify grapevines and leaves with 98% accuracy. Additionally, wheat ears were separated from other organs, and the ear volume was calculated and correlated with the ear weight, kernel number, and weight. Imaging techniques utilize depth cameras, Structure from Motion (SfM), multi-view stereo, or a combination to generate 3D architectures of plants. Han et al. [187] utilized a depth camera, Azure Kinect, mounted directly above melon seedlings to gather point clouds and proposed a novel point cloud segmentation framework for measuring leaf area.

The Microsoft Azure Kinect DK combines a high-precision time-of-flight (ToF) depth sensor with a 12 MP RGB color camera, a 7-microphone array for audio capture, and an Inertial Measurement Unit (IMU), including an accelerometer and gyroscope, all integrated into one device. It offers a depth resolution up to 1024 × 1024 pixels with a field of view of 120° × 120° and a depth range from 0.5 to 5 m. The 4K color camera captures at 3840 × 2160 pixels with a 75° × 65° field of view, supporting up to 30 frames per second. Intel RealSense depth cameras are also used in plant phenotyping [188]. Common models of Intel RealSense depth cameras include D415 and D435. D415 features a field of view of 69.4° × 42.5° × 77°, a depth resolution up to 1280 × 720 pixels, and a global shutter, making it ideal for capturing detailed scans of small objects or fast-moving features. In contrast, D435 offers a wider field of view at 91.2° × 65.5° × 100.6°, the same resolution, and a rolling shutter, which suits a broader scene understanding and provides a comprehensive view for larger environment reconstruction.

SfM is a photogrammetry and computer vision technique that uses a series of overlapping 2D images captured from multiple angles to reconstruct the 3D structure of an object [186,189,190]. Ghahremani et al. [5] used PhenoGreen rotary tables for capturing high-resolution multi-view images of grapevine and brassica plants. Collected images were processed using SfM algorithms to generate 3D point clouds. Rossi et al. [191] evaluated the performance of a low-cost phenotyping platform consisting of an RGB camera, rotating plates, and a sliding system. An RGB camera was installed on a sliding system consisting of two one-meter-long aluminum tubes, and plants were placed on the rotating plates. The images captured were further processed to generate 3D plants. A novel segmentation algorithm for the automatic collection of data capturing plant structural features such as height, angle, and area was proposed [192].

Gao et al. [193] performed a detailed study on the development of imaging systems for plant 3D modeling and compared factors such as the impacts of the number of cameras used to collect images, the number of images captured, the use of color checkboards during image capture, and the system’s stability. It was found that images captured using more than one camera resulted in better 3D models of plants. Multiple cameras at different heights were also used in systems developed by Tanabata et al. [194] and later adopted by Pongpiyapaiboon et al. [195]. Using a rotating system, the study found that at least 60 images are required to generate reliable 3D models. Additionally, increasing the number of images improves the model but comes with increased computational costs. Using a colored checkboard during data collection also improves the quality of 3D reconstruction. The system’s stability was higher when the camera was rotated compared to rotating the plants, especially when detecting non-rigid parts of the plants.

The evolution of 3D point cloud-based plant feature extraction has been significant as technology matures in fields like autonomous vehicles, robotics, and other automation applications. However, one of the primary challenges in plant 3D point cloud technology is incomplete data acquisition. Despite advancements in scanning technology, certain areas remain challenging or impossible to reach, resulting in gaps within the dataset and complicating the accurate analysis and understanding of plant structures. Additionally, extracting phenotypic parameters from point clouds presents challenges due to inaccuracies introduced by environmental factors and plant occlusions, which can adversely affect data accuracy. This is similar to measuring the size of a leaf while it is blown by the wind, complicating reliable parameter extraction, which in turn impacts plant breeding and crop management. Furthermore, point clouds exhibit rotation invariance, complicating the ascertainment of the point cloud’s orientation, akin to assembling a 3D puzzle without reference points. This poses a challenge in aligning point clouds with real-world coordinates, which is essential for precise analyses and measurements. The disordered and irregular nature of point clouds further complicates feature extraction, making it challenging to identify specific features, such as leaves or stems, which can result in inaccurate analyses and measurements.

Lastly, noise in data presents a significant challenge. Sensor errors and environmental factors can introduce noise, which complicates the extraction of accurate information. This noise can lead to incorrect analysis and measurement, with potential consequences for plant breeding and crop management. Gomathi et al. [196] introduced the Point Sampling Method to improve instance segmentation in point clouds, particularly for plant growth phenotyping in agriculture. Utilizing an occupancy grid representation and single symmetric function max pooling, they achieved semantic and leaf instance segmentation with remarkable accuracy, contributing to plant growth analysis and disease detection.

Continual semantic segmentation is essential for maintaining accurate spatial annotations in dynamic environments. Roggiolani et al. [197] proposed a self-supervised pre-training approach for 3D leaf instance segmentation, reducing labeling efforts in agricultural plant phenotyping. Their method combines this approach with automatic post-processing to handle overlapping petioles, showing consistent improvements in segmentation accuracy. LatticeNet [145] also addresses the challenges of 3D semantic segmentation of plant point clouds. The study employed data augmentation techniques to enhance plant part segmentation, demonstrating significant improvements. This work focuses on addressing leaf crossover challenges, contributing to plant architecture reconstruction. Li et al. [198] introduced PlantNet, a dual-function deep learning network for semantic and instance segmentation in multiple plant species from point clouds. Innovative strategies like 3D Edge-Preserving Sampling (3DEPS) and Local Feature Extraction Operation (LFEO) modules enhanced performance, as validated on datasets of tobacco, tomato, and sorghum, indicating its potential impacts on plant phenotype extraction and intelligent agriculture. Additional studies, such as Weyler et al. [7], further advanced the field with methods for joint plant instance detection and leaf count estimation in agricultural fields using mobile robots. They utilized a single-stage object detection method to localize crops and weeds while estimating the leaf count and tested it on sugar beet fields, where it outperformed traditional methods like Mask R-CNN.

Masuda [199] presents a novel approach for estimating the leaf area in tomato plants grown in a sunlight-type plant factory. Utilizing RGB-D sensors and point cloud segmentation techniques, Masuda achieves promising results, with a relative error of about 20% in the leaf area estimation, thereby contributing significantly to automated cultivation management. This method emphasizes the application of 3D point cloud technology in plant phenotyping, enabling more precise management of plant growth in controlled environments. Ghahremani et al. [5] focus on a direct point cloud analysis using the RANSAC algorithm. Their robust methodology achieves high accuracy in measuring critical plant attributes, such as branch diameters and leaf angles, further advancing the utility of point cloud technology in agricultural research. Gu et al. [199] introduce a method to analyze cabbage plant phenotypes through 3D point cloud segmentation. The ASAP-PointNet model, tested in this study, demonstrates improved semantic segmentation accuracy, with high IoU scores. The DBSCAN algorithm is employed to enhance cabbage instance segmentation, providing significant contributions to the field of phenotypic analysis.

Imabuchi et al. [200] propose a combined method for semantic and volumetric 3D modeling from point cloud data in plant environments. This method enhances accuracy in radiation dose distribution modeling, particularly in decommissioning scenarios, by integrating 2D image-based deep learning with volumetric reconstruction. Gu et al. [199] also present a virtual design-based 3D reconstruction technique for wheat plants, addressing limitations in traditional methods. By combining point cloud data with optimization algorithms, the study offers an effective approach to model the wheat plant architecture, despite challenges in precise reconstruction due to the absence of organ templates. Wang et al. [201] develop a method for segmenting tomato plant stems and leaves using multi-view image sequences from a solar greenhouse, extracting six key phenotypic parameters. The study highlights both superior segmentation accuracy and computational efficiency, though it also acknowledges the need for further research to address challenges like canopy adhesion.

Giang et al. [10] propose a method for estimating sweet pepper leaf area using semantic 3D point clouds generated from RGB-D images. Their technique achieves high accuracy and efficiency, with strong correlations between the point cloud data and actual leaf area. However, challenges remain in fully capturing taller plants and varying point cloud resolutions, indicating that further validation across diverse conditions is necessary.

Table 11. Three-dimensional (3D) point cloud-based plant feature extraction: contributions and limitations.

Technique	Contributions	Limitations	Authors
Point Sampling Method	Improves instance segmentation in point clouds for plant growth phenotyping in agriculture	May require large datasets and computational resources	[196]
RANSAC Extension	Extends RANSAC for complex plant morphologies, achieving high accuracy in measuring plant attributes	May be sensitive to imaging noise and cluttered backgrounds	[5]
Self-Supervised Pre-Training	Reduces the labeling effort for 3D leaf instance segmentation in agricultural plant phenotyping	May not work as well in highly occluded environments	[197]
LatticeNet	Data augmentation to enhance plant part segmentation, contributing to the plant architecture reconstruction	May require significant computational resources for semantic segmentation	[145]
PlantNet	Dual-function deep learning network for semantic and instance segmentation in multiple plant species	May require additional validation for diverse plant species	[198]
Joint Plant Instance Detection	Method for joint plant instance detection and leaf count estimations in agricultural fields	May have limited scalability to other types of plants or environments	[7]
Leaf Area Estimation	Estimates the leaf area in tomato plants using RGB-D sensors and point cloud segmentation	Relative error of about 20%, indicating possible accuracy issues	[199]
Robust RANSAC	Robust method for the direct analysis of plant point cloud data using RANSAC, achieving high accuracy	May not work well in highly noisy or cluttered environments	[5]
Virtual Design-Based Reconstruction	A 3D reconstruction technique for wheat plants, integrating point cloud data with optimization algorithms	Challenges in precise reconstruction due to the lack of organ templates	[202]
ASAP-PointNet	Method for analyzing cabbage plant phenotypes using 3D point cloud segmentation	Requires complex preprocessing and training	[200]
Semantic and Volumetric 3D Modeling	Method for semantic and volumetric 3D modeling from point cloud data for accurate radiation dose distributions	Can be computationally expensive and requires significant memory	[203]
Tomato Plant Segmentation	Method for segmenting tomato plant stems and leaves, extracting phenotypic parameters	Challenges with canopy adhesion and under-segmentation	[201]
Sweet Pepper Leaf Area Estimation	Method to estimate the sweet pepper leaf area using semantic 3D point clouds generated from RGB-D images	Issues in fully capturing tall plants and varying point cloud resolutions	[10]

summarizes the contributions and limitations of various 3D point cloud-based feature extraction techniques.

5. Remote Sensing and Plant Phenotyping: Insights

In this section, we present our experience in plant feature extraction, encompassing the entire process from RGB data collection and point cloud generation to 3D point cloud segmentation. Our aim is to provide valuable insights for scholars engaged in the study of plant structure classification and segmentation. We begin by detailing our methods for RGB data acquisition for both remote sensing and proximal sensing, followed by an explanation of image labeling and model training for 2D image-based segmentation. Finally, we discuss our approach to 3D point cloud-based plant structure segmentation, sharing the lessons learned and best practices identified throughout the process.

5.1. Canopy-Level Remote Sensing Data Collection and Analysis

To demonstrate remote sensing data collection and processing in the field, we captured raw RGB images of a sorghum field in Driscoll, Texas, and converted them into 3D point cloud data for plant phenotyping. The raw images obtained from the UAV were processed using Agisoft Metashape Software (2.1.3) (Agisoft, LLC, St. Petersburg, Russia), which utilizes a Structure from Motion (SfM) algorithm. This process generated geospatial data products, including a digital surface model (DSM) that represents the surface elevation of ground objects [204], orthomosaic images, and finally a 3D point cloud of plants (Figure 7).

The ground sampling distance (GSD) for the orthomosaic images was 2 cm per pixel. For the data collection experiment in the sorghum field, we used a DJI Phantom 4 Real-Time Kinematics (RTK) (SZ DJI Technology Co. Ltd., Shenzhen, China) as the primary platform. This UAV is equipped with a 1-inch 20 MP (megapixel) CMOS (complementary metal oxide-semiconductor) sensor (FC6310R) that captures RGB image data. Additionally, the Phantom 4 RTK features a GPS module that stores geospatial information for RTK corrections. To further enhance the accuracy of the geospatial data products, a ground control point (GCP) was installed in the field. The flight height during data collection was maintained at 120 m above the ground, with a 75 percent forward and side overlap. As show in Figure 8, raw RGD data are processed to generate 3D point cloud of a sorghum field. Environmental conditions, including wind and rainfall, are critical factors for ensuring high-quality data collection, essential for generating precise 3D point clouds and accurately extracting phenotypic traits from each plant section. Despite significant technical advancements in UAV technology, data collection across multi-acre fields remains a labor-intensive and high-cost operation. Further automation of UAV operations, including streamlined setup and autonomous data acquisition, will be a critical challenge to address in future remote sensing research.

In the following section, we demonstrate how to generate 3D point clouds and process 3D point cloud data for plant phenotyping.

5.2. Plant-Level 3D Data Collection for Phenotyping

In this study, a 3D data collection system was developed to capture detailed images of tomato plants at the plant level for a precise 3D reconstruction and canopy height analysis. The setup, as depicted in Figure 9, integrated a RAUBAY 360° motorized turntable with a 6 W, 110 V, 50/60 Hz motor operating at 15 rpm, paired with a speed controller for precise rotational control. To facilitate image capture at consistent angles, a Canon EOS 1000D camera with a 10 MP sensor was mounted on a rotating arm, which allowed for incremental rotations at various predefined angles around the plant (Figure 9a). The turntable rotation and image capture were synchronized through a custom Python script that communicated with an Arduino microcontroller via the PySerial library.

This script controlled a sequential “stop-take-go” action, ensuring that the camera captured high-quality images at set intervals as the turntable moved. To optimize image quality and minimize shadows or reflections, a professional-grade photo booth setup (Figure 9b) was constructed using a 47″ × 39″ × 78″ adjustable LED lightbox. The controlled lighting environment provided uniform illumination around the plant, reducing unwanted noise and artifacts during the image processing stage. Additionally, a target marker sheet from Alex Elvis Badillo [206] was placed on the base of the turntable, serving as a reference for improved alignment and key point detection in the subsequent photogrammetry process using Agisoft Metashape software.

The image acquisition process began by positioning the potted tomato plant on the RAUBAY turntable inside the photo booth. The Arduino microcontroller controlled the motor to rotate in small, incremental steps (5–10 degrees), ensuring sufficient overlap between consecutive images. After each rotational step, the Python script sent a command to the camera to capture a high-resolution image. This “stop-take-go” sequence was repeated for a complete 360-degree view around the plant, resulting in 60 to 70 high-quality overlapping images. The use of this automated system significantly enhanced the precision and efficiency of the image capture process, ensuring consistency in image overlap, angle increments, and focus.

The collected images shown in Figure 9c were processed using the Structure from Motion (SfM) technique in Agisoft Metashape software to generate a dense 3D point cloud of a plant (Figure 9d). Metashape aligns the images by identifying key features and matching points across overlapping regions. Once the sparse point cloud was generated, the software reconstructed a detailed dense point cloud using depth information from the matched points. This dense point cloud was then further refined to create an accurate 3D representation of the plant structure.

5.3. Two-Dimensional (2D) Image-Based Organ Level Feture Extraction

This section highlights the initial findings and future directions for improving the 2D image-based plant feature extraction process, addressing both the contributions and limitations identified during the study. The 2D image-based plant feature extraction processes are based on the experimental data captured in the indoor tomato farm at Texas A&M University—AgriLife Center. First, the collected images were sorted into a training dataset with manual annotation, followed by plant organ segmentation via CNN model training.

5.3.1. Image Annotation and Segmentation

The dataset images were annotated for semantic segmentation, focusing on branches, stems, and suckers, using LabelStudio (1.14.0). Each image annotation took approximately 2–4 min. Figure 10a illustrates the annotated image, where green, purple, and red masked areas represent the stem, branches, and suckers, respectively.

Model training was conducted over 50 epochs using a total of 262 images for training and validation. The training utilized full plant images, making suckers visible when observed closely. Consequently, the accuracy of the results, as measured by the F1 confidence score, did not exceed 0.65 in our experiment. The main issue was the lack of a sufficient dataset for training. Typically, a model needs to be trained with over 1000 images to achieve a higher F1 score

5.3.2. Model Training and Implementation

The YOLO v8 [167]-based machine learning model was trained on a 64-bit Windows 11 Pro operating system with 16 GB RAM on an Intel i7-9750H CPU. The training program was developed in Python 3.11, leveraging PyTorch 2.2.1, TorchVision 0.17.1, CUDA 1.21, and YOLO v8. Figure 10b demonstrates the F1–confidence curve for the trained model. The data obtained from the annotation process were used to train the machine learning model. Training was implemented on a high-performance computing system, ensuring efficient processing and model training. Despite the limited dataset, the initial results provided valuable insights into the segmentation capabilities of the model. The experimental study revealed several contributions and limitations, which are summarized in

Table 12. Contributions and limitations of the 2D image-based plant feature extraction process.

Aspect	Contribution	Limitation
Annotation Process	Efficient segmentation of plant parts using a annotation tool (stems, branches, and suckers) in LabelStudio	Time-consuming annotation process, approximately 2–4 min per image
Model Training	Successful training of a segmentation model with the available dataset	Limited using full plant images, resulting in an F1 confidence score of 0.65
Future Directions	Plan to refine training by focusing on specific plant organs (branches, stems, and suckers)	The current dataset and training approach are insufficient for higher accuracy

.

5.3.3. Future Directions in 2D Analysis

Future work will focus on enhancing the dataset by increasing the number of annotated images, specifically targeting branches, stems, and suckers. This will involve collecting additional images and annotating them to build a more comprehensive dataset. With a larger dataset, the model training can be improved, leading to higher accuracy in plant feature extraction. Additionally, exploring advanced segmentation techniques and integrating other machine learning models could further enhance the performance of the plant feature extraction process. For higher accuracy, training will need to focus specifically on images of branches, stems, and suckers rather than complete plant images, which is expected to enhance the prediction accuracy.

5.4. Organ-Level Feature Extraction Using High-Resolution 3D Data

PointNet, a pioneering deep learning architecture, directly processes point clouds without converting them into other representations like voxels or images, preserving inherent details but presenting unique challenges. By focusing on its application to the Pheno4D plant phenotyping dataset [207], we explored its advantages, limitations, and preprocessing steps to optimize performance in segmentation tasks. PointNet uses multi-layer perceptrons (MLPs) and a symmetric function to aggregate information from all points with classification and segmentation networks. The classification network classifies the entire point cloud into a single class, such as a chair or car, starting with a T-Net that learns an affine transformation matrix to align the input. Feature extraction is performed through multiple 1 × 1 convolutions, typically with increasing depths, followed by another T-Net for feature space alignment. A max-pooling operation aggregates the global feature vector, which then passes through fully connected layers with dropout and batch normalization, leading to the final classification layer. TransformNet includes the Input TransformNet and Feature TransformNet, outputting 3 × 3 and 64 × 64 affine transformation matrices, respectively. Both use convolutional layers, batch normalization, ReLU, fully connected layers, and regularization to maintain orthogonality. For segmentation, PointNet assigns specific labels to each point in the cloud. The architecture is similar to the classification model, involving input transform, feature extraction, max pooling for the global feature, and concatenation of global features with local features. Segmentation layers use 1 × 1 convolutions to predict semantic labels for each point, focusing on the semantic meaning of different regions.

PointNet’s advantages include preserving geometric details through direct processing, ensuring permutation invariance with symmetric functions, and robustness to noise. However, it initially struggled with capturing local geometric features, which was addressed later by PointNet++, and requires significant computational resources for large point clouds. Alternatives like PointNet++ improve local context capture through hierarchical feature learning. Graph Convolution Networks (GCNs) [107] perform convolutions based on dynamic point graphs, incorporating the neighboring point context but with higher computational complexity. Post-PointNet segmentation methods like Density-Based Spatial Clustering of Applications with Noise (DBSCAN) and Ordering points to identify the clustering structure (OPTICS) clustering refine and segment individual parts and are useful in complex scenarios like varying numbers of leaves on plants. Effective preprocessing is crucial for preparing point cloud data for PointNet, involving normalization, standardization, data augmentation, and downsampling.

Normalization centers point clouds at the origin and scales them to fit within a unit sphere, ensuring a consistent scale and uniform positioning, which is essential for geometric consistency. Standardization adjusts the point clouds so that each feature (x, y, and z coordinates) has a mean of zero and a standard deviation of one, stabilizing the learning process by reducing the impact of varying scales and distributions.

Data augmentation applies techniques like random rotation, scaling, and jittering to increase the diversity of the training data, helping to prevent overfitting and improving the model’s generalization ability. Downsampling reduces point clouds to a fixed number of points to lower computational complexity and ensure uniform input size. Voxel downsampling, where the 3D space is divided into a grid and points within each voxel are averaged, is a common method. The impact of preprocessing on PointNet’s performance is evident when comparing data distributions before and after normalization and standardization. The histograms in Figure 11 highlight the initial state of data consistency and distribution across training, validation, and test sets. In comparison, Figure 12 demonstrates significant improvements in these areas, which are crucial for ensuring that the model learns meaningful features and generalizes well to unseen data. PointNet represents a significant advancement in processing point cloud data directly, preserving raw geometric details, and achieving high performance. Its effectiveness relies on preprocessing steps like normalization, standardization, and augmentation, which ensure consistent and representative data, leading to improved performance and generalization. Future advancements like PointNet++ and Graph Convolution Networks (GCNs) build on these foundations, addressing PointNet’s limitations and pushing point cloud processing further.

In this approach, because plants have stems and leaves, training for the segmentation of every individual leaf is complex. We simplified this by dividing it into two classes: leaf and stem. Further individual segmentation of leaves is performed using clustering methods, which highlight unique characteristics and reveal the data structure. As the plant structure becomes more complex, the individual leaf segmentation is not performed properly (Figure 13). However, by integrating preprocessing techniques and leveraging PointNet for the initial segmentation followed by clustering methods for detailed segmentation, this approach is effective for complex scenarios like plant phenotyping, where accurately distinguishing individual parts is crucial. The training loss and accuracy of the leaf and stem classification via PointNet are shown in Figure 14.

6. Discussion

This paper presents a comprehensive review and demonstration of plant feature extraction and segmentation techniques utilizing multi-scale remote and proximal sensing platforms, ranging from canopy-level to organ-level resolution. The methods discussed emphasize significant technical advancements in phenotyping and agricultural monitoring, focusing on the integration of diverse sensing modalities to enhance the accuracy and efficiency of plant characterization. The integration of satellite, UAV, and ground-based sensors has enabled detailed analyses of plant traits, ranging from large-scale canopy assessments to precise organ-level phenotyping. Techniques such as multispectral and hyperspectral imaging, LiDAR, and thermal imaging have been particularly effective in providing multi-modal data for characterizing crop health, growth patterns, and the physiological status. However, despite these advancements, several challenges remain. For instance, while satellite-based systems offer extensive coverage and frequent data collection, they suffer from lower spatial resolution and are limited by cloud cover and weather conditions. On the other hand, UAV platforms provide high-resolution and flexible data collection capabilities but are constrained by the flight duration, battery life, and coverage area. Ground-based systems offer the most detailed phenotyping, but they are often labor-intensive and impractical for large-scale monitoring.

A critical gap in the current literature is the limited focus on integrating these diverse sensing platforms to create a unified phenotyping framework that can capture multi-scale plant traits across different spatial and temporal scales. While some studies have explored the combination of UAV and satellite data for canopy-level assessments, the integration of these methods with proximal sensing technologies, such as 3D point cloud-based segmentation for an organ-level analysis, is relatively underdeveloped.

The practical deployment of these advanced sensing and segmentation methods in real-world agricultural settings faces several challenges. Data processing, computation, and storage requirements are major constraints, particularly when dealing with high-resolution 3D data or dense point clouds from LiDAR and photogrammetry. High-throughput phenotyping generates vast amounts of data that require sophisticated storage solutions and a powerful computing infrastructure for a real-time analysis. Additionally, variability in environmental conditions, such as varying lighting, wind interference, and occlusions from overlapping plant organs, can introduce noise and reduce the accuracy of segmentation algorithms. Developing robust models that can adapt to these real-world conditions without compromising efficiency remains an open challenge.

7. Conclusions and Future Outlook

This literature review has presented a comprehensive analysis of plant feature extraction and segmentation techniques using 2D image-based and 3D point cloud-based data modalities, highlighting their contributions, strengths, and limitations. The study has shown how the use of advanced remote sensing technologies, such as multispectral, hyperspectral, LiDAR, and 3D imaging, enables high-throughput, multi-scale phenotyping from the canopy level down to the organ level. Techniques like image segmentation, shape analysis, and machine learning-based methods for 2D images, as well as direct 3D point cloud processing using neural networks, have made significant strides in precision agriculture. However, the practical implementation of these methods in real-world settings still faces numerous challenges, particularly in terms of data variability, computational requirements, and the integration of multi-modal datasets.

In addition to reviewing advancements in remote and proximal sensing for plant phenotyping, this paper demonstrates various plant feature extraction techniques, covering the entire process from data collection to feature extraction and segmentation at different levels of remote sensing studies. The insights gained from each experiment are expected to contribute to ongoing research in the field, inform future studies, and support the development of next-generation phenotyping platforms that integrate the strengths of both remote and proximal sensing technologies.

The significance of creating robust phenotyping models extends beyond improving crop yield and health monitoring. By linking high-resolution phenotypic traits to genomic data, researchers can better understand how different traits influence plant growth and resilience, ultimately leading to the development of new crop varieties that are more productive and adaptable to changing environmental conditions. Future research should also explore ways to optimize data processing and storage solutions for large-scale phenotyping studies, ensuring that these advanced methods can be deployed efficiently in diverse agricultural landscapes. Reducing the labor intensity of data collection in the field will also be essential to streamline the entire phenotyping process, enhancing efficiency and scalability in agricultural research.

Looking forward, the future of plant phenotyping should focus on creating integrated phenotyping platforms that combine the strengths of satellite, UAV, and ground-based systems with advanced data analytics and machine learning models. The use of deep learning frameworks, such as transformer-based architectures and graph neural networks for visual grounding, holds promise for achieving higher accuracy in complex phenotyping scenarios, such as dense canopies or intertwined plant organs. Furthermore, the integration of remote sensing data with in-depth genomic information will allow researchers to build predictive models that can link phenotypic traits to specific genetic markers, ultimately enhancing crop breeding efforts. By leveraging these interdisciplinary approaches, the field can move towards a more holistic and scalable phenotyping solution that supports both large-scale agricultural monitoring and detailed organ-level analysis for precision breeding and crop management.

One potential direction is the development of hybrid models that combine the strengths of 2D image-based approaches with 3D point cloud techniques to capture more comprehensive plant phenotypes. Developing hybrid approaches that integrate multi-modal data from diverse sources will enable more comprehensive and scalable phenotyping frameworks. Moreover, improving the segmentation performance for minority classes can be achieved through techniques such as synthetic data generation and adjusting class weights during training.

By overcoming these challenges, the field can move towards more scalable and comprehensive phenotyping frameworks that not only capture detailed plant traits but also contribute to sustainable agricultural management and breeding practices.