In-Field Estimation of Fruit Quality and Quantity

Every new tool gives humankind a new capability or capabilities, as a new tool finds a range of applications. For example, the production of a magnifying lens allowed a new look at the natural world. Every generation of scientists since has accessed some new tool allowing a different view of the same subjects, from a compound microscope to a confocal microscope and beyond. Each new view enables new interpretations.

This evolution also applies to agricultural applications. On a dairy farm, the production of every cow can now be monitored daily, and the herd can be managed at the level of the individual beast. In a wheat field, a satellite-acquired vegetation index map can be used for a variable-rate fertilizer application program. However, until now, fruit production has been based on a per-orchard basis rather than a per-tree basis. Technology advances, however, are throwing up a range of capabilities that allow for change in orchard management.

Agriculturalists are both an inventive and adaptive lot. They are inventive in the sense that technology advances sometimes come from agricultural applications and are then applied elsewhere. For example, the field of near-infrared spectroscopy was pioneered in the 1960s context of an agricultural product quality control by Karl Norris of the United States Department of Agriculture. The technology soon flourished in other application areas and is heavily used today in a range of disciplines, from petrochemical to pharmaceutical, as well as agricultural industries such as diary, forage, and grain. Its use in the assessment of intact fruit, both on fruit pack lines and in the orchard, using handheld devices came later, beginning in the 1990s.

However, agriculturalists are more commonly an adaptive lot, taking advantage of technologies developed for other applications, particularly from the medical, space, and defense sectors, given the well-funded R&D programs in those areas. Yield forecasts based on satellite- or drone-assessed vegetation indices are now commonplace for broadacre crops. Such technology-enabled forecasts of tree fruit harvest timing and load are yet to be widely commercially implemented, although advances in a range of technologies, including machine vision, image processing, LiDAR, and spectroscopy, hold promise for in-orchard assessment of various fruit crop attributes.

The need to forecast tree fruit harvest timing and load has increased due to scale and distance factors. Scale refers to the increase in size of fruit production systems, with a time-poor orchard manager relying on forecasts in harvest resourcing decisions. Distance refers to the increasing length of global value chains, with the managerial need to forecast harvest timing and volume increasing with the complexity of the value chain.

Relevant technologies used in the controlled environment of the packhouse are being shifted to use in the field environment of the orchard to address these needs. Examples include near-infrared spectroscopy (NIRS), used in assessment of fruit attributes relevant to the estimation of the optimum harvest timing, and machine vision techniques that allow for automated assessment of the flowering stage and level and fruit detection, sizing, and counting. These tools can inform farm management decisions on flower thinning, harvest timing, harvest resourcing (labour and materials), and marketing.

1. Contribution of the Special Edition

The Special Issue “In-Field Estimation of Fruit Quality and Quantity” presents a collection of eight articles. The topic is reviewed by Anderson et al. [1] with attention to the use of temperature monitoring and non-invasive NIRS assessment of fruit attributes in the forecasting of harvest timing, the use of in-field machine vision for a direct assessment of the fruit load, and UAV or satellite multispectral imaging, or extrapolation of historical yield data, for an indirect assessment of fruit load. Given that the different technologies measure harvest timing and load in different ways, the various estimates are complimentary.

An example of the use of NIRS technology is provided by Goke et al. [2], in a study involving the assessment of pear fruit dry matter content in the context of the pruning regime. NIRS is a relatively mature technology, established in commercial pack-line use for over two decades and instrumentation advances allowing its use in the orchard environment of variable light and temperature in recent years (Walsh et al. [3]). Attention is therefore moving from the development of the technology, per se, to its application uses.

In contrast, the introduction of deep learning (convolutional neural networks) to image-detection tasks is relatively recent, and the publication focus remains on the development of the method. For example, several papers in the Special Issue explored the use of machine vision in the estimation of either flowering or fruit load. Koirala et al. [4] reported on the machine-vision-based detection of mango flower panicles at three developmental stages and demonstrated a time course of the quantification of stages, using orchard images collected weekly. This is a step towards automation of the detection of flowering ‘events’ in an orchard, with heat units tallied from this date in forecast of fruit harvest maturation. Koirala et al. [4] also reported on the use of augmented image sets, although it was noted that the addition of rotated bounding boxes did not improve the training.

The primary limitation in use of in-field machine vision to estimate the fruit load is that the camera does not ‘see’ all fruit on the trees, especially for dense canopied trees. Anderson et al. [5] and Villacres and Cheein [6] both report issues with fruit occlusion in their respective documentation of the performance of machine-vision-based estimation of the fruit yield of whole cherry and mango orchards (Figure 1), respectively. Anderson et al. [5] documented the improvement in yield estimates made using a muti-view approach, involving multiple images per tree and tracking of fruit between images, over a ‘dual-view’ approach. The impact of the canopy architecture on yield estimates was also documented.

Koirala et al. [7] explored the use of techniques to correct for occluded fruit, for example, by the use of a machine-vision-based measure of the fraction of partially occluded to non-occluded fruit. None of the proposed techniques were recommended for adoption, and this topic remains open.

However, the fruit number is only part of the harvest load story, with the fruit size also being critical. The estimation of size requires information on distances, with several relevant candidate technologies being available. Neupane et al. [8] reported on an evaluation of depth cameras in the context of this application, while Mendez et al. [9] report on the use of LiDAR for fruit sizing, fruit counting, and canopy modelling. The application of LiDAR for fruit counting was also compromised by fruit occlusion in dense canopies, and acquisition time was also an issue, given the need for a high-density point cloud.

2. Conclusions

This collection thus presents a snapshot of the currently available techniques for the in-field estimation of fruit quality (maturity) and quantity. As for the NIRS technology, over the next five years, the machine vision and LiDAR techniques will mature, with the focus shifting from the assessment of the technology to the use of the technology to aid agronomic decision-making. As noted in the review by [1], “this is an exciting period to be involved in the development and application of tools for the forecast of tree fruit load and harvest timing”.