Machine Vision Retrofit System for Mechanical Weed Control in Precision Agriculture Applications

Abstract

This paper presents a machine vision retrofit system designed for upgrading used tractors to allow the control of the tillage implements and enable real-time field operation. The retrofit package comprises an acquisition system placed in the cabin, a front-mounted RGB camera sensor, and a rear-mounted Peiseler encoder wheel. The method combines shape analysis and colorimetric k-nearest neighbor (k-NN) clustering for in-field weed discrimination. This low-cost retrofit package can use interchangeable sensors, supplying flexibility of use with different farming implements. Field tests were conducted within lettuce and broccoli crops to develop the image analysis system for the autonomous control of an intra-row hoeing implement. The performance showed by the system in the trials was judged in terms of accuracy and speed. The system was capable of discriminating weed plants from crop with few errors, achieving a fairly high performance, given the severe degree of weed infestation encountered. The actuation time for image processing, currently implemented in MATLAB integrated with the retrofit kit, was about 7 s. The correct detection rate was higher for lettuce (from 69% to 96%) than for broccoli (from 65% to 79%), also considering the negative effect of shadows. To be implementable, the experimental code needs to be optimized to reduce acquisition and processing times. A software utility was developed in Java to reach a processing time of two images per second.

1. Introduction

Environmental pollution caused by the excessive use of herbicides is a global problem that requires the development of sustainable alternative methods for crop protection [1,2]. Recently, agriculture has advanced rapidly in the adoption of new technologies [3]. Precision agriculture through proximal sensing technologies (e.g., computer vision-based methods), uses intensive data and information collection to apply farm inputs more efficiently, thus improving crop production, environmental quality, and agricultural sustainability [4]. The European Union policy has been increasingly addressing the issues of agricultural sustainability, best practices, and food safety [5]. Precision farming techniques applied to different cultural practices can help reduce the use of agrochemicals, increasing their effectiveness and optimizing the doses. For some cultural practices, chemicals might be replaced altogether by other sustainable methods, such as mechanical weeding [1]. Mechanical cultivation is effectively used to remove weeds between crop rows, but selective weeding within the row (intra-row weeding) is more challenging, because the tool (hoe) must move in and out of the row and around the crop plant. Innovative technologies, including digital imaging, sensing, and robotics, are used to detect the crop plants and discriminate the weeds. Burgos-Artizzu et al. [6] developed a system to detect weed regions as image objects, by means of computer vision-based methods. Moreover, weeds can be detected by remote and proximal sensing [7]; in this last case, in-field machine-mounted (harvesters, tractors, robots) sensors are used. Usually, real-time proximal sensing (using mainly optical sensors) can be used to enable a system that will detect the weeds and simultaneously take action on them for control, as the vehicle moves forward at a constant speed [8]. While remote sensing is limited to map-based site-specific weed management, proximal sensing can also be applied for real-time weed control. In addition, remote sensing becomes expensive when used on small areas, often requires more intensive post-acquisition image processing, and suffers interferences from environmental factors (atmospheric correction such as clouds, fog, etc.). Therefore, an increasing number of farm implements designed with precision agriculture technology currently operate in the proximal sensing mode, mainly using optical sensors [4]. Some applications compare proximal and remote sensing data with soil characteristics [9]; however, these are more complex to manage for users. For site-specific control on a finer spatial resolution, the use of digital cameras, either spectral or optical sensor systems (non-imaging sensors), mounted on ground-based platforms is considered more interesting for monitoring weeds [10,11,12,13,14]. Proximal sensing mainly uses sensors that integrate an area to return a single joined value, often in a fast and efficient manner. Imaging techniques in this context have encountered minor consideration due to the complex analysis of the data obtained and the higher costs associated with the request of higher computing power and more complex optical devices.

On the other hand, imaging techniques (e.g., digital cameras, spectral, or optical sensor systems) offer greater flexibility for a wide range of applications and can contribute to the accuracy of agricultural practices, improving the performance of specific tasks, such as weeding.

This paper presents a retrofit package with machine vision functionality, designed to potentially enable real-time operation and control of farming implements connected to used tractors. The system comprises the acquisition system mounted inside the tractor cab, the RGB camera sensor mounted on a front frame, and the Peiseler encoder wheel mounted in the back. The method combines shape analysis and colorimetric k-nearest neighbor (k-NN) clustering to achieve in-field discrimination of weeds within lettuce and broccoli crops for control of intra-row hoeing implements. This proposed method can be conveniently used to upgrade farm machinery, considering that the farm machinery fleet in service in Italy is obsolete. Moreover, this low-cost system can use interchangeable types of sensors and can be applied on a wide range of agricultural machines to test different monitoring or control machine-vision applications.

2. Materials and Methods

2.1. Experimental Setup

This study was conducted in the experimental field of CREA-IT (42.099785° N; 12.623959° E). A test field was established for trails with lettuce and broccoli plants that were transplanted choosing a double-row arrangement, with 40 cm distance between two paired rows. For each crop, three double rows were transplanted, setting the distance between the row centers to 2.8 m and the row length to 30 m. For both crops, many empty spaces were observed along each row due to establishment failures caused by transplant shock. The acquisition of digital images of the crop rows was done once every two weeks, starting from the time of transplanting. No weeding at all was carried out in the test field, instead weeds were allowed to emerge and grow in numbers and size over time. This management scheme was intentionally chosen to test the retrofit system on increasingly challenging infestation levels. After completing the periodic field passes, the worst-case scenario (which occurred on 9 October 2015), showing the most severe infestation level of the whole acquisition period, was used for testing the image processing system. The authors visually assessed the degree of infestation in the image datasets and chose the most infested group (data not shown).

The final aim of this R&D project is to develop a machine vision retrofit system that can be mounted on any tractor to control mechanical weeding implements. The main steps planned for the development and testing of the optoelectronic system can be summarized as follows: (1) choice of the sensor; (2) assembly of the system and acquisition testing in the laboratory; (3) preparation of field trials; (4) preliminary field testing (image acquisition); (5) algorithm selection and coding based on the recorded data; (6) streamlining code for real-time application; (7) actuator control and machine testing. To date, steps from 1 to 4 have already been completed, step 5 is described herein, step 6 is currently under development, and step 7 will be developed in the coming future.

2.2. Hardware Setup

A preliminary step in the project was the evaluation of different sensing technologies (spectral, RGB, and black-and-white sensors) to select the most appropriate sensor for this specific application (data not shown). An RGB charge-coupled device (CCD) sensor was selected taking into account its operational flexibility and low cost in relation to its acquisition capability. In comparison, Vis-NIR matrix sensors are quite expensive, while black-and-white sensors can be used only for morphological shape analysis. Generally, the extra cost required to install this technology on automatic weeding implements would be considered to be affordable for adoption in commercial production if it is lower than 400 € per crop row (personal opinion provided by an Italian machinery manufacturer).

The tractor, model Landini Globus 80 (Landini ARGO TRACTORS S.p.A., Fabbrico, Italy), was equipped with the retrofit kit, placing the acquisition system inside the cabin, the camera on the front frame, and the Peiseler encoder wheel on the back. The setup was used to acquire images of the crops while the tractor was moving along the rows (Figure 1). The camera, model Allied Vision Mako G-125C RGB (Allied Vision Technologies GmbH, Stadtroda, Germany), was mounted on the front end of the tractor, via a custom-made supporting frame, at a height of ~2 m and orthogonal to the ground, obtaining a Field Of View (FOV) of 1.5 × 2.0 m (h × w). This model is an industrial Gigabit Ethernet (GigE) camera equipped with Sony’s ICX445 CCD sensor, having EXview HAD technology, with the following characteristics: ultra-compact design, affordable price (~500 €), 30.3 fps frame rate at full resolution including Power over Ethernet (PoE), three opto-isolated outputs, and a 64 MByte image buffer. The camera (C-Mount) was equipped with a 28 mm lens with F/1.4 max aperture. The Peiseler encoder wheel (40 cm diameter) was a custom-made device having a resolution of 1000 pulses per turn. The synchronization among encoder pulses, image acquisition, and actuator was achieved using dedicated multi-threading Java software, running on a compact high-performance PC (Intel NUC) equipped with a touch display (15″). The software, with respect to encoder management, was interfaced with an open-source Arduino™ microcontroller that was specifically programmed and configured for this study. The signals from the encoder wheel were used to link the image to its corresponding position in the plant row and to trigger the hoeing mechanism on that specific spot. Images were acquired every 150 pulses (equal to a length of 140 cm). A small spatial overlap between two consecutive pictures (~10 cm) was planned for the acquisition, to avoid data loss and to improve plant recognition. A total of 144 images were acquired (about 20 images for each passage). Arduino™ was programmed in C++.

2.3. Experimental Design

In the field tests, the tractor was always used in the forward drive direction, moving along the rows from end to end, making runs down the rows (forward pass) and back (return pass or inverse direction). This traffic pattern was purposely chosen to test two opposite lighting conditions, determined by the position of the sun in the sky at midday (time of day chosen for testing) with respect to the tractor. The row orientation in the test field was north–south, and this had an effect on the shadows projected by the tractor on the ground. In one case, the sun was high up and in front of the tractor, therefore there was no shadow cast on the acquisition area within the row (full-light illumination); in the opposite case, the sun was behind the tractor, and its shadow was cast in front over the acquisition area. In this latter case, the changing shadows cast over the acquisition spot caused additional lighting variability. This traffic pattern was therefore chosen to add robustness to the tests and to develop a less restricted process. The experimental design was set up as follows: for each crop, images were acquired in both driving directions, forward and inverse, with the sun in both positions, front and back, respectively. In the forward pass, two ground speeds were tested, slow (1 km h⁻¹) and fast (3 km h⁻¹), but only the slow speed was used for the testing in the inverse direction mode.

2.4. Image Processing

The image processing procedures were all developed in MATLAB (rel 7.1; Mathworks, Natick, MA, USA). The following workflow was established for the processing of the images: (a) illumination correction for an increased uniformity; (b) transformation of the color coordinates from RGB to Hue Saturation Value (HSV; more efficient for the specific purpose) [15]; (c) extraction of the H and S channels; (d) filtering for noise reduction (e.g., low-band filter and morphological filters: erode and fill to avoid empty space within connected components). These operations were followed by three morphometric operations on the binarized images: (1) selection on the basis of image area and of shape factors, that is the relationship between the main axes calculated once the centroid is determined (e.g., 0.75 = elliptic shape, 1 = perfect circle); (2) classification; (3) depending on the images, some additional operations may be required (the program automatically rejects the object only partially included in the image). The classification of objects was achieved by thresholding with a k-NN clustering algorithm. This is a supervised clustering technique, used for pattern recognition, that classifies each pixel of the input image following the majority vote of its neighbors, with the pixel being assigned to the class most common among its k nearest neighbors [16,17], being, in this case, k = 3. The neighbors were taken from a set of pixels (one per class) for which the correct classification was known (training set). The minimum area for consideration of an object was established at 500 px (0.12 cm²). The training set was built by averaging the values of each color channel of representative patches extracted from the original images (30 for both weeds and soil). t-tests were conducted to observe the effect of the two factors (speed and crop) on the percentage of correct classification. The null hypothesis of the test is the equality of the mean percentages.

A software utility (k-PE—k-NN Patches Extraction) has been developed in Java to facilitate the training operations. The software acquires square patches of a chosen size and assigns them to clustering classes. This preliminary operation was used to optimize the performance of weed discrimination in different light conditions.

3. Results and Discussion

Unsupervised algorithms were preliminarily tested to evaluate their efficiency in object classification. These algorithms failed for different reasons, firstly because of the differences in ambient illumination conditions (data not shown), i.e., the different shadows cast by the tractor on the ground. Instead, the selected k-NN achieved good object recognition performances even in severe conditions (e.g., adverse weather conditions, shadows, etc.).

In these field tests, the retrofit system required about 7 s for each single acquisition, while the total time was divided into the following time fractions for the succeeding steps: training test 1.43 s, k-NN processing 4.5 s, morphological extraction 0.48 s (in total about 7 s). At a 3 km h⁻¹ speed, the time delay between image acquisition and the actuation of the hoeing tools (5 m longitudinal offset) was about 4.2 s. Code streamlining is currently under development to further reduce the operational times and reach the target timing necessary for real-time control. The quadrants in the following figures (Figure 2A–C) represent the image processing flow for different conditions of weed infestation and shadowing: (1) Low band filtering for noise reduction; (2) Segmentation (binarization) of the image by k-NN with consequent reduction of the chromatic dynamics from RGB to two classes (background and object); (3) Morphological operations for exclusion of objects that touch the edge (smaller or larger than a certain threshold) and filling of holes with a shape factor >0.75 (1 = perfect circle); (4) Composition of the original image with the selected crop object contour from quadrant 3.

Figure 2C shows a challenging situation with severe shadows (inverse direction) and a high infestation, underling the power of the k-NN clustering technique. The shadows cast by the tractor, created two distinct lighting conditions on the row (Figure 2C-4). In addition, the number of weeds around each plant was very variable. Therefore, the system would be inclined to remove the lettuce plant not completely present in the image and to correctly extract the remaining four plants. The drawback of this supervised technique is the need to make a training set for each class. However, this step could be simplified by installing a touchscreen in the cabin, thus enabling the operator to acquire a few representative images intended for a rapid selection of the training set before the hoeing operation.

Table 1. Number of patches (light and shadow) for the different types of objects (crops, soil, and weeds) used to train the k-nearest neighbor (k-NN) algorithm.

Typology	Training	N Patches
Broccoli	Light	43
	Shadow	20
Lettuce	Light	26
	Shadow	20
Soil	Light	22
	Shadow	13
Weed	Light	20
	Shadow	20

shows that few patches (from 13 to 43) were needed to train the k-NN algorithm subdivided into the considered classes: broccoli, lettuce, soil and weed in both light and shadows (8 classes).

A limited number of patches was sufficient for training purposes. These patches were not automatically extracted, instead the operator assigned a class to the image directly on the touchscreen. The procedure could be easily implemented by semi-automated routines in a software application. Figure 3 shows the software utility screenshot. The software facilitates the training operations and allows the acquisition of square patches of a chosen size, assigning them to the clustering classes. The control panel is shown on the left side of the image. From top to bottom, the user finds: (i) image folder (input path) selection button; (ii) class creation input field together with the update control add button (these operations add a new folder in the input path for each class containing the patches created); (iii) list of the classes and the number of patches acquired for each class; (iv) space to set the dimension of the square patches; (v) zoom control and image navigation; (vi) image preview; (vii) arrows to select images in the input path. Different light and shadow conditions are clearly present in the image. For example, an entire shaded area created by the moving tractor is clearly visible on the right side of the image. A square patch is visible on the central lettuce.

The automated routines [18] could facilitate the attribution of patches for the training phase, increasing the general performance of the algorithm.

Table 2. Results for the two different crops (broccoli and lettuce) at two different speeds (1 and 3 km h⁻¹) of the k-NN tests: total objects analyzed; number and percentage of correctly identified objects; number and percentage of misclassified objects; number of weed identified as crops (errors). At the speed of 1 km h⁻¹, also the inverse direction was considered.

Crop	Speed (km h−1)	Total Objects	Correctly Identified	% Correctly Identified	Misclassified	% Misclassified	Errors
Broccoli	1	62	40	64.5	22	35.5	1
	1 (inverse)	62	49	79.0	13	21.0	4
	3	62	44	71.0	18	29.0	0
Lettuce	1	68	65	95.6	3	4.4	1
	1 (inverse)	68	47	69.1	21	30.9	2
	3	68	62	91.2	6	8.8	4

shows the results obtained in the k-NN classification tests, as objects correctly classified, misclassified, and not considered (errors), for broccoli and lettuce at two different speeds (1 and 3 km h⁻¹).

At the speed of 1 km h⁻¹, also the inverse direction (presence of shadows) was considered.

The errors (number of weeds identified as a crop) were few (from 0 to 4), also considering the severe weeding condition of the experimental field. The percentage of correctly identified crops was higher for lettuce (ranging from 69.1 to 95.6) than for broccoli (ranging from 64.5 to 79.0) but not significant (t-test p = 0.21), probably due to the low number of replicates. The best performance was achieved for lettuce at slow speed (1 km h⁻¹) in direct direction: 65 out of 68 objects were correctly identified (95.6% with only one error and three misclassified objects). The t-test performed on the speed factor did not show significant differences in terms of percentages of correct classification (p = 0.96) confirming the null hypothesis. This study represents an application of precision agriculture which can improve farm management and agricultural production and sustainability [19].

To avoid the influence of sunlight and shadows on the acquisition of images, some commercial automatic hoeing machines (e.g., the Steketee IC Weeder or the Ferrari Remoweed) use active illumination devices (high-power LED lights, illumination box), mounted close to the ground to provide constant lighting over the acquisition spot. These illumination systems require specifically designed light emitters and are associated with additional manufacturing costs. Many other applications of RGB sensors were used to minimize the volume of herbicides, implementing an on-the-go site-specific weed management systems. An example is the study by Hague et al. [11], in which the automatic assessment of crop and weed areas in images of widely spaced (0.25 m) cereal crops, was performed using a tractor-mounted RGB camera. In addition, in their work, Downey et al. [10] used a digital RGB camera for continuous image capture along the crop seedline from a moving vehicle equipped with a GPS receiver to map the weeds directly in the field. Moreover, the light emission geometry makes these systems less flexible in comparison to the ones based on matrices sensors.

4. Conclusions

The results of this study demonstrated that the RGB imaging retrofit kit that we described can be implemented on a system to carry out mechanical weed control in row crops with different degrees of weed infestation. The fulfilment of this goals will result in considerable savings in both ecological (i.e., eliminate the need for chemical weed control) and economic terms. In an economical context, the white circle in Figure 2B represents a misclassified broccoli plant. The plant was misclassified by the system probably because of the partial shadow in the acquisition image and because of its small size. In real hoeing, this will result in an economic loss for the farmer and thus needs to be considered when evaluating the system’s general performance.

In this study, the total processing time for each image, integrated with the retrofit kit, was about 7 s. To enable a working speed of up to 3 km h⁻¹, the image processing time must be further reduced. This optimization will be implemented by exploiting algorithms and computational hardware graphics capabilities using development libraries (e.g., OpenGL), achieving a processing speed of two images per second.

This work shows the preliminary results obtained with a machine vision retrofit system tested on crops under real field conditions. The system could detect quite accurately the position of the single plants, extracting the image of crop plants from the soil background and from the surrounding weeds. The system was not capable of classifying all the crops correctly, underlining the need to carefully determine the percentage of correct classifications for a deep evaluation. The development of environmentally friendly and sustainable solutions for weed control in crops is necessary to reduce environmental pollution from farming and to adopt better management decisions. Moreover, to make the system truly implementable for use on weeding machinery, the code streamlining should significantly cut the time needed for image acquisition and elaboration. Once the development and calibration of the kit are completed, the kit could be mounted on different farm implements and be used for different crops and purposes.