Opportunities for Robotic Systems and Automation in Cotton Production

Abstract

Automation continues to play a greater role in agricultural production with commercial systems now available for machine vision identification of weeds and other pests, autonomous weed control, and robotic harvesters for fruits and vegetables. The growing availability of autonomous machines in agriculture indicates that there are opportunities to increase automation in cotton production. This article considers how current and future advances in automation has, could, or will impact cotton production practices. The results are organized to follow the cotton production process from land preparation to planting to within season management through harvesting and ginning. For each step, current and potential opportunities to automate processes are discussed. Specific examples include advances in automated weed control and progress made in the use of robotic systems for cotton harvesting.

1. Introduction

The concept of robotic applications for agricultural operations has been discussed for over three decades [1], and recent advances in machine vision, computer processing, and controllers have led to an increase in agricultural robotic systems, including applications for weed control, row crop planting, and fruit and vegetable harvest [2]. Some agricultural sectors have already embraced automation, such as the dairy industry’s estimated 35,000 automated milking systems currently in use globally [3]. For row crops, much of the commercial focus is on weed control with the rise of herbicide-resistant weeds and the lack of new herbicide modes of action [4]. Major agricultural machinery companies have announced intentions to develop autonomous machinery, have prototype machines, and/or have filed patents on autonomous robotic systems for agriculture [5]. The concept of an autonomous platform with several interchangeable implements is emerging as a preferred concept for agricultural robots [6,7]. Additional evidence of the proliferation of robotics systems for agriculture is the number of robotic operating system (ROS) open-source tools that are available [8]. A recent review that highlights challenges and opportunities for agricultural robotics in general is provided by [9], and the authors note that automation could be a disruptive technology as “farming-as-a-service” could become a possibility in the future. Cotton is a unique crop as it has an indeterminate flowering period, is a perennial plant that is managed as an annual, and requires specialized equipment for mechanical harvest and post-harvest processing [10]. For those not familiar with cotton production, additional background information is provided in Appendix A. Thus, there are many within season and post season opportunities for automation that will differ from other crops. The overall objective of this paper is to inform both the research community and those providing automated services for agriculture of the opportunities for current and future application of machine vision, automation, and robotic systems in the cotton production process.

2. Materials and Methods

Based on the authors range of agricultural disciplines and discussion with primarily U.S. cotton producers, possible scenarios where automation and/or machine vision technologies could increase the efficiency of cotton production were theorized. The results of these discussions and theory follow the cotton production process from pre-planting to harvest and ginning discussion areas where there is potential for automation; automation applications in other crops that could be adapted to cotton production; or where automated systems are in use in the cotton production process. Citations and examples of where automation has taken place in cotton or other crops that could be directly applied to cotton are included. Additionally, in the areas of advanced technologies for weed control and automated harvest, Cotton Incorporated has initiated studies with several U.S. universities, and updates from those projects are included in this paper.

Leveraging Open-Source Libraries

The recent explosion of open-source libraries has been a boon to research in that it allows for rapid development in highly specialized and technologically advanced subjects that would otherwise take years or even decades of extensive research to bring a team up to speed on the subject matter. For robotics, there are several open-source packages that provide a middleware communication tool designed to help developers streamline their development by off-loading some of the time-consuming development centered around the problem of communication between remotely distributed sensors and systems; Elkady and Sobh provide an extensive literature review of these various open-source software libraries [11]. One of the most popular of these middleware communication libraries is the Robotic Operating System (ROS) middleware library. ROS is a key example of an open-source library enabling developers to leverage a software package for rapid deployment through the use of an extensive library of routines and test programs that are all targeted at the communication layer between discrete systems and sensors. ROS also provides an extensive set of library packages, provided by a highly active open-source community, which one can readily pull into their development. One of the most useful ROS auxiliary packages provides for incorporating advanced simultaneous localization and mapping algorithms (SLAM). With all the ROS advantages, there is one notable disadvantage that ROS 1.0 does not address, which is in the region of hard real-time control where latency between communications has to occur or damage will occur. This is a difficult subject and it is unsurprising that ROS 1.0 did not address it. However, in omitting it, there is a major hole in the application layer whereby the developers must turn to other lower software layers for hard real-time coordinated control and can only leverage ROS for non-time-critical sensor inputs. ROS 2.0 is starting to look at solving this; however, it is far from complete and it is uncertain how well or if it will be able to fill this void, given the complexity of hard real-time control. One alternative source of control libraries that can be used to fill in this gap is used in the computer numerical-control (CNC) industry. There are some notable open-source CNC firmware libraries in this area, which provide the ability to control up to five axes of simultaneous control, and leveraging these libraries is fairly straight forward. They are typically target-hardware optimized in order to achieve the hard real-time control targets required by CNC operations. There are versions available for several micro-controller architectures, including Arduino platforms and, for more demanding systems, there are a few that several ARM based 32-bit micro-controllers. The most notable of which is heavily leveraged by the open-source 3D printing developments, which, at the heart, are leveraging the open-source project Grbl [12]. Another library for hard real-time control that is more demanding from an integration perspective, yet provides up to nine-axis control, is Linux-CNC, which is targeted at the x86 series [13]. However, it is quite dated and is not regularly updated. At the time of this writing, the known set of x86 motherboards that are proven to work with LinuxCNC are no longer available, and many modern motherboards have been shown not to work. Furthermore, there is little support for embedded target platforms, so LinuxCNC is not well suited for embedded robots because of a lack support for embedded processors, including x86 single-board computers. There is potential with an off-shoot project called Machine-Kit, which is a subset of LinuxCNC and which was ported to the BeagleBone embedded 720 MHz ARM Cortex-A8 micro-controller. There are several open-source CNC and robotic projects that leverage this open-source library.

Given the advantages for rapid development that open-source systems provide, many of the projects discussed herein have designed their software by leveraging open-source libraries where possible. The main primary libraries utilized were the ROS library originally written by Intel Corp. Research and OpenCV (machine-vision library; originally written by Intel Corp. Research) [14].

3. Results

3.1. Preplant and Planting Operations

3.1.1. Soil Sampling

Based on a survey of over 900 U.S. cotton producers, it was determined that 80% of the producers perform soil sampling to determine fertilizer rates [15]. The company, Robo Ag, has adapted a Bobcat T450 platform to collect soil samples autonomously [16]. The system is able to cover 32 ha an hour when sampling a 1 ha grid. A high-speed auger is used to collect samples and automatically bags the soil and stores up to 250 samples on board. Currently, the company is using the automated system as part of their soil sample service, and they are not selling individual units to producers or consultants. Valjaots et al. [17] report on a similar platform developed for research use and also discuss the possibility of conducting real-time soil measurements in addition to sampling. Given the current progress in this area, autonomous soil sample collection is likely to become more widespread in the next five years. As robotics become more common on farms, soil sampling will be an ideal off-season task for multi-use robots.

3.1.2. Planting a Cover Crop

The American Cotton Producers of the National Cotton Council have set 2025 goals to increase soil carbon and decrease soil erosion. A key tactic to meet those goals is the increased use of winter cover crops [18]. One challenge to the use of cover crops in cotton is planting the cover crop early enough so that it produces enough biomass to suppress weeds, reduce soil erosion, and build soil carbon. There have been attempts in corn to use a small robot to seed the cover crop between the corn rows before it is harvested, allowing for a much earlier planting date for the cover crop [19]. Another barrier to the increased use of cover crops by U.S. cotton producers is the added management and labor needed near and during harvest, one of the busiest times of the season. If the process could be completely automated, including refilling seed, it would make cover crops more feasible for a greater number of producers.

If frequent autonomous harvest becomes possible in the future, during peak boll opening the robot will be tasked with cotton harvest, but the robot would have time to plant the cover crop either late or early in the boll opening process. Other precision application uses of robots in cover crop management could include: selective N fertilization where cover crop growth is slow, early or late chemical termination of cover crops depending on soil moisture levels, mowing of cover crops to reduce herbicide use, gap filling of cover crops, selective planting dates for cover crops to avoid excess biomass in some parts of the field, a variable seeding rate, and species blend planting of cover crops.

3.1.3. Preplant Weed Control

The ability to automatically segregate green vegetation from bare soil and crop residue is well established using red (~680 nm) and near infrared (~800 nm) spectral regions, as actively growing plants strongly absorb red light and have very high reflectance in the NIR (about 50% reflectance). One of the first commercial sensor-control herbicide application was the WeedSeeker^®, which used a modulated light source to detect green vegetation material and then activated a solenoid valve to turn on the spray nozzle and which was found to work successfully in cotton [20]. Swarm Farm, an Australian autonomous vehicle company [21], has used Weedit technology (similar to the Weedseeker [22]) for preplant weed control on autonomous sprayers.

3.1.4. Planting

Cotton planting is currently done by large multirow systems to cover as much acreage as possible during narrow planting windows when soil moisture, soil temperature and forecast weather are favorable. However, the ability of small all-wheel-drive robots to navigate wet fields without severe compaction or ruts may complement current planters when parts of a field are too wet for large equipment to enter. This occurs frequently around playa lakes in West Texas and on delta clay soils where drainage is poor. Fendt has proposed a swarm robotic concept, referred to as “Project Xaver,” for crop planting that may be useful for cotton in such wet field conditions [23]. As described in the vision of Project Xaver, precise geolocation of early season plants could be the first step in managing the inputs for each individual plant. It could also be an important data layer to assist in weed control decisions later in the season by ensuring no tillage or herbicides are applied to that point in the field.

3.1.5. Gap Fill Planting

Poor conditions at the time of planting (cold and/or dry soil) and extreme weather events such as wind and hails storms while the plant is small can result in poor plant stands that may justify replanting [24]. The ability to image fields for delayed emergence, skippy stands, seedling desiccation, or death has already been demonstrated [25,26]. In the future, it would be useful to combine these data with a robotic planter that is guided by a drone to focus solely on the parts of the fields with poor stands, offering growers a timely tool to substantially increase a uniform healthy stand. With large planters, growers must wait multiple weeks after emergence to assess stands before considering the difficult replant decision for large sections or entire fields. With small robotic planters, growers could elect, at the earliest possible time. to put additional seed in the ground without removing the original plants. This decision could be made multiple times during the planting window with software that records replant seed placement and models its progress to emergence. This targeted planting could be a significant saving on seed costs, seed treatment costs, crop termination herbicides, labor, and equipment operation costs.

3.1.6. Uncapping after Planting

One of the successful tools for stand establishment under dry-windy, poor seedbed, or saline conditions is to cap the bedded row by hipping a small (~4 inch tall) soil behind the planter. This can be successfully uncapped by dragging a medium weight chain anytime between 1 day after planting until 1 day before normal emergence (without a cap). The weight of the cap keeps the hypocotyl from unfurling. When the cap is removed close to a normal emergence time, emergence is observed within 1 day as the hypocotyl quickly unfurls due to the built-up turgor pressure [27]. This method is not adopted because of the labor required to uncap at a time when labor is being used to plant and because of the risk of rain after planting that would prevent entry of tractors to uncap. Therefore, the development of robotic uncappers would solve both the labor and field access problems and could handle several hectares in a day since they can enter wet fields, uncapping energy use is low, and since there is no need for in-row precision or seed/chemical refilling.

There may be other planting innovations that are possible with robots that we have not considered. Growers have been moving away from applying herbicides, fungicides, or starter fertilizers at planting because they deem it necessary to focus only on planting. There may be a role for robots in applying pre-emergent herbicides or other starter chemicals once new options become available.

3.2. Within-Season Management

3.2.1. Stand Evaluation

Many studies have demonstrated the potential value of the use of multispectral data for detection of crop stress, but one limitation has been the frequency with which images are collected [28]. An autonomous ground-based robot could collect frequent and high spatial resolution images of the crop throughout the season while engaged in other production tasks or when not assigned other task similar to what has been done in crop phenotyping studies [29]. For example, a narrow-focus thermal camera could also calculate a crop water stress index (CWSI) [30] on each plant. Ideally, a lack of growth or a high CWSI would be paired with replanting capabilities allowing additional seed placement where early plant growth had stalled, suggesting root injury. A benefit of running imaging robots in the field on a weekly or near-continuous basis is the ability to detect nutrient deficiency or drought early enough to make a correction before significant yield loss resulted.

Another potentially valuable cotton growth evaluation tool is light detection and ranging (LIDAR) scanning, which enables accurate mapping of all plant heights, widths, and branching geometry. LIDAR data can measure positions of plant stems, branches, leaves, and bolls to sub-centimeter accuracy, which could be used in near real time to evaluate stands with respect to norms established for each plant variety [31]. When combined with conventional and thermal imaging, LIDAR data could be used to locate and possibly correct problematic sections within fields.

3.2.2. Crust Busting

In some soil types, a rainfall after planting can lead to a crust at the soil surface that prevents crop emergence, and sometimes a rotary hoe is used to break the crust to allow emergence to occur [32]. Robots would be ideal at crust busting. If they had a precise GPS, a variable down-pressure rolling spike that also served as a soil penetrometer, with the ability to sense emerged cotton seedlings and both forward and reverse imaging, then they could detect skips and apply a very precise force just to the side of the drill row that only breaks the crust and pushes no further. Since this kind of robot would not damage emerged cotton, it could be deployed earlier than current broad area crust busting practices, which damage some emerged plants. Since soil moisture content is critical for ease of crust busting, the back-facing camera compared with the front-facing camera could be used to determine if the crust was being broken and could either adjust more down pressure or delay a day until the crust had dried more.

3.2.3. Sand Fighting

In areas of the Southwest where lack of water prevents the use of winter cover crops, young cotton plants can be damaged by blowing sand [33]. Traditional sand fighting is currently done after a rain when the soil surface has lost its roughness, and a tillage implement is run down the soil furrow to increase soil roughness [34]. However, robots could offer the new possibility during a rain to build surface roughness from the wet soil. There may even be utility for a robot to create roughness prior to a rain or paired with a drone to focus sand fighting where the greatest amount of sand is blowing.

3.2.4. Weed Control

There is substantial public and private sector activity in robotic systems for within-season weed control. Slaughter et al. [35] provide a comprehensive review of past efforts, and several current systems are summarized by Pandey et al. [36]. One of the first applications of a robotic system for within-season weed control was developed for a cotton crop [37]. Using a machine vision system, they were able to correctly spray 89% of weeds in the field and misapplied herbicide to cotton 21% of the time. Since then, many systems have transitioned to using machine vision for weed identification, coupled with a wide range of weed removal methods. Distinguishing weeds from crops is a challenge even for today’s best machine vision systems, but several prototypes from both industry and universities are showing promise.

Multiple efforts are taking a “see and spray” approach using computer vision and machine learning to detect weeds between rows. The tractor-mounted equipment from Blue River Technology (Sunnyvale, CA, USA) utilizes a controlled lighting cover and two sets of cameras to identify and spray weeds in real-time. Although originally developed for lettuce, both the Robovator (F. Poulsen Engineering, Denmark) and the Robocrop InRow Weeder (Garford Farm Machinery Ltd., Peterborough, UK) use vision-based techniques for mechanical weed removal, and this technology could be adapted for future use in cotton.

In addition to tractor-mounted autonomous weeding implements, multiple companies are developing small, standalone autonomous robots capable of weeding. Two different companies have developed a small platform for weed control using a delta arm and machine vision. Nexus Robotics (Halifax, Nova Scotia, CA, USA) has a small platform, the R2-Weed2, that uses a neural network to identify and either mechanically remove weeds or apply herbicide. That system is similar to a commercial prototype from ecoRobotix (Vaud, Switzerland), whose first prototype used a delta manipulator to apply a small amount of herbicide to weeds and which adds the use of solar panels to recharge the robot’s battery while in the field [38]. The startup Small Robot Company (Salisbury, UK) is developing an autonomous weeding robot that will use electricity from a system developed by RootWave (Warwick, UK) to kill weeds. Another non-herbicide weed removal robot was developed by Deepfield Robotics (Bosch, Gerlingen, Germany) and is now managed by Farming Revolution GmbH. Their BoniRob platform uses a mechanical stamping mechanism to remove small weeds at an early growth stage.

A similar weed detection and control system focused on machine vision to classify weeds in images collected while moving through cotton and peanut fields is in development [39]. The actual operation of the system will include a diode laser, an herbicide spot-spraying nozzle, and a mechanical weeder to control weeds when in the seedling stage. Identification of the weed species will allow the selection of the best control method. For example, weeds in the row with cotton can be controlled using the laser. Weeds that are between rows and known to have herbicide resistance could be controlled with the laser or the mechanical weeder. Control can also be rotated between tactics to help reduce resistance to a specific control method.

To enable the system for cotton, training images of 12 weed species were collected: crowfoot grass, goosegrass, crabgrass, Texas panicum, yellow and purple nutsedge, pigweed, pitted morning-glory, ivyleaf morning-glory, smallflower morning-glory, and sickelpod were collected. Additional work is being conducted by researchers at North Carolina State University and Mississippi State University to increase the number of images available to develop an open-source image database of weeds important to cotton. Similar databases have been developed for other crops and environments [40]. United States Department of Agriculture-Agricultural Research Service (USDA-ARS) engineers are also working to develop a simulated three-dimensional cotton field that can be used to adjust lighting and background conditions for the training of machine vision systems. Maja et al. [41] have evaluated the use of the ClearPath Husky robot as a platform to tow tillage implements that do not require weed detection capabilities. Two weeder/tiller prototypes were tested in 2019 and 2020. The first module has six individual prongs on each side, where each prong measured approximately 15 cm. The prong was designed to penetrate about 3.8 cm into the soil. Two wheels were used to ensure the prongs would be kept at a constant depth into the ground. A slider mechanism was designed to make the width of the two-prong holder adjustable. The second weeder/tiller was an adjustable harrow disk, where the disk holder can be adjusted at a certain angle. Since the disk used was off the shelf and heavy, it was retrofitted with two wheels to minimize the mobile robot’s force to pull the weeder.

3.2.5. Insect & Disease Management

The use of robotic systems to scout fields to identify problems has been demonstrated in several studies, such as Nagasaka et al. [42] who developed a “dog” robot using a camera, laser system, and controller area network (CAN) bus to find problems in the field. A large portion of the Cotton Belt fields are visited at least twice a week by a field scout to determine whether insect populations are exceeding thresholds or whether disease symptoms warrant a pesticide application at the cost of around $22.00 ha⁻¹ [43]. As such, there is great future potential to use robotic scouts to alert growers of the infiltration of pests before populations exceed levels known to justify a pesticide spray. With insects, a significant challenge exists in identifying the species present and to distinguish between beneficial insects and pests as well as to distinguish between the different pests to determine which are over threshold. However, systems could be developed relying on imaging of plant damage and/or insects, pest DNA sampling, or volatile detection to determine which species are present [44]. As an example, the ability to determine through imaging whether new leaf area is being added at a rate commensurate with heat units would add to the precision of a spray for thrips, as a decrease in the rate of early season canopy development is a known symptom of thrips damage [45]. Growers occasionally “revenge spray” thrips at a time when they are no longer an impediment to adequate leaf area expansion. The presence of thrips in the field would also need to be determined in addition to reduced canopy development rates.

As with herbicide applications, land- and air-based robotic systems could also be used to make insecticide and fungicide applications. Often, diseases are confined to certain areas of field as are certain pests such as spider mites, which would be good candidates for spot spraying applications. These applications could improve pest control by penetrating deeper into the canopy given their proximity and orientation to the plant. These systems could also release semiochemicals for mating disruption, beneficial attraction, or pest repellency while performing other tasks such as weed control or scouting [46].

3.2.6. Nuisance Animal Deterrent

Feral hogs, deer, rabbits, and bears are becoming another pest for cotton producers in different regions of the Cotton Belt. A robot designed to autonomously deter nuisance animals has been patented [47]. While not currently used by cotton producers, the concept is that once the animal is detected the robot is designed to simulate a predator of the target animal.

3.2.7. Fertility

Past research has shown that multi-spectral reflectance data can be used to detect nutrient deficiencies in cotton, particularly nitrogen [48,49]. Small swarm robots equipped with multi-spectral sensors could provide the ability to monitor cotton plant foliage to identify plants or portions of the field before the onset of nutrient deficiencies or if excessive nutrients are available. The robots would be able to make real-time prescription applications, or prescription maps could be developed for use in traditional fertilizer applicators. This approach would allow growers to become less dependent on preplant fertilizer applications, which lead to more upfront expenses and increased environmental risks, especially for nitrogen.

3.2.8. Plant Growth Regulation

The mainstem growth rate is used to precisely time plant growth regulator (PGR) applications to reduce internode length and to instigate fruiting, and multi-spectral vegetation indices have been used as a tool to create site-specific PGR rates in cotton [50]. If robots become powered by inexpensive solar energy, there may be value in running a spraying robot through the field continuously to optimize PGR rates in the crop during times of excessive vegetative growth. Early applications of PGRs can be highly effective if low rates are applied regularly, which would lead to increased crop uniformity and would optimize harvest efficiency and fiber quality. The key is to avoid PGR applications to slowly growing cotton, which can cause additional stress, and robotic sensors or drone imaging could assess plant stress during multiple passes through the field as descripted in Section 3.2.1; prescriptions could be applied by swarm robots.

3.2.9. Mid-Season Leaf Removal

When labor was inexpensive in China (during the 1990s), lower leaves were hand-removed once they were shaded [51]. This was done to minimize boll rot during early boll opening. In addition, vegetative branches were also removed if the mainstem was well established. There may be utility in this practice for target spot, hard lock, and boll rot in the humid Southeast [52]. This method may also set up boll conditioning and a better plant architecture for robotic harvesting. Although a robotic harvester will gather early opening bolls, unless they fluff out, it may be difficult for a robot to pick them. Cotton generates many more squares on vegetative branches than needed to contribute to final yield [53]. Removal of these branches should not affect yield, assuming an adequate plant population, but will help narrow the fruiting window and reduce the need for a late-season insecticide application.

3.3. Harvest

3.3.1. Plastic Trash Removal

There are currently challenges in some cotton fields located near highways and urban areas with plastic trash littering fields, which is ultimately harvested with the cotton [54]. A potential near-term application for robotics systems is the use of high-resolution unpiloted aerial system (UAS) imagery to identify plastic and other contaminants present and the deployment of an unpiloted ground vehicle (UGV) or UAS to remove those items from the field prior to harvest [55]. Detection of plastic by UAS is currently based on multi-spectral cameras and thus can only detect plastic not obscured by the crop canopy; it is more feasible after the crop has been defoliated. Some improvement in detection would be expected for cameras deployed on agricultural equipment already making passes through the field.

3.3.2. Automated Yield Monitor Calibration

Current cotton yield monitors indirectly measure cotton mass flow based on light attenuation or microwave reflectance of seed cotton in the convey ducts and thus can require a variety of specific calibration factors [56]. Automation of cotton yield monitor calibration has been accomplished by the use of pressure sensors to measure the weight of the basket by monitoring the static pressure in the hydraulic lift cylinder circuit of a traditional basket stripper harvester. The software running the system was split into two parts that were run on an embedded low-level micro-controller and a mobile computer located in the harvester cab. The system was field tested under commercial conditions and found to measure basket load weights within 2.5% of the reference scale [57,58]. As such, the system was proven to be capable of providing an on-board auto-correction to a yield monitor for use in multi-variety field trials. The implementation sub-systems, electronic, micro-controller firmware and human-machine-interface (HMI) software designs are provided in Pelletier et al. [59,60,61]. Ongoing research is currently being conducted in a joint research effort between USDA-ARS and Texas A&M University (TAMU) to extend this system to include an optical cotton yield monitor that estimates mass flow of cotton bolls in the pneumatic air ducts. Such integrated systems promise to continue the trend of “smarter” agricultural equipment in the future.

3.3.3. Automated Material Tracking

Automated identification of cotton modules is already a possibility because of radio frequency identification (RFID) tags incorporated into the plastic wrap used to cover cylindrical cotton modules formed by John Deere’s harvesters. Each RFID tag contains a module identifier (module ID) that is unique to that module. Harvesters equipped with the HID Cotton Pro system from John Deere create a database of harvest-related data for each module using the module identifier as the primary key. The data files generated on the harvester can be manually downloaded onto a USB memory drive or wirelessly transmitted to a John Deere website for later retrieval. The module ID can be read from the RFID tag using electronic scanning tools and used to help growers and ginners manage modules and associated information gathered during the harvesting, storage, transportation, and ginning processes. To demonstrate the utility in this new identification system, an electronic module management system was developed that incorporates several RFID interrogation tools: (1) a mobile application for scanning modules by hand in the field or at the gin yard [62]; (2) a system for use on module trucks that automates the process of scanning modules when loaded or unloaded [63]; (3) a stationary bridge utility for scanning modules at the truck scale; and (4) a stationary bridge utility for scanning modules at the gin module feeder. Each time a module is scanned by one of these tools, the module ID is associated with a GPS location and client/farm/field ownership information. A data management utility was developed as part of the electronic module management system and compiles module-specific information from all data sources into one location for analysis and use by producers and ginners [62]. Two additional tools were developed that provide module and lint bale data to the electronic module management system: the Cotton Harvest File Download Utility and a PBI Logger Utility. The Cotton Harvest File Download Utility was developed by Cotton Incorporated and utilizes an API from John Deere to automatically download, unzip, and sort HID files into a file structure easily utilized by gin office staff and which can be easily imported into the data management utility. The PBI Logger Utility is a tool used in the gin to automatically scan the 1D barcode on the Permanent Bale Identification (PBI) tag affixed to each lint bale as it exits the bale press. The PBI Logger associates a timestamp and bale weight with each PBI when the tag is scanned. An algorithm titled “PBI to Round Module Mapping” was developed to automate the process of associating lint bales with the round module from which they were ginned. Associating lint bale PBIs back to the round module opens the door for fiber quality mapping at the field level once lint grade information is obtained from USDA-Agricultural Marketing Service (AMS) Cotton Classing Offices.

3.3.4. Frequent Harvest System

The ability to conduct multiple harvests after the first open boll could improve fiber quality and reduce yield loss due to extreme weather events [64]. The first cotton boll on a plant will be mature and ready for removal on average 50 days before the field is harvested under the current mechanical harvester system. The ability to frequently harvest the plant (5 to 10 times during the season) will reduce the risk of fiber damage and/or yield reduction due to extreme weather events. It will also limit the time white fly or aphid secretions result in what is referred to as “sticky cotton”. Finally, as the bolls harvested during the season will have developed under similar environmental conditions, the uniformity of fiber properties such as micronaire and length is likely to increase.

An enabling technology for frequent harvest will be the ability to use machine vision tools to detect cotton boll under various lighting conditions. The use of machine vision to identify cotton bolls is underway in the U.S. [65], India [66], and China [67]. Cotton is easier than other crops to identify for harvest as, when ready for harvest, it is white in contrast to green and brown backgrounds. Additionally, if the boll is open, it is ready to harvest, unlike some fruits where a specific color indicates the fruit is mature and ready for harvest.

3.3.5. End Effector for Cotton Harvest

A key requirement of a robotic cotton harvester is an appropriate end effector. Lab tests were conducted to estimate the power requirements for a suction end effector and found a minimum of 1 kW was required [68]. Because solar-recharged batteries would be ideal for multiple field robots, this level of power requirements appears to be excessive for in-field solar robots. Multiple potential versions of an energy-efficient end effector based on mechanical picking have been considered by those trying to develop new mechanisms to selectively remove seed cotton from the boll. Each require doffing and transfer of picked seed-cotton. One approach used in a prototype autonomous cotton harvest system in India has been to use a set of three cylindrical rotating pins at the end of a suction tube controlled by a robotic arm [69]. Another has been the use of four equally spaced mechanical fingers to grasp the cotton and remove it from the boll [70].

A challenge for both suction and mechanical approaches is the cotton boll orientation. Three potential solutions exist to deal with this issue: (1) utilizing a high degree of freedom manipulator that can face cotton boll along with artificial intelligence to calculate control actions such as is done for robotic fruit [71] and pepper [72] harvesting; (2) adding auxiliary components to the end effector to force the cotton boll to change its orientation; and (3) designing an end effector that can pick seed cotton without considering cotton boll orientation. A design that shows promise involves the use of rotating pins that could pull a boll into a suction device (Figure 1).

The concept for the finger end-effector is rotating pins. Multiple prototypes of this concept including one, two, and three fingers were manufactured and evaluated (Figure 2). The one-finger configuration has the lowest picking speed and it has difficulty transferring the seed cotton to the doffing mechanism. The three-finger configuration has the highest picking speed and the best penetration through the plant, but it tends to ingest not only the seed cotton but also the calyx and even leaves and branches. A potential solution is to move the end-effector forward and backward a few times during picking of the seed cotton so that the end-effector cannot ingest undesired material. The two-finger configuration does not tend to ingest undesired material, but it is more sensitive to cotton boll orientation. Therefore, if the open cotton boll is not facing the end-effector, the end-effector must rotate until its lower surface (not the tip) is face to face with the cotton boll. In future work, the two and three-finger end-effectors will be further tested and compared by attaching them to a system of linear actuators to conduct more precise tests to optimize the system.

3.3.6. Autonomous Cotton Boll Removal

Fue et al. [74] used a stereoscopic camera, machine vision processing, a deep learning network model (YOLOv3), and an embedded computer to manage computation of the images to identify cotton bolls in the field. A red rover was used as the platform for testing the system, as illustrated in Figure 3. Results have shown that bolls were identified and located with a high level of confidence using one camera looking downward with sparse foliage later in the season.

Cotton boll images used for training the YOLOv3 deep neural network (DNN) model were augmented 27 times using CLoDSA. CLoDSA is an open-source image augmentation library for object classification, localization, detection, semantic segmentation, and instance segmentation [75]. A total of 2085 images were collected and labeled, and images were augmented to provide a new labeled dataset of 56,295 images. The YOLOv3 model was used to train the dataset using a Lambda server (Intel Core i9-9960X (16 Cores, 3.10 GHz) with two RTX 2080 Ti GPUs, blowers with NVLink, and a memory of 128 GB, Lambda Computers, San Francisco, CA, USA). One thousand iterations provided the optimal performance for YOLOv3, and the training took only 4 h.

The platform (Husky from Clearpath Robotics) noted in [36] for weed control is designed to be retrofitted with different manifolds that perform specific tasks, e.g., spraying, scouting (having multiple sensors), phenotyping, weeding, harvesting, etc. Performance evaluation for the cotton harvesting was performed in terms of how effectively the harvester removed the cotton bolls and the effective distance [76,77]. Preliminary results on the performance of the developed mobile robot platform for cotton harvesting showed an average success rate of 57.4% in harvesting locks that are about half an inch close to the harvester nozzle. Further design enhancement was done in 2020 where a stripper mechanism was added and placed on the side of the Husky (Figure 4). The new design replaced the suction cap with a rolling stripper similar to a stripping machinery and used the same suction motor to move the harvested bolls to the bucket previously used in the first harvester prototype. The stripper on the side is driven by a single 24 V motor. Current efforts are examining the possibility of using the Husky as a once-over harvester.

Currently, there are additional harvest concepts being discussed. One example is one where there is non-selective harvest of a limited part of the plant (for example, the bottom fives nodes in the first harvest cycle) at a high rate of speed (no boll detection). A slower “gleaner” robot then follows using a machine vision system to collect any bolls that were not captured. Another concept is, rather than having multiple single row units, “intelligent” headers could be developed to allow some similar to the “red rover” in Figure 3, which could be adapted to a multiple row system still capable of multiple harvest passes through the same field. Additional concepts are under development and the economic models described in a later section will be an important component in ranking various concepts.

There is a clear interest in developing a harvest system that is capable of multiple passes through the field as it is anticipated that more frequent harvests will increase the quality of the cotton and reduce the risk of harvest loss due to severe weather events relative to a single end-of-year harvest event. Even with the most efficient harvesters operating in the timeliest manner, lower-positioned mature cotton bolls are left exposed to weathering for over 50 days while the upper position bolls mature and wait to open. This is particularly important in the areas of the U.S. where tropical storms that commonly occur in the fall can drastically limit lint yield. Multiple timing harvesting events will also allow for a greater uniformity of cotton fiber quality characteristics and fiber grades from each harvest event [78]. This fiber uniformity should provide additional market opportunities and premiums for farmers.

It may also be desirable to prune lower leaves and vegetative branches (“suckers” in viticulture lingo) once the lower bolls have been harvested. Selective application of a defoliant or boll opener during the robot harvesting to facilitate the next week’s passes may be advisable. In stripper-harvested areas, the use of robotic harvesters may allow for the avoidance of the desiccation pass, which in turn would lower production costs.

To better quantify the potential value of frequent harvests to calibrate economic models, frequent hand-harvest studies (with the goal to harvest two times per week after the first open boll) were conducted at two sites in Texas (an irrigated site near College Station and a non-irrigated location near Vernon), and one near Tifton, Georgia in 2018 and 2019 and also in west Tennessee in 2019. A similar experimental protocol was followed at all four sites. The primary treatments were: (1) frequent harvesting by hand throughout the season (no defoliants applied); (2) hand harvesting one time at the end of the season; and (3) machine harvesting one time at the end of the season following accepted defoliation practices. In 2018, all sites except for Georgia conducted the harvest treatments across two varieties, and, in 2019, all sites had two varieties. Additional details can be found in [79,80]. In a majority of the sites and years, color grades were consistently higher for the frequently harvested cotton relative to a single harvest at the end of the season. Yield impacts were not as consistent; however, in 2018, there was a significant yield advantage to the frequently harvested treatment compared to the end-of-season harvest treatments as Hurricane Michael impacted the Georgia site. All of the data sets from this study can be used in economic modeling of different cotton harvest systems.

3.3.7. Economic Models of Cotton Harvest

Two distinct methodologies have been applied to support economic analysis of robotic versus current mechanical harvest systems. One was based on potential capacity with a financial analysis. The second incorporated stochastic nature of weather and yield probabilities rather than relying upon deterministic metrics. Both perspectives have been developed into interactive dashboards such that interested individuals can enter farm-specific parameters.

Evaluation of the number of robots needed to replace the status quo systems relies upon a range of machinery and environmental parameters. Many of these attributes are farm-specific such that a single use-case would not be sufficient to provide global recommendations across cotton producing areas; therefore, interactive dashboards were created so that the end-user could not only enter their own parameters but change those parameters for a series of their own sensitivity analyses. A deterministic capacity calculator is currently available on the development site at [81], and a dynamic analysis dashboard is available at [82].

The deterministic model dashboard allows the user to select their chosen state to populate the calculator with data specific to their state from USDA-NASS. Days suitable for fieldwork [83] and crop progress for planting, percent open bolls, and harvest are collected and presented for exploratory analysis on the first table. The third tab allows the user to select the time window for two separate harvest systems (such as basket versus modulating picker, or modulating picker versus autonomous robotics). In addition to the harvest window that populates the calculator with the number of days to harvest given long term probabilities, the user can select machinery parameters such as field efficiency, ground speed, swath width, hours worked per day, days worked per week, etc. for each system. The user can also select the number of machinery units for either harvest system; this is specifically useful for comparing a single cotton picker, i.e., status quo, to swarms of modular robotics. Interactive graphs allow the user to change any of the aforementioned parameters to receive visual assessment of the probabilistic area potentially harvested. This partially answers the question of “how many small robots are needed to replace the status quo such that harvest is completed on the same date?” Parameterizing a whole-farm linear programming model using capacity metrics from the comparison above, returns to fixed costs for a series of scenarios can be calculated and then compared to a base scenario farm.

3.4. Ginning

As autonomous tractors become available [84], the ability to automate management of cotton modules on the gin yard could not only reduce labor requirements at the gin but also reduce human errors. Fiber bales leaving the gin could be automatically loaded in a truck or warehouse. There are already other industries making use of robotic systems for warehouse management [85], and these systems could be adapted for managing cotton bales at the gin.

There is a growing amount of automation occurring in the ginning industry including wide use of automated strap applicators and baggers at the bale press [86]. Systems have also been developed to automatically retrieve the classing sample, but it still requires a person to insert the barcoded tag and place it in the container for the classing office. Automatic baggers to wrap finished cotton bales are also gaining adoption by U.S. gins [87].

The removal of plastic contamination from cotton lint is an issue of top priority to the U.S. cotton industry. One of the main sources of plastic contamination showing up in marketable cotton bales is plastic used to wrap cotton modules produced by John Deere round module harvesters [88]. Despite diligent efforts by cotton ginning personnel to remove all plastic encountered during module unwrapping, plastic still finds a way into the cotton gin’s processing system. In order to help address this revenue-loss, engineers at the USDA-ARS Cotton Production and Processing Research Unit developed an automated robotic machine-vision-based detection-ejection system that was designed to rapidly identify and then remove these pieces of plastic from the cotton flow [89,90]. The location selected was on the gin-stand feeder apron, just prior to entering the gin stand, where the cotton is spread out to the thinnest stream in the ginning process. Thus, the feeder apron is the ideal location to apply a vision system to detect and remove plastic contamination flowing in seed cotton. One of the challenges to this location is that the detection operation must be high-speed as there is only a 0.5 m length of feeder apron on which to detect and then blow the plastic out of the seed cotton. Furthermore, at this location, the cotton flows down the feeder-apron at greater than 3 m s⁻¹, leaving the machine-vision software only about 25 ms in which to capture an image, analyze, and then set a digital output line that will actuate a solenoid to blow the plastic out of the seed cotton stream. To accomplish all of this within the short time-constraint, the software was written in C++ using a combination of custom software while leveraging high-quality open-source machine-vision libraries, with the primary library being OpenCV.

A prototype of the plastic-inspection-detection-ejection-system (PIDES) is shown in Figure 5 with unit in action shown in Figure 6. The system is now being marketed and sold by Lummus Corp. under the brand-name Visual Inspection Plastic Removal (VIPR) system.

3.5. Warehouse Operations

Several industries have automated warehouse operations that range from completely autonomous materials handling to collaborative robots [91]. Automated control of cotton warehouse operations would be ideal in the U.S., as every bale of cotton is uniquely identified, and the fiber properties (e.g., fiber length, strength, and color) are measured. The ability to automatically find bales has promise to increase the efficiency of warehouse operations both for cotton merchants and textile spinning mills. An initial study has been conducted to evaluate the efficiency of various bale handling methods in cotton warehouses [92].

4. Discussion of Potential Challenges to Cotton Automation

In discussions with U.S. cotton producers, potential challenges have been identified for autonomous farm applications. It is important to remember that any system designed for on-farm use must be reliable and extremely durable. Almost all field operations, particularly planting, pest control, and harvest, must occur within a narrow time window, and any delays due to equipment failure will be impediments to long-term adoption. It is also important to recognize there will be obstacles in the field such as large weeds, rocks, and deep ruts created by pivot tracks or tractors; therefore, robots with small-diameter wheels will not be able to reach all areas of the field.

Autonomous systems must be comparable in terms of efficiency and performance to agricultural machinery currently in use. While several smaller systems may be able to cover the same area as a larger piece of machinery, the operational velocities and efficiencies of many autonomous robotics are limited by the speed and performance of real-time vision-based detection and actuation algorithms. Additionally, for multiple smaller systems to adequately and efficiently cover the field, multi-robot planning algorithms and real-time coordination would be needed.

For equipment that will be unattended, vandalism and theft are concerns. Geofencing, the ability to transmit real-time images of any unauthorized personnel attempting to interact with the equipment is needed.

Field sizes vary across the country and it is common in the southeastern U.S. to have fields that range in size from 2 to 250 ha [15]. Furthermore, farm operations have fields that are commonly spread over a wide geography, such as 30 km or more in radius, and over poorly accessible roads. So, for applications where the equipment is expected to cover large areas, the logistics of transportation between fields must be carefully considered, especially if the vehicle must be transported on public roads as opposed to transported as a kit to be assembled on site.

In addition to being cost competitive with traditional systems, robotic systems need to have a short payback time, as it is anticipated that some of these technologies will change rapidly.

Another key point in considering automated systems for large area applications is that the system needs to be truly autonomous and must require minimal management time. Using the cotton harvest example, while many cotton producers find the risk reduction to extreme weather events that frequent harvest with a robot could bring, there is significant concern about executing a complicated system at harvest time. The current harvest system used in the U.S. is a once-over, round-module building harvester that can allow one person to harvest as much as 4 ha per hour. The dependability and simplicity of this system partially explains its rather quick adoption in the U.S., Brazil, and Australia. If multiple automated machines were needed to replace this single machine, they must be just as dependable and self-sufficient as the current system in addition to being economically competitive.

There is a great deal of societal concern about the use of robots in replacing human labor and creating employment crises across many disciplines, not just agriculture. In the case of labor for cotton harvest where mechanization is already prevalent, the impact on labor requirements will not be significant, as one to two employees are currently all that are needed to harvest approximately 2000 acres per year [83]. However, in parts of the world such as eastern China and much of India where cotton is still hand harvested, the impacts on farm labor are of more concern. Some lessons learned from the first wave of cotton harvest mechanization could help inform how robotic harvest may impact hand-harvested areas of the world. Several have speculated about the forces that drove the ultimate adoption of machine harvesting for U.S. cotton, with labor shortages being a significant, but only partial, explanation for the early adoption in some regions [93]. Ultimately, there is evidence that when yields became high and the per acre costs of the technology was lowered, workers were likely displaced by mechanization [94]. The World Bank recently completed a study on the potential societal impacts of cotton harvest mechanization in Uzbekistan [95]. From that study they concluded there would be situations where alternative incoming-earning opportunities would need to be created in the short-term, particularly to support women in rural communities if mechanical cotton harvesting is widely adopted. The report offers several measures that could be taken to minimize the negative impacts of mechanization, and the overall report does suggest the societal impact of robotic harvest on non-mechanized rural communities will have to be considered. A summary of challenges and potential solutions to increasing the automation of cotton production is provided in

Table 1. Summary of potential challenges to increasing the automation of cotton production systems.

Challenge	Potential Solution
Reliability/Durability	Service-based system so service company maintains the system. Modular parts for fast & easy replacement.
Field Obstacles	Collision avoidance and automatic path correction. Robust suspension system.
Vandalism/Theft	Mounted camera monitoring surroundings. Geofence.
Timely operations	Machine-to-machine coordination. Provide time-in motion data.
Small Fields	Assign a single machine to field. Automate transportation between fields.
Cost	Must be at least equal to current system. Decision aids needed to help compare.
Labor	Must decrease labor in mechanized production systems. The social implications of displaced labor needs to be considered for non-mechanized production systems.
Management	Must not increase farmers’ management requirements, so, must be a truly automated system.

.

5. Summary and Conclusions

Every aspect of the cotton production system could benefit from automation and/or robotics. Current uses of robotics in agriculture are aimed at weed control and also automating intensive sampling and scouting tasks, such as soil sample collection and plant monitoring. A summary of several of the commercial services or prototypes discussed in this review is provided in

Table 2. Examples of autonomous systems currently used in agriculture or with advance prototypes with application to cotton.

Automated System	Company, Country	Function	Web Site
Modified Bobcat T450	Robo Ag, Wolcott, IN, USA	Soil sampling	https://rogoag.com/ (Accessed 27 May 2021)
Between row gas powered track UGV	Rowbot Systems, Minneapolis, MN, USA	Cover crop planting In-season fertilizer	https://www.rowbot.com/ (Accessed 27 May 2021)
Multi-row autonomous gas platform with selective spray technology	Swarm Farm (platform), Gindie, Qld, Australia Weedit (spray control), CJ Steenderen, The Netherlands	Weed control as a service	https://www.swarmfarm.com/(Accessed 27 May 2021) https://www.weed-it.com/ (Accessed 27 May 2021)
UAS	Multiple. Examples: Precision Hawk, Raleigh, NC, USA. Drone Deploy, San Francisco, CA, USA.	Management zone development Stress detection Plastic detection	https://www.precisionhawk.com/(Accessed 27 May 2021) https://www.dronedeploy.com/ (Accessed 27 May 2021)
Xaver swarm between row electric units	Fendt (AGCO), Marktoberdorf, Germany	Planting	https://www.fendt.com/int/xaver (Accessed 27 May 2021)
Multi-row electric	Nexus Robotics, Halifax, NS, Canada	Weed control	https://nexusrobotics.ca/ (Accessed 27 May 2021)
AVO solar/electric multirow	ecoRobotix, Yverdon-les-Bains, Switzerland	Weed control	https://www.ecorobotix.com/ (Accessed 27 May 2021)
Multi-row electric	Farming Revolution GmbH, Ludwigsburg, Germany	Weed control as a service	https://www.farming-revolution.com/ (Accessed 27 May 2021)
Husky between row electric	ClearPath, Kitchener, ON, Cananda	Development platform	https://clearpathrobotics.com/ (Accessed 27 May 2021)
Electric between row or over row	Rabbit Tractors, Cedar Lake, IN, USA	Cover planting Soil sampling Spraying	https://www.rabbittractors.com/ (Accessed 27 May 2021)
VIPR automated plastic removal	Lummus Corp, Savannah, GA, USA	Removing plastic in ginning process	https://www.lummus.com/cottonginning (Accessed 27 May 2021)

and illustrates that the application of automated systems in agriculture has already started. The production of cotton, particularly harvest and ginning, have additional requirements beyond that of many agricultural commodities such as corn and soybeans.

Table 3. Field and gin activities that could benefit from automation and technology and/or hardware needed for implementation.

	Enabling Technology or Hardware
Field Activity	Forward Camera	Back Camera	Implement	RTK GPS	Machine Vision	Thermal Imaging
Initial Planting	X		Planter	X
Gap Fill Planting	X		Planter	X	X	X
Uncapping	X		Tillage	X
Stand Evaluation	X		Sensor	X	X	X
Curst Busting	X	X	Tillage	X
Sand Fighting	X		Tillage	X
Insect Control	X		Sprayer	X	X
PGR 1	X		Sprayer	X	X
Weed Control	X		Sprayer/Tillage	X	X
Harvesting	X		Rapid arm with end effector	X	X

¹ Plant Growth Regulator.

summarizes many of the operations during the cotton season that could benefit from robotic systems. Under certain weather conditions the ability to frequently harvest will increase yield and fiber quality. Key questions on the benefit side of the equation come from speed potential and reduced vulnerability to breakdown, especially considering harvest-related uncertainty coming from weather-driven threats to yield and quality. On the cost side, questions arise from the number of robots envisioned and the variety of tasks that each robot can be expected to support. Enabling technologies that will accelerate the speed of agricultural automation include open-source codes systems such as ROS, the increased availability of image data sets specific to cotton to train machine vision systems for pest detection and identification of cotton bolls, and advancements in deep learning architectures for efficient object classification, detection, and segmentation.

In the future, support will be needed to continue to enable autonomous weed control and harvest applications for cotton. Open source sharing of code and image libraries is a key strategy to accelerate agricultural automation. An increased effort to create open-source image libraries of cotton parts and weed species important to cotton to encourage commercial interest working in these areas to adapt their systems to cotton applications is needed. The ability to simulate different field conditions (e.g., lighting, crop size, and soil background) to augment image data sets is also a priority.