A Soft Gripper Design for Apple Harvesting with Force Feedback and Fruit Slip Detection

Round 1

Reviewer 1 Report

Review 

 

This paper presents a soft gripper in a well known fin ray structure with a commercialized thin film pressure sensor on the contact surface. 

Although authors provide mechanical analysis of the fin ray soft gripper, it is not well connected with the authors' claim in the experiment.

Many claims that authors made in the experiment senction are not well supported with analysis or with data.  

 

The soft gripper of fin ray structure is used extensively since its commercialization by Festo [1].

THe mechanical properties of apple skin is well studied, for example, in [2].

The reported damage seems to be caused by the rigid part of the gripper, which is nothing to do with the analysis in Sec. 2 or the slipp detection.

It could be improved by using a larger gripper to avoid the solid-gripper-part to the apple contact or a better sensing system so that the apple can be grasped by safe soft part of the gripper. 

There is no direct evidence that shows the improvement of apple grasping is due to the slip detection because the soft gripper is not hard enough to give a damage to the apple skin. 

 

Apple picking mechanics is well studied in [44], but authors cite this only for the gripper alignment. 

 

In order to verify the effectiveness of the gripper more properly, the test should be performed with a real apple tree in outdoor orchard.

 

 

[1] Wilson, M. (2011), "Festo drives automation forwards", Assembly Automation, Vol. 31 No. 1, pp. 12-16. https://doi.org/10.1108/01445151111104128

 

[2] M. Grotte , F. Duprat , D. Loonis & E. Piétri (2001) MECHANICAL

PROPERTIES OF THE SKIN AND THE FLESH OF APPLES, International Journal of Food

Properties, 4:1, 149-161, DOI: 10.1081/JFP-100002193

Author Response

Point 1: This paper presents a soft gripper in a well known fin ray structure with a commercialized thin film pressure sensor on the contact surface.

Although authors provide mechanical analysis of the fin ray soft gripper, it is not well connected with the authors' claim in the experiment.

Many claims that authors made in the experiment section are not well supported with analysis or with data.

Response 1: Thank you for the reviewer’s suggestion. This paper proposes an apple harvesting gripper with force feedback, and the force feedback function is realized through the torque feedback of the servo instead of an additional sensing system. The purpose of this is to simplify the structure of the gripper. Therefore, we conducted a mechanical analysis of the soft gripper to obtain the relationship between the output torque of the servo M_d and the force acting on the surface of the apple by the fingers f_c. We hope to construct the sensing system of the soft gripper through the relationship between them. Therefore, in the experimental section, our force feedback system is based on the conclusions obtained from the above mechanical analysis. The commercialized thin film pressure sensor is a tool for our experiments, and since it produces errors as the finger bends, we only perform a qualitative analysis on the linear sensing system in Sec. 5.2.1.

Thanks to the reviewer’s comment, we notice there are some flaws in the experiment section. In the revised manuscript, we modified the experimental design to further demonstrate the effectiveness of slip detection in Sec.5.2.2.

Point 2: The soft gripper of fin ray structure is used extensively since its commercialization by Festo [1].

The mechanical properties of apple skin is well studied, for example, in [2].

Response 2: Frankly speaking, the structure of Fin Ray gripper has been widely used in many fields since its commercialization by Festo [1], as well as has been widely investigated thanks to its adaptively wrapping ability to avoid damage.

Different from many existing researches, e.g., a multi-material structure gripper in one piece with Fin-Ray fingers [23]; the impact of each structure parameter on the finger gripping force [25-27], we investigate the effect of the front and rear beam thickness, the finger width, and the number of cross beams on the finger gripping force and bending degree in this work, as shown in Table 2.

Table 2. Comparison of previous research and our study on the structure of the Fin Ray structure.

Literature	Independent variable				Variable
[25]	Material		Rib orientation		Stress	Displacement
[26]	Angle	Infill density		Rib thickness	Contact force
[27]	The number of ribs	Outer Wall Slope		The slope of the ribs	Stress	Displacement
Our study	The front and rear beam thickness	The finger width		The number of cross beams	Stress	Displacement

The front beam is in direct contact with the fruit, the width of the finger is related to the coverage of the fruit, and the cross beams are the core rigid elements of the finger with the Fin-Ray structure, so these three structural parameters directly affect the gripping force and contact area of the fruit, which is also the reason why we chose these three structural parameters as our study. There is no report found yet which considers the above aspects to the authors’ best effort. In addition, we also deduce the analytical model of the gripper, construct the sensing system of the soft gripper by solving the relationship between the output torque of the servo M_d and the gripping force f_c., and verify its effectiveness through experiments.

Moreover, the analysis of the properties of apple skin is not the core content of this paper, and the description in Sec.5.1 is mainly based on the consideration of the content integrity of the paper. The test results show that they were consistent with those in literature [2].

To address the above concerns, we made the following revisions:

• To make the paper more rigorous, the relevant literature of Fin Ray gripper and mechanical properties of apple skin have been supplemented in the introduction section;
• To make the purpose of the study clearer, we further clarified the substantive contribution of this paper in contribution;
• To increase readability, we have simplified Section 5.

Point 3: The reported damage seems to be caused by the rigid part of the gripper, which is nothing to do with the analysis in Sec. 2 or the slip detection.

There is no direct evidence that shows the improvement of apple grasping is due to the slip detection because the soft gripper is not hard enough to give a damage to the apple skin.

Response 3: Indeed, the rigid part of the gripper is one aspect that causes apple skin damage. But the same is true of slippage. We can clearly see from Figure 27 that slippage can lead to damage. This is because during harvesting, when the gripping force is increased but not enough to separate the fruit stem, the gripper and the fruit inevitably occur friction. Due to the rough surface of the fingers, it is easy to cause damage due to scratches and bruises caused by the fingers’ rough surface, as shown in Figure 11.

The reviewer‘s concern is important in proving the effectiveness of our designed gripper. Although soft fingers can cushion the stress damage caused by gripping, there is no 100% guarantee that sliding damage will not occur, which is the main motivation of our research in this paper. We feel sorry for failing to clarify the improvement of apple grasping is due to slip detection. In the revised manuscript, we supplemented the experimental details and added a remark in Sec.5.2.2 to further clarify this issue. The relevant content has been highlighted in yellow.

Point 4: It could be improved by using a larger gripper to avoid the solid-gripper-part to the apple contact or a better sensing system so that the apple can be grasped by safe soft part of the gripper.

Response 4: The reviewer made a very insightful comment. On the one hand, due to the complex growth environment of the fruits, including the blocking of branches and leaves, and the overlapping of fruits, the large gripper doesn't seem to be very beneficial to fruit harvesting, as shown in Figure 4. On the other hand, using a larger gripper cannot completely guarantee that the fruit will not be damaged. When the fruit is completely freed from the gripper, the local stress will still cause damage to the fruit, which can be proved from Sec.5.2.2 in Figure 27.

Figure 4. A large gripper harvests apples.(The figure please see the attachment).

At the same time, the purpose of this paper is to simplify the sensing system, so we choose to realize the sensing system of the gripper through the feedback information of the servo itself rather than adding additional sensing systems.

Point 5: Apple picking mechanics is well studied in [44], but authors cite this only for the gripper alignment.

Response 5: We are sorry for the citation error in reference [44] ([42] in the revised manuscript), and we have modified it in the revised manuscript. Thanks to the reviewer’s comment, we notice that we also missed some information from literature [42]. Bu [42] et al. studied the significant effects of TPU stiffness, grasp distance and fruit size on the tension of the end-effector, and we learned a lot from them, not just limited to the gripper alignment. When the grasp distance is 65mm, the tension performance is at its best, so we chose it as our gripper alignment both in the finite-element analysis、mechanical analysis and grasping. In addition, from the results of their response surface analysis, we can get that with the TPU stiffness larger, the tensile increases. However, the TPU stiffness should not be too large, which is easy to cause fruit damage. Therefore, we choose TPU 95A as the material for the fingers of our gripper. To clarify, we have indicated it in the text.

Point 6: In order to verify the effectiveness of the gripper more properly, the test should be performed with a real apple tree in outdoor orchard.

Response 6: Thank you for the reviewer’s constructive suggestion. To verify the effectiveness of the gripper more properly, we have added the orchard experiments in the revised manuscript.

Author Response File: Author Response.docx

Reviewer 2 Report

 

A soft gripper for apple harvesting is presented in this paper. The mathematical model for the pulling force is obtained, and the control strategies for the constant pressure feedback and slip detection are discussed. The gripping ability was experimentally investigated. Although the Fin Ray design is not novel, its specific application to gripping apples is interesting. The paper can be accepted for publication after careful revision. Several criticisms arise, which are as follows.

On Page 2, there is a gap between lines 62 and 63.

On Page 10, the authors called F either resultant or pulling force, and F_c either concentrated or gripping force. The definitions should be consistent to prevent confusion.

In Fig. 17, there seems to be some bias between the experimental data and the poly fitting line.

 

Author Response

Point 1: On Page 2, there is a gap between lines 62 and 63.

Response 1: We are sorry for the mistake. The issue of the gap has been addressed in the revised manuscript.

Point 2: On Page 10, the authors called F either resultant or pulling force, and F_c either concentrated or gripping force. The definitions should be consistent to prevent confusion.

Response 2: Thank you for the reviewer’s suggestion. To prevent confusion, we define the concentration force as F_c and the grasping force as f_c in the revised manuscript. In the kinematics analysis of the gripper, we define the resultant force in the x-axis direction as F; and in the actual grasp of the gripper, the pulling force is that the resultant force in the x-axis direction, so they are the same. To clarify, we explain it in lines 349 and 350 in the revised manuscript.

Point 3: In Fig. 17, there seems to be some bias between the experimental data and the poly fitting line.

Response 3: Thanks to the reviewer’s comment, we notice the experimental flaws. According to the comment, we have expanded the experimental data to 25 groups in the revised manuscript. After experimental data supplementation, the experimental data and the fitting line can be well matched. The R² can reach 0.92, which can be seen in Figure 17 (Fig 16 in the revised manuscript).

Author Response File: Author Response.docx

Reviewer 3 Report

This manuscript presents a three-finger soft gripper based on the Fin Ray structure for apple harvesting. Authors firstly determine the structure parameters influencing the grasping action by finite element analysis. Then, authors propose a control strategy with pressure feedback and slip detection based on the kinematic model of the soft gripper. The experiments show that the soft gripper can achieve satisfactory picking efficiency without compromising the quality of apples. I think this work is very interesting and meaningful for robot picking. The manuscript as a whole is logical, but some details need to be minor revised before publication.

 

1.      The abstract Section is suggested to explain the research background or significance in a short sentence.

2.      Line 61, there is an incorrect text format.

3.      Line 72, the Ref.18 in this paragraph is an incorrect citation.

4.      Fig. 5 has a weak self-explanation. It is recommended to add a subgraph of the physical structure similar to Fig. 4, to map the correspondence between the kinematic model and the physical structure.

5.      Line 263, “The servo drives the rocker to rotate clockwise when grabbing”, Fig. 5 shows that the rocker rotates counterclockwise.

6.      The formula 8, what is the term L_FC?

7.      The formula 9, what is the term L_R?

8.      In the Section 3.1, the formula derivation is not clear (maybe some variables are not explained clearly), please check carefully the correctness of the all formulas in the manuscript.

9.      The terms of all of the variables in the text are in italics.

10.  Line 327, what is the term φ?

11.  Please check the text carefully for grammatical errors.

Author Response

Point 1: The abstract Section is suggested to explain the research background or significance in a short sentence.

Response 1: Thank you for the reviewer’s suggestion. We have explained the research background and significance in detail in the introduction Section.

Point 2: Line 61, there is an incorrect text format.

Response 2: We are sorry for the mistake. The issue of the incorrect text format has been addressed in the revised manuscript.

Point 3: Line 72, the Ref.18 in this paragraph is an incorrect citation.

Response 3: The incorrect citation of the Ref.18 has been addressed in the revised manuscript.

Point 4: Fig. 5 has a weak self-explanation. It is recommended to add a subgraph of the physical structure similar to Fig. 4, to map the correspondence between the kinematic model and the physical structure.

Response 4: The comment is very helpful for the improvement of our manuscript. According to the suggestion, to better explain the kinematic model, we have added a subgraph of the physical structure in Fig. 5 (a), as a comparison of the kinematic model (Fig. 5 (b)).

Point 5: Line 263, “The servo drives the rocker to rotate clockwise when grabbing”, Fig. 5 shows that the rocker rotates counterclockwise.

Response 5: The representation in Fig.5 is correct, and we have modified the “clockwise” to “counterclockwise” in the text of the revised manuscript.

Points 6 and 7: The formula 8, what is the term L_Fc? The formula 9, what is the term L_R?

Responses 6 and 7: We are sorry for failing to clarify the variables L_Fc and L_R. L_Fc is the length of the Finger connector, and L_R is the length of the Rocker. To clarify, we explain them in the revised manuscript.

Point 8: In the Section 3.1, the formula derivation is not clear (maybe some variables are not explained clearly), please check carefully the correctness of the all formulas in the manuscript.

Response 8: According to the constructive comment from the reviewer, we have rechecked all formulas for correctness, and explained each variable in more detail in the revised manuscript.

Point 9: The terms of all of the variables in the text are in italics.

Response 9: We agree with the reviewer. In the revised manuscript, we have changed the font of all variables in formulas, figures and text to italic.

Point 10: Line 327, what is the term φ?

Response 10: φ is the angle between the uniform load and the y-axis, we differentiate them to φi (i = 1, 2, ……, m). To clarify, we indicate them in the revised manuscript.

Point 11: Please check the text carefully for grammatical errors.

Response 11: Thank you for the reviewer’s constructive suggestion. We have rechecked the text carefully for English writing errors.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

There are still three main concerns.

 

1) Literature study can be still improved further. For example, recently slip detection with the tactile sensor was studied by [1,3], FEA analysis [2]. 

 

2) I do not find a good relationship between the analysis and the slip detection and their experimental verification. Authors argue that their contribution is the analysis, but the damage detection is more related to slip detection. These two were never rigorously compared in Sec. 2. Although the analysis in Sec. 2 is well presented, in the result of Sec. 5, where is the finding of Sec. 2 used and how? 

 

3) Authors argue that they have performed real orchard experiments. However, no evidence is given in the revision. 

 

 

 

[1] Liu, Sandra Q., and Edward H. Adelson. "GelSight Fin Ray: Incorporating Tactile Sensing into a Soft Compliant Robotic Gripper." 2022 IEEE 5th International Conference on Soft Robotics (RoboSoft). IEEE, 2022.

 

[2] De Barrie, Daniel, et al. "A Deep Learning Method for Vision Based Force Prediction of a Soft Fin ray Gripper Using Simulation Data." Frontiers in Robotics and AI (2021): 104.

 

[3] Zhou, Hongyu, et al. "A Tactile-enabled Grasping Method for Robotic Fruit Harvesting." arXiv preprint arXiv:2110.09051 (2021).

Author Response

To indicate how we address the received criticism, the point-to-point responses are as follows and the corresponding revisions in the resubmitted manuscript were highlighted in yellow with the notation "Rx-Cy", where ’x’ denotes the number of the reviewers, and ’y’ denotes the number of the comments. (Note: the mark will appear when the mouse cursor is over the highlighted texts.)

Comment 1: Literature study can be still improved further. For example, recently slip detection with the tactile sensor was studied by [1,3], FEA analysis [2].

[1] Liu, Sandra Q., and Edward H. Adelson. "GelSight Fin Ray: Incorporating Tactile Sensing into a Soft Compliant Robotic Gripper." 2022 IEEE 5th International Conference on Soft Robotics (RoboSoft). IEEE, 2022.

[2] De Barrie, Daniel, et al. "A Deep Learning Method for Vision Based Force Prediction of a Soft Fin ray Gripper Using Simulation Data." Frontiers in Robotics and AI (2021): 104.

[3] Zhou, Hongyu, et al. "A Tactile-enabled Grasping Method for Robotic Fruit Harvesting." arXiv preprint arXiv:2110.09051 (2021).

Response 1: Thank you for the reviewer’s insightful comment. To improve the quality of our paper, we have carefully read the three literatures recommended by the reviewer. And to enrich the article content, we have cited them and added some comments in Sec. 1. In our recent study, the control strategy of the slip detection proposed with the distance sensor cooperating with the servo in this paper has performed well in experiments, which is sufficient to support our contributions. Thanks again for the reviewer's suggestion, and we are also interested in these related novel studies. In the future research, we would like to draw on the reviewer's suggestion to further improve the accuracy of force feedback and slip detection.

Comment 2: I do not find a good relationship between the analysis and the slip detection and their experimental verification. Authors argue that their contribution is the analysis, but the damage detection is more related to slip detection. These two were never rigorously compared in Sec. 2. Although the analysis in Sec. 2 is well presented, in the result of Sec. 5, where is the finding of Sec. 2 used and how?

Response 2: Thanks for the reviewer’s suggestion. We agree with the reviewer’s opinion that the relationship between the analysis and the slip detection did not seem to be well in the previous manuscript. To help the reviewer better understand this paper, the article context is shown in Figure 1.

Figure 1. The article context.

First, the finite element analysis (FEA) in Sec.2 is aiming at the optimization of Fin-Ray finger structure. Due to the nonlinearity and complexity of soft finger deformation, it is more difficult to conduct research through theoretical analysis. The purpose of the finger’s structure study is to ensure the grasping stability of the gripper. That is, there should be enough gripping force on the surface of the fruit to separate it from the tree and a large amount of the finger deformation for the better covering of the fruit. (In the FEA method, we used contact stress and fingertip displacement to characterize these two parameters.) In Sec. 2, we analyze the structural parameters of fingers is only to provide a stable mechanical structure design of the gripper, but did not elaborate much on fruit damage. And according to the conclusion obtained after FEA optimization, the 3D-printing fingers are carried out, which provides a basis for the subsequent research. The problem of avoiding fruit damage due to clamping is controlled by the force feedback system. Second, slip detection is realized by the control strategy of the distance sensor cooperating with the servo to protect the fruit from damage, which is on the basis of the gripper design. Therefore, in our study, finite element analysis (FEA) and slip detection are two nearly independent parts. In other words, the slip detection proposed in this paper can also be used on other grippers. The finite element analysis (FEA) provides a stable working platform (the gripper) for the realization of slip detection.

On the other hand, the analysis in this paper consists of two parts: one is the finite element analysis (FEA) in Sec. 2.1, and the other is kinematic mechanics analysis in Sec. 3. The former is to provide a stable gripper design, and the latter is to build the gripper's force feedback system. As mentioned before, we did not study fruit damage in Sec. 2. By observing experimental phenomena and reading related literatures (for example, literature [39] and [40] in the revised manuscript), the fruit damage is mainly caused by excessive gripping force and fruit slippage. Therefore, the contributions of this paper are mainly carried out around them, that are gripping force feedback and control strategy of fruit slippage. In addition, the experimental design is also mainly focused on these two points. First, we verified the effectiveness of the force feedback system in preventing fruit pinching in Sec. 5.1, and verified the linear relationship of the force feedback control system. In Sec. 5.2 of the outdoor orchard experiment, the cross-analysis of fruit damage due to clamping and slippage was conducted, and it was found that no fruit was damaged due to clamping, which further proved the effectiveness of the designed force feedback system in preventing fruit damage. That is, the validity of the mechanical analysis on it. At the same time, the effectiveness of slip detection in preventing fruit damage was also demonstrated.

Finally, as mentioned before, we characterize the grasping performance of fingers by contact stress and fingertip displacement in Sec. 2. Therefore, there are mainly two reasons why we did not perform the physical experiments on FEA in Sec. 5:

• When finite element analysis (FEA) is used to optimize the structural design, the cross-parameters of the orthogonal test will produce many situations. In actual experiments, it is impossible for us to do physical experiments for every situation. In our study, if all the finger structures of each parameter permutation and combination are 3d-printed out for physical experiments, it will consume a lot of manpower and material resources.
• The contact stress and fingertip deformation are difficult to measure in practical experiments, especially fingertip deformation. In the actual deformation of the finger, we have no way to calibrate its initial position and final position and measure its displacement.

To sum up, we have not performed practical experiments on finite element analysis (FEA) in Sec. 5. And to the best of our knowledge, there are very few examples (for example, literature [25] and [27] in the revised manuscript) of practical experiments on the finite element analysis (FEA) method in the related studies. Universally, the experiments are usually carried out based on the experimental effects of the involved grippers, such as testing the weight and effectiveness of grasping objects, etc. In our study, both force feedback and slip detection are carried out on the basis of optimized harvesting gripper, so the results of the experimental part in Sec. 5.2 can be regarded as the effect of combining these three parts. Moreover, the optimized harvesting gripper with force feedback and slip detection has a good picking effect from the experimental results and analysis.

Thanks to the reviewer’s comment, we notice the importance of clarifying these issues and have added some explanations and remarks in the revised manuscript. The relevant content has been highlighted in yellow.

Comment 3: Authors argue that they have performed real orchard experiments. However, no evidence is given in the revision.

Response 3: We feel sorry for failing to clarify the real orchard experiments. In the revised manuscript, we have added the details of the real orchard scene, and there were some pictures of the outdoor experiment scene that day in Figure 2. Due to the season, we can see that the apples are not fully ripe. And we can also see through the injured fruits in Figure 3 (Figure 27 in the revised manuscript) that the fruits are still a little green in August. However, it's a pity that we didn't make some videos and pictures of the end of the experiment at that day.

Figure 2. The outdoor experiment scene.

Figure 3. Two damaged apple conditions in the outdoor test.

We are sorry that the pictures can not seem to show up here, and we have added a response letter in the attachment.

Author Response File: Author Response.pdf