Why Is Deep Learning Challenging for Printed Circuit Board (PCB) Component Recognition and How Can We Address It?

Round 1

Reviewer 1 Report

This report is useful to recognize especially different shape of capacitor and resistor by using ECLAD Net.

Please check the following parts.

1. In the title, how about addition of “components” or “surface mounted parts”?

Why is Deep Learning Challenging for Components of Printed Circuit Board (PVB) Assurance and・・・

2. Please mention the relationship between pixels number (data amounts) and time to classify the components. Also please write the effects of shape, size and color.
3. There are many components on PCB. Please mention the recognition possibility about the cylindrical capacitor and coils.
4. Please write somewhere “ This method is useful to reuse and recycle of components on PCB”.
5. There are many abbreviations. How about writing the abbreviation lists?

Author Response

Authors’ comments for Review Report 1

Open Review

(x) I would not like to sign my review report
( ) I would like to sign my review report

English language and style

( ) Extensive editing of English language and style required
( ) Moderate English changes required
(x) English language and style are fine/minor spell check required
( ) I don't feel qualified to judge about the English language and style

Authors’ Comments for Rating:

Thank you for the suggestion. The manuscript has been checked for grammatical and formatting errors.

Reviewer’s Comments

Authors’ Comments for Rating:

We thank you for your comments, and thus we have worked on improving the entire document with special focus on the methods section and the introduction. The revised manuscript includes a more detailed introduction as well as explanations and new cited material.

Comments and Suggestions for Authors

Reviewer’s Comments

This report is useful to recognize especially different shape of capacitor and resistor by using ECLAD Net.

Please check the following parts.

Reviewer Comment 1:

1. In the title, how about addition of “components” or “surface mounted parts”?

Thanks for the comments, we have modified the title to reflect this idea. The new title reads “Why is Deep Learning Challenging for PCB Component Recognition and How to Address it?”

Reviewer Comment 2:

1. Please mention the relationship between pixels number (data amounts) and time to classify the components. Also please write the effects of shape, size and color.

Thanks for the comments. Great suggestion, and we are looking into them as future work for the manuscript. While we have provided information about the image size in the manuscript (Lines 194-195, and also Table 1), the effect of shape, size, texture and so forth and how they impact not only ECLAD-Net but PCB Assurance in general is a separate branch that we are currently exploring.

With regards to this paper, we believe this comment lies outside the scope; while we have presented the edge cases and the challenges in the results section, we do not claim to have the best method. We only introduce problem to both the security and the computer vision community and present a solution that succeeds over current methods, but it is still far from perfect. Analysis such as the ones you have suggested, and others are currently under our investigation and will be presented in future works.

 The execution time of classification cannot be compared as the methods are very different in terms of computational complexity and computing elements. While ECLAD-Net is the fastest amongst the discussed methods, the other methods have deeper networks that will naturally take a long time (possibly in days). Our discussion in this work is about the capabilities of the present system over the other networks despite the longer time taken.

Reviewer Comment 3:

1. There are many components on PCB. Please mention the recognition possibility about the cylindrical capacitor and coils.

Thanks for the comments. We have included a few statements about this comment, and in fact cited one of the papers that have done recognition of cylindrical capacitors. We also discuss about coils and other larger components. Lines 158-161 specifically and the same paragraph in general describe this revision. However, we believe that we have to consider comprehensive analysis one at a time. Some of the works in the literature have tried to perform PCB assurance with multiple components, however they have poor performance for small components such as resistors and capacitors. Our focus is to encompass every aspect, therefore we start with the challenging ones and tackling them ourselves and then build up towards the larger components either by direct/indirect utilization of the existing literature or enhancing their contributions.

Reviewer Comment 4:

1. Please write somewhere “This method is useful to reuse and recycle of components on PCB”.

Thank you for the comment. We include a discussion in paragraphs 1 and 2 of the Introduction and in line 36 about the effects of counterfeit and recycled components in. In addition, we also have mentioned about the application of reused and recycled components in lines 482-483 under the Conclusion section.

Reviewer Comment 5:

1. There are many abbreviations. How about writing the abbreviation lists?

Thanks for the comments. We have included an abbreviation list at the end of the document with every abbreviation we have used.

 Reviewer 2 Report

The paper presents an interesting image classification application. The structure of the document should be improved in some parts:

- the datasets used should be described through tables with information such as number of images, number of classes, rate of imbalance, to improve readability.
- The experiments should be measured with additional performance measures such as F-measure (to be included in tables 1 and 2).
- Further comparison approaches different from deep learning should be included.
- Details on software development should be included.
- It would be worth mentioning the following work which concerns the same approach but with different application 

Manzo, M., & Pellino, S. (2020). Bucket of deep transfer learning features and classification models for melanoma detection. Journal of Imaging, 6(12), 129.

Author Response

Author’s comments for Review Report Form 2

Open Review

(x) I would not like to sign my review report
( ) I would like to sign my review report

English language and style

( ) Extensive editing of English language and style required
(x) Moderate English changes required
( ) English language and style are fine/minor spell check required
( ) I don't feel qualified to judge about the English language and style

Authors’ Comments for Rating:

Thank you for the suggestion. The manuscript has been checked for grammatical and formatting errors.

Authors’ Comments for Rating:

We thank you for your comments. We have worked on improving the suggested sections including changes to the methods, and the results. We have included new material in both the introduction as well as the conclusion to provide the necessary information and details.

With regards to the research design, we believe that we have introduced the challenges of deep learning to hardware security and presented an initial proof of concept on the possibility to address problems in hardware assurance with specialized work. We also compare our work with existing research and discuss challenging scenarios and edge cases for our work. In our future works, we will proceed to work on ECLAD-Net with more innovations and insights.

Comments and Suggestions for Authors:

 The paper presents an interesting image classification application. The structure of the document should be improved in some parts:

Reviewer Comment 1:

- the datasets used should be described through tables with information such as number of images, number of classes, rate of imbalance, to improve readability.

Thank you for the suggestion. We have consolidated the essential image statistics in Table 1 including the number of original images, the number of samples after class balancing, and the number of augmented images. To tackle class imbalance, we made sure to select a subset with the same number of images, which is now presented in the revised manuscript.

Reviewer Comment 2:
- The experiments should be measured with additional performance measures such as F-measure (to be included in tables 1 and 2).

Thanks for the comments. Some of the commonly used metrics include True Positive Rate, False Positive Rate, True Negative Rate, False Negative Rate etc. These are not utilized in our work since the recall and precision scores can adequately demonstrate the same information. However, we describe Recall and Precision and their meaning in the context of the work better now. Also, we have included their equations in the manuscript under section 4.5, from lines 340 to 358. ROC and AUC measures are not fitting for our methods since there are two classifiers present, meaning there is no possibility of iterating over thresholds and preparing such curves. However, F1 Score, as you have suggested is a fitting evaluation metric for our case and we have incorporated it in the manuscript, along with a discussion about it in lines 422 -423. The entire results section has also been slightly modified to reflect the new changes.

Reviewer Comment 3:
- Further comparison approaches different from deep learning should be included.

Conventional image processing methods and their application in PCB assurance have mainly been for defect detection. There haven’t been many implementations for PCB component detection using conventional methods that don’t require a reference image. We have described these works in paragraphs 1 and 2 of the related works section, with additional information now, starting from lines 134; however, their methods are in a different direction to be compared with our study.

We have tried to keep the focus of this manuscript on Deep Learning and its challenges for PCB assurance. We want to answer the common question that is raised in the security community “Why not use Deep Learning to solve PCB assurance and component detection problem?”. To this end, we present the problem, its challenges, compare them with existing popular object detection techniques, and also present an initial solution to the problem that can be explored and developed on further in the future.

Reviewer Comment 4:
- Details on software development should be included.

Thank you for the suggestion. We have now included details on the programming language as well as the libraries used for developing the algorithms in lines 205 to 207 under the Experimental Specifications section.

Reviewer Comment 5:

It would be worth mentioning the following work which concerns the same approach but with different application: Manzo, M., & Pellino, S. (2020). Bucket of deep transfer learning features and classification models for melanoma detection. Journal of Imaging, 6(12), 129.

Thank you for the suggestion, we have now included similar problem scenarios that ECLAD-Net can be applied for, including the above paper and its details as concluding points in the Results section from lines 470 to 480.

 Round 2

Reviewer 2 Report

The paper can be accepted with the changes made. Despite this, it should be improved from the point of view of writing and form but not in contents.