A Graphical Model to Diagnose Product Defects with Partially Shuffled Equipment Data
Round 1
Reviewer 1 Report
This paper focused on the diagnosis of product defects, especially the case when manufacturing industry has a data collection issue of partially shuffled data problem. The authors propose a graphical model to diagnose product defects with partially shuffled equipment data. The proposed graphical model calculates the probabilities that products are shuffled, and the expectation of the probability that a product is defective considering the shuffling probability. In this context they introduce latent variables to formulate a supervised learning model to solve the problem. The authors also provide a numerical example to show the application of the proposed model. Classification performance of the defective diagnosis is compared between the proposed model and the one that ignores the shuffling. Performance measure for the comparison is F1-score.
Overall, the paper presents key components of a research paper, and well meets the goal of the special issue: “Fault Detection and Process Diagnostics by Using Big Data Analytics in Industrial Applications.” It provides many meaningful results to the readers in both industry and academia and presents a guidance for obtaining the framework in fault detection and process diagnosis when there exist some invisible processes among the entire manufacturing procedure.
The paper, however, still requires some minor revise works. After revising them, it can be positively considered to be included in the special issue. The detailed comments are listed as follows:
1. This paper uses both terms “sample” and “record”, but it is recommended to choose one of them and use it only for simplicity.
2. Decision trees and support vector machine are not probabilistic classifiers, but the authors say that “other classification models (e.g., decision trees, support vector machines) should yield similar results” in subsection 4.4. It is necessary to explain how these models and a logistic regression can yield similar results.
3. Indent level of line 178 is different with other lines, and it should be corrected.
4. The author should explain why F1 score is employed as a performance measure in the experiment.
Author Response
Please see the attachment: 191202_Review_Answer_1.pdf
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Paper Summary.
In this paper, the authors proposed a probabilistic graphical model to diagnose product defects with partially shuffled equipment data. The unique feature of this approach is that the model consists of two layers: the first layer calculates the probabilities that products are shuffled, and the second layer calculates the likelihood that a product is defective considering the shuffling probability.
Strengths.
- The topic is relevant to the theme of this journal.
- The model definition is exact, and it contributes to the replicability of the proposed method.
Weaknesses.
- The reviewer cannot understand the effectiveness of this work. The authors seem to use pseudo or manually generated data rather than raw data obtained from the wild. The method is well-defined, but there is no evidence for clarifying the effectiveness of this model.
Author Response
Please see the attachment: 191202_Review_Answer_2.pdf
Author Response File: Author Response.pdf
Reviewer 3 Report
In the present paper the authors raise a very interesting problem of training data inconsistency in machine learning. This problem is very general, and the applications could go far beyond defect analysis in manufacturing. The authors mention it in the conclusion (“in addition, we can apply this model to other fields such as traffic management and bio-informatics”). It would be really nice to see more precise ideas and descriptions here.
Besides, the paper lacks a review on whether a similar problem has already been studied in machine learning in other contexts.
The advantage of the paper is a clear explanation of a rather complicated notation system. However, a few things can possibly be improved:
- the authors never explain the idea of parallel independent machines at each workstation, the reader has to guess it from the figures and formulas. Figure 4 explains it well, but is closer to the ending of the paper.
- one key concept here is the shuffle measure R, which appears in line 199 for the first time and is explained in line 202 with a formula. I guess this concept needs more attention and should be introduced from the beginning. Also, the explanation from lines 153-154 could be moved either to the beginning of the 3rd section or even to the introduction.
- time increment in line 99 and onwards is denoted as 1, this needs either a comment or a different notation.
- In Algorithm 1, the external loop over the product p is not written explicitly, but only implied, and probably not all the updates of Ij,k are stated clearly.
The authors present F1 scores as the criterion of the method efficiency. It would be useful to also demonstrate other measures which reflect the performance of the classifier on true negatives. A reflection on the potential harm of “false alarms” and false positives in different cases and on how to chose a proper classifier in each case would also be advantageous here.
The authors emphasize the evident fact that the model not taking into account possible shuffling yields worse results that the one which does. Perhaps some other, less naive approaches could be considered, such as e.g. omitting features in case of uncertainty. At least a general reflection on this subject could be added.
Some minor language corrections/corrections of misprints are needed:
- line 83: “this is the first study”
- line 124: check the grammar of the second part of the sentence;
- line 127: “one can only know if each machine is busy”
- lines 130-131: two parts of the sentence are not connected, it looks unnatural to join them;
- line 156: check the grammar.
Author Response
Please see the attachment: 191202_Review_Answer_3.pdf
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
well-revised.
