A Comparative Study of Machine Learning and Deep Learning Techniques for Fake News Detection
Round 1
Reviewer 1 Report
The paper presents a comparative study of standard machine learning algorithms and deep learning algorithms.
The paper is well-written and has an extensive analysis of results in addition to a solid discussion.
My comment is about mentioning both accuracy and F1-score in the results Usually, if the dataset is balanced, accuracy can be used. Otherwsie, F1-score is a better score. I recommend that you look into this again.
Another comment is about the references. The topic handled in this paper is a hot topic in research and many papers are published everyday about it.
I recommend that the literature mentions some of the new papers such as:
https://link.springer.com/chapter/10.1007/978-3-030-90087-8_1
and:
https://www.researchgate.net/profile/Martin-Sarnovsky/publication/358326822_Fake_News_Detection_Related_to_the_COVID-19_in_Slovak_Language_Using_Deep_Learning_Methods/links/61fcf45b1e98d168d7ed15d5/Fake-News-Detection-Related-to-the-COVID-19-in-Slovak-Language-Using-Deep-Learning-Methods.pdf
One more point that should be raised either in the introduction or in the future work is that you are not considering word sense. It means, all these approaches work only on the structure. This article can be cited when you want to mention this limitation:
https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.1447
away from these comments, the paper handles good work,
Author Response
Thank you for your comments:
My comment is about mentioning both accuracy and F1-score in the results Usually, if the dataset is balanced, accuracy can be used. Otherwsie, F1-score is a better score. I recommend that you look into this again.
I have used F1 score to compare the results on GossipCop dataset since this dataset is imbalanced. The results section is modified. Please see the attachment.
Another comment is about the references. The topic handled in this paper is a hot topic in research and many papers are published everyday about it.
Those two papers have been added in the literature review section. Please see the attachment [Line 182 and Line 213]
Line 182: A systematic mapping study is conducted in [ 61 ] to analyze and
synthesize studies regarding the use of machine learning techniques for detecting fake news.
Line 213: A deep learning model was presented by the authors in [ 73 ] for the automatic detection of fake news written in Slovak. Several local online news sources were used to gather data related to the COVID-19 pandemic to train and evaluate a variety of deep learning models. A model combining a bidirectional long-short-term memory network with one-dimensional convolutional layers achieved an average macro F1 score of 94% on an independent test set.
One more point that should be raised either in the introduction or in the future work is that you are not considering word sense. It means, all these approaches work only on the structure. This article can be cited when you want to mention this limitation.
The limitation is mentioned at the end of the introduction, Line 61: Note that the presented approaches concentrate only on structure without considering word sense (i.e., resolving ambiguities in context).
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The paper presents a comparison of different methods of analyzing news content in order to determine whether it is fake.
The paper, while interesting, should be improved in the presentation.
The authors present a related work that does not seem to be coordinated with the choices of what to compare in the rest of the paper.
How are the 17 methods presented chosen?
The conclusions should explain the results and explain their meaning,
Author Response
Thank you for your comments:
The authors present a related work that does not seem to be coordinated with the choices of what to compare in the rest of the paper.
The related work is restructured, and we added a sentence at the beginning of the section to demonstrate that we want to summarize previous studies on fake news detection using ML and DL models.
How are the 17 methods presented chosen?
We added a small paragraph at line 324 to explain how we select these methods. Please see the attachment.
The conclusions should explain the results and explain their meaning
A section named "Discussion" before the conclusion is added and we revised the conclusion. Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
The article (i) reviews studies concerned with fake news classification and (ii) compares several classification models on four fake news datasets.
The authors did a good job of finding related work. However, the section is not so easy to read as it is a sequence of different studies (A did that and B did that). Moreover, often the performance of the machine learning algorithm is reported but not the underlying data (meaning, one cannot compare if e.g. random forest performed better than SVM if different data are used).
Only at the end of the section one finds four tables that summarize the main facts for different datasets. My recommendation would be to move these tables to the beginning of the section.
At the end of the article, the authors finally describe these datasets. It is confusing that they are not described earlier, at least a summary should be provided (i.e.., report e.g, size of the data, where it can be accessed, the timeframe of the data collection, etc.) with a reference to the sections, where the datasets are described in more detail.
Moreover, the tables should be referenced in the text, close to where they appear (now the tables are at the beginning of the manuscript, while they are referenced at the end of the manuscript.
Another confusing thing throughout the whole article is that usually, one speaks about a deep network if the network has at least two hidden layers. In this article, the authors claim that their networks with only one hidden layer are deep (see, e.g., Table 5, but also all other content in the article).
Moreover, the authors report related work on the same data but they do not summarize the conclusions on how their own efforts perform in comparison to related work. This is also found in the abstract, which is not clear and very vague: "We hope this study can provide useful insights for researchers working on fake news detection." -> Just summarize your insights and findings. For example, which model and feature extraction techniques are the best? And what are the advantages of the models by the authors compared to what was published already?
In addition, I have several smaller comments:
line 27: opinions. [2]. -> remove the serial stop before [2] (opinions [2].)
Figure 1: what is the "Note" bar? Please explain. Moreover, the x-axis has only four weird points. Please change it, so that it shows e.g., every half a year.
line 38: causes -> caused
line 39: article missing. Should be "the 2016 U.S. presidential election.."
line 57: In addition, We -> we
In the literature review, some studies (e.g. 18) are described in the present but they should be described in the past tense. For example, the following sentences should be changed: "First, they analyze various features of the news articles (including n-grams, LIWC, punctuation, grammar, and readability). Then, based on these features, a linear SVM classifier is trained." -> First, they analyzed .. Then, .. a linear SVM classifier was trained
Some paragraphs are way too long and very uneven (e.g. "Section 4.1. Embeddings" consists of only two paragraphs with the first paragraph taking almost one page, and the second being only a few lines long).
- between lines 363 and 364 is a large paragraph without line numbers
- line 462: "An ensemble of decision trees with labels is predicted using a tree-like model." -> Explain more clearly, this sentence does not make sense.
- line 464-466: "Using this method, the first model constructed using training data is used to construct a strong classifier, followed by a second model that attempts to correct the errors of the first model. " -> check the grammar and logic of this sentence. It does not make sense as it is.
line 494: The point is called BiLSTM, but only LSTM is described. The authors should at least define BiLSTM as bidirectional LSTM and explain the main difference between BiLSTM and LSTM
Figure 3 should be explained
- Repetitions should be removed (e.g., BERT standing for Bidirectional Encoder Representation from Transformer is defined several times in the article)
- What is the superscript of W in equation (15)? Please define/explain.
- Please refer to BERT base and BERT large consistently throughout the manuscript (they are referred to differently in the text, even in the same paragraphs). I prefer if "base" (and "large" respectively) are attached in subscript to BERT.
- Figure 4 is not helpful as it is. For example, it does not include the preprocessing of the news data. Please add the "whole story" or remove the figure.
- 4.6.1. LIAR -> LIAR
Author Response
Thank you for your comments:
Only at the end of the section one finds four tables that summarize the main facts for different datasets. My recommendation would be to move these tables to the beginning of the section.
The tables are moved to the appropriate place in the section (after introducing the work on each dataset used in our study).
At the end of the article, the authors finally describe these datasets. It is confusing that they are not described earlier, at least a summary should be provided (i.e.., report e.g, size of the data, where it can be accessed, the timeframe of the data collection, etc.) with a reference to the sections, where the datasets are described in more detail.
A reference to the table discerption of each dataset is referenced in the text so it is easier for the reader to navigate easily to the section where the datasets are described.
Moreover, the tables should be referenced in the text, close to where they appear (now the tables are at the beginning of the manuscript, while they are referenced at the end of the manuscript.
The tables are moved to the suitable place and are being referenced in the text.
Another confusing thing throughout the whole article is that usually, one speaks about a deep network if the network has at least two hidden layers. In this article, the authors claim that their networks with only one hidden layer are deep (see, e.g., Table 5, but also all other content in the article).
I have changed that to advanced ML models instead of DL models throughout the article.
Moreover, the authors report related work on the same data but they do not summarize the conclusions on how their own efforts perform in comparison to related work. This is also found in the abstract, which is not clear and very vague: "We hope this study can provide useful insights for researchers working on fake news detection." -> Just summarize your insights and findings. For example, which model and feature extraction techniques are the best? And what are the advantages of the models by the authors compared to what was published already?
We added, in the abstract, a sentence that demonstrates the effectiveness of our approaches: Compared to state-of-the-art results across the used datasets, we achieve better results with just news text and minimal text preprocessing.
In addition, I have several smaller comments:
line 27: opinions. [2]. -> remove the serial stop before [2] (opinions [2].)
Removed
Figure 1: what is the "Note" bar? Please explain. Moreover, the x-axis has only four weird points. Please change it, so that it shows e.g., every half a year.
The figure has been updated.
line 38: causes -> caused
Addressed
line 39: article missing. Should be "the 2016 U.S. presidential election.."
Addressed
line 57: In addition, We -> we
Addressed
In the literature review, some studies (e.g. 18) are described in the present but they should be described in the past tense. For example, the following sentences should be changed: "First, they analyze various features of the news articles (including n-grams, LIWC, punctuation, grammar, and readability). Then, based on these features, a linear SVM classifier is trained." -> First, they analyzed .. Then, .. a linear SVM classifier was trained
Addressed
Some paragraphs are way too long and very uneven (e.g. "Section 4.1. Embeddings" consists of only two paragraphs with the first paragraph taking almost one page, and the second being only a few lines long).
Addressed
- between lines 363 and 364 is a large paragraph without line numbers
Addressed
- line 462: "An ensemble of decision trees with labels is predicted using a tree-like model." -> Explain more clearly, this sentence does not make sense.
Updated. Please see the attachment
- line 464-466: "Using this method, the first model constructed using training data is used to construct a strong classifier, followed by a second model that attempts to correct the errors of the first model. " -> check the grammar and logic of this sentence. It does not make sense as it is.
Addressed please see line 492
line 494: The point is called BiLSTM, but only LSTM is described. The authors should at least define BiLSTM as bidirectional LSTM and explain the main difference between BiLSTM and LSTM
Updated. The difference is explained at line 530
Figure 3 should be explained
Figure 3 is explained at line 510
- Repetitions should be removed (e.g., BERT standing for Bidirectional Encoder Representation from Transformer is defined several times in the article)
The repetitions are removed
- What is the superscript of W in equation (15)? Please define/explain.
Addressed
- Please refer to BERT base and BERT large consistently throughout the manuscript (they are referred to differently in the text, even in the same paragraphs). I prefer if "base" (and "large" respectively) are attached in subscript to BERT.
The manuscript is updated accordingly.
- Figure 4 is not helpful as it is. For example, it does not include the preprocessing of the news data. Please add the "whole story" or remove the figure.
The figure is removed
- 4.6.1. LIAR -> LIAR
Addressed
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
The authors have, in this version, exceeded many of the limits highlighted in previous revisions.
The paper is therefore publishable in this form although I would suggest that authors do other experiments with other data sets.
Reviewer 3 Report
The authors addressed all comments sufficiently.
