ULAN: A Universal Local Adversarial Network for SAR Target Recognition Based on Layer-Wise Relevance Propagation
Round 1
Reviewer 1 Report
In general, I think it is worthy of publishing. Some points should be included within the manuscript in order to improve the publication.
1- The caption of figure 1 and figure 3 are so long.
2- Equations 1, 2 ,and 3 need more explanation.
3- In table 3 and table 4 , the highest accuracy number may be in bold text.
4- Although the presented application is interesting, the experimental study needs some improvements.
You need to apply your experiments on another dataset.
Author Response
Dear Editor and Reviewer:
Thanks for your letter and the reviewer’s comments concerning our manuscript entitled “ULAN: A Universal Local Adversarial Network for SAR Target Recognition Based on Layer-wise Relevance Propagation” (Manuscript ID: remotesensing-2061736). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our research. We have studied all comments carefully and have made conscientious corrections. Revised portions are marked in red on the paper. The main corrections in the paper and the responses to the reviewer’s comments are as follows:
Point 1: The caption of figure 1 and figure 3 are so long.
Response 1: We thank the reviewer for pointing this out. Figure 3 has been adjusted to Figure 2, and the captions have been revised as follows:
The caption of Figure 1 has been abbreviated to “The adversarial attacks with (bottom) and without (top) perturbation offset. Suppose the green box region is the perturbed region. The adversarial attack without perturbation offset means that the perturbed region must be exactly fed to the model. However, if the model takes as input the red box region that has an offset from the perturbed region, the incomplete adversarial perturbation is likely to make the attack fail.”
The caption of Figure 2 has been abbreviated to “Framework of the ULAN. The generator G(·) crafts the local UAP δ. The attention heatmap (hmap) of the surrogate model fs (·) locates the target (green box) region. Attackers obtain the adversarial example x* by adding δ to the target region and utilize it to attack the victim model fv (·). Finally, the total loss Lt formed by the attack loss La and norm loss Ln is used to update G(·).”
Point 2: Equations 1, 2, and 3 need more explanation.
Response 2: Thank you for your valuable suggestion. We provide the following supplement explanations for the equations, respectively.
In Equation 1, the universal adversarial perturbation (UAP) is a single perturbation that attacks the model independently of the input data. In brief, for most of the samples in the dataset plus the UAP, the generated adversarial examples can easily fool the model. (See lines 123-126)
In Equations 2 and 3, the discriminant function D(·) equals one if the equation holds; otherwise equals zero. Ctr and Cta represent the true and target classes of the input data. N is the total number of images in the dataset. By traversing all the samples in the dataset, Equation 2 can design a minor δ to minimize the probability that DNN models correctly recognize samples in non-targeted attacks. In contrast, δ designed by Equation 3 is to maximize the probability of models identifying samples as target classes in targeted attacks. (See lines 139-144)
Point 3: In table 3 and table 4, the highest accuracy number may be in bold text.
Response 3: We gratefully appreciate your rigorous advice. The best and critical experimental results have been changed to bold text in all the experimental tables. Meanwhile, to show the experimental data more clearly, we adjust the structure of the tables in Section 4.4, Section 4.5, and Section 4.7.3 by adding the row of mean values, which helps readers compare the attack performance of different methods more intuitively.
Point 4: Although the presented application is interesting, the experimental study needs some improvements. You need to apply your experiments on another dataset.
Response 4: Thank you so much for your valuable comment, and it is indeed necessary to verify the validity of our method on another dataset. Thus, we add Section 5 to discuss the attack of the ULAN against DNN models on the FUSAR-Ship dataset. The details of the dataset are displayed in Table A2, and the attack results under different conditions are shown in Table A3 and Table A4. The conclusion is that our method can fool DNN models on the FUSAR-Ship dataset by perturbing the target regions of SAR images, and the generated adversarial examples prevent perturbation offset effectively. (See lines 528-541)
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The paper is well written, and contributes a semi-white box attack network Universal Local Adversarial Network (ULAN). And the proposed method prevents perturbation offset and achieves comparable attack performance to conventional global UAPs by perturbing only a quarter or less of SAR image areas. But there are some problems, which must be solved before it is considered for publication. 1. There are some details that are not clearly illustrated: e.g. in Figure 10, it is not clearly illustrate that how the attack performance fluctuates when the amount of training data changes. There should be exact data support. 2. Part 4.6 will be more convincing if the experimental results of other methods are added 3. In 4.4 and 4.5, the L2-norm values are listed in Table 3 and Table 4, but the reason is explained at the end (4.7.3). Adjusting the order of this part will allow readers to better understand this paper. In summary, this paper made exact contribution to universal adversarial perturbation (UAP), but there are some problems to deal with.
Author Response
Dear Editor and Reviewer:
Thanks for your letter and the reviewer’s comments concerning our manuscript entitled “ULAN: A Universal Local Adversarial Network for SAR Target Recognition Based on Layer-wise Relevance Propagation” (Manuscript ID: remotesensing-2061736). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our research. We have studied all comments carefully and have made conscientious corrections. Revised portions are marked in red on the paper. The main corrections in the paper and the responses to the reviewer’s comments are as follows:
Point 1: There are some details that are not clearly illustrated: e.g. in Figure 10, it is not clearly illustrate that how the attack performance fluctuates when the amount of training data changes. There should be exact data support.
Response 1: Thank you so much for your valuable comment. We totally agree with you and provide data support in Table 6 and Table 7. For space reasons, the data only includes the comparison of the attack performance under extreme (a subset of 50 samples) and ideal (full dataset) conditions.
Point 2: Part 4.6 will be more convincing if the experimental results of other methods are added.
Response 2: We gratefully appreciate your rigorous advice. It is necessary to compare the experimental results of the ULAN with other baseline methods.
So, we consider an extreme situation where attack networks are trained on a subset containing only 50 samples (5 per class). Specifically, we uniformly sample 50 images from the full training dataset to form the subset and compare the attack performance of attack networks trained on the subset and full training dataset against different DNN models. The results of non-targeted and targeted attacks based on different size datasets are shown in Table 6 and Table 7, respectively.
As we can see, the reduction of training data seriously impacts the attack performance of the UAN and ResNet Genenrator. Although a slight deterioration in the Acc metric can be tolerated, the average decrease in the SSIM metric is nearly 0.2. It means that the above methods severely sacrifice the attack stealthiness for better attack effectiveness, which makes the generated adversarial examples can be easily detected by defenders. However, the ULAN and U-Net still maintain good attack effectiveness and stealthiness under small sample conditions. The average change of the Acc metric in both attack modes is less than 8%, and the mean decrease in the SSIM metric is within 0.07.
The reasons for the above results might be due to the skip connection structure of the network and the fixation structure of the SAR image. The decoder of the ULAN and U-Net fuses the features from different layers through the skip connection structure, which can help the generator learn the data distribution sufficiently. Moreover, the low dependence on the training data also attributes to the fixation structure of the SAR image itself such that its semantic features are easier extracted and represented than natural images. Thus, our approach can work well in the situation where attackers have difficulty obtaining sufficient training data. (See lines 446-472)
Point 3: In 4.4 and 4.5, the L2-norm values are listed in Table 3 and Table 4, but the reason is explained at the end (4.7.3). Adjusting the order of this part will allow readers to better understand this paper.
Response 3: We feel sorry for the inconvenience brought to the reviewer. In order to avoid major adjustments to the structure of the paper and help readers better understand our ideas, we declare the default parameter settings that can achieve excellent attack performance in advance in Section 4.1.2. We will discuss the influence of parameters on UAPs in detail in Section 4.7. (See lines 320-322)
Author Response File: Author Response.pdf
Reviewer 3 Report
This paper proposes a semi-whitebox attack network—Universal Local Adversarial Network (ULAN) to generate local perturbations, adding to the target regions of synthetic aperture radar (SAR) images to craft adversarial examples. The manuscript is well written with clear structural framework and detailed data, including training data, operating environment and other descriptions, formula annotations are clear. However, the weakness of the manuscript include some details are confusing throughout the manuscript. The comments are attached.
Comments for author File: Comments.pdf
Author Response
Dear Editor and Reviewer:
Thanks for your letter and the reviewer’s comments concerning our manuscript entitled “ULAN: A Universal Local Adversarial Network for SAR Target Recognition Based on Layer-wise Relevance Propagation” (Manuscript ID: remotesensing-2061736). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our research. We have studied all comments carefully and have made conscientious corrections. Revised portions are marked in red on the paper. The main corrections in the paper and the responses to the reviewer’s comments are as follows:
Point 1: Line 92-94. “the proposed method also ensures that the well-designed perturbations can be fully fed to the victim model along with the targets such that perturbation offset is fundamentally prevented.” This sentence is overstated.
Response 1: We completely agree with you. The expression is indeed overstated and inaccurate. Thus, we modify the sentence to “the proposed method also can make the adversarial perturbations be fed to the victim model as completely as possible, preventing perturbation offset to the greatest extent.” (See lines 92-94, 442-444, and 547-549)
Point 2: Line 109. “the results shows that given five images per class, our method can cause a misclassification rate over 70%.” According to the Figure10 (a) and (b), in the case of non-targeted attacks, the attack accuracy of the ULAN is under 30%, i.e. misclassification rate over 70%. But in the case of targeted attacks, the attack accuracy of the ULAN is more than 80%. So this conclusion should be subject to preconditions. Please correct it.
Response 2: Thank you for pointing this out. We revise the above conclusion: “Furthermore, we evaluate the attack performance of the ULAN under small sample conditions. The results show that given five images per class, our method can cause a misclassification rate over 70% in non-targeted attacks and make the probability of victim models outputting specified results in targeted attacks close to 80%.” (See lines 108-112)
Point 3: Line 197-198. “we suppose fs(·) is an l-layer DNN without the softmax output layer.” Why emphasize that fs(·) is a DNN without softmax output layer?
Response 3: Thank you for your rigorous comment. The reason we emphasize this is that layer-wise relevance propagation (LRP) [32] takes as input the model’s logits outputs and outputs the pixel-wise attention heatmaps of the surrogate model fs(·). Note that the logits output is the model’s output before the softmax layer. Thus, for an easy explanation, we suppose fs(·) is an l-layer DNN without the softmax output layer. (See lines 203-206)
Point 4: Line 280. According to the above, the non-targeted attack can decrease the confidence of true classes, i.e., increase the confidence of others, and the targeted attack is to increase the confidence of target classes. From this view, both are similar. Please explain “contrary to the non-targeted attack”.
Response 4: We feel sorry for the confusion brought to the reviewer. Actually, this word is to express that the decrease of the true class confidence in non-targeted attacks is exactly contrary to the increase of the target class confidence in targeted attacks. However, there is no doubt that this sentence is ambiguous. Therefore, we believe that the word "contrary" should be removed, and the sentence should be modified to “Thus, the attack loss La of targeted attacks can be expressed as (24).” (See lines 289-290)
Point 5: Line 326-328. “The non-targeted attack effect is inversely proportional to the classification accuracy, while the targeted attack performance is proportional to the Acc metric.” How is this conclusion reached? Please clarify it.
Response 5: We feel sorry for the poor explanation of the above conclusion and clarify it as follows:
In non-targeted attacks, the Acc metric reflects the probability that victim models correctly recognize adversarial examples. The lower the classification accuracy of the victim model on adversarial examples, the better the non-targeted attacks. While in targeted attacks, the Acc metric represents the probability of victim models identifying adversarial examples as target classes. The higher the Acc metric, the stronger the targeted attacks. In conclusion, the non-targeted attack effectiveness is inversely proportional to the Acc metric, and the targeted attack effectiveness is proportional to this metric. (See lines 336-343)
Point 6: Line 407-410. “we first recover the adversarial examples generated in Section 4.4 to 128 × 128, and next random-crop the recovered images to 88 × 88 again, such that the perturbation offset condition is constructed.” Why is the condition of perturbation offset established after restore-crop?
Response 6: We appreciate you pointing this out and will describe the construction detail of the perturbation offset condition more clearly. Specifically, we first recover the adversarial examples generated in Section 4.4 to 128×128, and next obtain the input data by random-cropping the recovered images to 88×88 again. In this way, we cause a mismatch between the input and perturbed regions. As shown in Figure 1, the input and perturbed regions correspond to the red and green box regions such that the adversarial perturbations cannot be fed to the victim model completely, and thus the perturbation offset condition is constructed. (See lines 423-429)
Point 7: Line 480-481. “the attack stealthiness is improved as ω increasing, while the attack effectiveness is getting worse.” If Acc represents the attack effectiveness, the attack effectiveness is getting better under non-targeted attacks. So the above conclusions are not universal. Please consider it.
Response 7: We feel sorry for the confusion brought to the reviewer. This comment is quite similar to comment 5, and thus we will explain it on the basis of the response to comment 5. According to Figure 8, we can see that for non-targeted attacks, the Acc and SSIM metrics are rising as ω gets larger. In targeted attacks, the Acc metric declines as ω grows, while the SSIM metric is still increasing. Since the non-targeted attack effectiveness is inversely proportional to the Acc metric and the targeted attack effectiveness is proportional to this metric, the effectiveness of adversarial attacks is getting worse as ω increases. However, in both attack modes, the SSIM metric is always proportional to the attack stealthiness such that UAPs become more imperceptible as ω gets larger. (See lines 504-511)
Author Response File: Author Response.pdf
Reviewer 4 Report
This paper proposes a semi-Whitebox attack network to generate universal adversarial perturbations for the target regions of SAR images. The organization of the paper is pretty good, and the methodology and results are described well. However, a careful minor revision must be achieved before publication.
· In Tables 3 and 4, the SSIM metric is being used, how it has been computed? And how is it compared to the attack performance?
· Model evaluation always has a subjective element. How is the proposed model designed to generate real-time adversarial examples without requiring access to the model itself?
· There are too many tables and figures. It would be better to put some tables and figures in the supplementary material.
· There are some grammatical and sentence errors. The authors are advised to carefully revise the whole manuscript to fix the language and grammatical errors.
Comments for author File: Comments.pdf
Author Response
Dear Editor and Reviewer:
Thanks for your letter and the reviewer’s comments concerning our manuscript entitled “ULAN: A Universal Local Adversarial Network for SAR Target Recognition Based on Layer-wise Relevance Propagation” (Manuscript ID: remotesensing-2061736). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our research. We have studied all comments carefully and have made conscientious corrections. Revised portions are marked in red on the paper. The main corrections in the paper and the responses to the reviewer’s comments are as follows:
Point 1: In Tables 3 and 4, the SSIM metric is being used, how it has been computed? And how is it compared to the attack performance?
Response 1: We feel sorry for the confusion brought to the reviewer. Thus, we modify the form of (27) to match our paper (Please see the pdf file for details). It calculates the mean of the SSIM value between all the samples in the dataset and the corresponding adversarial examples, which ranges from −1 to 1. The higher the SSIM, the more imperceptible the UAPs, and the better the attack stealthiness. (See lines 349-354)
Point 2: Model evaluation always has a subjective element. How is the proposed model designed to generate real-time adversarial examples without requiring access to the model itself?
Response 2: Thank you for pointing this out. We will explain it in detail. This paper designs a generative network to craft UAPs for the target regions of SAR images under semi-whitebox conditions. The proposed method requires model information only during the training phase. Once the network is trained and given inputs, it can real-time generate adversarial examples for the victim model through one-step forward mapping without requiring access to the model itself anymore. Thus, our method possesses higher application potential than traditional iterative methods. (See lines 100-105 and 549-553)
Point 3: There are too many tables and figures. It would be better to put some tables and figures in the supplementary material.
Response 3: Thank you so much for your valuable advice. We have added an appendix at the end of the paper. Some details of the datasets and supplementary experiments have been moved to Appendix A.
Point 4: There are some grammatical and sentence errors. The authors are advised to carefully revise the whole manuscript to fix the language and grammatical errors.
Response 4: We regret that there are problems with English writing. The manuscript has been carefully revised by a native English speaker to fix the grammatical and sentence errors.
Author Response File: Author Response.pdf
