A Virtual Machine Platform Providing Machine Learning as a Programmable and Distributed Service for IoT and Edge On-Device Computing: Architecture, Transformation, and Evaluation of Integer Discretization
Round 1
Reviewer 1 Report
Comments and Suggestions for Authors
Abstract: Some information is missing. Please add that two tests based on simulated and synthetic data are provided, and the effects of discretization on regression accuracy are illustrated.
20 and 40: Are classification tasks truly addressed in this work? If not, drop them and focus on regression only.
153: What is the rationale behind the adoption of FORTH, a language today regarded by many as suitable for niche applications and legacy systems? Why not use, for instance, C/C++? Is the ISA of the VM really almost 40 years old? Are there truly no alternatives to working with such an aged technology?
190-191: Check the phrase structure.
328: Are pooling operations also provided in 6 (lines 331-332)?
378: The definition of conv(a,c) is unclear to me, particularly the sums over i and j with the notation i=1,+s and j=1,+s. Can you elaborate on that? Additionally, please state that the case of multi-filter convolution operations will be discussed later.
398: "Tangents hyperbolic" ---> "hyperbolic tangent"
479-482: In the caption add that the hyperbolic tangent is computed, too
527: I believe it is actually Figure 7
542: I believe it is actually Figure 7
544: "The small error plots show only positive x values." ---> "The small error plots show only positive x values with zoomed y values.
574: I guess it is ConvNetJS instead of convent.js. Could you rather use PyTorch or TensorFlow?
1000, 1002: Always use EMAX or E_max
1017 and 1021: That notation is quite unusual. Please use the standard notation instead: F(x) = k0 + k1 x + k2 x^2 + k3 x^3 + ... + kn x^n
1066-1067: The second part of the caption is unclear. Please elaborate on it.
1142: provide the URL
1143: provide the URL
Comments on the Quality of English Language
The quality of the English language is generally good, but a final check is needed to eliminate a few typos in the text.
Author Response
Dear reviewer,
thank you for the valuable comments that we addressed carefully in our revision.
Abstract: Some information is missing. Please add that two tests based on simulated and synthetic data are provided, and the effects of discretization on regression accuracy are illustrated.
> Was added and clarified
20 and 40: Are classification tasks truly addressed in this work? If not, drop them and focus on regression only.
> Yes, the ML-ISA provides operations that can be used for classification as well as regression, and the first use-case "CNN for Damage Location Regression and Classification" is a hybrid model providing binary damage classification and damage location regression. This was clarified in the introduction and the use-case sections.
153: What is the rationale behind the adoption of FORTH, a language today regarded by many as suitable for niche applications and legacy systems? Why not use, for instance, C/C++? Is the ISA of the VM really almost 40 years old? Are there truly no alternatives to working with such an aged technology?
> The REXA VM is a stack processor. Any ISA can be implemented addressing stack-based computation, but FORTH is a well known and long-standing programming language that 1.) Is high- and low-level 2.) Can be implemented efficiently on low-resource systems 3.) Is extensible (ML-ISA!) 4.) Requires only simple compilers. C/C++ is a compiled language not intended von script execution as intended in this work. C/C++ compilers are complex; Forth compilers are not. This was clarified in Sec. "REXA VM Architecture and Programming Language".
190-191: Check the phrase structure.
> corrected
328: Are pooling operations also provided in 6 (lines 331-332)?
> Yes, pooling as being similar to convolution is handled by the convolution operation, too. This was clarified by an additional paragraph in the numbered list.
378: The definition of conv(a,c) is unclear to me, particularly the sums over i and j with the notation i=1,+s and j=1,+s. Can you elaborate on that? Additionally, please state that the case of multi-filter convolution operations will be discussed later.
> indeed, conv formula was nonsense, formula error was fixed, formulas were revised and clarified
398: "Tangents hyperbolic" ---> "hyperbolic tangent"
> fixed
479-482: In the caption add that the hyperbolic tangent is computed, too
> added
527: I believe it is actually Figure 7
> corrected
542: I believe it is actually Figure 7
> corrected
544: "The small error plots show only positive x values." ---> "The small error plots show only positive x values with zoomed y values.
574: I guess it is ConvNetJS instead of convent.js. Could you rather use PyTorch or TensorFlow?
> Correct! References were updated, too. In principle, our approach can be used with PyTorch and Tensorflow, too, although it is more difficult to implement our algorithms.
1000, 1002: Always use EMAX or E_max
> corrected
1017 and 1021: That notation is quite unusual. Please use the standard notation instead: F(x) = k0 + k1 x + k2 x\^2 + k3 x\^3 + ... + kn x\^n
> changed
1066-1067: The second part of the caption is unclear. Please elaborate on it.
> Caption was corrected and clarified
1142: provide the URL
1143: provide the URL
> URLs were added, references were updated.
Reviewer 2 Report
Comments and Suggestions for Authors
The manuscript seems to address a valid problem for practical application of ML to data acquired from IoT devices but it contains many passages that require rewriting as it is difficult to comprehend them and determine the author's point.
For instance, in the abstract:
"Distributed sensor networks like the IoT and new trends in Edge and Material-integrated Computing provide often only low-resource embedded computers, and sensor data is acquired locally and should be processed locally."
or in the introduction:
"The computation of Deep Learning (DL) models is further limited by ultra low-power devices [4], and hardware designs are becoming more popular [5], both of which are topics covered in this work."
The title seems to not exactly mirror the paper'a contents. Both the title and abstract mention Machine Learning as a Service which is a cloud-based provisioning of ML algorithms, whereas the proposal IIUC is supposed to be implemented as edge on-device computing (as suggested by the author saying that "sensor data is acquired locally and should be processed locally").
In the conclusion, the author writes "This work investigated the effect of scaled integer discretization for regression task with two use-cases."
Is it really the key contribution of the paper? Then the title should say so. Otherwise, please change the conclusion section (also, see below).
The paper is very long and, in my opinion, the paper should be restructured to get its contents better organized and thus approachable by the reader.
The Introduction should briefly present the context and author's motivation without delving into technical details (like the paragraph starting with "There is ongoing work to implement computation of ML models..." does now).
It should also clearly state the research goal and key contributions (also explaining the extension in comparison to the conference paper).
It should be followed by Relevant Work section that would describe what the others achieved so far in this area.
Subsections and/or subsubsections could be used for tackling the different aspects of prior work.
This should be followed by the main section, presenting the solution: its concept, architecture and implementation/algorithms.
This should be followed by the Evaluation section - subsections and/or subsubsections could be used for tackling the different parts of it.
Then should be the Discussion section, comparing the findings to what was achieved by the prior work.
The final section Conclusion should not include detailed observations (these belong rather to the Discussion) but just a few key summary phrases of what was done, why, and what are the most important results plus the description of research limitations and planned future work.
Similarity check note:
- iThenticate reports 22 percent match, but considering this is an extended version of a conference paper, this is within the acceptable similarity threshold.
Comments on the Quality of English Language
See the main comments: many sentences are too complex and difficult to understand.
Language notes:
- common nouns in English (e.g. bits) are written in lowercase
- the commas are misplaced (e.g., "Although, this is")
Author Response
Dear reviewer,
thank you for the valuable comments that we addressed in our revision.
The manuscript seems to address a valid problem for practical application of ML to data acquired from IoT devices but it contains many passages that require rewriting as it is difficult to comprehend them and determine the author's point.
For instance, in the abstract:
"Distributed sensor networks like the IoT and new trends in Edge and Material-integrated Computing provide often only low-resource embedded computers, and sensor data is acquired locally and should be processed locally."
> Confusing sentence was revised: We address constraints in distributed sensor networks like the IoT, Edge and Material-integrated Computing providing only low-resource embedded computers, with sensor data that is acquired and processed locally.
or in the introduction:
"The computation of Deep Learning (DL) models is further limited by ultra low-power devices [4], and hardware designs are becoming more popular [5], both of which are topics covered in this work."
> Confusing sentence was revised: The computation of complex Deep Learning (DL) models is further limited by memory and computing power constraints of ultra low-power devices [ALA22]. To overcome software limitations and limited computability hardware designs are becoming more popular [JAI23]. We focus on software processing of ML models on low-resource and low-power devices by model transformation fitting low-resource devices.
The title seems to not exactly mirror the paper'a contents. Both the title and abstract mention Machine Learning as a Service which is a cloud-based provisioning of ML algorithms, whereas the proposal IIUC is supposed to be implemented as edge on-device computing (as suggested by the author saying that "sensor data is acquired locally and should be processed locally").
> Although, "as a service" is a common Cloud-based paradigm, we use this term in terms of virtualization on node and single computer level.
In the conclusion, the author writes "This work investigated the effect of scaled integer discretization for regression task with two use-cases."
> The title was adapted to reflect one major part of this paper: A Virtual Machine Platform providing Machine Learning as a programmable and distributed service for IoT and Edge On-device Computing: Architecture and Evaluation of Integer Discretization
Is it really the key contribution of the paper? Then the title should say so. Otherwise, please change the conclusion section (also, see below).
> Yes, it is a key contribution (or result) of the experimental work, but the final result is the demonstration of the suitability of the proposed VM-ML ISA and the operations.
The paper is very long and, in my opinion, the paper should be restructured to get its contents better organized and thus approachable by the reader.
> It is an extension of a conference paper (showing only selected architecture descriptions and results) and should elaborate the whole story including the VM itself.
The Introduction should briefly present the context and author's motivation without delving into technical details (like the paragraph starting with "There is ongoing work to implement computation of ML models..." does now).
> The computation of ML models on low-resource devices is the main driving force of this work and must be elaborated in some technical detail level to understand the hard constraints arising by these devices.
It should also clearly state the research goal and key contributions (also explaining the extension in comparison to the conference paper).
> The extension of the journal paper with respect to the conference paper was given in the introduction. "Compared with [BOS23C] we provide new algorithms and ML applications including ANN/CNN-based regression and classification tasks with a rigorous evaluation of discretization errors."
It should be followed by Relevant Work section that would describe what the others achieved so far in this area.
Subsections and/or subsubsections could be used for tackling the different aspects of prior work.
This should be followed by the main section, presenting the solution: its concept, architecture and implementation/algorithms.
This should be followed by the Evaluation section - subsections and/or subsubsections could be used for tackling the different parts of it.
> We believe that the current structure gives a better overview than the standard form. We divided the paper into VM architecture, transformation algorithms, and use-cases.
Then should be the Discussion section, comparing the findings to what was achieved by the prior work.
The final section Conclusion should not include detailed observations (these belong rather to the Discussion) but just a few key summary phrases of what was done, why, and what are the most important results plus the description of research limitations and planned future work.
> The conclusion highlights the key results and summarizes the required computing resources for typical ML tasks (as demonstrated in this work). An outlook and future work is given.
Similarity check note:
- iThenticate reports 22 percent match, but considering this is an extended version of a conference paper, this is within the acceptable similarity threshold.
> Please note that the overlap with the FedCSIS conference proceeding paper is only 8%, followed by 6% overlap with an arXiv pre-print 8from the authors), which is not considered in sense of plagiarism (since it is a pre-print, that can be published 1:1, in principle). But you are right, copy&paste is bad practice, so we rewrote and restructured some of the passages from the arXiv paper. Finally, there is a small overlap of 5% of one of our previous conference papers. We tried to reduce this overlap, but the overlap is a result of auxiliary information (required here). The remaining 3% is noise.
Comments on the Quality of English Language
See the main comments: many sentences are too complex and difficult to understand.
Language notes:
- common nouns in English (e.g. bits) are written in lowercase
> fixed
- the commas are misplaced (e.g., "Although, this is")
Round 2
Reviewer 2 Report
Comments and Suggestions for Authors
Dear Authors,
thank you for your corrections.
Out of my three main comments, three were addressed (some sentences were simplified and the title was changed).
The third (the suggestion to restructure the paper into a classic form) was not. I do not insist you should do it (although - as I wrote before - it would improve the clarity of the text), but there is one thing that in my opinion deserves your effort: the final part. From the first round review:
The final section Conclusion should not include detailed observations (these belong rather to the Discussion) but just a few key summary phrases of what was done, why, and what are the most important results plus the description of research limitations and planned future work.
Once again, I suggest moving the highlights (as you call them. but they are detailed) to Discussion (best if they could put in context of prior work) and leaving only high-level conclusions in the Conclusion section (along with research limitations and planned future work).
Comments on the Quality of English Language
The following issue mentioned in the first round of review persists:
- the commas are misplaced (e.g., "Although, this is")
Please correct comma placement.
Author Response
Once again, I suggest moving the highlights (as you call them. but they are detailed) to Discussion (best if they could put in context of prior work) and leaving only high-level conclusions in the Conclusion section (along with research limitations and planned future work).
> The enumerated list of results was shifted to a new Discussion section. The Conclusion section now contains only some key facts.
The commas are misplaced (e.g., "Although, this is")
> corrected
Some more minor grammar errors and sentences were corrected.
