Synergistic Verification of Hardware Peripherals through Virtual Prototype Aided Cross-Level Methodology Leveraging Coverage-Guided Fuzzing and Co-Simulation
Round 1
Reviewer 1 Report
The authors propose a verification methodology for hardware peripherals, where they utilise a virtual prototype and then employing both coverage-guided fuzzing and co-simulation approaches. These combinations allows the methodology to uncover different mismatches such as timing and behavioural. Generally, the paper is well written and the overall methodology is well described. However, there are some issues that have to be addressed. More specifically my comments for the authors are the following:
- Authors must provide some analysis on how they extend the state-of-the-art and what are the specific contributions that appear on the paper. It seems that they just combine two methodologies in one framework.
- Authors should provide some complexity analysis regarding their methodology. Is it feasible to be employed in large-scale benchmarks.
- For all the benchmarks authors must report the number of registers and input and describe the benchmarks in terms of gates and variables.
- What is the runtime of the methodology for all the different experiments.
- It would be interesting if the authors could compare their methodology with other established methods in terms of runtimes and errors found.
Author Response
The authors propose a verification methodology for hardware peripherals, where they utilise a virtual prototype and then employing both coverage-guided fuzzing and co-simulation approaches. These combinations allows the methodology to uncover different mismatches such as timing and behavioural. Generally, the paper is well written and the overall methodology is well described. However, there are some issues that have to be addressed. More specifically my comments for the authors are the following:
- Authors must provide some analysis on how they extend the state-of-the-art and what are the specific contributions that appear on the paper. It seems that they just combine two methodologies in one framework.
We thank the reviewer, encouraging us to position our work more directly.
As positioned in the related work, our work differentiates from state-of-the-art for the coverage-guided fuzzing methodology by utilizing a TLM based reference for mismatch detection. Furthermore, the VP integrated RTL peripherals have experienced comparatively little attention. We additionally believe that presenting a combined approach also paves the path to a holistic approach to modern verification.
- Authors should provide some complexity analysis regarding their methodology. Is it feasible to be employed in large-scale benchmarks.
As the methodology makes use of the high-level abstraction introduced with SystemC TLM based Virtual Prototypes (VP) and Verilator for RTL simulations, scalability is not an issue. SystemC TLM based models are in use in research and industry for their fast speed simulation speeds while allowing engineers to model details accurately in trade-off for simulation speed. In the paper, we utilized coverage-guided fuzzing as a method proven in the domain of software verification, which compares to the corresponding method of Constrained Random Verification (CRV). Fuzzing has proven to be orders of magnitudes faster than CRV which is also pointed out in the papers that do this comparison thoroughly. To furhter accomodate the detail about scalability/speed, we added further information in the manuscript (Section 2, page 3). As for the VP-based application driven co-simulation, we utilized an application built with the industry standard real-time operating system FreeRTOS, which incoorperates a multitude of functions representing larger-scale applications compared to bare-metal applications.
- For all the benchmarks authors must report the number of registers and input and describe the benchmarks in terms of gates and variables.
We think the reviewer points toward an intersting aspect. Since we utilize an abstraction above RTL, not all information (e.g., number of registers) can be carried over into the higher abstraction. With TLM occuring in the C++/SystemC domain, registers are split into variables and similar constructs. For the SystemC RTL generated from Verilator, a similar issue can be faced. As we utilized a RISC-V Interrupt Controller, we included a comprehensive summary of the specification (including the number of interrupts, i.e. inputs, handled by the controller as well as context space hanlded by the controller). Additionally, we added further specifics of the utilized RTL interrupt controller on the basis of the reviewer's comment (Section 5, Page 8 bottom, Page 9 top).
- What is the runtime of the methodology for all the different experiments.
The runtime of the coverage-guided fuzzer is given with 1 hour, for further clarity the hour unit is written out fully in the manuscript (see Section 5.1, Page 9). For the application driven co-simulation, the evaluation took around 7 minutes. In order to clarify this point further, we added the information to the manuscript (Section 5.2, Page 10).
- It would be interesting if the authors could compare their methodology with other established methods in terms of runtimes and errors found.
While we agree with the comment of the reviewer, we think a comparison is difficult to achieve for a multitude of reasons. None of the other works combine the different models of abstraction for the verification of peripherals. On those works which could compare to parts of our methodology, the information given in the works is not enough to faithfully reproduce them. We believe that it is important for comparisons to be fair and assessable, hence we ommited this part. Regardless of this, our work presents the first methodology combining the different abstractions in a unit-level for a coverage-guided fuzzing verification and at the system-level at which application driven co-simulation is utilized.
Reviewer 2 Report
The authors propose a Virtual Prototype driven verification methodology for Hardware peripherals, combining Coverage-Guided Fuzzing and an application-driven co-simulation-based approach that enables verification of the HW peripheral at the system-level. The work is very interesting and the paper is well written, with a few minor errors listed in sequence. The detailes of the work as given in a complete way, and the results obtained are satisfactory.
More general comments and minor errors are listed as follows.
"an RISC-V" -> "a RISC-V"
"With this approach" -> "With this approach,"
" in the PLIC" -> " in the PLIC."
"RTL/TLM development" -> "RTL/TLM development."
Author Response
The authors propose a Virtual Prototype driven verification methodology for Hardware peripherals, combining Coverage-Guided Fuzzing and an application-driven co-simulation-based approach that enables verification of the HW peripheral at the system-level. The work is very interesting and the paper is well written, with a few minor errors listed in sequence. The detailes of the work as given in a complete way, and the results obtained are satisfactory.
More general comments and minor errors are listed as follows.
"an RISC-V" -> "a RISC-V"
"With this approach" -> "With this approach,"
" in the PLIC" -> " in the PLIC."
"RTL/TLM development" -> "RTL/TLM development."
We thank the reviewer for the list of comments. We corrected the minor errors accordingly.
Reviewer 3 Report
The goal of this paper, as exposed by the authors, is to present a Virtual Prototype (VP) driven verification methodology for HW peripherals.
Sections 1 and 2 contain a well-structured introduction and an in-depth review of state of the art. The authors successfully describe problems related to transaction-level modeling, system verification, RISC-V, etc.
The authors summarize their main contributions in this study at the end of the Introduction section. For each point mentioned in the contribution paragraph, identify which part in the resubmitted manuscript considers that point. The authors should discuss the research gap and existing problems in this step as the research motivation.
What is the jitter introduced by the FreeRTOS real-time operating system? What is the response time of the interrupt controller and the RTOS scheduler, respectively the RTOS synchronization and communication mechanisms? Are these aspects considered in the simulations in Figure 4?
The experimental data (sections 4 and 5) are presented and analyzed in detail. The reference section is good, citing new and relevant articles in the research area.
Author Response
The goal of this paper, as exposed by the authors, is to present a Virtual Prototype (VP) driven verification methodology for HW peripherals.
Sections 1 and 2 contain a well-structured introduction and an in-depth review of state of the art. The authors successfully describe problems related to transaction-level modeling, system verification, RISC-V, etc.
The authors summarize their main contributions in this study at the end of the Introduction section. For each point mentioned in the contribution paragraph, identify which part in the resubmitted manuscript considers that point. The authors should discuss the research gap and existing problems in this step as the research motivation.
What is the jitter introduced by the FreeRTOS real-time operating system?
We thank the reviewer for the aspects pointed out by the questions. As the VP emulates the whole hardware platform, any aspect of the software and operating system running on the platform will be considered. This includes regular interactions between the hardware as well as the possible timing discrepancies. As we used FreeRTOS as an application for the second methodology, the jitter of FreeRTOS task scheduling is beyond the scope of the paper. Prior works have looked into specific effects of the FreeRTOS real-time operating system (See DOI:10.1016/j.jss.2016.04.063 and DOI:10.3390/sym12040592)
What is the response time of the interrupt controller and the RTOS scheduler, respectively the RTOS synchronization and communication mechanisms?
The response time of the interrupt controller is one cycle, upon arrival of an external interrupt. The RISC-V PLIC specification defines an arbitrary latency (Also see the RISC-V PLIC Specification -- Section 3 Interrupt Notifications). To further clarify this point in the paper we added this information (Section 3.1 Page 4). For this RTOS specific response times, we like to highlight the aforementioned references (See DOI:10.1016/j.jss.2016.04.063 and DOI:10.3390/sym12040592) as they thoroughly analyze the response time of these aspects of FreeRTOS.
Are these aspects considered in the simulations in Figure 4?
As the VP fully supports FreeRTOS execution, the synchronization, and communication mechanisms are considered as well for the simulation.
Round 2
Reviewer 1 Report
I believe that the manuscript is ready for publication.
Reviewer 3 Report
The paper was improved by the revision process.
