Fine-Grained Modeling of ROP Vulnerability Exploitation Process under Stack Overflow Based on Petri Nets

Abstract

Software vulnerability discovery is currently a hot topic, and buffer overflow remains a prevalent security vulnerability. One of the key issues in vulnerability discovery and analysis is how to quickly analyze buffer overflow vulnerabilities and select critical exploitation paths. Existing modeling methods for vulnerability exploitation cannot accurately reflect the fine-grained execution process of stack overflow exploitation paths. This paper, based on the discussion of buffer overflow exploitation techniques, proposes a fine-grained modeling and analysis method based on Petri nets for the selection and execution of exploitation processes, specifically focusing on the return-oriented programming in stack overflow. Through qualitative analysis, we compared the simulated time of the software with the execution time of existing exploitation tools, achieving timeout-based simulation experiments. We validated the model’s effectiveness using symbolic execution and dynamic analysis techniques. The results indicate that this model performs well for vulnerable programs with Position Independent Executable (PIE) protection enabled and has an advantage in selecting exploitation paths, enabling timeout-based simulation. This method provides a reference for rapidly constructing exploitation implementations.

1. Introduction

Buffer overflow [1] is a common vulnerability found in various applications and operating systems [2]. Due to the lack of automatic buffer overflow detection instructions in languages like C/C++ and the high time cost of manually writing code to check for buffer overflow occurrences in real time, these vulnerabilities are easily exploitable by attackers. Although fuzz testing techniques helped us automatically discover the program vulnerabilities [3], and parallel fuzz testing platforms efficiently uncover buffer overflow vulnerabilities in programs, both defenders and attackers are more concerned about whether these program vulnerabilities or errors can be exploited. Rapidly analyzing buffer overflow vulnerabilities and selecting critical paths for exploitation is one of the key issues in current vulnerability discovery and analysis. Petri nets are modeling and analysis tools with a strict mathematical definition and powerful graphical representation capabilities [4]. They can simulate processes that describe the static structure and micro-dynamic behavior. While there has been extensive research on buffer overflow in the past, these studies did not delve into the micro-level processes of stack overflow exploitation, and they failed to provide an in-depth and detailed analysis of vulnerability mechanisms that reflect the essence of exploitation. Therefore, this paper discusses the use of Petri nets to model the process of buffer overflow exploitation, analyzing factors such as register states and memory information relevant to buffer overflow exploitation. This establishes a fine-grained Petri net model for vulnerability exploitation in stack overflows, specifically focusing on Return-Oriented Programming (ROP) exploitation [5]. The paper also describes the path selection method during vulnerability exploitation, effectively enabling buffer overflow exploitation, and enhancing research on vulnerability discovery and software vulnerability exploitability issues.

In this study, our objective is to achieve the analysis and rapid construction of vulnerability exploitation by establishing Petri nets on the ROP vulnerability exploitation process. The main contributions are as follows:

(1)

The Petri nets model for ROP vulnerability exploitation under stack overflow is proposed. In comparison to other studies, this paper delves into the fundamental principles of stack overflow and provides a detailed description of the process of constructing vulnerability exploitation using ROP techniques for programs with stack overflow. It employs Petri nets to model the payload construction process, offering a fine-grained description of the exploitation process.

(2)

The model is compared with the existing vulnerability exploitation tools, facilitating real-time simulation experiments. The average time for 10 sets of experiments is calculated for two ROP vulnerability exploitation methods, and it is found that this model takes less time and has an advantage in path selection.

(3)

Based on the model’s approach to vulnerability exploitation, the model’s effectiveness is verified using symbolic execution and dynamic analysis. The results indicate that this model performs well for vulnerable programs with PIE protection enabled and provides a reference method for rapidly constructing vulnerability exploitation implementations.

The remaining parts of this paper are organized as follows. Section 2 presents Petri nets and related work on buffer overflow exploitation. Section 3 introduces the Petri net modeling of ROP buffer overflow exploitation. Section 4 provides a detailed analysis of the model. Section 5 presents an analysis of experimental results for the model. Section 6 summarizes the relevant research in this paper.

2. Related Work

Buffer overflow attacks are initiated by overwriting the return address saved in the function stack frame of the called function that redirects the execution to arbitrary code injected by the attacker. Stack overflow is one of the most common vulnerabilities in buffer overflow. Aleph One [6] detailed the architecture of the Linux stack in 1996 and proposed a method for exploiting buffer overflow vulnerabilities based on the stack to gain control of the program’s control flow. Attackers can remotely gain administrative privileges through this vulnerability and then execute malicious instructions with those privileges. Buffer overflow has been identified as one of the most common and dangerous security vulnerabilities to date [7]. Many detection and mitigation techniques have been developed and deployed to reduce the likelihood of buffer overflow attacks [8,9,10]. Li, S et al. [11] proposed a machine learning-based buffer overflow detection method, which demonstrates good accuracy in detecting remote buffer overflow attacks. Piromsopa, K [12] introduced a framework to prevent buffer overflow attacks and created an effective buffer overflow prevention tool using this framework. Ozdoganoglu et al. [13] introduced Smashguard, a hardware stack implemented in the Central Processing Unit to protect return addresses, first implemented in the Alpha architecture. No-Execute (NX) [14] and Address Space Layout Randomization (ASLR) [15] at the hardware and operating system levels also provide some mitigation for buffer overflow vulnerabilities. Although current detection and mitigation techniques help us address the discovery of program vulnerabilities [1,16,17,18,19], these mitigation measures are not foolproof. Return-oriented Programming (ROP) [20] attacks can effectively bypass NX. ROP is an advanced form of code reuse attack that achieves control flow bypassing protection mechanisms by using pre-existing small instruction sequences. Therefore, research on ROP exploitation under buffer overflow is of great significance.

Vulnerability exploitation refers to leveraging security flaws within software to achieve objectives that cannot be attained through normal software operation. The primary method for exploiting buffer overflow vulnerabilities relies on the absence of boundary checks for reading and writing data in C and C++ programming languages. This allows data placement outside the boundaries of a buffer, potentially overwriting the return address of a program function, thereby hijacking control flow and leading to program crashes or gaining system control privileges. D. Brumley [21] and others introduced an Automatic Exploit Generation (APEG) method based on binary patch comparison. Its core idea involves adding filtering conditions to patched programs to prevent triggering the original program’s crashes. For example, a patch might increase the length of a buffer to fix a buffer overflow vulnerability. A D. Federico [22] and colleagues proposed a vulnerability exploitation technique capable of bypassing NX (No-Execute) and ASLR (Address Space Layout Randomization) protections. This technique exploits the dynamic linking process of ELF executable files to overcome complex security mechanisms. T. Avgerinos [23] and his team introduced an effective method for automated vulnerability discovery and exploitation called AEG. The core concept of this method is to use program verification techniques to identify inputs that lead the program into an insecure state that can be exploited. Such insecure states may involve memory overflows, malicious format string usage, and more. Various AEG solutions have emerged for different vulnerabilities and exploitation methods [24,25]. This research extends beyond the software layer, delving deep into the Linux kernel to enhance security [26,27,28,29].

Existing research on vulnerability exploitation can efficiently discover a large number of program errors, and the description of exploitation methods has become more refined [30,31,32]. However, current research lacks fine-grained modeling and analysis of vulnerability exploitation scenarios, overlooks details in attack scenarios, and has vague path selection analysis for exploitation. Most approaches directly explore program paths, making it challenging to address the problem of path explosion.

The control flow and sensitive data of a program can be disrupted by successful buffer overflow exploitation. This paper models the micro-level process of Return-Oriented Programming (ROP) vulnerability exploitation using Petri nets, with states and operations as the basic granularity. A Petri net model for stack overflow exploitation is established, addressing the issue of traditional exploitation method models being too macroscopic and providing a detailed description of the micro-level processes of stack overflow vulnerabilities. By exploring the path selection method for vulnerability exploitation in this Petri net model, efficient identification of exploitable vulnerabilities in programs can be achieved, offering a more fine-grained representation for enhancing the understanding of the buffer overflow exploitation process.

3. Petri Net Model for Stack Overflow Vulnerability Exploitation

In this section, we describe a Petri net model for vulnerability exploitation based on stack overflow. The Petri net model for stack overflow automatic vulnerability exploitation is established based on the Return to C Library (Ret to libc) [33] and Return to C Start-Up (Ret to csu) [34] vulnerability exploitation methods. Through a fine-grained analysis of the exploitation process, states and behaviors in the exploitation process are mapped one to one to the places and transitions of the Petri net, creating a model that reflects the stack overflow exploitation process. This model detects stack buffer overflows by checking whether the EIP (Extended Instruction Pointer) register is contaminated and collects relevant information during runtime to automatically generate the vulnerability exploitation method paths.

3.1. Petri Net Modeling Rules

Petri nets are directed graphs used to describe relationships between events and conditions. They consist of places, transitions, and directed arcs. Places are represented by circles to describe possible local states of the system. Transitions are represented by rectangles to describe events that modify the system’s state. Directed arcs connect from Place nodes to Transition nodes or from Transition nodes to Place nodes, representing connections between Places and Transitions. The following is the definition of Petri nets.

Definition 1.

A triple PN = (P, T, F) is called a Petri net if it satisfies the following conditions:

• P is a finite set of places, T is a finite set of transitions;
• Where $P\ne \varnothing$, $T\ne \varnothing$, $P\cap T=\varnothing$;
• F = $(P\times T)\cap (T\times P)$ represents the flow relationship of PN.

The mapping relationship between the established Petri net model and automatic vulnerability exploitation is shown in

Table 1. Petri Net Rules.

Petri	Elements Automatic Vulnerability Exploitation
Place	States
Transition	Events
Direct arc	State Transition Process

.

In formal description and modeling, states primarily include the state of return address contamination, stack protection status, the Ret to libc exploitability status, library function offset address acquisition status, and shellcode execution capability status. Operations primarily encompass dynamic analysis, overwriting return addresses, and invoking a shell, among other behaviors.

In this study, places are used to represent various system states or states within the stack in the model. For example, states like library function offset address acquisition, exploitation construction in progress, and stack protection enabled are represented as places. Transitions are used to indicate operations during the construction of the vulnerability exploitation. For instance, operations like reading the offset address and overwriting the return address are indicated by transitions. Transitions in the model possess the characteristic of describing fine-grained execution operations in the automatic vulnerability exploitation process. Therefore, aspects related to states in various vulnerability exploitation operation scenarios are represented by places, while aspects related to operations are represented by transitions.

Within places, token values can be set based on the scenario of the actual vulnerability exploitation process. Token values represent the quantity of resources available in the model system or the extent of information leakage. These values are determined based on the specific exploitation scenario. According to the rules of the Petri net, when the model is running, a transition can occur only if all the places corresponding to that transition have tokens and the number of tokens satisfies the weight of the directed arcs. Token values decrease in the group of places preceding the transition according to the weight of the directed arcs, and token values increase in the group of places following the transition according to the weight of the directed arcs.

3.2. Petri Net Modeling for Vulnerability Exploitation

The process of constructing a stack overflow vulnerability exploitation can be divided into two main parts: accessing information within the stack and constructing the payload. According to the Petri net modeling rules, the entire system process is divided into sets of places and transitions, and an automatic vulnerability exploitation Petri net model is established. In this model, the main places include the state of overwriting the return address in the stack, the exploitability state of the vulnerability, various states related to vulnerability construction, and the state of acquiring library function offset addresses. These states are represented by circles in the Petri net model. States such as the stack-based return address overwrite and the stack protection enabled represent information access within the stack, while states related to vulnerability construction and library function offset address leakage represent payload construction.

The main transitions include operations for retrieving information from the stack, reading function offset addresses, and overwriting return addresses. These operations are represented by squares in the Petri net model. Specific token values can be set at the initial state, and in the system, the weight of directed arcs represents the number of times an action is performed and the quantity of resources.

According to the Petri net modeling rules proposed in this paper, the Petri net model for stack overflow automatic vulnerability exploitation is shown in Figure 1.

To address the selection of specific and appropriate exploitation methods during vulnerability exploitation, various branches are used in the model described above. The start and end states of each stage or substage of the construction are indicated using place nodes. The meanings of places in the vulnerability exploitation model are as shown in

Table 2. Meaning of Place in Vulnerability Exploitation Petri Net.

Place	Description
P0	Initial state
P1	PIE contamination state
P2	State Transition Process
P3	Stack Protection Disabled State
P4	Other Vulnerability Exploitation States
P5	Vulnerability Exploitation State
P6	Ret to libc construction
P7	Exploitation Construction State
P8	Ret to csu Construction
P9	Ret to libc Vulnerability Point Exploitation Information Acquisition State
P10	libc Library Function Offset Address Leakage State
P11	Ret to csu Vulnerability Point Exploitation Information Acquisition State
P12	write Function Offset Address Leakage State
P13	Library Function Offset Address Acquisition State
P14	BSS Segment Exploitable Vulnerability State
P15	Shell Establishment Exploitable State

.

This model can finely describe various states and changes during the construction of different vulnerability exploitation processes related to stack overflow. From the model in Figure 1, it can be observed that P6 and P8 represent two different states of vulnerability exploitation construction, mainly resulting in root privilege leakage to unauthorized users through the acquisition of library function offset addresses, return address contamination, and the construction of the system (/bin/sh) function. P4 represents the state of a failed stack overflow vulnerability exploitation, while P15 represents the state of a successfully established shell, indicating the successful completion of the exploitation.

In the vulnerability exploitation model, the meanings of transitions are as shown in

Table 3. Meaning of Transitions in Vulnerability Exploitation Petri Net.

Transition	Description
T0	Symbolic execution
T1	Return-oriented exploitation construction
T2	Exploitation construction
T3	Dynamic analysis for path selection
T4	Query initial address of got-puts
T5	Acquire csu return address
T6	Acquire got-puts offset address using the puts function
T7	Read system offset address and bin/sh using the read function
T8	Leak libc library offset address
T9	Overwrite return address to hijack control flow to libc-csu-init
T10	Acquire libc library offset address
T11	Read function reads the system offset address and bin/sh
T12	Exploit using system /bin/sh

.

From the model perspective, Transition T3 is the dynamic analysis path selection operation and a crucial step in constructing suitable vulnerability exploitation. Transitions T7 and T12 represent the operations of invoking the shell using the system function, indicating the successful exploitation of the stack overflow vulnerability. P9 in the model is the state required when enabling the Ret to libc exploitation method, while P11 and P14 are the states required when enabling the Ret to csu exploitation method. The quantity of tokens set for P0 indicates the number of times the model selects vulnerability exploitation.

4. Model Analysis

This section analyzes how to construct vulnerability exploitation scenarios in the presence of stack overflow vulnerabilities in a program. It explores the construction of suitable vulnerability exploitation methods, such as Ret to libc exploitation, and discusses the path selection during construction. It provides an in-depth explanation of the Petri net model for stack overflow vulnerability exploitation, describing the meanings represented by Places and Transitions in the model. It maps the process of constructing payloads for stack overflow scenarios to the Petri net model of vulnerability exploitation to illustrate the effectiveness of the model. Finally, it explains how to set token values in Places and the meanings of some of the weights on directed arcs.

4.1. Ret to Libc Method Model Analysis

Ret to libc is a major attack technique primarily targeting dynamically linked programs. When a program is dynamically linked, the libc library functions are loaded, and these library functions offer a wide range of system functionalities for program use. Therefore, it is possible to obtain control of the target program by searching for usable system function addresses in memory and combining stack overflow to overwrite the function’s return address. This model divides the details of the program’s control flow hijacking process in the context of stack overflow vulnerability exploitation and sets a Place node at the beginning and end of each stage or substage to mark its start and end states. The stack’s internal structure for the Ret to libc method is shown in Figure 2.

When conducting vulnerability exploitation attacks on a target program with a stack overflow vulnerability using the Ret to libc method, obtaining stack frame information and library function address offsets is essential to prepare for further payload construction. In Figure 2, vulnerability exploitation is achieved through two stages of payload construction. First, by overflowing the stack and overwriting the return address, control flow is hijacked to the “puts” function, with “puts-got” set as the parameter. This allows the retrieval of the actual address of the “puts” function from the “got” table, calculating the leaked library function offset address. After obtaining the necessary information for exploitation, the return address is overwritten to point to “system-addr”, and the address of “/bin/sh” string is placed on the stack. The program is then hijacked to the "system" function, ultimately leading to the acquisition of a shell.

The Petri net model for stack overflow vulnerability exploitation proposed in this paper is abstracted from the vulnerability exploitation methods of ROP, Ret to libc and Ret to csu. It is particularly well suited for the detection and generation of vulnerabilities in the context of stack overflow.

4.2. Vulnerability Exploitation Path Selection Analysis

The path selection analysis process primarily involves two components: path selection and branch acquisition. The goal is to reproduce the crash path, capture the program’s state when it crashes, and obtain information such as path constraints and memory snapshots.

With PIE and NX enabled in the stack space, and considering that the base address of the libc library is random, the model utilizes symbolic execution methods for the path selection in vulnerability exploitation. It aims to detect input that triggers stack buffer overflow vulnerabilities and employs dynamic analysis methods for the dynamic execution and analysis of base address offsets in binary programs.

First, the path selection program assesses whether stack protection is enabled. It determines the status of PIE and NX protection and selects the execution path for Return-oriented Programming (ROP) attacks. After obtaining addresses for library functions and character sets, the “puts” function activates the Ret to libc vulnerability exploitation path, as depicted in Figure 3 and

Table 4. Table of Place and Transition variations of the Petri model.

Place	Description	Transition	Description
P0	Ret to libc construction begins	T0	Obtaining Vulnerability Point Information
P1	Ret to libc vulnerability point information retrieval state	T1	puts function retrieves the got-puts offset address
P2	Libc Library Function Offset Address Leak State	T2	Leaking libc library offset address
P3	Establishing a Shell-Exploitable State	T3	call a shell

.

Formalizing the path selection of Ret to libc in a Petri net abstraction, after obtaining token resources at the Place P0 node, Transition T0 has the authority to fire. Following the occurrence of behavior in T0, state P1 is triggered. However, at this point, since the library function address offset cannot be obtained, it is necessary to execute behavior in T1, constructing the first payload. After obtaining the function address offset, state P2 has token resources. By constructing the payload once again, Transition T3 gains the authority to fire, ultimately activating state P3 to call the shell and acquire program control flow. The payloads constructed in two iterations are shown in

Table 5. Construction of Payloads for Ret to libc.

Ret to Libc	Attack Code
Payload1	‘a’*n + p64 (puts_plt) + p64 (start_addr) + p64 (put_got)
Payload2	‘a’*n + p64 (system_addr) + P64 (‘a’*8) + p64 (bin/sh_addr)

.

5. Experimental Analysis

The experimental environment configured for implementing the model in this paper is as follows: Operating System: Ubuntu 22.04, AMD R7-5800H CPU 64-bit operating system; Memory: 16 GB; Modeling tools and runtime environment: Tina v3.6.0, angr v9.0.10576 symbolic execution engine [35], and radare2 v5.62 dynamic analysis module [36]. Target programs: CTF and CVE programs with stack buffer overflow vulnerabilities. Comparison tool: Zeratool vulnerability exploitation tool [37].

5.1. Model Analysis

In this section, this article provides statistics for the time taken to reach the first successful exploitation state in Petri net models, the time taken to reach the first Ret to libc vulnerability exploitation state, and the time taken to reach the first Ret to csu vulnerability exploitation state. It also includes the actual time for successful exploitation using the Zeratool tool. Zeratool is a Python script that takes a binary file as a parameter and focuses on exploiting buffer overflow and format string vulnerabilities.

Table 6. The environment of the experiment.

Environment	Configuration
OS	Windows10
CPU	AMD® Ryzen™ 7-5800H
Tina tools	3.7.0
Python	3.10.12
Libc	2.35-0ubuntu3-amd64

presents the corresponding experimental system environments, simulation software, and system version for the verification experiments.

According to the Petri net model in the third section, a model was established and simulation experiments were conducted using the Tina tool [38]. The representation of reaching the vulnerability exploitation state through the Ret to libc exploitation method in the Petri net model is shown in Figure 4. The value of P0 represents the number of attempts to exploit the vulnerability in the target program. In the experiment, the token value of P0 was set to eight. Colored transitions indicate states waiting for resources to become available. By selecting the exploitation method, a payload is constructed and executed. When T0 becomes active, P2 and P3 can acquire tokens, representing the acquisition of the EIP register status and the execution of the ROP exploitation method. Resources flow to P5, allowing T3 acquisition of tokens for path selection, obtaining memory status, function address offsets, and other information. After P9 acquires resources, both T6 and T7 become active, executing the Ret to libc exploitation method path. In the end, P9 and P10 jointly possess the necessary resources, completing the payload construction. The payload is injected into the target program, and upon execution of T7, resources flow to P15, granting control over the program, thus resulting in a successful vulnerability exploitation.

In the Petri net model diagram, the number of tokens possessed by all Places is represented by the quantity of black dots. Transitions with a red color indicate transitions with the right to occur. Throughout the entire model, transitions are designed to be executed randomly, so the time it takes to reach the successful exploitation state P15 using the exploitation method is random with each attempt. Similarly, P0’s token value is set to eight. Figure 5 illustrates the selection of the Ret to csu exploitation method to reach the exploitation state.

The duration for which tokens are emitted in the model can be manually adjusted. The triggering duration is expressed as the rate of token propagation within the model. The faster the tokens flow, the shorter the duration. In our experiments, we set the triggering duration for tokens to 1 s and enabled the random, unpredictable timing option. Figure 6 displays ten sets of experiments. The average time for reaching the exploitation state for both the Ret to libc method and the Ret to csu method pathways is 24.2 s and 29.3 s, respectively. This article also uses the Zeratool vulnerability exploitation tool to simulate the time required for vulnerability exploitation in stack overflow scenarios. The simulated programs used are “downunderctf2020_return” and “utctf2021_resolve”. The time required for both symbolic execution and dynamic analysis processes is calculated, including the time needed to compute path constraints. Figure 7 presents data from eight sets of experiments, with average times of 33.4 s and 36.6 s, respectively. The experimental results indicate that the runtime of Petri net vulnerability exploitation simulation proposed in this article is shorter than the runtime of existing vulnerability exploitation tools using Zeratool. The simulation environment allows for near real-time simulation.

5.2. Comparative Analysis of Experiments

Based on the abstract concept of the stack overflow automatic vulnerability exploitation model in Section 3 and Section 4, a program is written in the Python language to validate the effectiveness of this model. The experiments are conducted on a 64-bit Ubuntu 22.04 system under VMware. Symbolic execution is performed using the angr framework [39], and dynamic analysis is carried out using radare2 [40]. The experiments focus on CTF (Capture The Flag) programs with stack buffer overflow vulnerabilities, most of which can be found on CTFTime [41]. CTF is a cybersecurity competition that is used to test and develop computer security skills. CTF programs can be considered simplified versions of real-world programs and are used to demonstrate the principles of vulnerabilities more concisely. Since CTF programs are designed to test participants’ skills in competitions, their vulnerabilities can be exploited.

Based on the established ROP vulnerability exploitation Petri net model, whether the program is protected by PIE (Position-Independent Executable) is used as a selection criterion to demonstrate the effectiveness of this model in selecting and executing vulnerability exploitation methods as efficiently as possible. The experiments are conducted in environments with ASLR (Address Space Layout Randomization) and NX (No-Execute) protection enabled [5].

Table 7. Results of the Petri Model Program on CTF.

Program	NX	ASLR	PIE	Rop Branch
downunderctf2021_deadcode	√	×	√	Ret to libc
csictf2020_pwn0x1	√	×	√	Ret to libc
csictf2020_pwn0x2	√	√	√	Ret to libc
csictf2020_pwn0x3	√	√	√	Ret to libc
dctf2021_babybof	√	√	√	N/A
utctf2021_resolve	√	√	√	Ret to csu
dicectf2021_babyrop	√	√	√	Ret to csu
nahamconctf2021_smol	√	√	√	Ret to csu
redpwnctf2020_coffer	√	√	×	Other
downunderctf2021_out	√	√	×	Other

shows the results of this model on 10 CTF programs. This model can automatically select paths and construct vulnerabilities efficiently and effectively.

For programs with PIE (Position-Independent Executable) protection enabled, this model can effectively implement Return-Oriented Programming (ROP) exploitation techniques and choose different vulnerability exploitation paths. However, in programs that do not have PIE protection enabled, the model opts for executing other vulnerability exploitation techniques, such as “Ret to Shellcode”.

Table 8. Qualitative Analysis Table.

Modeling Method	Code Level	Automatic Operation	Software Simulation	Clear State	Formal Analysis
Liu, Z et al. [42]	√	×	√	×	×
Biswas et al. [43]	×	√	×	×	×
Bu, W et al. [44]	√	√	√	×	×
This article	√	√	√	√	√

provides a fine-grained qualitative analysis and comparison of several typical vulnerability exploitation methods. Liu, Z et al. [42] proposed a vulnerability exploitation generation framework that uses a crash reproduction algorithm to mitigate the path explosion problem. Biswas et al. [43] introduced a two-level branch predictor data structure reuse technique for buffer overflow or return-oriented programming, but it did not involve code analysis. Bu, W et al. [44] proposed a method to monitor the program execution process through dynamic binary instrumentation, analyzing the buffer overflow vulnerability exploitation process and achieving transplantation and utilization of vulnerability samples. However, this method lacks in-depth analysis of key information such as program loading addresses. Overall, existing research has not provided a fine-grained description and in-depth analysis of ROP vulnerability exploitation processes under stack overflow. This paper, exemplifying the Ret to libc exploitation method under ROP vulnerability exploitation, achieves the mapping of path selection and payload construction to a fine-grained formalized model. It extends the analysis of exploitation details, addressing the shortcomings of existing methods.

6. Conclusions

In this paper, we consider the stack buffer overflow vulnerability, which occurs when the number of bytes written by a program to a buffer variable on the stack exceeds the buffer’s requested size. This can lead to the overwriting of a function’s return address, thereby hijacking the program’s control flow. Based on this characteristic and according to the modeling rules defined in this paper, we establish a fine-grained Petri nets model for ROP vulnerability exploitation. Recording exploit path selection and branch acquisition, we conduct timeout-based simulation experiments, qualitatively analyzing and separately calculating the simulation time of the software and the execution time of existing exploit tools. Because it does not involve the processing of actual data, the model operates at a faster speed. Consequently, the results of the model’s execution can serve as a theoretical decision reference for constructing practical exploit implementations. Through the fine-grained partitioning of the manipulation of function stack frames and state changes, the model can provide a theoretical reference for building Petri net models in the context of exploiting vulnerabilities using other methods. Using symbolic execution and dynamic analysis techniques to implement simulation experiments, the experimental results indicate that the model performs well in exploiting programs with PIE protection enabled and demonstrates advantages in exploit path selection. However, for programs that do not have PIE protection enabled and programs where obtaining library function offset addresses is not possible, there are limitations to the vulnerability exploitation method of this model. Model deadlock issues may be encountered when choosing to execute the Ret to CSU path, and inevitable path explosion problems may arise when using symbolic execution to verify the model. In the future, we plan to extend our research to programs without PIE protection and further optimize path selection based on this Petri nets model. We aim to address the issue of path explosion in symbolic execution and implement more exploitation methods, thus providing a more comprehensive and systematic approach to program vulnerability exploitation.