Symmetrical Simulation Scheme for Anomaly Detection in Autonomous Vehicles Based on LSTM Model

Abstract

Technological advancement has transformed traditional vehicles into autonomous vehicles. Autonomous vehicles play an important role since they are considered an essential component of smart cities. The autonomous vehicle is an intelligent vehicle capable of maintaining safe driving by avoiding crashes caused by drivers. Unlike traditional vehicles, which are fully controlled and operated by humans, autonomous vehicles collect information about the outside environment using sensors to ensure safe navigation. Autonomous vehicles reduce environmental impact because they usually use electricity to operate instead of fossil fuel, thus decreasing the greenhouse gasses. However, autonomous vehicles could be threatened by cyberattacks, posing risks to human life. For example, researchers reported that Wi-Fi technology could be vulnerable to cyberattacks through Tesla and BMW autonomous vehicles. Therefore, further research is needed to detect cyberattacks targeting the control components of autonomous vehicles to mitigate their negative consequences. This research will contribute to the security of autonomous vehicles by detecting cyberattacks in the early stages. First, we inject False Data Injection (FDI) attacks into an autonomous vehicle simulation-based system developed by MathWorks. Inc. Second, we collect the dataset generated from the simulation model after integrating the cyberattack. Third, we implement an intelligent symmetrical anomaly detection method to identify false data cyber-attacks targeting the control system of autonomous vehicles through a compromised sensor. We utilize long short-term memory (LSTM) deep networks to detect False Data Injection (FDI) attacks in the early stage to ensure the stability of the operation of autonomous vehicles. Our method classifies the collected dataset into two classifications: normal and anomaly data. The experimental result shows that our proposed model’s accuracy is 99.95%. To this end, the proposed model outperforms other state-of-the-art models in the same study area.

1. Introduction

Recently, the market value for autonomous vehicles (AVs) has been growing rap-idly, and it is estimated to reach 100 billion in the near future [1]. In India, fossil fuel cars will be prohibited by 2030 because of their negative impact on the environment [2]. While the production and demand for the green energy source is an increase with the addition of renewable energy capacity in the subsequent years [3], A.V. is one such technology that uses renewable energy sources instead of fossil fuel sources which reduces greenhouse gas emissions produced by conventional fossil fuel vehicles [4].

Similar to any cyber-physical system (C.P.S.), A.V.s imply communication network infrastructure to transmit critical information in real-time. Therefore, integrating communication networks in A.V. introduces many benefits, such as exchanging information between embedded devices, sensors, actuators, and other technology. This is important to ensure the requirement for high connectivity between such synchronous cyber-physical systems [5]. In addition, integrating a communication network into physical components allows remote control and resource management [6]. This will enhance energy efficiency and consumption and make it much more convenient for A.V. owners to find available charging stations [6]. However, A.V.s can be exposed to cyber-attacks which cause negative impacts on the stability of the system. Cyberattacks can be launched stochastically anytime and anywhere [7] to target the connected devices of the A.V. whenever attackers find vulnerabilities in the system. Their impact is not limited to a single component of A.V., such as the control system but can involve the whole powertrain [8]. Besides, the effect of the attack can be observed in the short and long term.

Like any man-made object, in which symmetry is one of its main signatures [9], a typical A.V. includes symmetrical sensors to perceive the nearby environment, which must communicate effectively [10,11]. Therefore, sensors are the eyes of the A.V. onboard computer, which regularly provide the location, speed, and updates on nearby environments. In addition to that, A.V. is capable of exchanging data with other vehicles (V2V), pedestrians (V2P), and Infrastructures (V2I) [12]. The electronic control unit (E.C.U.) processes the measurements coming from sensors and transmits commands to actuators to control the devices nearby the vehicles [13]. E.C.U. Processes those measurements via software, which is vulnerable to an adversary. Imagine that E.V. contains many E.C.U.s, making it harder now to detect flaws in the software. Such adversaries, with their diversity, are collectively known as cyber-attacks or intrusions.

Network-based attacks, communication-based attacks, cyber-based attacks, and physical-based attacks are the taxonomy of Cyber-physical (C.P.) attacks, as displayed in Figure 1 [14]. The network-based attack involves passive and active attacks. A threat actor can compromise network security by gaining access to a node or nodes other than those under hijack. A.V.s networks also are susceptible to different types of attacks, such as access attacks, ransomware (RANSOM) attacks [15], denial of service (DoS) attacks, and reconnaissance attacks [14]. In communication-based attacks, the A.V.s rely heavily on sensors to exchange data with other sensors through a compromised communication channel. The attacker compromises the communication channel utilized for data exchange and transmits false data. Suppose the false data is shared over a network and reaches the relayed data node to the controller. In that case, a network-based attack takes place and affects the whole nodes associated with the relayed data node [13]. This kind of attack leads to cyber and physical catastrophic impacts. Therefore, it breaches the data integrity and shares false data with the affected nodes [13,16]. The cyber-based attack involves changing the system’s code to a new code that can serve the adversary’s plan. The most common attacks in A.V.s are malware injection attacks, FDI attacks, supply chain attacks, database manipulation attacks, and password cracking attacks [17]. In a physical-based attack, the adversary attempts to provide abnormal measurements to damage the physical device, such as the control system of an A.V. Many types of research have been conducted in anomaly detecting cyber-physical attacks and can be found in [18]. Understanding how a physical system works is crucial to building a predictive model to detect any malicious data that can damage the system. For instance, the programmable logic controller is widely used in A.V.s and is susceptible to an attack on Iran’s nuclear power, known as the Stuxnet attack.

An intrusion detection system (I.D.S.) can be implemented into the onboard computer to detect any security flaws, monitor the system’s events, and report incidents that violate the security policy [19]. However, in cybersecurity, more than 99% of new intrusions are symmetrical with very small mutations of previously existing ones [20]. This requires the development of very accurate I.D.S.s with high sensitivity in detecting cyber-attacks.

Advances have greatly influenced the development of self-driving vehicles in computing. For safety and speed reasons, all constraints must be considered when simulating the self-driving vehicle model. This simulation extensively uses deep learning models and strategies, making it possible to test an automated driving model. An approach such as long short-term memory (LSTM) has efficiently simulated the system. The LSTM is based on the symmetry of recurrent neural networks (R.N.N.).

As a result, academics in the autonomous system have quickly adopted it as a problem-solving method using deep learning. For example, LSTM can be used to predict a pedestrian’s path and vehicle destination at an intersection. The effectiveness of LSTMs in time series prediction has been well established [11]. The ability of an LSTM network to predict the path of on-road vehicles is required for safe autonomous overtaking or lane changes. Furthermore, as shown in [12,13,14], several research articles investigated the advantages of using the LSTM in various systems.

One of the most common C.P. attacks is the FDI attack, which involves fabricating the data and keeping the system’s code the same. FDI attack is present in all classifications of C.P. attacks and can threaten A.V.s’ applications, systems, and network layers [11]. Generally, it is challenging to detect the FDI attack because, for example, some of its effects cannot be noticed in the short term [20].

1.1. Our Contributions

This research aims to develop a resilient cybersecurity method to mitigate the impact of False Data Injection (FDI) attacks on autonomous vehicles by detecting such attacks at earlier stages of communication. The main contribution of this research can be summarized as follows:

• We present a new dataset to simulate the False Data Injection (FDI) attacks on autonomous vehicles. The dataset was generated from the simulation model after integrating the cyberattack. False Data Injection (FDI) attacks were injected into an autonomous vehicle (A.V.) simulation-based system developed by MathWorks Inc. for research purposes. We assumed an attacker compromised a smart sensor.
• We propose an intelligent anomaly detection method based on long short-term memory (LSTM) neural networks to identify False Data Injection (FDI) attacks targeting the control system of the autonomous vehicle through a compromised sensor. The proposed anomaly detection system can classify communication traffic of autonomous vehicles into normal or anomaly data.
• We provide extensive experimental evaluation results using standard performance indication factors such as detection accuracy, precision, recall, and F Score. Ultimately the prosed system achieved an overall accuracy equal to 99.95%.

1.2. Paper Organization

This paper is organized as follows: the recent literature review is represented in Section 2. Section 3 discusses the autonomous vehicle simulation model used in this research. Then, Section 4 explains the system development and specifications. Finally, Section 5 provides conclusions and future work related to this research.

2. Literature Review

Due to the rapid development in engineering and technology, cities have become increasingly smart. This can be achieved while relying on data and technology to improve several sectors such as mobility and transportation. As such, autonomous vehicles are an indispensable part of smart mobility that emerged to improve the life quality inside smart cities. Nevertheless, autonomous vehicles are vulnerable to a wide range of cyberattack vectors that might severely impact humans’ life quality and safety. Therefore, several research studies have been conducted to analyze, identify, and mitigate autonomous vehicle cyberattacks and defense mechanisms.

2.1. Existing Related Models

For instance, in [21], the authors suggested a preemptive classification scheme for the cyber risk categories of connected and autonomous vehicles. Their predictive model uses Bayesian Networks (BN) to utilize the variables and fundamental relationships from the Common Vulnerability Scoring Scheme (CVSS) to parameterize the cyber risk of connected and autonomous vehicles. As a result of evaluating their model on an out-of-sample test, their B.N. predictive scheme exhibited high prediction accuracy for several risk scores and levels, scoring approximately 100%.

Also, in [22], the authors proposed a conceptual framework to classify the potential vulnerabilities of connected and autonomous vehicle systems. The suggested conceptual framework was developed using Uniform Modeling Language (UML) using the KDD99 dataset to produce a new dataset modeling the cyberattacks targeting the communication processes of connected and autonomous vehicles, known as CAV-KDD-2020. CAV-KDD-2020 dataset is a communication-oriented dataset covering several types of attacks targeting different possible attack points of the connected and autonomous vehicle’s systems, including the hardware parts of the autonomous vehicle such as LIDAR sensor, Camera, power system, software parts of the autonomous vehicle such as in-vehicle system, data processing system, and decision-making system, the data itself of the autonomous vehicle such as vehicle id, vehicle’s speed, users personal information, and brake status, and the communication network protocols of the autonomous vehicle such as the vehicle to infrastructure communication, vehicle to cloud communication and vehicle to vehicle communication. To evaluate the new dataset, the authors employed two supervised machine learning classifiers, including a Decision Tree Classifier (D.T.C.) and a Naive Bayes Classifier (N.B.C.), to classify the cyberattacks on the autonomous vehicles into four attack groups, including probe Dos attacks, Root to Local (R2L) attacks, and User to Root (U2R) attacks. Accordingly, the experimental results revealed that the D.T.C. model scored higher accuracy and precision proportions with a shorter runtime and thus is more applicable for the attack detection of autonomous vehicle communication.

Moreover, in [23], the authors proposed a real-time multi-stage deep-learning-based I.D.S. structure designed to recognize cyberattacks from the Intelligent Transportation Systems (ITS) to generate minimal False Alarm Rates (FAR). Their system employs the normal state-based and the Long Short-Term Memory (LSTM) deep learning model in a bidirectional mode to detect the potential attacks of connected and autonomous vehicle systems. To assess their implemented method’s performance, the authors evaluated their model on two standard datasets: the UNSWNB-15 dataset and the CAR-HACK dataset. Consequently, their empirical investigations pointed out that their proposed multi-stage I.D.S. system surpassed other models scoring higher accuracy levels with 98.88% and 99.11% for UNSWNB-15 and CAR-HACK datasets, respectively. Such outcomes may enhance the cybersecurity for autonomous vehicles at both levels, the in-vehicle communications and out-vehicle communications (exterior).

Similarly, in [24], the authors developed a deep learning-based I.D.S. for autonomous vehicles in a real-time fashion. The proposed system is composed of two main stages. The first stage is responsible for features extraction leveraging auto-encoder-based long short-term memory. The second stage is responsible for anomaly detection and classification for every signal sequence using a Convolutional Neural Network (CNN) in a real-time environment. Their experimental results reported 95.5% and 94.2% for model accuracy and precision, respectively.

Furthermore, in [25], the authors researched the cyber-security vulnerabilities of autonomous vehicles under sensor attacks. Specifically, they proposed a new rule-based I.D.S. system to identify the sensor attacks and sources for connected and autonomous vehicles. The proposed I.D.S. uses a combination of an extended Kalman filter (E.K.F.) to estimate the vehicle’s location and a Cumulative Sum (CUSUM) discriminator to identify the possible variation of the sensor measurement. For higher resiliency against intrusion, multiple sensors were deployed to deliver real-time postures of the autonomous vehicle states. Besides, an auxiliary detector to examine the irregularity between multiple sensor measurements. Finally, a rule-based separation system is employed to analyze the detectors’ results and provide information about the abnormal sensor. Extensive experimental results were reported, showing the developed model’s usefulness in actual autonomous vehicle data.

Besides, in [26], the authors investigated and studied the threat classification concerning autonomous vehicles targeting three major security services: authentication, accountability, and availability. The authors elaborated on the various countermeasures for autonomous vehicle intrusions and their developmental aspects in this study. Specifically, the authors emphasized the vital role of blockchain to prevail over and mitigate such security and privacy concerns (for autonomous vehicles). Lastly, they end their investigational study by delving into the genuine concerns and questions of blockchain-based security systems for autonomous vehicles.

The authors in [27] proposed two deep learning algorithms to detect Denial of Service (DoS) attacks committed to the Electric Vehicle Charging Station (EVCS). The authors used python’s Long-Short Term Memory (LSTM) and Deep Neural Network (D.N.N.) algorithms to classify the DoS attacks. It was assumed that attackers could use any weak network link to establish the DoS attack. The D.N.N. and LSTM algorithms were trained, tested, and validated. 50% of the data was used for training, 20% for validation, and 30% for testing. According to the authors, the accuracy of both deep learning algorithms has recorded high accuracy rates.

Finally, another noticeable work was observed in [28]. In this paper, the authors investigated the prospective and pragmatic encounters in the use of artificial intelligence (AI) to analyze the cyber threat and risks (AI-enabled dynamic cyber risk analytics at the edge), to enhance toughness against risks and threats in connected devices such as Internet of Things (IoT) and Cyber-Physical System (CPS) devices. Besides, the authors have applied the grounded theory to group the requirements for AI in CPS risk analytics which was then constructed into a conceptual diagram, representing a cascading hierarchy of processes.

2.2. Research Gap and Novelty

Unlike the studies mentioned above, where the presented models are developed through the learning-based scheme (training and testing) using predefined systematic Attack-Aware datasets [28] that contain features of common cyber-attacks (intrusions). However, such models lack to detect newly developed false data injected against the control system of the autonomous vehicle. This research contributes to the cybersecurity of autonomous vehicles by detecting the False Data Injection (FDI) attacks developed in this research. First, we develop and inject new FDI attacks into an autonomous vehicle simulation-based system developed by MathWorks. Second, we collect the dataset generated from the simulation model after integrating the cyberattack. Third, we implement an intelligent symmetrical anomaly detection method to identify FDI attacks targeting the control system of the autonomous vehicle. through a compromised sensor. We use long short-term memory (LSTM) deep networks to detect FDI attacks in the early stage to ensure the stability of the operation of the autonomous vehicle. Our method classifies the collected dataset into two classifications: normal and anomaly data. The experimental result revealed a high-performant model outperforming other state-of-the-art models in the same study area.

3. Autonomous Vehicles Simulation Model

MathWorks developed the A.V. software system used in this research. Inc. It is a simulation based-model built using MATLAB, and Simulink focuses on using adaptive cruise control (A.C.C.) to regulate the A.V. velocity. It consists of two cars (1) the ego car and (2) the lead car. The ego car is a self-driving car that needs to maintain its speed and distance from the lead car in the same lane. Therefore, the ego car relies on an A.C.C. to regulate speed and distance. The A.C.C. system composes speed control and distance control to adjust the dynamic of the ego car to be symmetrically commensurate with the lead car [29].

The ego car uses sensors such as radar to collect information about the position of the ego car and the lead car. The sensor readings are fed to the A.C.C. system to regulate the speed of the ego car to maintain its position according to the lead car. The default safe distance between the ego and lead car is symmetrically set to 10 m. Therefore, if the safe distance between the ego and the lead car is smaller than or equal to the relative distance (i.e., Asymmetric), the ego car needs to increase its speed ($D_{rel}\ge D_{Safe})$, as shown in Figure 2. However, if the safe distance is larger than the relative distance, the ego car needs to decrease its speed $(D_{rel}<D_{Safe})$, as shown in Figure 3.

Figure 4 shows the velocity of the ego car and the lead car. The initial velocity of the ego car is 20 m/s, and the initial velocity of the lead car is 25 m/s. The desired velocity, in this case, is 30 m/s. It can be observed that the ego car symmetrically maintains its speed according to the lead car during the entire simulation time. Therefore, when the lead car reduces or increases its speed, the ego car follows accordingly.

Figure 5 illustrates the actual distance and the safe distance between the ego car and the lead car measured in meters during 81 s of the simulation. The red line in the figure refers to the actual distance between the two cars. The blue line refers to the safe distance, which is the distance the ego car should maintain while following the lead car. Overall, we can observe that the actual and safe distance have comparable results, which means the distance between the two cars is maintained symmetrically. Furthermore, we notice that when the actual distance decreases, the safe distance also decreases, whereas when the actual distance increases, the safe distance increase accordingly. That is because when the actual distance is larger than the safe distance, the ego car accelerates its speed to follow the lead car, as shown in Figure 4. Also, when the actual distance is smaller than the safe distance, the ego car decelerates its speed to avoid collision with the lead car. As mentioned earlier, the ACC controller is responsible for adjusting the distance and velocity of the ego car to safely follow the lead car.

4. System Development and Specifications

This section discusses generating the dataset used in this research and provides information about the data preprocessing procedure. In addition, it provides a detailed description of the implementation of the LSTM model used for the classification’s procedure. Finally, it evaluates the performance of the LSTM model. Figure 6 illustrates the overall architecture of the system model used in this research.

4.1. A Scheme for Generating Dataset

This section will discuss the testbed used to generate a dataset for anomaly detection for the A.V. system. First, the development and integration of the cyberattack are explained. Second, it shows the collected dataset features. Third, it discusses the calculation of the anomaly detection feature.

4.1.1. Implementation of Cyberattack

The MATLAB/Simulink simulation model used in this research does not include cyberattacks. Therefore, FDI attacks were implemented and injected into the sensor responsible for measuring the position of the ego car. The equation of the FDI attack is shown in Equation (1). The $D_{act}$ refers to the relative distance between the ego car and the lead car measured by the sensor. The attack percentage refers to the strength of the attack in percentage. The value of the attack percentage starts from 0.00001% to 100% to include a maximum number of possible attack strengths.

Figure 7 showed the velocity performance of the ego car and the lead car when the FDI attack was injected into the sensor responsible for reading the position of the ego car. The simulation ran for 81 s, and the velocity was measured in meters per second. The attack percentage value was 60%. That means the actual position of the ego car read by the sensor was increased by 60% from the original value.

As a result, the ACC reduced the ego car speed, which is a normal response of the controller because the position value read by the sensor fed to the ACC controller was risen by 60% due to the attack. Therefore, the ACC reduced the ego car speed because the ACC assumed that the distance between the two car was very close. According to Figure 7, the velocity of the ego car was not stable due to the impact of the FDI attack, and the ego car velocity could not reach the desired speed (30 m/s) compared with Figure 4.

Therefore, the actual sensor value can be modeled according to the following Algorithm 1 (Calculating Actual Sensor Value Under FDI):

Algorithm 1. Calculating Actual Sensor Value Under FDI

Input_1: Original Sensor Reading$(D_{act })$in meter

Input_2: Attack Percentage (0.00001% to 100%)

Processing: Get Original Sensor Reading$(D_{act })$

Assume Attack Percentage = $(At{t}_{\% })$

Compute: $Temp=\left(At{t}_{\% }\times D_{act }\right)$

Then: $D_{act+1}=D_{act }+\mathrm{Temp}$

Output: Actual Sensor Value$(D_{act+1}) in meter$

Figure 8 depicts the safe distance and actual distance between the two cars under the same attack percentage (60%). Comparing Figure 8 with Figure 4, we found that they no longer have comparable performance because the ACC has a different response in each figure. The reason is that the values of the sensor measuring the position of the ego car given to the controller were alike. As a result, the ACC responded differently when calculating the actual and the safe distance between the ego car and the lead car. In Figure 4, the actual and safe distances have very close results because there was no attack. However, in Figure 8, during the first 20 s of driving, the actual and safe distances were not similar. Also, the actual and safe distances have zigzag waves, which means the ego car performance was not stable due to the attack.

4.1.2. Dataset Features

The dataset used in this research was collected from a real-time simulation using the A.V. model with the integrated FDI attacks. We collected the response of the following parameters: the actual position of the ego car, the actual velocity of the ego car, the actual position of the lead car, and the actual velocity of the lead car. In addition, we extracted a new feature we called the anomaly detection label with two classifications label normal and anomaly, using the last four features. Therefore, our dataset consists of five features listed in

Table 1. Dataset Features.

Feature No.	Feature Name	Unit	Data Type
1	Actual position of the ego car	m	Double
2	Actual velocity of the ego car	m/s	Double
3	Actual position of the lead car	m	Double
4	Actual velocity of the lead car	m/s	Double
5	Anomaly detection label	Normal, Anomaly	Binary

.

4.1.3. Anomaly Detection Label

The anomaly detection label was calculated using features 1, 2, 3, and 4. We injected FDI attacks into the sensor responsible for measuring the actual position of the ego car. According to our observation, the ego car has a stable performance when the attacks’ strength is between 0.00001% to 0.01%. For this reason, this anomaly detection label is marked as normal. The size of the normal dataset is 10,000 records, and each record has four features with 81 data lengths. However, when the strength of the FDI attack is larger than 0.01%, the ego car’s performance is considered unstable. Therefore, we inject FDI attacks into the actual position of the ego car, but the strength of the FDI attack this time is between 0.011% to 100%. The size of the injected dataset (anomaly) is 10,000 records, and each record has four features with 81 data lengths. Therefore, the total size of the dataset is 20,000 records.

4.2. Implementation of LSTM

LSTM is a deep neural network first proposed by Hochreiter in 1997 [30,31]. LSTM uses time-series data for classification by keeping track of cell states to preserve certain memory trends across time [32]. LSTM block consists of three gates: forget, update, and output, which works with the input for a time series, as shown in Figure 9 [33,34]. The model decides whether to forget, update, or output new data at each state stage. Therefore, LSTM is made to avoid the issue of long-term dependence. The forget gate determines whether a piece of data should be saved. In the LSTM, the input gate refreshes the cells, while the output gate always determines the hidden state. As a result, they determine which data should be shared with other cells and which should be ignored based on the outcome, which ranges from zero to one. Zero indicates rejection, but one indicates inclusion.

The component of the LSTM architecture used for classification is illustrated in Figure 10. Initially, the data is fed to the sequence input layer, followed by the LSTM layer. Next, the prediction procedure is performed in the fully connected Softmax layers. Finally, the output is produced in the classification layer [35].

4.3. Training Procedure

After preprocessing the dataset, we trained it using the LSTM model. We split our dataset into two groups; the first group is used for training with 70% of the dataset, and the second group is 30%, which will be used for testing. Therefore 17,000 records were used for training, and 3000 were used for testing. We used the Adam optimization algorithm to train our LSTM networks, as discussed in the following section [36]. To evaluate or training procedure, we used the Cross-Validation technique [35]. It is a technique that can be used as a validation scheme to solve over-fitting problems. Basically, the dataset is randomly subsetted into groups of data. Specifically, in the training procedure, we used 5-fold cross-validation, meaning the entire dataset is divided into 5 sets of almost equal size. Each set of the 5 sets is trained and tested separately. Also, the error is calculated for each set to avoid overfitting problems.

Adaptive Moment Estimation Optimization (ADAM)

Classification can be difficult when dealing with problems relating to the learning process. Several approaches have been proposed to help us arrive at an optimal learning level. The Adaptive moment estimation (ADAM) optimization algorithm is a recent deep learning extension of the stochastic gradient descent algorithm, which has recently been used in a variety of applications like on the Internet of Things, text detection, and so on [37,38,39,40]. According to empirical outcomes, the method has performed well in practice and compares favorably to other stochastic optimization approaches [40]. Stochastic gradient descent is an efficient and effective optimization technique that has played an important role in many machines learning. According to the concept of the method, individual adaptive learning rates for distinct parameters are calculated using estimates of the gradient’s first and second moments based on combining both RmsProp and AdaGrad [39]. RmsProp computes the average of recent changes in the magnitude of the internal signal gradient, while AdaGrad handles sparse gradients with uncentered variance [38]. These algorithms can be calculated according to the following Equations, Equations (1) and (2) [38]:

$$m_t=\beta m_t-1+\left(1-bet{a}_1\right)g_t$$

(1)

$$v_t={\beta }_2{v}_t-1+\left(1-bet{a}_2\right)g^2{}_t$$

(2)

4.4. Testing Procedure

To ensure the high validity of our experimental evaluation, we have performed the following techniques:

• K-Fold Cross-Validation (already discussed in Section 4.3)
• Confusion matrix
• Evaluation metrics (precision, recall, and F1-score)
• Comparison with existing methods.

We evaluated our model using the confusion matrix shown in Figure 11, which depends on the True Positive (TP), False Positive (FP), False Negative (FN), and True Negative (TN). Then the accuracy is calculated using Equation (4) [41]. TP refers to the number of positive data classified correctly. FN refers to the number of positive misclassified data. Meanwhile, FP refers to the number of negative misclassified data, and TN refers to the number of negative data classified correctly. As was mentioned above, 30% of the data was used for testing. Figure 12 illustrates the confusion matrix of our proposed models. We can observe that majority of observations were correctly classified with only three observations are incorrectly classified.

In addition, we evaluated our test method using detection accuracy, detection precision, detection recall, and the F1-score metrics, as represented in

Table 2. Precision, recall, and F1-score metrics.

Accuracy Parameter	Value
Precision	99.93%
Recall	99.97%
F1-Score	99.95%
Accuracy	99.95%

[42]. The accuracy of our model reached 99.95%. The detection accuracy, precision, recall and the F1-Score are calculated using the standard Equations (Equations (3)–(6)) [41].

$$Accuarcy=\frac{TP+TN}{TP+TN+FP+FN}$$

(3)

$$\mathrm{Precision}=\frac{TP}{TP+FP}$$

(4)

$$\mathrm{Recall}=\frac{TP}{TP+FN}$$

(5)

$$\mathrm{F}1-\mathrm{Score}=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

(6)

Table 3. Comparing our proposed model’s accuracy with existing deep learning models’ accuracy.

Research	Task	No. of Features	ML Model	Accuracy
Hamza et al. [45]	Detection	NA	COSBO-BiLSTM	98.81%
Almasoud et al. [46]	Detection	24	RNN-GLSTM	96.7%
Roh et al. [47]	Detection	64	CNN-LSTM	92.03%
Sarwar et al. [48]	Detection	83	Random Forest	83%
Song et al. [49]	Classification	77	Deep-learning	97.4%
Alkahtani et al. [50]	Classification	80	CNN-LSTM	98.90
Al-Haija et al. [51]	Classification	43	CNN	98.2%
Ullah et al. [52]	Detection	83	SVM	80%
Proposed method	Detection	4	LSTM	99.95%

lists the accuracy of current existing deep learning methods developed by researchers compared with our results. According to the table which considers the performance comparison between our proposed model against several state-of-the-art models. The comparisons revealed the supremacy of our proposed model in terms of accuracy and the data generation process. In addition, it shows the number of features that were used in each study, and it can be observed that our proposed method used a lower number of features (only four) compared with other models. Generally, training a deep learning model with a few features can be challenging since most deep learning models require a sufficient number of features to reach higher accuracy [43]. While deep learning models’ accuracy could also suffer from many number features [44].

5. Conclusions

Rapid computing advances have significantly impacted the study and development of autonomous vehicles in various fields. Autonomous vehicles are broadly deemed as an indispensable system of smart city development due to their significant roles in improving the lifestyle in developed cities. Nevertheless, autonomous vehicles are susceptible to a range of cyberattack vectors that endanger human lives. Hence, an intelligent anomaly detection system for autonomous vehicles is proposed, developed, and evaluated in this paper. In this system, we developed and presented a new simulated dataset for False Data Injection (FDI) attacks on autonomous vehicles. Then, we implemented an intelligent anomaly detection method based using a symmetrical LSTM neural network to detect the injected FDI attacks targeting the control system of the autonomous vehicles through a compromised sensor. Finally, to certify the trustworthiness of the validation process, the evaluation process has undergone 5-fold cross-validation and the average performance indicators have been observed and recorded using the confusion matrix, the detect accuracy, the detect precision, the detect recall, and the F1-Score measurements to gain more insights into the solution approach. Accordingly, the performance evaluation of our anomaly-based LSTM model exhibits outstanding results. Specifically, the d model recoded an average detection accuracy of 99.95%, average detection precision of 99.93%, average detection sensitivity (recall) of 99.97%, and average detection F-Score of 99.95%. Thus, the comparison with existing recent models revealed the supremacy of our model in terms of detection performance and detection ability for new false data attacks.