Next Article in Journal
A Data-Driven Analysis of Electric Vehicle Adoption Barriers in the Philippines: Combining SEM and ANNs
Previous Article in Journal
RBCKF-Based Vehicle State Estimation by Adaptive Weighted Fusion Strategy Considering Composite-State Tire Model
Previous Article in Special Issue
Enhanced Vehicle Logo Detection Method Based on Self-Attention Mechanism for Electric Vehicle Application
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
WEVJ
Volume 15
Issue 11
10.3390/wevj15110518
wevj-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

A Study to Investigate the Role and Challenges Associated with the Use of Deep Learning in Autonomous Vehicles

by
Nojood O. Aljehane
College of Computer Science and Information Technology, Tabuk University, Tabuk 71491, Saudi Arabia
World Electr. Veh. J. 2024, 15(11), 518; https://doi.org/10.3390/wevj15110518
Submission received: 30 September 2024 / Revised: 23 October 2024 / Accepted: 4 November 2024 / Published: 12 November 2024
(This article belongs to the Special Issue Deep Learning Applications for Electric Vehicles)
Versions Notes

Abstract

:
The application of deep learning in autonomous vehicles has surged over the years with advancements in technology. This research explores the integration of deep learning algorithms into autonomous vehicles (AVs), focusing on their role in perception, decision-making, localization, mapping, and navigation. It shows how deep learning, as a part of machine learning, mimics the human brain’s neural networks, enabling advancements in perception, decision-making, localization, mapping, and overall navigation. Techniques like convolutional neural networks are used for image detection and steering control, while deep learning is crucial for path planning, automated parking, and traffic maneuvering. Localization and mapping are essential for AVs’ navigation, with deep learning-based object detection mechanisms like Faster R-CNN and YOLO proving effective in real-time obstacle detection. Apart from the roles, this study also revealed that the integration of deep learning in AVs faces challenges such as dataset uncertainty, sensor challenges, and model training intricacies. However, these issues can be addressed through the increased standardization of sensors and real-life testing for model training, and advancements in model compression technologies can optimize the performance of deep learning in AVs. This study concludes that deep learning plays a crucial role in enhancing the safety and reliability of AV navigation. This study contributes to the ongoing discourse on the optimal integration of deep learning in AVs, aiming to foster their safety, reliability, and societal acceptance.

1. Introduction

Deep learning is a form of machine learning that applies neural networks to mimic the structural and functional dynamics of the human brain [1]. The operational aspects of autonomous cars rely heavily on analyzing vast amounts of environmental data, from which operational decisions and situational awareness are derived [1,2]. This ability to learn from the environment through the collection and analysis of data in autonomous cars is enabled by deep learning [2]. Initially, the idea of autonomous vehicles was a fictional idea. However, due to the availability and accessibility of advanced technologies like deep learning, autonomous vehicles are now a reality [3]. Therefore, it is vital to understand how AI-based technologies like deep learning work in AVs as a primary step towards level 5 automation.
The thesis of this research is based on the argument that deep learning algorithms have been extensively used in optimizing the technical and operational architecture of autonomous vehicles (AVs). In autonomous vehicles, the current research postulates that deep learning algorithms are used to enable perception, decision-making, localization, and mapping in autonomous navigation. However, the application of deep learning algorithms is also hindered by challenges ranging from the complexity of model training to sensor challenges and the complexity and uncertainty of deep learning systems. This paper concludes that addressing these challenges will optimize the accuracy and robustness of deep learning systems in AVs.
The main objective of this study is to analyze the uses and challenges of deep learning in autonomous vehicles (AVs). Through the analysis of how deep learning is applied in autonomous vehicles, this research paper will enhance the existing understanding of AI-based technologies used in AVs. This paper will also highlight reasons why the role of technologies like deep learning in AVs is indispensable. By highlighting some of the barriers and challenges involved in the application of deep learning models in AVs, this study will also spur and inspire future research directions targeted at upscaling the application of deep learning in AVs.
Subsequently, this study will extensively contribute towards the feasible and large-scale adoption of deep learning in AVs.

2. Research Method

Therefore, to satisfy these objectives, this research paper employs a qualitative research method in this systematic and dynamic literature review of the literature on deep learning in autonomous vehicles. A comprehensive review of relevant peer-reviewed literature published between 2017 and 2023 was conducted, focusing on real-world applications of deep learning in autonomous vehicles.
Terms like ‘deep learning’, ‘autonomous vehicles’, ‘application challenges’, and ‘systematic literature review’ were used in the search. These keywords were deemed most appropriate to select the literature for the present study that was allied to the major research themes: “Deep learning algorithms significantly enhance the perception, localization, and navigation capabilities of autonomous vehicles, optimizing their overall operational framework”.
The reason for choosing a systematic literature review as the legal research method is that as it provides an accurate, comprehensive, and quite detailed identification and definition of the research variables [4,5]. To improve the validity of the review, the selection process included only articles published between 2017 and 2023, which reflects the recent state of the issue. This period was selected deliberately because it covers the modern developments and issues in the utilization of deep learning in self-driving automobiles.
However, to select the articles, several criteria were used aside from the publication date, among which include the following: In particular, only articles published in peer-reviewed journals were considered, which allowed us to focus only on reliable sources. Papers that were excluded were those that were purely theoretical, with little computational or field data support, or those where the real-world implementation of the theories was not clearly explored. Furthermore, only those papers that focused on deep learning and self-driving cars were included in this review.
As far as perspective awareness is concerned, it can be assumed that the literature review collected from the studies published during the last five years will give a fresh outlook on the state-of-art technologies and methodologies on deep learning in AV systems, along with the identification of open problems and challenges in this regard.

3. Literature Review

3.1. Autonomous Vehicles

The wave of modernization and technological development is responsible for the paradigm shift being witnessed in the automotive industry. By 2030, level 2 AVs will represent 92% of the market share and level 3 AVs will represent 8% [6]. Additionally, the AV market is expected to grow by 39.47% from USD 54.23$ billion in 2019 to USD 75.6 billion in 2026 and ultimately aggregate USD 87 billion by 2030 [7]. By 2035, self-driving cars are also expected to account for 25% of total car sales. Between 2019 and 2026, Europe is expected to have the highest growth rate in the AV market at 42.6%, with North America also being expected to be a leader in the AV industry [7,8]. Nevertheless, whereas AI systems like deep learning have enhanced the technical architecture of AVs, the large-scale adoption of AVs is also hindered by social acceptability, adverse road conditions, weather, data privacy, and cybersecurity among others [1].
Despite the current development, Biswas and Wang [9] argue that the practicality of level 5 autonomous vehicles is still under development. The primary impetus factors that cause these phenomena include unaddressed technological barriers besides trust, safety, and ethical issues. Nevertheless, technological giants like Tesla, Google, Audi, BMW, and Mercedes-Benz among others, through ongoing road-testing, have extensively influenced current research designed to address the AV technological architectural barriers [10]. Through such efforts, giants like Tesla and Google have managed to incorporate self-driving features in current AVs. Furthermore, with the increasing availability of data and advanced technology, the detection accuracy, latency, and response time of AVs are expected to be optimized.
The six automation levels used to categorize autonomous vehicles are summarized in Table 1.

3.2. The Need for Autonomous Vehicles

There are various reasons why autonomous cars are relevant and significant in the backdrop of changing transportation. Besides alleviating the economic and environmental issues related to transportation, autonomous vehicles are promising solutions to congestion, accidents, and emissions [12].
Notably, Fayyad et al. [10] agree that autonomous vehicles will provide a safe, efficient, cost-effective, and accessible means of transport. Autonomous cars are also expected to alleviate the impact of undesirable impacts of carbon emissions on climate change. For example, Ercan et al. [13] illustrate that a 1% increase in the sale of electric vehicles has the potential to reduce carbon emissions in a city by 0.096% and 0.087% in a nearby city. Additionally, electronic vehicles also reduce carbon emissions indirectly through substitution, consumption, and technological effects. Overall, the results of Ercan et al. [13], which analyzed data from more than 929 metro/metropolitan areas in the US, showed that the adoption of autonomous vehicles could reduce greenhouse gases by 34% by 2050.
However, another study undertaken at the Massachusetts Institute of Technology (MIT) revealed that the powerful onboard computers programmed to use deep learning and neural networks are not environmentally friendly [14]. Therefore, widespread and global adoption of autonomous vehicles is likely to generate over 0.14 gigatons of greenhouse emissions annually, similar to the annual greenhouse emissions of Argentina [14]. Therefore, enhancing the theoretical, technical, and operational understanding and challenges of deep learning in autonomous vehicles is likely to alleviate such undesirable environmental impacts.
Autonomous vehicles are also expected to solve other transport-related issues like accidents and congestion. For instance, 93% of accidents, especially crashes, are caused by human errors [15]. Autonomous vehicles will reduce such statistics by reducing human involvement in driving, which will subsequently minimize human errors like speeding, distraction, and driving under the influence [15]. This impact on minimizing accidents has already been realized in semi-autonomous vehicles. A Survey by the Insurance Institute of Highway Safety showed that partially autonomous features like forward collision avoidance, side view assistance, and lane departure warning reduced road accident crashes, accidents, and fatalities by at least 33% [16]. Karnati et al. [1] also agree that the application of AI in AVs will optimize the ability of self-driving vehicles to address some of the problems affiliated with conventional cars like road safety, limited independence for people with disabilities, low efficiency, traffic congestion, and environmental pollution.
Self-driving cars are said to enhance traffic flow but have difficulties in simulating congestion. Human-like AVs are explicitly programmed to drive patiently and safely; thus, they may over-compensate on this by going slower. “Deep learning applications face several challenges, including model training complexity and the need for highly accurate sensor data. These challenges directly impact the decision-making processes of autonomous vehicles”.
The second set of issues is linked to fleet management at ride-hailing platforms, where AVs drive around without passengers, thus contributing to congestion. In urban environments, reactions to pedestrians and cyclists may also hinder AVs as they may have to make several braking and slow movements for reasons of safety [16].
However, the opportunity exists for AVs to be coordinated in the management of traffic in a way that is likely to control congestion in the long term. Elements include vehicle “platooning” of cars, where shifting from one route to another could improve traffic flow by better aligning the distances between vehicles and their speeds. Potential modifications include better connectivity for AVs, including smart traffic lights and dedicated AV lanes to reduce congestion points. Although the use of AVs might result in small congestion initially, their adoption is likely to lead to long-term opportunities of achieving enhanced traffic systems [14].

3.3. Deep Learning

Deep learning is a specialized form of machine learning based on artificial neural networks (ANNs), whose structure is derived from the human brain. Deep learning algorithms comprise multiple layers of ANNs that are trained to extract and learn relevant features from vast amounts of data [17,18]. This ability to learn and extract relevant features makes deep learning algorithms applicable in different AV features like natural language processing, image and speech recognition, and autonomous navigation [19]. One of the turning points in deep learning that fostered its application in self-driving cars, among other fields, was the achievement of state-of-the-art results in the ImageNet visual recognition challenges by a deep convolutional neural network called AlexNet [20,21].
Ultimately, the comprehensive application of deep learning has been influenced by various factors. Such factors include the advancements in powerful computing resources and the availing of large, quality, and reliable training datasets [22,23]. Some of the existing deep learning structures are summarized in Table 2. “Recent advancements in end-to-end deep learning have further streamlined the decision-making process in AVs by eliminating intermediate steps, allowing models to directly map sensory inputs to control actions. This approach has shown promise in enhancing the accuracy of AV systems, particularly in path planning and obstacle detection”.

4. Results

Deep learning has multiple uses in AVs as demonstrated by a wide scope of the literature being related to the research topic. These uses are affiliated with components/aspects of AVs like perception, decision making, motion planning, and safety validation.

4.1. Perception

Perception refers to the ability of the AV to continuously scan and track the surrounding environment. Perception also involves the semantic segmentation of roads with different drivable surfaces like off-road and tarmacked surfaces. For this purpose, the AV uses LiDar and radar sensors besides cameras to mimic human vision [28]. The existing deep learning algorithms enable both mediated and direct perception.
Mediated perception applies both deep learning and convolutional neural networks to detect images of the surrounding environment. The detailed map of the surroundings is developed from the analysis of distance and coordinates from other vehicles and other physical obstacles like trees and road signs [29]. The study of Tong et al. [30] sought to establish the perception accuracy of deep learning algorithms. The study showed that deep learning enabled AVs to detect traffic signs with an accuracy of 99.46%, which was higher than humans in some tests [30]. Other deep learning models like YOLO Darknet v2 detect 40–70 frames per second, which is an 80% detection accuracy rate in real-time AV driving [30]. Ultimately, high-definition images are expected to enhance the detection accuracy of deep learning algorithms. Additionally, Guan et al. [31] acknowledge that advanced techniques like salient analysis and edge detection have been developed to derive high-definition images.
On the other hand, direct perception involves decision-making and integrated scene awareness. Hence, direct perception focuses on immediate AV aspects like the immediate steering wheel motion and speed while avoiding preliminary localization and mapping [32]. Therefore, instead of using a detailed local map, the AV uses deep learning to develop sections of maps required to acquire immediate scene awareness components like the distance from immediate vehicles and lane markings [33].
One of the most recommended deep learning algorithms used for direct perception in AVs is PilotNet. The deep learning model is efficient because it comprises a single normalization layer and five convolutional layers besides three fully connected layers [34]. Using sensor and camera data as the input, the primary output of the model is steering parameters, which help to steer the AV.
Odometry is also an important aspect of perception enabled by deep learning algorithms. It involves identifying shifts in position and orientation relative to surroundings during autonomous navigation [35,36]. Notably, Li et al. [37] and Mohamed et al. [38] established visual odometry algorithms like UnDeepVO, which significantly relied on unlabeled data, unsupervised learning, and deep neural networks to enhance accuracy and robustness. Others like probabilistic visual odometry (ESP-VO) also use deep learning, recurrent convolutional neural networks (RCNNs), and monocular cameras to estimate pose and generate depth maps [39,40]. These examples demonstrate the extensive application of deep learning algorithms in fostering perception during autonomous navigation in AVs.

4.2. Decision Making

Given that autonomous vehicles (AVs) are rapidly developing and placing a new kind of focus to the transport’s future, such opportunities and challenges exist [41]. Some of the most relevant concerns connected to AVs include platooning, car sharing, as well as relocation considerations. All of these concepts form the core of how self-driving cars will operate on the roads and interact with other traffic systems [42]. All of these topics convey a potential to yield substantial benefits but provoke issues such as traffic jam and ineffectiveness if proper planning is not performed.
  • Platooning
The technology of platooning is critical in AV use, wherein vehicles have the ability to travel in series with little distance between them. Based on the information exchange between vehicles and other reference vehicles, as well as vehicle-to-vehicle communications, i.e., (V2V), speeds, brake actions, and even the steering actions can automatically be synchronized to reduce the inter-vehicle distances to be very small. Through this formation, it enhances fuel economy as less drag is created, and in increasing the number of vehicles to be transported within a given space, the efficiency of highways is enhanced [43].
The main advantage that can be received by implementing platooning is a possible improvement in traffic conditions, reducing sudden lane changes or jerks caused by sudden braking or fluctuations in speed, which are often exhibited by human drivers in a convoy, are eliminated, hence allowing platooning AVs to maintain the best speed in order to avoid traffic congestion [44]. However, this very notion presupposes certain difficulties as well. If AVs were to share the roads with other conventional automobiles, human drivers could sometimes interfere with platoons by joining the highly compact formation. This could disrupt the communication and coordination of autonomous vehicles on the road, meaning that everyone will be moving slowly and simultaneously pose risks to one another [45].
The realization of the above benefits of platooning may require enhancements in the road infrastructure such as allocating exclusive AV lanes. These lanes would enable the grouped operation of AVs without hindrance by other human-operated automobiles. Thirdly, the standards of V2V communication must be set to enforce all forms of AV to be compatible with one another and be able to interact and integrate into platooning systems [46].
  • Car Sharing
Another promising phenomenon associated with the emergence of the use of self-driving cars is car sharing. Possessive handling of a car could be replaced by shared models, where users obtain access to a flotilla of self-driving cars, as needed. It may cut the circulation of cars, lessen emissions, and perhaps even eliminate some of the traffic congestions seen in cities today, especially when it comes to hunting for parking lots. As there is no human driver in an AV, it can operate for twenty-four hours, picking up and dropping off passengers, thus enhancing transportation systems [47].
However, car sharing with AVs offer the following challenges: An emerging problem is the question of how to coordinate the arrival of client demands to shared cars. Car-sharing AVs must be located throughout a city to facilitate passenger demands, but they are not always required at all times because passenger demands rise and fall with time, geographic location, and specific events. Combined, these vehicles can be condensed in certain areas, leaving other areas a little or not at all serviced. Self-driving fleets must be able to deploy complex predicative algorithms for the demand and deployment of the cars [48].
Another issue is what’s commonly known as deadheading, where self-driving cars travel empty, moving around without any passengers on board. If not accomplished effectively, this could worsen traffic jams, especially in large cities. With such factors as distance, cost, and demand density being essential ingredients in the relocation policy, greatest efforts should be made to ensure that a fleet is not left idle in a particular location for many hours on end while, on the other hand, a vehicle is required urgently in another location [49].
  • Relocation Strategies
Location solutions are important in managing AV fleets, especially in car-sharing/ride-hailing businesses. AVs require a shuttle between trips to serve customers and can be problematic if they are not well managed in terms of spatial needs and traffic jam. The movement of vehicles without passengers, a practice referred to as ‘dead running’, can reinforce traffic in already traffic-troubled zones should several fleets be running all at once [50].
To overcome this, AVs have to process detailed algorithms that will help them anticipate the influx of passengers and move vehicles to these parts quickly. Such algorithms should include inputs like real-time traffic flow, weather conditions, or demand so as to ensure that more of these relocations do not just add to traffic congestion for example, which would be a waste of resources. Moreover, relations of cooperation between fleets of AVs and urban transport systems imply the potential fine-tuning of mobility shifts in relation to traffic conditions [51].

4.3. Localization and Mapping

Localization is the ability of the AV to effectively use its sensors in precisely detecting and perceiving the environmental features based on the developed environmental map [52]. It involves identifying, classifying, and integrating physical obstacles and features into an actual navigational map using sensor data and deep learning among other AI-based systems [53]. The navigational capabilities of AVs are extensively dependent on localization. Ultimately, Li et al. [54] agree that localization is a major indicator of an autonomous system’s reliability as it is one of the primary sources of autonomous driving challenges.
By applying the sensor data, deep learning algorithms, and other AI-based systems, the AV should be able to not only estimate its location but also detect and assess the proximity of physical obstacles and other vehicles [55]. The deep learning algorithm relies on a diverse scope of sensor data to enable localization in AVs. For example, the point clouds generated by LiDAR are analyzed to develop a map of the environment [56]. Additionally, features like particle filters enable deep learning models to enhance the accuracy of the data collected by sensors by comparing the observed environmental description with a known map that is used as part of the algorithm training dataset [57]. Additionally, features that cannot be precisely identified through this comparison are used by the deep learning algorithm to update new features on the existing map held in the algorithm [58,59].
The sensor data and deep learning algorithms are used to develop absolute and relative maps. Firstly, absolute maps describe a geographical location based on its fixed point on a worldwide coordinate frame [42]. It shows stationary landmarks defined by two parameters that show their location on a Cartesian plane relative to the worldwide coordinate frame [42]. On the other hand, relative maps are used by AVs to derive awareness about the distance between two landmarks [60]. Golroudbari and Sabour [19] also show that deep learning algorithms like convolutional neural networks are effective in object detection through their abilities to acquire a comprehensive representation of the object under detection.
Notably, deep learning-based object detection and localization mechanisms like Faster R-CNN and YOLO have demonstrated high accuracy, robustness, and speed in the real-time detection of obstacles, regardless of factors like adverse weather conditions or darkness [59]. This accuracy in object detection during autonomous navigation has also been ascertained by several studies. For example, Afif et al. [61] assessed the effectiveness of lightweight EfficientDet in autonomous navigation. The study established that this deep learning object detection approach, among others like TensorFlow and OpenCV, optimized obstacle detection by providing a high-resolution binary image of the obstacle [61]. Therefore, despite the challenges affiliated with the acquisition of adequate training data, it is evident that deep learning-based systems are highly effective in AV localization and mapping.

5. Challenges

5.1. Complexity and Uncertainty

The issues are similar to the previous problems in the application of cerebral learning in autonomous vehicles, which demonstrate continual difficulties. Grigorescu et al. [62] explain that ambiguities can be framed in two main ways: Firstly, there are issues that are explored when the sensors are unable to perform their functions properly, as they are affected by environmental factors. The outdoor environment is another factor that may affect the sensor as climate change will inevitably affect the quality of the collected data. Second, even learning algorithms themselves might be problematic when they are used for application in the real world. Lack of clarity regarding the relationships between these models and their particular roles—interactions between object detection and decision-making modules—may result in suboptimal work and inconvenience [63].
Moreover, because the environment is unpredictable, sensors cannot consistently provide high-quality data required for accurate models [39]. This is worsened by the fact that deep learning models depend on large amounts of quality data to make accurate decisions.
However, there are other aspects that cause deep learning models to produce poor driving outcomes for autonomous cars and trucks apart from environmental conditions beyond environmental aspects; other aspects hinder deep learning models from delivering good outcomes for self-driving cars and trucks. For instance, lane detection is a challenge at night due to the nature of data used in deep learning models, which is mainly obtained during the daytime. While a human driver is able to anticipate and successfully drive a car in low visibility, today’s deep learning models are unable to do the same. The aim should be to move on to the next generation of deep learning models capable of operating with a high degree of accuracy when operating in complex and potentially volatile conditions.
Regarding the drawbacks of deep learning models, it is also worth mentioning that users, in terms of the autonomy of automobiles, are risky and shaky as well. Biswas and Wang [9] wrote that any changes to the environment could impact deep learning systems and hence the behavior of progressive supports and drivers incorporated into cars. Furthermore, ISO 26262 [64], the current industrial standards for automotive functional safety, give no consideration to the incorporation of deep learning into automated systems [19]. This necessitates new, better frameworks and standards relevant to various challenges that AI presents in self-driving vehicles.

5.2. Sensor Challenges

The detection accuracy and latency of deep learning algorithms significantly depend on the quality of data obtained from the multiple sensors embedded in AVs. Notably, one of the approaches used to foster accuracy in AVs is sensor fusion. It involves the integration of data from different sensors to increase the quantity and quality of data available for deep learning algorithms to make better and more accurate decisions [64,65]. For example, the integration of LiDar and camera data optimizes AV performance at night [65]. However, adverse weather conditions are likely to affect the performance of data collection sensors, which can be slightly improved through sensor fusion.
Biswas and Wang [9] also acknowledge that a challenge for AV manufacturers emerges from the tradeoff between the cost of sensors and their accuracy. The outcome is different manufacturers opting for different sensors. Such sensor inconsistencies, among others, lead to heterogeneous datasets that might have undesirable effects on accuracy. Besides the varying reliability and quality of sensors, Yeong et al. [66] note that the different frequencies and timestamps of sensors also affect the synchronization accuracy and subsequent safety of AVs.
Another issue is the lack of universal standards and comprehensive research regarding the aspect of sensor failure. Sensor failure is an important factor as the safety and reliability of AVs significantly rely on the presence and optimal functionality of fundamental sensors [67]. Therefore, undetected sensor failure might influence severe technical failures like accidents. Besides technical failure, sensor failure due to external factors like dirt, deviation, and blockage might also lead to the communication of false data within the AV’s architecture [68].

5.3. The Complexity of Model Training

For deep learning algorithms to offer their best in AV deployment, there is the need to feed the models with data that acknowledges the variety of conditions. However, the environment for the deployment of AVs is dynamic and complex and may at times include scenarios that were not covered in the training dataset [9]. This complexity can reduce the efficiency of the most essential AV fronts, including detection, perception, SLAM, and decision-making [9].
Furthermore, the construction of a proper training set is a laborious task because it requires proper coordinates for pedestrians, vehicles, lanes, and other obstacles. The unpredictability of a temporal driving environment and the variability of possible scenarios enforce the difficulties of training deep learning models by temporal data [9]. However, it is noteworthy that various training strategies have been developed to surmount these training difficulties, such as collaborative training, lightweight deep learning algorithms, and model compression techniques.
The second major problem is linked with the impossibility of training in realistic conditions. Training deep learning algorithms in AVs typically employs three major approaches, using the help of car simulations, experiments with miniature car models, or real-life experiments. Among the three methods, the first two have been used frequently, whereas the use of real-world experiments has been limited due to technical and infrastructural challenges. The lack of kinetic and stochastic actual field exercise undermines the real-world accuracy and variability of the deep learning training sample.
In a bid to buttress the relevance of real-life training, Ni et al. [69], have estimated in their study that about 109 h of vehicle operations are needed to obtain a correct estimate of the failure rate. Moreover, improved statistical significance would require running such tests serially to provide the needed results. However, many practitioners, such as Tesla, have performed multiple tests of real-world training and have noted the current shortcomings of AV architectures and the importance of better approaches [70].

6. Conclusions

Ultimately, it is prudent that deep-learning-based systems have enhanced the safety and reliability of navigation in AVs. The analysis showed that deep learning algorithms have been applied in major AV components like perception, localization, mapping, path planning, and navigation. Future advancements in deep learning algorithms are expected to enhance the accuracy of AVs in decision making, perception, localization, and mapping. Therefore, to optimize the use of deep learning in AVs, the current study recommends an increased standardization of the sensors to enhance synchronization and accuracy. Real-life testing should also be actively incorporated in model training to ensure that deep learning algorithms adapt to the dynamic nature of real driving. These recommendations along with future research will enhance the safety, reliability, and social acceptability of autonomous vehicle systems.

Funding

This research received no external funding.

Conflicts of Interest

The author declares no conflicts of interest.

References

  1. Karnati, A.; Mehta, D. Artificial Intelligence in Self-Driving Cars: Applications, Implications and Challenges. Ushus J. Bus. Manag. 2022, 21, 1–28. [Google Scholar] [CrossRef]
  2. Miglani, A.; Kumar, N. Deep learning models for traffic flow prediction in autonomous vehicles: A review, solutions, and challenges. Veh. Commun. 2019, 20, 100184. [Google Scholar] [CrossRef]
  3. Kisačanin, B. Deep learning for autonomous vehicles. In Proceedings of the 2017 IEEE 47th International Symposium on Multiple-Valued Logic (ISMVL), Novi Sad, Serbia, 22–24 May 2017; p. 142. [Google Scholar]
  4. Tikito, I.; Souissi, N. Meta-analysis of systematic literature review methods. Int. J. Mod. Educ. Comput. Sci. 2019, 12, 17–25. [Google Scholar] [CrossRef]
  5. Davies, A. Carrying out systematic literature reviews: An introduction. Br. J. Nurs. 2019, 28, 1008–1014. [Google Scholar] [CrossRef] [PubMed]
  6. GreyB. Top 30 Self-Driving Technology and Car Companies. 23 July 2022. Available online: https://www.greyb.com/autonomous-vehicle-companies/ (accessed on 23 October 2022).
  7. León, L.F.A.; Aoyama, Y. Industry emergence and market capture: The rise of autonomous vehicles. Technol. Forecast. Soc. Chang. 2022, 180, 121661. [Google Scholar] [CrossRef]
  8. Pütz, F.; Murphy, F.; Mullins, M.; O′Malley, L. Connected automated vehicles and insurance: Analysing future market-structure from a business ecosystem perspective. Technol. Soc. 2019, 59, 101182. [Google Scholar] [CrossRef]
  9. Biswas, A.; Wang, H.C. Autonomous vehicles enabled by the integration of IoT, edge intelligence, 5G, and blockchain. Sensors 2023, 23, 1963. Available online: https://www.researchgate.net/publication/368436881_Autonomous_Vehicles_Enabled_by_the_Integration_of_IoT_Edge_Intelligence_5G_and_Blockchain (accessed on 1 December 2023). [CrossRef]
  10. Fayyad, J.; Jaradat, M.A.; Gruyer, D.; Najjaran, H. Deep learning sensor fusion for autonomous vehicle perception and localization: A review. Sensors 2020, 20, 4220. [Google Scholar] [CrossRef]
  11. Qian, J.; Zeleznikow, J. Who shares legal liability for road accidents caused by drivers assisted by artificial intelligence software? Canberra Law Rev. 2021, 18, 18–35. [Google Scholar]
  12. Coalition for Future Mobility. Benefits of Self-Driving Vehicles. 19 March 2018. Available online: https://coalitionforfuturemobility.com/benefits-of-self-driving-vehicles/ (accessed on 23 October 2022).
  13. Ercan, T.; Onat, N.C.; Keya, N.; Tatari, O.; Eluru, N.; Kucukvar, M. Autonomous electric vehicles can reduce carbon emissions and air pollution in cities. Transp. Res. Part D Transp. Environ. 2022, 112, 103472. [Google Scholar] [CrossRef]
  14. Zipper, D. Electric Vehicles Are Bringing Out the Worst in Us. The Atlantic, 4 January 2023. [Google Scholar]
  15. Fagnant, D.J.; Kockelman, K. Preparing a nation for autonomous vehicles: Opportunities, barriers and policy recommendations. Transp. Res. Part A Policy Pract. 2015, 77, 167–181. [Google Scholar] [CrossRef]
  16. Anderson, J.M.; Nidhi, K.; Stanley, K.D.; Sorensen, P.; Samaras, C.; Oluwatola, O.A. Autonomous Vehicle Technology: A Guide for Policymakers; Rand Corporation: Santa Monica, CA, USA, 2014. [Google Scholar]
  17. Heaton, J. Ian Goodfellow, Yoshua Bengio, and Aaron Courville: Deep learning. Genet. Program. Evolvable Mach. 2018, 19, 305–307. [Google Scholar] [CrossRef]
  18. Jebamikyous, H.-H.; Kashef, R. Autonomous vehicles perception (AVP) using deep learning: Modeling, assessment, and challenges. IEEE Access 2022, 10, 10523–10535. [Google Scholar] [CrossRef]
  19. Golroudbari, A.A.; Sabour, M.H. Recent Advancements in Deep Learning Applications and Methods for Autonomous Navigation—A Comprehensive Review. arXiv 2023, arXiv:2302.11089. [Google Scholar]
  20. Rao, Q.; Frtunikj, J. Deep learning for self-driving cars: Chances and challenges. In Proceedings of the 1st International Workshop on Software Engineering for AI in Autonomous Systems, Gothenburg, Sweden, 28 May 2018; pp. 35–38. [Google Scholar]
  21. Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. Commun. ACM 2017, 60, 84–90. [Google Scholar] [CrossRef]
  22. Ren, J.; Gaber, H.; Al Jabar, S.S. Applying deep learning to autonomous vehicles: A survey. In Proceedings of the 2021 4th International Conference on Artificial Intelligence and Big Data (ICAIBD), Chengdu, China, 28–31 May 2021; pp. 247–252. [Google Scholar]
  23. Smaldone, A.M.; Kyro, G.W.; Batista, V.S. Quantum Convolutional Neural Networks for Multi-Channel Supervised Learning. arXiv 2023, arXiv:2305.18961. [Google Scholar] [CrossRef]
  24. Pak, A.; Manjunatha, H.; Filev, D.; Tsiotras, P. Carnet: A dynamic autoencoder for learning latent dynamics in autonomous driving tasks. arXiv 2022, arXiv:2205.08712. [Google Scholar]
  25. Chen, L.; Lin, S.; Lu, X.; Cao, D.; Wu, H.; Guo, C.; Liu, C.; Wang, F.-Y. Deep neural network based vehicle and pedestrian detection for autonomous driving: A survey. IEEE Trans. Intell. Transp. Syst. 2021, 22, 3234–3246. [Google Scholar] [CrossRef]
  26. Huang, Y.; Panahi, A.; Krim, H.; Yu, Y.; Smith, S.L. Deep adversarial belief networks. arXiv 2019, arXiv:1909.06134. [Google Scholar]
  27. Ren, J.; Green, M.; Huang, X. From traditional to deep learning: Fault diagnosis for autonomous vehicles. In Learning Control; Elsevier: Amsterdam, The Netherlands, 2021; pp. 205–219. [Google Scholar]
  28. Ivanov, S.A.; Rasheed, B. Predicting the Behavior of Road Users in Rural Areas for Self-Driving Cars. Adv. Eng. Res. 2023, 23, 169–179. [Google Scholar] [CrossRef]
  29. Kenesei, Z.; Ásványi, K.; Kökény, L.; Jászberényi, M.; Miskolczi, M.; Gyulavári, T.; Syahrivar, J. Trust and perceived risk: How different manifestations affect the adoption of autonomous vehicles. Transp. Res. Part A Policy Pract. 2022, 164, 379–393. [Google Scholar] [CrossRef]
  30. Tong, Q.; Li, X.; Lin, K.; Li, C.; Si, W.; Yuan, Z. Cascade-LSTM-based visual-inertial navigation for magnetic levitation haptic interaction. IEEE Netw. 2019, 33, 74–80. [Google Scholar] [CrossRef]
  31. Guan, W.; Wang, T.; Qi, J.; Zhang, L.; Lu, H. Edge-aware convolution neural network based salient object detection. IEEE Signal Process. Lett. 2018, 26, 114–118. [Google Scholar] [CrossRef]
  32. Lee, D.-H.; Chen, K.-L.; Liou, K.-H.; Liu, C.-L.; Liu, J.-L. Deep learning and control algorithms of direct perception for autonomous driving. Appl. Intell. 2020, 51, 237–247. [Google Scholar] [CrossRef]
  33. Bojarski, M.; Yeres, P.; Choromanska, A.; Choromanski, K.; Firner, B.; Jackel, L.; Muller, U. Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv 2017, arXiv:1704.07911. [Google Scholar]
  34. Pavel, M.I.; Tan, S.Y.; Abdullah, A. Vision-based autonomous vehicle systems based on deep learning: A systematic literature review. Appl. Sci. 2022, 12, 6831. [Google Scholar] [CrossRef]
  35. Aqel, M.O.A.; Marhaban, M.H.; Saripan, M.I.; Ismail, N.B. Review of visual odometry: Types, approaches, challenges, and applications. SpringerPlus 2016, 5, 1897. [Google Scholar] [CrossRef]
  36. Péter, G.; Kiss, B.; Tihanyi, V. Vision and odometry based autonomous vehicle lane changing. ICT Express 2019, 5, 219–226. [Google Scholar] [CrossRef]
  37. Li, R.; Wang, S.; Long, Z.; Gu, D. Undeepvo: Monocular visual odometry through unsupervised deep learning. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia, 21–25 May 2018; pp. 7286–7291. [Google Scholar]
  38. Mohamed, S.A.S.; Haghbayan, M.-H.; Westerlund, T.; Heikkonen, J.; Tenhunen, H.; Plosila, J. A survey on odometry for autonomous navigation systems. IEEE Access 2019, 7, 97466–97486. [Google Scholar] [CrossRef]
  39. Wang, S.; Clark, R.; Wen, H.; Trigoni, N. End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks. Int. J. Robot. Res. 2018, 37, 513–542. [Google Scholar] [CrossRef]
  40. Xue, F.; Wang, X.; Wang, J.; Zha, H. Deep visual odometry with adaptive memory. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 44, 940–954. [Google Scholar] [CrossRef] [PubMed]
  41. Tang, X.; Yang, K.; Wang, H.; Wu, J.; Qin, Y.; Yu, W.; Cao, D. Prediction-uncertainty-aware decision-making for autonomous vehicles. IEEE Trans. Intell. Veh. 2022, 7, 849–862. [Google Scholar] [CrossRef]
  42. Gomes, T.; Matias, D.; Campos, A.; Cunha, L.; Roriz, R. A Survey on ground segmentation methods for automotive LiDAR sensors. Sensors 2023, 23, 601. [Google Scholar] [CrossRef] [PubMed]
  43. Hoel, C.-J.; Driggs-Campbell, K.; Wolff, K.; Laine, L.; Kochenderfer, M.J. Combining planning and deep reinforcement learning in tactical decision making for autonomous driving. IEEE Trans. Intell. Veh. 2019, 5, 294–305. [Google Scholar] [CrossRef]
  44. Ayawli, B.B.K.; Appiah, A.Y.; Nti, I.K.; Kyeremeh, F.; Ayawli, E.I. Path planning for mobile robots using Morphological Dilation Voronoi Diagram Roadmap algorithm. Sci. Afr. 2021, 12, e00745. [Google Scholar] [CrossRef]
  45. Orthey, A.; Chamzas, C.; Kavraki, L.E. Sampling-Based Motion Planning: A Comparative Review. Annu. Rev. Control Robot. Auton. Syst. 2023, 7, 285–310. [Google Scholar] [CrossRef]
  46. Meng, T.; Yang, T.; Huang, J.; Jin, W.; Zhang, W.; Jia, Y.; Wan, K.; Xiao, G.; Yang, D.; Zhong, Z. Improved Hybrid A-Star Algorithm for Path Planning in Autonomous Parking System Based on Multi-Stage Dynamic Optimization. Int. J. Automot. Technol. 2023, 24, 459–468. [Google Scholar] [CrossRef]
  47. Gupta, A.; Anpalagan, A.; Guan, L.; Khwaja, A.S. Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues. Array 2021, 10, 100057. [Google Scholar] [CrossRef]
  48. Wang, Z.; Sun, K.; Ma, S.; Sun, L.; Gao, W.; Dong, Z. Improved Linear Quadratic Regulator Lateral Path Tracking Approach Based on a Real-Time Updated Algorithm with Fuzzy Control and Cosine Similarity for Autonomous Vehicles. Electronics 2022, 11, 3703. [Google Scholar] [CrossRef]
  49. Wang, X.; Gilliam, C.; Kealy, A.; Close, J.; Moran, B. Probabilistic map matching for robust inertial navigation aiding. NAVIGATION J. Inst. Navig. 2023, 70, 4–18. [Google Scholar] [CrossRef]
  50. Li, Q.; Queralta, J.P.; Gia, T.N.; Zou, Z.; Westerlund, T. Multi-sensor fusion for navigation and mapping in autonomous vehicles: Accurate localization in urban environments. Unmanned Syst. 2020, 8, 229–237. [Google Scholar] [CrossRef]
  51. Berntorp, K.; Hoang, T.; Di Cairano, S. Motion planning of autonomous road vehicles by particle filtering. IEEE Trans. Intell. Veh. 2019, 4, 197–210. [Google Scholar] [CrossRef]
  52. Wong, K.; Gu, Y.; Kamijo, S. Mapping for autonomous driving: Opportunities and challenges. IEEE Intell. Transp. Syst. Mag. 2020, 13, 91–106. [Google Scholar] [CrossRef]
  53. Milanes, V.; Shladover, S.E.; Spring, J.; Nowakowski, C.; Kawazoe, H.; Nakamura, M. Cooperative Adaptive Cruise Control in Real Traffic Situations. IEEE Trans. Intell. Transp. Syst. 2014, 15, 296–305. [Google Scholar] [CrossRef]
  54. Li, S.E.; Zheng, Y.; Li, K.; Wang, L.Y.; Zhang, H. Platoon Control of Connected Vehicles from a Networked Control Perspective: Literature Review, Component Modeling, and Control Strategies. IEEE Trans. Veh. Technol. 2017, 66, 10659–10678. [Google Scholar] [CrossRef]
  55. Shladover, S.E.; Su, D.; Lu, X.-Y. Impacts of Cooperative Adaptive Cruise Control on Freeway Traffic Flow. Transp. Res. Rec. J. Transp. Res. Board 2012, 2324, 63–70. [Google Scholar] [CrossRef]
  56. Talebpour, A.; Mahmassani, H.S. Influence of Connected and Autonomous Vehicles on Traffic Flow Stability and Throughput. Transp. Res. Part C Emerg. Technol. 2016, 71, 143–163. [Google Scholar] [CrossRef]
  57. Joubert, N.; Reid, T.G.; Noble, F. Developments in modern GNSS and its impact on autonomous vehicle architectures. In Proceedings of the 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA, 19 October–13 November 2020; pp. 2029–2036. [Google Scholar]
  58. Kuutti, S.; Bowden, R.; Jin, Y.; Barber, P.; Fallah, S. A survey of deep learning applications to autonomous vehicle control. IEEE Trans. Intell. Transp. Syst. 2020, 22, 712–733. [Google Scholar] [CrossRef]
  59. Muhammad, K.; Ullah, A.; Lloret, J.; Del Ser, J.; de Albuquerque, V.H.C. Deep Learning for Safe Autonomous Driving: Current Challenges and Future Directions. IEEE Trans. Intell. Transp. Syst. 2020, 22, 4316–4336. [Google Scholar] [CrossRef]
  60. Kristo, M.; Ivasic-Kos, M.; Pobar, M. Thermal object detection in difficult weather conditions using YOLO. IEEE Access 2020, 8, 125459–125476. [Google Scholar] [CrossRef]
  61. Afif, M.; Ayachi, R.; Said, Y.; Atri, M. An evaluation of EfficientDet for object detection used for indoor robots assistance navigation. J. Real-Time Image Process. 2022, 19, 651–661. [Google Scholar] [CrossRef]
  62. Grigorescu, S.; Trasnea, B.; Cocias, T.; Macesanu, G. A survey of deep learning techniques for autonomous driving. J. Field Robot. 2020, 37, 362–386. [Google Scholar] [CrossRef]
  63. Li, Y.; Chen, R.; Niu, X.; Zhuang, Y.; Gao, Z.; Hu, X.; El-Sheimy, N. Inertial sensing meets machine learning: Opportunity or challenge? IEEE Trans. Intell. Transp. Syst. 2021, 23, 9995–10011. [Google Scholar] [CrossRef]
  64. Hommes, Q.V.E. Review and Assessment of the ISO 26262 Draft Road Vehicle-Functional Safety; SAE Technical Paper; SAE: Warrendale, PA, USA, 2012. [Google Scholar]
  65. Jo, J.; Tsunoda, Y.; Stantic, B.; Liew, A.W.C. A likelihood-based data fusion model for the integration of multiple sensor data: A case study with vision and lidar sensors. In Robot Intelligence Technology and Applications 4: Results from the 4th International Conference on Robot Intelligence Technology and Applications; Springer International Publishing: Cham, Switzerland, 2017; pp. 489–500. [Google Scholar]
  66. Yeong, D.J.; Velasco-Hernandez, G.; Barry, J.; Walsh, J. Sensor and sensor fusion technology in autonomous vehicles: A review. Sensors 2021, 21, 2140. [Google Scholar] [CrossRef] [PubMed]
  67. Sabaliauskaite, G.; Liew, L.S.; Cui, J. Integrating autonomous vehicle safety and security analysis using STPA method and the six-step model. Int. J. Adv. Secur. 2018, 11, 160–169. [Google Scholar]
  68. Abdulkhaleq, A.; Lammering, D.; Wagner, S.; Röder, J.; Balbierer, N.; Ramsauer, L.; Raste, T.; Boehmert, H. A systematic approach based on STPA for developing a dependable architecture for fully automated driving vehicles. Procedia Eng. 2017, 179, 41–51. [Google Scholar] [CrossRef]
  69. Ni, J.; Chen, Y.; Chen, Y.; Zhu, J.; Ali, D.; Cao, W. A survey on theories and applications for self-driving cars based on deep learning methods. Appl. Sci. 2020, 10, 2749. [Google Scholar] [CrossRef]
  70. Bachute, M.R.; Subhedar, J.M. Autonomous driving architectures: Insights of machine learning and deep learning algorithms. Mach. Learn. Appl. 2021, 6, 100164. [Google Scholar] [CrossRef]
Table 1. The automation levels are used to categorize autonomous vehicles.
Table 1. The automation levels are used to categorize autonomous vehicles.
LevelsDescription
Level 0 (no automation)The dynamic driving task (DDT) is fully controlled by human beings [11].
Level 1 (driver assistance)It is the lowest level of automation that incorporates mild driver assistance systems like adaptive cruise control.
Level 2 (partial driving automation) It incorporates an advanced driver assistance system that controls aspects like speed and steering. Human intervention is still required.
Level 3 (conditional driving automation)Advanced autonomy with numerous sensors to analyze the environment and make informed decisions. They incorporate autonomous systems like automated emergency breaking (AED), traffic jam assist, and driver monitoring (DM) among other functionalities [11].
Level 4 (high driving automation)They can operate in self-driving mode, but due to geo-fencing, they are limited to certain low-speed urban areas. Incomprehensive legislation and inadequate infrastructure required for such AVs also limits self-driving [11].
Level 5 (fully autonomous driving)The dynamic driving task is eliminated, and hence, such AVs do not require human intervention. They will not be limited by geo-fencing. Despite the ongoing extensive research on actualizing level 5 AVs, the universal adoption of such AVs is a long-term objective [12].
Table 2. Deep learning structures.
Table 2. Deep learning structures.
Deep Learning TypeDescription
Autoencoder Composed of an encoder and a decoder. It is also designed to learn a compressed version of input data from which the original input data can be recreated [19].
Autoencoders are incorporated with end-to-end deep learning strategies to help AVs determine the appropriate steering angle during autonomous navigation [24].
Convolutional neural networks (CNNs)The CNN uses convolution operations to extract and learn relevant features from data. It helps in the identification of data patterns that could have been challenging to detect using traditional algorithms. It has a hierarchical structure, whereby the lower layers learn simple data features whereas the high layers extract complex data features [25].
Deep belief networks (DBNs)Comprises multiple layers of the restricted Boltzmann machine (RBM). The shallow and two-layered RBMs are stacked on top of each other to form a deep DBN network [26].
Besides being trained through unsupervised learning, DBNs can be applied in AV functions like natural language processing, speech recognition, and computer vision relevant in the detection and classification of images during autonomous navigation [27].
Recurrent neural networks (RNNs)RNNs could analyze sequential data as input. This ability to model temporal dependencies and patterns has enabled RNNs to be used for different AV functions like natural language processing, speech recognition, and time series predictions [19]. However, RNNs are also sensitive to the order of input data.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Aljehane, N.O. A Study to Investigate the Role and Challenges Associated with the Use of Deep Learning in Autonomous Vehicles. World Electr. Veh. J. 2024, 15, 518. https://doi.org/10.3390/wevj15110518

AMA Style

Aljehane NO. A Study to Investigate the Role and Challenges Associated with the Use of Deep Learning in Autonomous Vehicles. World Electric Vehicle Journal. 2024; 15(11):518. https://doi.org/10.3390/wevj15110518

Chicago/Turabian Style

Aljehane, Nojood O. 2024. "A Study to Investigate the Role and Challenges Associated with the Use of Deep Learning in Autonomous Vehicles" World Electric Vehicle Journal 15, no. 11: 518. https://doi.org/10.3390/wevj15110518

APA Style

Aljehane, N. O. (2024). A Study to Investigate the Role and Challenges Associated with the Use of Deep Learning in Autonomous Vehicles. World Electric Vehicle Journal, 15(11), 518. https://doi.org/10.3390/wevj15110518

Article Metrics

Back to TopTop