1. Introduction
Due to the rapid development of Advanced Driver Assistance System (ADAS) technology, users can drive safely even under challenging conditions. In the near future, it is expected that a majority of the cars in the world will be Autonomous Vehicles (AVs). It is expected that about 90 percent of vehicles that would be commercially available in 2030 would be fully autonomous and almost 100 percent of vehicles would be conditionally autonomous [1]. According to the Department of Transportation (DOT) and the National Highway Traffic Safety Administration (NHTSA), almost 94% of accidents on US roads occur due to human error [1]. Also, about 99% of accidents occurred when using self-driving cars are caused by pedestrians and other vehicles driven by humans [2]. In other words, since most accidents on the road are caused by humans, road safety will be more secured if the number of self-driving cars (and drones) on the road increases. Therefore, self-driving vehicles would drastically reduce the number of accidents and crashes that occur on the roads today.
However, most people are not ready to embrace self-driving technology yet. As a result, the actual use of AVs is still significantly low, as people have low confidence in self-driving cars and feel less comfortable. According to prior studies, pacceptance of an AV is mainly determined by trust, which is yet to convince people to explore and fully utilize new autonomous technologies. In order to increase people’s reliability in self-driving cars, it is necessary to introduce more advanced ADAS technology designed according to each person’s driving style.
(Figure 1) Structure map of CARLA
Therefore, in this paper, a new attempt at a personalized ADAS is proposed, by designing a personalized AV algorithm based on each driver’s driving style. Each driver’s behavior is divided into two categories: assertive and defensive. In general, aggressive drivers tend to speed up and make rapid direction changes, while defensive drivers actively try to avoid potentially dangerous driving situations. In this case, aggressive drivers are defined as assertive drivers, who prefer aggressive driving habits but try to remain in safe conditions. Based on this classification, the ADAS sensors can be made more sensitive for the case of assertive drivers, enabling more advanced driving safety support.
A novel simulation-based personalized ADAS algorithm with high classification accuracy is designed in this paper, which divides each driver based on their driving style. Each driver’s driving data is first collected and simulated using CARLA. In addition, Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) machine learning algorithms are used to optimize the ADAS parameters. In addition, among the vast amount of feature data extracted from the drivers, distinguishable driving features are collected selectively using Support Vector Machine (SVM) by comparing the amount of influence on the classification of the two categories.
As a result, the main contributions of this research can be summarized as follows:
1. Currently, self-driving technology is developing rapidly, but people’s self-driving car usage rate has not reached it. In addition, so far little research has been conducted to address this problem. In this paper, for the first time, the impact of assertive drivers and defensive drivers on each ADAS parameter is analyzed, and the derived results are used to classify them, thereby laying the foundation for autonomous vehicle technology that drivers can utilize with sufficient confidence with the development of personalized ADAS. In addition, the reliability of this algorithm was increased by using stable and reliable data, which is collected by using CARLA.
2. The proposed algorithm used LSTM and GRU for time-series driving data to classify drivers according to their driving tendencies with high accuracy. In addition, the classification accuracy was further improved by additionally utilizing SVM to extract and analyze features that can better reflect drivers' driving tendencies in advance. By additionally going through this process, a classification model using refined driving features is created so that it can show more stable and higher performance.
This paper consists of the following sections. The overall explanation of ADAS technology and simulator with the related work is described in section 2. Section 3 describes system configuration and feature extraction techniques using SVM, and section 4 explains driver classification techniques using LSTM and GRU. Finally, the paper is concluded in section 5.
2. Background & Related Work
2.1 ADAS
ADAS is a built-in system within the vehicle that assists the driver to enhance driving performance and help prevent accidents. The ADAS can aid the drivers or can support the autonomous driving system while keeping the driver in the flow. The functions of ADAS consist of the following: traffic signals recognition, adaptive cruise control, and forward-collision warning. Among these functionalities, this paper introduced a user-based self-driving mode recommendation system by classifying each driver’s tendencies based on cruise control.
According to SAE International (formerly known as the Society of Automotive Engineers), driving automation levels are categorized into 6 levels. Level 0 indicates zero autonomy, and full autonomy where the vehicle can conduct all the driving is represented as level 5. Currently, the state-of-the-art level of autonomous driving is level 3, which requires advanced functions that can assist drivers using ADAS technology.
As the stage of autonomous driving technology increases, the importance of the ptechnology increases, as it can enhance the safety of AVs. In addition, it is shown that advanced safety systems increase safety by reducing the overall number of traffic accidents. Thus, the increase in the usage of AVs becomes one of the most important factors to protect public road safety, and this paper proposes a novel method to significantly improve overall performance.
2.2 CARLA
CARLA is a widely used open-source autonomous driving simulator [3], which serves as a modular and flexible application programming interface (API) to address problems of autonomous driving. It is used as a comprehensive tool for simulating real-world scenarios. One of the biggest advantages of CARLA is that users can conveniently set up their own environment. Since CARLA runs simulations based on an Unreal Engine, it is one of the most realistic simulators to imitate real road and traffic conditions, including various weather conditions. The operation of CARLA consists of a sequence of several steps: localization, collision detection, path planning, and control stages. The overall CARLA structure diagram is shown in Figure 1. In this paper, CARLA is used to generate drivers’ datasets, which include diverse sensor data (e.g., GPS data, inertial measurement unit (IMU) sensor data, accelerator data, etc.) in complex urban driving situations.
(Figure 1) Flowchart of the proposed model.
2.3 Related Work
In recent years, studies on ADAS including personalized ADAS have been actively conducted. The correlation between the degree of personalization of self-driving cars and the reliability of people was analyzed in [4]. In [5], the relationship between the values of various sensors, including acceleration and jerk, was analyzed, on the comfort of people driving. The authors of [4]-[6] analyzed each driver’s driving style by directly collecting and conducting the experiment with the driving data of actual human drivers. However, collecting data by driving themselves would result in inconsistent data, which varies slightly from time to time. In contrast, simulator-based driving data can be easily customized in diverse driving situations with a relatively small amount of outliers.
Therefore, in this paper, a newly designed model is proposed, which analyzes and classifies the driving styles of each driver and recommends driving modes according to their propensity. Driving data for this personalized ADAS system was collected by using the CARLA simulator. In addition, various machine learning-based algorithms were applied to improve the accuracy and performance of the classification model.
3. System Model
In this section, a classification model is introduced to distinguish drivers based on their driving skills and behaviors.
3.1 Driving Style-based Classification
According to previous human driving studies, drivers are divided into two groups considering their driving styles and habits, namely ‘assertive’ and ‘defensive’. Generally, assertive drivers tend to drive at the speed limit and pursue rapid accelerations. On the other hand, defensive drivers tend to enjoy less risky driving, in a cautious manner. Since this classifying method best reflects the driver’s driving characteristics and preferences, drivers are classified into two groups based on this classification criterion.
3.2 SVM
SVM is a supervised machine learning algorithm that decides the boundary between the classes of training data, which is typically used in classification and regression problems. [7] The hyperplane or line between two datasets is determined to guarantee a maximum margin. This gap is described as the distance between the hyperplane and the observations which are nearest in either class. In this paper, SVM is used to distinguish each driver as an assertive driver or defensive driver using the given driving features.
In Table 1, the relationship between the data classified by the machine learning model and the actual answer is presented, where the definitions of the relationship are defined as follows.
(Table 1) The relationship between the classified data and the actual answer.
True Positive (TP): TP represents the case where the correct answer that is actually true is predicted as true.
True Negative (TN): TN represents the case where the correct answer that is actually false is predicted as false.
False Positive (FP): FP represents the case where the correct answer that is actually false is predicted as true.
False Negative (FN): FN represents the case where the correct answer that is actually true is predicted as false.
3.3 LSTM and GRU
Recurrent Neural Networks (RNN) are excellent in analyzing sequential data, as they can analyze semantic correlation and predict the features of time series data, including natural language data. LSTM and GRU are special kinds of RNNs, which can also learn long-term dependent characteristics very well. The key difference between LSTM and GRU is that GRU is much simpler and more efficient than LSTM. LSTM has three gates: input, output, and forget gates, but GRU has only two gates, which include input gates and output gates. Therefore, due to its complex structure, LSTM is advantageous when processing data of long sequences with large dependencies [8]. On the other hand, since GRU has a much simpler structure than LSTM, it works well with less training data and can provide higher efficiency in terms of computation costs.
3.4 Flowchart of Driver Classification Model
The model proposed in this paper has the structure described in Figure 2. First, the driving data is collected using the CARLA simulator. A vast amount of driving data was collected from assertive drivers and defensive drivers. It has features including various sensor values such as IMU sensors, accelerators, and speeds of the CARLA simulator. After that, each sensor value is inserted as an input of the LSTM and GRU models and goes through the process of classifying them into assertive drivers and defensive drivers. Furthermore, the classification performance and accuracy were improved by additionally utilizing the SVM classifier to extract the top distinguishable features that particularly well reflect the driving tendency among the ADAS parameters. In this paper, experiments were conducted on a total of four models. First of all, an experiment was conducted to classify driving data using LSTM and GRU, respectively, and experiments with ‘SVM+LSTM’ and ‘SVM+GRU’ models were also conducted using a trained SVM classifier. At this time, in order to further increase the feature selection accuracy in the process of training the SVM, the ROC curve is used to verify the accuracy of the SVM model when selecting the top significant features that well reflect the driving preferences of each driver.
4. Performance Analysis
In the previous section, SVM classification was applied to figure out which driving features which mainly influence each driver’s driving style. Using LSTM and GRU, the proposed algorithm classifies each driver considering their driving preferences.
4.1 Environment Settings
According to prior studies, there are several classification standards that can be used to describe how to classify drivers into two categories: assertive drivers and defensive drivers [9]. Since assertive drivers are classified as drivers who are relatively quick-tempered, they are characas people who enjoy tailgating, cutting others off in a lane, speeding up, rapid acceleration, and late braking.
Therefore, this paper generates simulation-based driving data considering these features. Drivers are labeled according to their predetermined feature values. Then, the data of each driver are then automatically extracted by using the CARLA simulator. CARLA is a simulator based on Unreal Engine. CARLA has a great advantage in that it can be experimented in diverse driving situations by customizing the driving environment including buildings, urban layouts, and traffic lights. In real driving conditions, traffic situations, including the location and driving directions of surrounding vehicles, and weather are all different each time the experiment is performed. However, using a simulator when conducting an experiment has a great advantage in that it can reduce the range of randomness or environmental change that can be applied in an actual driving environment. In this paper, drivers are classified into two groups: assertive drivers and defensive drivers. By collecting data using CARLA simulator that provides a constant driving environment, ADAS parameters that are affected by drivers' driving tendencies are correctly extracted and the influence they receive according to drivers’ driving preferences are relatively accurately measured.
The driving experiment was conducted using the route officially provided by the CARLA leaderboard. Town01, Town02, Town03, Town04, Town05, Town06, Town07, and Town10 are the official driving routes provided by the CARLA leaderboard. Among them, the experiment was conducted in Town05, which contains the most complex driving environment in the urban area. At this time, several buildings or people appearing on the road were fixed with the same seed value, so that the experiment could be conducted in the same traffic situation. The data includes about 30 sensor-based driving features with about 8300 rows of time series.
4.2 SVM Classification Method
Table 2 shows the overall performance of the SVM model.
(Table 2) The accuracy of the SVM classifier.
Precision: Precision is defined as the number of TP over the sum of the number of TP and FP, which means the number of actual true values among those classified as true.
\(\begin{aligned}\text {Precision}=\frac{T P}{T P+F P} \#\end{aligned}\) (1)
Recall: Recall is defined as the number of TP over the sum of the number of TP and FP, which means the number of what the model classified as true among the actual true values.
\(\begin{aligned}\text {Recall}=\frac{T P}{T P+F N} \#\end{aligned}\) (2)
F1 score: F1 score is the harmonic average of precision and recall, which can accurately evaluate the performance of the model when the data label has an unbalanced structure.
\(\begin{aligned}F 1\; score=2 \times \frac{1}{\left(\frac{1}{\text { Precision }}+\frac{1}{\text { Recall }}\right)} \#\end{aligned}\) (3)
Macro Average Precision: Macro Averaged Precision is typically used when multi-class classification. It is defined as an average of precision of all different classes.
Weighted Average Precision: Weighted Average Precision is used by giving weight to the class that should be emphasized when the number or ratio of each class is different.
It is shown that the SVM scheme shows high classification accuracy. The average cross-validation score is about 0.9733, and the classification accuracy of the SVM model is about 0.9734, which is very close to 1.
4.3 ROC Curve
The SVM algorithm can be evaluated through the ROC curve. The ROC curve shows the rate of true positives against false positives over the possible threshold values range. It explicitly shows how accurate the prediction of the model is. Since the most ideal binary classifier is formed when the slope of the graph faces the upper left direction, it can be said that the classifier implemented in this paper is working well. The performance of the implemented classifier demonstrated by the ROC curve is shown in Figure 3.
(Figure 3) ROC curve of the binary classifier using SVM.
4.4 Feature Selection Method
In the simulations, each driver drives a certain route scenario in CARLA. The driving data includes about 30 driving features and 8300 rows of time slots. Speed, acceleration, location of the vehicle, brake position, and steering angle are included in driving features.
Figure 4 shows the top eight driving characteristics that are most affected by the driver’s propensity to drive. Speed, throttle, steer angle, acceleration in the x-axis direction, acceleration in the y-axis direction, brake, and location of the vehicle are the top significant features that are extracted using SVM. Since people’s driving tendencies are well described in a complex driving environment, this experiment was conducted in a city with many vehicles, pedestrians, and complex routes.
(Figure 4) Feature selection results.
Assertive drivers are basically fast drivers and they frequently change lanes to get to their destination as quickly and efficiently as possible. In addition, they enjoy tailgating and turning rapidly when turning a corner. On the other hand, defensive drivers who seek safety tend to drive slowly while maintaining a constant speed limit and distance from other vehicles. Therefore, driving features that are related to these behaviors are selected as distinguishable features.
(Table 3) Performance comparison of the classifiers.
Driving features including throttle, steer, acceleration, and brake are explicitly significant features since these features show a large difference between drivers when they are turning a corner. In addition, the absolute coordinates of drivers are also one of the factors that best represent drivers’ propensity. Assertive drivers’ x and y coordinates of locations are much more varied than defensive drivers in the x and y coordinates of locations since they frequently change lanes when driving. Therefore, it is important to pay attention to the sensors and parameters that include the above-mentioned data, which best represent drivers’ driving tendencies. By doing this, it is possible to approach personalized ADAS technology that can provide a higher reliability to drivers.
4.5 Comparison of Driver Classification Methods
LSTM and GRU, which were described in the previous section, were used to classify drivers into two categories, assertive and defensive. Since the driving data is time-series data, which means the data highly depends on time, it is efficient to train RNN-based LSTM and GRU models and use them to make predictions.
The experiment was conducted using four different classifiers. First, LSTM and GRU were used to measure the classification accuracy of each driver. In addition, a feature selection step was added for the ‘SVM+LSTM’ model and the ‘SVM+GRU’ model. By combining SVM to LSTM or GRU, respectively, both models were trained again with the most significant features that are most affected by the driver's behaviors. Table 2 shows the overall performance of the four different models considering accuracy and loss.
The CPU used to conduct the experiment is AMD Ryzen 3 3300X 4-Core Processor and the GPU used is NVIDIA GeForce RTX 3080Ti. Also, the operating system used for training and evaluation is Ubuntu 18.04. LSTM and GRU are models that propagate while being influenced by previous states. In the case of LSTM, it is proposed to solve the long-term dependency problem in the basic RNN model and consists of three gates: an input gate, an output gate, and a forget gate. On the other hand, GRU consists of a total of two gates, a reset gate, and an update gate, and is a slightly simplified version of the time-step cell constituting LSTM. In the actual experiment, it took about two hours to train both models, but since the structure of GRU is a little simpler than LSTM, GRU took a little shorter time to learn. The total data consists of 8300 rows with 30 features. Accordingly, it took about 867.46 ms to process one-row data containing 30 features and about 28.92 ms was required to train 1 feature data in one-row data. In addition, since SVM is a model that linearly solves classification or regression problems, it takes much less time to train and evaluate compared to LSTM and GRU that sequentially process time-series data. Accordingly, despite a large amount of data, the time required was less than 15 minutes. In other words, SVM took about 108.43 ms to process one-row data containing 30 features, and 3.61 ms to learn 1 feature in one row, which is about 0.125 times longer than LSTM or GRU training and processing time.
Figure 5 describes the convergence of classification accuracy of the four different proposed models as the epoch increases. It is shown that the overall train accuracy is high, which means that the proposed systems perform well as binary classifiers. Among these models, the ‘SVM+GRU’ combined model shows the highest accuracy of 0.89. In general, models combined with SVM show a higher performance compared to the models that don’t include the feature extraction step supported by SVM. This is mainly because features that are somewhat irrelevant to driving behaviors are eliminated when training LSTM and GRU if the additional SVM step is introduced. In addition, since the experiment was conducted in a way that was driven in heavy traffic road conditions, the actual continuous time series were relatively short, resulting in higher performance on more efficient and simple GRUs than on complex LSTMs.
(Figure 5) Accuracy comparison of classifiers.
5. Conclusion
In this paper, a novel personalized ADAS scheme is proposed to improve the current state-of-the-art level 3 AV systems. The proposed scheme analyzes each driver’s driving behavior and preference to classify them into two groups: assertive and defensive drivers. The biggest challenge is that people’s trust or acceptance of AVs is low, so the actual AV usage is also significantly low. Therefore, this paper proposes a novel method to classify each driver considering their driving behaviors so the drivers who utilize this technology would feel much more comfortable and have more trust in personalized ADAS driving performance and safety. In the proposed scheme, the driver’s driving data is generated using the CARLA simulator. The proposed scheme uses SVM to first extract the distinguishable driving features to increase the classifying performance, and then LSTM and GRU models were used to implement the classifier, based on their RNN characteristics to perform well in conducting sequential data analysis and optimized control.
References
- W. D. Montgomery, R. Mudge, E. L. Groshen, S. Helper, J. P. MacDuffie, and C. Carson, "America's workforce and the self-driving future: Realizing productivity gains and spurring economic growth," Securing America's Future Energy, Washington, DC, USA, Tech. Rep., 2018. https://avworkforce.secureenergy.org/wp-content/uploads/2018/06/Americas-Workforce-and-the-Self-Driving-Future_Realizing-Productivity-Gains-and-Spurring-Economic-Growth.pdf
- J. Jeffs, "Autonomous Cars, Robotaxis & Sensors 2022-2042", IDTechEx, 2021. https://www.idtechex.com/en/research-report/autonomous-cars-robotaxis-and-sensors-2022-2042/832
- A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, "CARLA: An Open Urban Driving Simulator," in Proc. of 1st Conference on Robot Learning (CoRL 2017), California, USA, pp. 1-16, 2017. http://proceedings.mlr.press/v78/dosovitskiy17a/dosovitskiy17a.pdf
- X. Sun, J. Li, P. Tang, S. Zhou, X. Peng, H. N. Li, and Q. Wang, "Exploring Personalised Autonomous Vehicles to Influence User Trust," Cognitive Computation, Honolulu, HI, USA, pp. 1170-1186, 2020. https://doi.org/10.1007/s12559-020-09757-x
- I. Bae, J. Mon, J. Jhung, H. Suk, T. Kim, H. Park, J. Cha, J. Kim, D. Kim, and S. Kim, "Self-Driving like a Human driver instead of a Robocar: Personalized comfortable driving experience for autonomous vehicles," in NeurIps 2019 Workshop: Machine Learning for Autonomous Driving, Vancouver, Canada, Jan. 2020. https://doi.org/10.48550/arXiv.2001.03908
- M. Hasenjager, M. Heckmann and H. Wersing, "A Survey of Personalization for Advanced Driver Assistance Systems," IEEE Transactions on Intelligent Vehicles, Vol.5, Issue2, 2020. https://doi.org/10.1109/TIV.2019.2955910
- W. Xu, H. Zheng, Z. Yang, and Y. Yang, "Micro-Expression Recognition Base on Optical Flow Features and Improved MobileNetV2," KSII Trans. Internet and Inform. Syst., vol. 15, no. 6, pp. 1981-1995, Jun. 2021. https://doi.org/10.3837/tiis.2021.06.002
- J. Hong, "LSTM-based Sales Forecasting Model," KSII Trans. Internet and Inform. Syst., vol. 15, no. 4, pp. 1232-1245, Apr. 2021. https://doi.org/10.3837/tiis.2021.04.003
- E. M. Szumska and R. Jurecki, "The Effect of Aggressive Driving on Vehicle Parameters," 2020, (24), 6675, 2020. https://doi.org/10.3390/en13246675
- G. Hwang, D. Jung, Y. Goh, and J.-M. Chung, "Personal Driving Style based ADAS Customization using Machine Learning for Public Driving Safety," in Proc. of APIC-IST, Gangneung, Republic of Korea, Jun. 2022, pp. 201-202.