Keywords: Automated Vehicles, Autonomous / Intelligent Robotic Vehicles, Advanced Driver Assistance Systems
Abstract: This paper presents a method for generating coupling topologies for multi-agent systems. Our method is based on a non-cooperative game in which each agent chooses couplings to activate or deactivate using a utility function. The utility function measures the importance of agents to one another and enables conflict avoidance in distributed decision-making. Depending on the application’s needs, our method is able to generate unidirectional or bidirectional couplings. In our evaluation, we used car-like robots in a simulation environment. It shows that the generated coupling topologies are applicable to the domain of networked and autonomous vehicles.

Keywords: Traffic Flow and Management, Automated Vehicles, Vehicle Control
Abstract: The Cyber-Physical Mobility (CPM) Remote project provides web-based access to a simulation service and the CPM Lab itself, which is provided by the Chair of Embedded Software at RWTH Aachen University. With this approach, users no longer need special hardware, software, or physical access to use the simulation tool and the CPM Lab. This reduces the effort required to participate in research compared to the conventional simulation and lab setup.

CPM Remote already offers uniform hardware by outsourcing the simulation to our Chair’s servers and uniform software through the computer simulation of the CPM Lab. This paper proposes extending the CPM Remote framework by a set of formalized problems and a unified evaluation method. These benchmarks detect whether the vehicles have successfully passed through the scenario and evaluate the quality of the driven trajectories. We compiled the extracted scenarios into an event, the CPM Olympics. Thus, CPM Remote now has a fully defined stack for the objective comparison of algorithms for networked and autonomous driving.

Keywords: Intelligent Vehicle Software Infrastructure, Vehicle Control, Self-Driving Vehicles
Abstract: Agent-based simulation is an important testing tool for the development of autonomous vehicle software. Simulators enable engineers to test autonomous driving behavior in virtual environments, which is cheaper, faster, and safer than using a physical vehicle. An important aspect of autonomous driving software is its real-time capability, i.e. its ability to react to unforeseen events and new sensor inputs within a very short amount of time to prevent accidents. In this paper, we present a modular agent-based simulator architecture, which not only simulates the physical behavior of the vehicle, controlled by the software under test, but also its ac{E/E} network. In particular, each ECU is simulated using a hardware emulator, which enables us to test the software as if it is run on the actual target hardware. Furthermore, the hardware emulator estimates the execution delays for the software under test, which enables more realistic approximations of the real behavior. In an evaluation example we analyze empirically how well the timing estimates reflect the reality. We show that modeling the memory hierarchy and instruction decoding has a crucial effect on the precision of this estimation.

Keywords: Reinforcement Learning, Automated Vehicles, Cooperative ITS
Abstract: Cooperative trajectory planning methods for automated vehicles can solve traffic scenarios that require a high degree of cooperation between traffic participants. However, for cooperative systems to integrate into human-centered traffic, the automated systems must behave human-like so that humans can anticipate the system's decisions. While Reinforcement Learning has made remarkable progress in solving the decision-making part, it is non-trivial to parameterize a reward model that yields predictable actions. This work employs feature-based Maximum Entropy Inverse Reinforcement Learning combined with Monte Carlo Tree Search to learn reward models that maximize the likelihood of recorded multi-agent cooperative expert trajectories. The evaluation demonstrates that the approach can recover a reasonable reward model that mimics the expert and performs similarly to a manually tuned baseline reward model.

Keywords: Cooperative ITS, V2X Communication, Cooperative Systems (V2X)
Abstract: Connected and Automated Vehicles (CAVs) are highly expected to improve traffic throughput and safety at road intersections, single-track lanes, and construction zones. However, multiple CAVs can block each other and create a mutual deadlock around these road segments (i) when vehicle systems have a failure, such as a communication failure, control failure, or localization failure and/or (ii) when vehicles use a long shared road segment. In this paper, we present an Autonomous Deadlock Detection and Recovery Protocol at Intersections for Automated Vehicles named A-DRIVE that is a decentralized and time-sensitive technique to improve traffic throughput and shorten worst-case recovery time. To enable the deadlock recovery with automated vehicles and with human-driven vehicles, A-DRIVE includes two components: V2V communication-based A-DRIVE and Local perception-based A-DRIVE. V2V communication-based A-DRIVE is designed for homogeneous traffic environments in which all the vehicles are connected and automated. Local perception-based A-DRIVE is for mixed traffic, where CAVs, non-connected automated vehicles, and human-driven vehicles co-exist and cooperate with one another. Since these two components are not exclusive, CAVs inclusively and seamlessly use them in practice. Finally, our simulation results show that A-DRIVE improves traffic throughput compared to a baseline protocol.

Keywords: Vehicle Environment Perception, Lidar Sensing and Perception, Active and Passive Vehicle Safety
Abstract: Robust sensing and perception in adverse weather conditions remain one of the biggest challenges for realizing reliable autonomous vehicle mobility services. Prior work has established that rainfall rate is a useful measure for the adversity of atmospheric weather conditions. This work presents a probabilistic hierarchical Bayesian model that infers rainfall rate from automotive lidar point cloud sequences with high accuracy and reliability. The model is a hierarchical mixture of experts model, or a probabilistic decision tree, with gating and expert nodes consisting of variational logistic and linear regression models. Experimental data used to train and evaluate the model is collected in a large-scale rainfall experiment facility from both stationary and moving vehicle platforms. The results show prediction accuracy comparable to the measurement resolution of a disdrometer, and the soundness and usefulness of the uncertainty estimation. The model achieves RMSE 2.42,mm/h after filtering out uncertain predictions. The error is comparable to the mean rainfall rate change of 3.5,mm/h between measurements. Model parameter studies show how predictive performance changes with tree depth, sampling duration, and crop box dimension. A second experiment demonstrates the predictability of higher rainfall above 300,mm/h using a different lidar sensor, demonstrating sensor independence.

Keywords: Automated Vehicles, Cooperative Systems (V2X), Cooperative ITS
Abstract: Connected and Automated Vehicles (CAVs) have received a lot of attention in recent years. However, there are still numerous challenges in this field. In this paper, we investigated the effects of dynamic-flexible platooning on travel time by considering real-world trips data. For this purpose we extended the platooning capabilities of the 3DCoAutosim simulation platform, and proposed a dynamic-flexible model that we validated by creating use cases on traffic efficiency. We studied our dynamic-flexible platooning case for three electric vans with an autonomous leader, a semi-autonomous first follower with a driver, and an autonomous last follower. Results showed that the model developed in this study is efficient to investigate the effects of dynamic-flexible platooning on travel time.

Keywords: Autonomous / Intelligent Robotic Vehicles, Deep Learning
Abstract: Precisely modeling interactions and accurately predicting trajectories of surrounding vehicles are essential to the decision-making and path-planning of intelligent vehicles. This paper proposes a novel framework based on ensemble learning to improve the performance of trajectory predictions in interactive scenarios. The framework is termed Interactive Ensemble Trajectory Predictor (IETP). IETP assembles interaction-aware trajectory predictors as base learners to build an ensemble learner. Firstly, each base learner in IETP observes historical trajectories of vehicles in the scene. Then each base learner handles interactions between vehicles to predict trajectories. Finally, an ensemble learner is built to predict trajectories by applying two ensemble strategies on the predictions from all base learners. Predictions generated by the ensemble learner are final outputs of IETP. In this study, three experiments using different data are conducted based on the NGSIM dataset. Experimental results show that IETP improves the predicting accuracy and decreases the variance of errors compared to base learners. In addition, IETP exceeds baseline models with 50% of the training data, indicating that IETP is data-efficient. Moreover, the implementation of IETP is publicly available at https://github.com/BIT-Jack/IETP.

Keywords: Automated Vehicles, Advanced Driver Assistance Systems, Vehicle Control
Abstract: The Steer-by-Wire (SbW) steering system is a key technology for highly automated driving. For automated lateral vehicle guidance, the precise position control of the SbW Front Axle Actuator is an essential prerequisite. This paper presents the modeling, control synthesis, control loop analysis, and vehicle performance evaluation of the position control for the SbW Front Axle Actuator. Based on a nonlinear model of the plant a simplified linear system model is derived. This model yields the basis for the design of a Two-Degrees of Freedom Linear Quadratic Gaussian Control (2DOF LQG control), which allows an independent design of the command and the disturbance response. Besides a linear analysis of the control performance and stability, real vehicle tests for different driving maneuvers are conducted to verify simulation results and get a representative picture of the controller performance.

Keywords: Eco-driving and Energy-efficient Vehicles, Intelligent Vehicle Software Infrastructure, Vehicle Control
Abstract: With high line adaptability, low noise and vibration, and potential in super-high operation speed compared to the wheel-rail (WR) train, the magnetic levitation (Maglev) train has attracted extensive attention from academia and industry. Although the Maglev train eliminates the energy consumption of wheel-rail friction, the huge aerodynamic energy consumption caused by its high-speed operation cannot be ignored. Due to the difference of operation mechanism and dynamic model between Maglev and WR train, the energy-efficient train operation for Maglev train considering peculiar characteristics needs to be further studied. This paper proposed a speed trajectory optimization model formulated by mixed-integer linear programming (MILP) to minimize the energy consumption of the Maglev system. Besides, piecewise linear (PWL) was utilized to deal with the nonlinear terms involved in the Maglev model. Finally, commercial software was applied to solve the model with consideration of various practical constraints of the Maglev system. The comparative result indicates that the flexible MILP model can obtain an optimal strategy efficiently with low linearization errors which are less than 0.1 %.

Keywords: Electric and Hybrid Technologies, Situation Analysis and Planning, Information Fusion
Abstract: This paper proposes to use Gaussian process regression to predict the consumption of a plug-in electric hybrid vehicle from low-quality data. We specify background knowledge regarding new operating points and information regarding the noise process. This makes it possible to adapt the original (`naive') model. Experiments realized using dynamic and energetic models simulated electrified vehicle show the interest of our approach in order to improve robustness against scarce and noisy data.

Keywords: Advanced Driver Assistance Systems, Autonomous / Intelligent Robotic Vehicles, Driver Recognition
Abstract: In order to meet the demand for safety, usability, comfortability, and entertainment for rear seat passenger service, we introduce Skeleton motion dataset of Vehicle Rear seat Passenger (SVRP) which is a world first skeleton motion dataset for rear seat passenger with 22 different actions publicly available. The dataset was trained and tested by a neural network with CTR-GCN for action inference. The result shows the accuracy is 78.3 percent for 25 joint 2D skeleton and 80.2 percent for 32 joint 3D skeleton by sliding 4 second observation window. We also found that a longer observation window is crucial for a stable inference while time frame resolution can be reduced to 5 frames per second for lightweight computation without much accuracy drop. The number of skeleton joints can be also reduced with same accuracy from 25 points to 10 points, which is a mostly upper body part, by a proposed heatmap correlation method.

Keywords: Vulnerable Road-User Safety, Collision Avoidance, Autonomous / Intelligent Robotic Vehicles
Abstract: In this paper, we present a microscopic agent-based pedestrian behavior model Intend-Wait-Cross. The model is comprised of rules representing behaviors of pedestrians as a series of decisions that depend on their individual characteristics (e.g. demographics, walking speed, law obedience) and environmental conditions (e.g. traffic flow, road structure). The model's main focus is on generating realistic crossing decision-model, which incorporates an improved formulation of time-to-collision (TTC) computation accounting for context, vehicle dynamics, and perceptual noise.

Our model generates a diverse population of agents acting in a highly configurable environment. All model components, including individual characteristics of pedestrians, types of decisions they make, and environmental factors, are motivated by studies on pedestrian traffic behavior. Model parameters are calibrated using a combination of naturalistic driving data and estimates from the literature to maximize the realism of the simulated behaviors. A number of experiments validate various aspects of the model, such as pedestrian crossing patterns, and individual characteristics of pedestrians.

Keywords: Vulnerable Road-User Safety, Autonomous / Intelligent Robotic Vehicles, Collision Avoidance
Abstract: Pedestrian behavior prediction is one of the major challenges for intelligent driving systems in urban environments. Pedestrians often exhibit a wide range of behaviors and adequate interpretations of those depend on various sources of information such as pedestrian appearance, states of other road users, the environment layout, etc. To address this problem, we propose a novel multi-modal prediction algorithm that incorporates different sources of information captured from the environment to predict future crossing actions of pedestrians. The proposed model benefits from a hybrid learning architecture consisting of feedforward and recurrent networks for analyzing visual features of the environment and dynamics of the scene. Using the existing 2D pedestrian behavior benchmarks and a 3D driving dataset, we show that our proposed model achieves state-of-the-art performance in pedestrian crossing prediction.

Keywords: Driver State and Intent Recognition, Automated Vehicles
Abstract: The microscopic behavior of the vehicle can be decomposed into car following and lane changing, and can be described by the longitudinal and lateral movement. The longitudinal movement has long been studied, while the lateral counterpart, especially the stochastic lateral movement, has rarely been investigated. The lacking of an understanding of the lateral behavior makes current microscopic simulation results deviate from real-world observations. Besides, many behavior identification algorithms which rely on lateral displacement are not robust, if the lateral stochastic nature is not well studied. To fill in this gap, a stochastic differential equation approach is employed. Firstly, the lateral noise is modeled by a transformed Brownian motion. Then the noise is embedded into a differential lateral movement model. The parameters in the lateral noise and movement models all have clear physical meaning. The Fokker-Planck equation, which describes the distribution evolution of the lateral displacement, is derived. A parameters calibration procedure is derived using the Euler discretization scheme. The model is calibrated using real world data. The results show that the proposed model can well describe the lateral movement distribution.

Keywords: Autonomous / Intelligent Robotic Vehicles, Self-Driving Vehicles, Situation Analysis and Planning
Abstract: Ensuring the safety of autonomous vehicles remains challenging given the uncertainty in sensing other road users. Moreover, separate safety specifications for perception and planning components impede assessing the overall system safety. This work provides a probabilistic approach to calculate safety envelopes under perception uncertainty. The probabilistic envelope definition is based on a risk threshold. It limits the cumulative probability that the actual safety envelope in a fully observable environment is physically more extensive than the applied envelope and is solved using iterative worst-case analysis of envelopes. Our approach extends non-probabilistic envelopes -- in this work, the Responsibility-Sensitive Safety (RSS) -- to handle uncertainties. To evaluate the probabilistic envelope approach, we compare it in a simulated highway merging scenario against several baseline safety architectures. Our evaluation shows that our model allows adjusting safety and performance based on a chosen risk level and the level of sensing uncertainty. We conclude with an outline of how to formally argue safety under perception uncertainty using our formulation of envelope violation risk.

Keywords: Radar Sensing and Perception, Deep Learning, Automated Vehicles
Abstract: Due to the small number of raw data automotive radar datasets and the low resolution of such radar sensors, automotive radar object detection has been little explored with deep learning models in comparison to camera and lidar-based approaches. However, radars are low-cost sensors able to accurately sense surrounding object characteristics (e.g., distance, radial velocity, direction of arrival, radar cross-section) regardless of weather conditions (e.g., rain, snow, fog). Recent open-source datasets such as CARRADA, RADDet or CRUW have opened up research on several topics ranging from object classification to object detection and segmentation. In this paper, we present DAROD, an adaptation of Faster R-CNN object detector for automotive radar on the range-Doppler spectra. We propose a light architecture for features extraction, which shows an increased performance compare to heavier vision-based backbone architectures. Our models reach respectively an mAP@0.5 of 55.83 and 46.57 on CARRADA and RADDet datasets, outperforming competing methods.

Keywords: Self-Driving Vehicles, Mapping and Localization, Deep Learning
Abstract: Lane boundaries, as the main component of high definition maps (HD maps), are difficult to auto-generate accurately in various scenarios. In this paper, a general lane boundary extraction method is proposed for HD mapping in both highway and urban scenarios. Firstly, a learning-based heatmap regression network is applied to estimate the center of lane boundaries in bird’s eye view (BEV) images from light detection and ranging (LiDAR). Secondly, the geometry of various lane boundaries is extracted accurately in a coarse-to-fine strategy. Given the regression results, the geometry generation method initially extracts kinds of lane boundaries coarsely, including highway boundaries and complex cases in urban scenarios, such as splitting lane boundaries, lane boundaries in arbitrary directions, etc. Subsequently, the fine adjustment method increases the accuracy of the lane boundary geometry by inserting and adjusting the keypoints recursively according to the regression heatmap. To handle large-scale mapping, additional methods are presented to merge the same lane boundary including the connection priority strategy and adaptive lane vertex downsampling. Experiments demonstrate that the proposed method manages to generate accurate lane boundaries in both highway and urban scenarios with limited storage consumption, and therefore is an effective and storage-saving method for large-scale HD mapping.

Keywords: Advanced Driver Assistance Systems, Convolutional Neural Networks, Vision Sensing and Perception
Abstract: Whether it's a driver warning or an autonomous driving system, ADAS needs to decide when to alert the driver of danger or take over control. This research formulates the problem as an alert-scene categorization one and proposes a method using contrastive learning. Given a front-view video of a driving scene, a set of anchor points is marked by a human driver, where an anchor point indicates that the semantic attribute of the current scene is different from that of the previous one. The anchor frames are then used to generate contrastive image pairs to train a feature encoder and obtain a scene similarity measure, so as to expand the distance of the scenes of different categories in the feature space. Each scene category is explicitly modeled to capture the meta pattern on the distribution of scene similarity values, which is then used to infer scene categories. Experiments are conducted using front-view videos that were collected during driving at a cluttered dynamic campus. The scenes are categorized into no alert, longitudinal alert, and lateral alert. The results are studied at both feature encoding, category modeling, and reasoning aspects. By comparing precision with two full supervised end-to-end baseline models, the proposed method demonstrates competitive or superior performance. However, it remains still questions: how to generate ground truth data and how to evaluate performance in ambiguous situations, which leads to future works.

Keywords: Reinforcement Learning, Smart Infrastructure, Automated Vehicles
Abstract: Autonomous intersection management has the ability to reduce congestion at intersections significantly, compared to classical traffic signal control in the era of connected autonomous vehicles. Autonomous intersection management requires time and speed adjustment for vehicles arriving at an intersection for collision-free passing through the intersection. Due to its computational complexity, this problem has been studied only when vehicle arrival times towards the vicinity of the intersection are known beforehand or with other simplifying scenarios which limits the applicability of these solutions for real-time settings. To solve the real-time autonomous traffic intersection management problem, we propose a reinforcement learning (RL) based multiagent architecture and a novel RL algorithm coined multi-discount Q-learning. In multi-discount Q-learning, we introduce a simple yet effective way to solve a Markov Decision Process by preserving both short-term and long-term goals, which is crucial for collision-free speed control. Our experimental results using microscopic simulations show that our RL-based multiagent solution can achieve near-optimal performance efficiently when minimizing the travel time through an intersection.

Keywords: Self-Driving Vehicles, Situation Analysis and Planning, Automated Vehicles
Abstract: State trajectory planning is one of the primary self-driving cars technology enablers. However, state trajectory planning is a more complex and computationally demanding task compared to path planning. The vehicle's east and north position, yaw, yaw rate, velocity, and battery state of charge variables trajectory planning with a particular focus on the safety and economy of the vehicle operation is concerned in this paper. Comparison of Model Predictive Control (MPC) and Minimum Violation Planning (MVP) used for trajectory planning is brought in this paper. The latter is a sampling-based algorithm based on the RRT* algorithm compared to the other optimization-based algorithm. A heuristic is introduced to convert the complex non-convex optimization planning task to a convex optimization problem. Next, MVP algorithm enhancement is proposed to reduce the calculation time. Both algorithms are tested on a selected testing scenario using a high fidelity nonlinear single-track model implemented in Matlab & Simulink environment.

Keywords: Human-Machine Interface, Automated Vehicles, Novel Interfaces and Displays
Abstract: Higher levels of vehicle automation come with new challenges for designing safe systems. The Human Machine-Interface (HMI) plays a key role in mediating the interaction between the human driver and vehicle automation. By providing the driver with appropriate feedback, the HMI has the potential to increase mode awareness and situational awareness. For the development of appropriate HMI solutions, usability assessments are essential. Immersive Virtual Reality (VR) technology enables researchers and designers to construct realistic virtual prototypes and immersive evaluation scenarios with less time and resources. The current study presents a VR evaluation tool called VRHEAD, which is designed to facilitate an iterative design process and support the rapid implementation of virtual prototypes to evaluate of an automated vehicle’s HMI. Initial results indicate that VRHEAD is a promising approach for the rapid implementation and evaluation of design concepts. The use of VR tools, like VRHEAD, can reduce the time and costs associated with developing high-fidelity prototypes and provide more flexibility in modifying a design according to new research findings, thus broadening the exploration of the HMI design space.

Keywords: Sensor and Data Fusion, Vehicle Environment Perception, Active and Passive Vehicle Safety
Abstract: Tire wear conditions are crucial for safe vehicle control and should be managed properly. The demand for technologies for self-diagnosis of wear conditions is expected to increase to save time and effort required for daily tire checking in fleet management companies or to enable self-notification of the tire replacement requirement in autonomous vehicles. Although several methodologies based on vibration or acceleration analysis have been proven to be feasible, their functionality is critically limited under low-speed conditions. To solve this problem, this study proposes a novel methodology using inner liner strain measurement. We experimentally confirmed that the ratio of the circumstantial strain around the trailing edge to that around the leading edge of a tire decreases according to tire wear. The strain reduction rates in the trailing edge were stable under various speeds, loads, and tire inflation pressure conditions, suggesting that this feature could be a reliable indicator for estimating tire wear conditions under various situations, including slow-driving autonomous vehicles. The mechanism hypothesis for the results is based on the fundamental knowledge of pneumatic tire mechanics, indicating that our observation is generally applicable and not unique to our system.

Keywords: Cooperative Systems (V2X), Traffic Flow and Management, Reinforcement Learning
Abstract: Urban intersections are prone to delays and inefficiencies due to static precedence rules and occlusions limiting the view on prioritized traffic. Existing approaches to improve traffic flow, widely known as automatic intersection management systems, are mostly based on non-learning reservation schemes or optimization algorithms. Machine learning-based techniques show promising results in planning for a single ego vehicle. This work proposes to leverage machine learning algorithms to optimize traffic flow at urban intersections by jointly planning for multiple vehicles. Learning-based behavior planning poses several challenges, demanding for a suited input and output representation as well as large amounts of ground-truth data. We address the former issue by using a flexible graph-based input representation accompanied by a graph neural network. This allows to efficiently encode the scene and inherently provide individual outputs for all involved vehicles. To learn a sensible policy, without relying on the imitation of expert demonstrations, the cooperative planning task is considered as a reinforcement learning problem. We train and evaluate the proposed method in an open-source simulation environment for decision making in automated driving. Compared to a first-in-first-out scheme and traffic governed by static priority rules, the learned planner shows a significant gain in flow rate, while reducing the number of induced stops. In addition to synthetic simulations, the approach is also evaluated based on real-world traffic data taken from the publicly available inD dataset.

Keywords: Self-Driving Vehicles, Automated Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: In urban driving scenarios, it is a key component for autonomous vehicles to generate a smooth, kinodynamically feasible, and collision-free path. We present an optimization-based path planning method for autonomous vehicles navigating in cluttered environment, e.g., roads partially blocked by static or moving obstacles. Our method first computes a collision-free reference line using quadratic programming(QP), and then using the reference line as initial guess to generate a smooth and feasible path by iterative optimization using sequential quadratic programming(SQP). It works within a fractions of a second, thus permitting efficient regeneration.

Keywords: Deep Learning, Vision Sensing and Perception, Automated Vehicles
Abstract: Autonomous driving (AD) and advanced driver assistance systems (ADAS) increasingly utilize deep neural networks (DNNs) for improved perception or planning. Nevertheless, DNNs are quite brittle when the data distribution during inference deviates from the data distribution during training. This represents a challenge when deploying in partly unknown environments like in the case of ADAS. At the same time, the standard confidence of DNNs remains high even if the classification reliability decreases. This is problematic since following motion control algorithms consider the apparently confident prediction as reliable even though it might be considerably wrong. To reduce this problem real-time capable confidence estimation is required that better aligns with the actual reliability of the DNN classification. Additionally, the need exists for black-box confidence estimation to enable the homogeneous inclusion of externally developed components to an entire system. In this work we explore this use case and introduce the neighborhood confidence (NHC) which estimates the confidence of an arbitrary DNN for classification. The metric can be used for black-box systems since only the top-1 class output is required and does not need access to the gradients, the training dataset or a hold-out validation dataset. Evaluation on different data distributions, including small in-domain distribution shifts, out-of-domain data or adversarial attacks, shows that the NHC performs better or on par with a comparable method for online white-box confidence estimation in low data regimes which is required for real-time capable AD/ADAS.

Keywords: Vision Sensing and Perception, Mapping and Localization, Vehicle Environment Perception
Abstract: In this paper, a complete pipeline for image-based 3D reconstruction of urban scenarios is proposed, based on PatchMatch Multi-View Stereo (MVS). Input images are firstly fed into an off-the-shelf visual SLAM system to extract camera poses and sparse keypoints, which are used to initialize PatchMatch optimization. Then, pixelwise depths and normals are iteratively computed in a multi-scale framework with a novel depth-normal consistency loss term and a global refinement algorithm to balance the inherently local nature of PatchMatch. Finally, a large-scale point cloud is generated by back-projecting multi-view consistent estimates in 3D. The proposed approach is carefully evaluated against both classical MVS algorithms and monocular depth networks on the KITTI dataset, showing state of the art performances.

Keywords: Convolutional Neural Networks, Lidar Sensing and Perception, Deep Learning
Abstract: Recent advances in 3D object detection are made by developing the refinement stage for voxel-based Region Proposal Networks (RPN) to better strike the balance between accuracy and efficiency. A popular approach among state-of- the-art frameworks is to divide proposals, or Regions of Interest (ROI), into grids and extract features for each grid location before synthesizing them to form ROI features. While achieving impressive performances, such an approach involves several hand-crafted components (e.g. grid sampling, set abstraction) which requires expert knowledge to be tuned correctly. This paper proposes a data-driven approach to ROI feature computing named APRO3D-Net which consists of a voxel-based RPN and a refinement stage made of Vector Attention. Unlike the original multi-head attention, Vector Attention assigns different weights to different channels within a point feature, thus being able to capture a more sophisticated relation between pooled points and ROI. Our method achieves a competitive performance of 84.85 AP for class Car at moderate difficulty on the validation set of KITTI and 47.03 mAP (average over 10 classes) on NuScenes while having the least parameters compared to closely related methods and attaining an inference speed at 15 FPS on NVIDIA V100 GPU. The code is released.

Keywords: Recurrent Networks, Electric and Hybrid Technologies
Abstract: Lithium-ion batteries (LIBs) will accelerate the degradation of capacity after long-term cycling, showing nonlinear aging features. The onset of the nonlinear aging feature is called the “knee-point”. The appearance of nonlinear aging will not only lead to a rapid drop in the overall performance of LIBs, but also the serious collapse of battery safety, which makes the identification and early prediction of nonlinear aging knee-point particularly important. In this paper, we propose a nonlinear aging knee-point prediction method of LIBs based on sliding window with stacked long short-term memory (LSTM) neural network. We compared our method with other common machine learning algorithms, and found that the prediction results of knee-point under this method are significantly better than other algorithms. When the sliding window size is 20, the root mean square error (RMSE) of prediction result is 72.3 and the mean absolute error (MAE) is 51.6. In addition, we further study the effect of different sliding window sizes on the prediction results. By predicting the knee-point, it can be recognized when the nonlinear aging begins so that the user can be reminded whether to replace the battery, which greatly reduces the risk of battery safety problems.

Keywords: Automated Vehicles, Active and Passive Vehicle Safety, Image, Radar, Lidar Signal Processing
Abstract: The safety of autonomous vehicles is hard to ensure in adverse weather since the sensors will degrade drastically. Setting a variable speed limit based on real-time weather condition is the most efficient method to make the vehicle safe. But most current speed limit methods are based on human visibility rather than the sensor, which is not suitable for autonomous vehicles. Thus, it is necessary to explore the performance of sensors in different weathers and propose a speed limit method based on sensor performance. Safety decisions will be made based on the calculated speed limit to ensure safety. This paper describes how to make safety decisions based on sensor performance and road conditions in real-time. The experiment explores the degradation of different sensors, and variable speed limit methods are proposed for rainy and foggy days. MPC controller is used to generate safety decisions.

Keywords: Electric and Hybrid Technologies, Eco-driving and Energy-efficient Vehicles
Abstract: Reasonable power distribution between battery and supercapacitor in electric vehicles is a crucial problem to improve energy consumption and economy. An online energy management strategy based on model predictive control (MPC) is proposed in this paper. Firstly, a radial basis function neural network optimized by particle swarm algorithm is presented to generate the short-term future velocity, i.e., the reference trajectory of the MPC. Then, a cost function considering the battery degradation cost and the electricity cost is constructed and optimized within each prediction horizon while maintaining the state of charge of the supercapacitor. Simulation results on the UDDS driving cycle show that the total cost of the proposed strategy is reduced by 6.3% and 3.9% compared with the near-optimal rule-based strategy and the none optimized velocity predictor-MPC, respectively, indicating that the velocity prediction accuracy has a significant impact on the performance of real-time energy management.

Keywords: Information Fusion, Automated Vehicles, Mapping and Localization
Abstract: A state observer, robust to road surface conditions, is designed to estimate the longitudinal speed (and slip) which is essential for controls and safety-critical decision making in autonomous driving. The novel approach estimates slip at each wheel, and can be integrated with the existing visual-inertial navigation systems. The wheel-level observer, which uses proprioceptive sensor data, fuses vehicle kinematic states, tire internal states, and the wheel dynamics to estimate the speed at each tire, without any information of the road surface friction or global navigation satellite systems (GNSS). Then, a wheel-vehicle dynamical model, which augments estimates at each tire with the vehicle dynamics, is developed to design an integrated slip-aware framework for speed estimation. The stability of the augmented error dynamics is studied and the mean square estimation error is proved to be uniformly bounded. Experimental tests have been conducted to validate the proposed framework in pure- and combined-slip driving scenarios on various surface friction conditions. As confirmed by several road experiments, the designed observer provides consistent and accurate speed (and slip) estimates at each tire for high-slip scenarios, which are essential for safe navigation, motion planning, and path following in automated driving systems.

Keywords: Electric and Hybrid Technologies, Recurrent Networks, Security
Abstract: Battery management system (BMS) plays an important role in ensuring the safe and stable operation of batteries. In BMS, the State of health (SOH) status as a measure of the battery storage and release of the ability to change, in essence reflects the aging and damage of batteries. However, in online applications, it is difficult to directly measure the capacity of batteries. In this paper, the voltage, current and temperature data extracted from the charging and discharging process of the battery are directly used as Health Factors(HF), which are divided into training set verification set and test set. The battery capacity estimation model is established based on the Long Short-term Memory Recurrent Neural Network (LSTM, RNN) to estimate SOH.

Keywords: Reinforcement Learning, Deep Learning
Abstract: 随着机器学习的兴起，强化学习（RL）逐渐应用于插电式混合动力电动汽车（PHEV）的能源管理策略（EMS）。一些旧算法通过与强化学习相结合也取得了更好的效果。为了学习现有算法的优势，探索强化学习算法的应用潜力，提出一种将等效消耗最小化策略（ECMS）知识与近端策略优化（PPO）相结合的自适应分层管理策略。该系统是目前先进的数据驱动RL算法。为了进行更全面的比较，本文将所提出的EMS与动态规划（DP）、具有恒定等效因子和q-learning的ECMS进行了比较。结果表明，所提控制策略的油耗与基于DP的控制策略非常接近&

Keywords: Eco-driving and Energy-efficient Vehicles, Cooperative ITS, Cooperative Systems (V2X)
Abstract: Advances in connected automated technology allow for more efficient driving in heavy-duty transportation. By well coordinating the longitudinal movements of multiple vehicles driving in a string, eco-platooning control can significantly improve the driving comfort and fuel economy. Moreover, benefitting from the short following distances of the platoon members, the aerodynamic effects are believed to further reduce the overall energy consumption in heavy-duty applications. In this paper, we develop an aerodynamically aware cooperative adaptive cruise control (CACC) strategy based on nonlinear model predictive control (NMPC). The proposed strategy is implemented under different communication topologies: 1) predecessor following (PF), 2) leader following (LF), and 3) predecessor-leader following (PLF). The performance of three communication topologies is evaluated through several indexes, and the simulation results indicate that when the information of the platoon leader is broadcast to the other platoon members, resulting in a so-called LF or PLF topology, the string stability would be guaranteed, and the proposed strategy can improve the driving comfort of all three trucks by eliminating unnecessary accelerations. On the other hand, a remarkable decrement on demanded power can be derived due to the effect of air-drag reduction.

Keywords: Advanced Driver Assistance Systems, Vehicle Control, Active and Passive Vehicle Safety
Abstract: Vehicle stability control is the core technology required for improving driving safety of advanced driver assistance systems (ADAS). In this paper, vehicle dynamic stability characteristics are investigated, and an improved vehicle stability controller is proposed to enhance the vehicle’s performance. The sum-of-squares programming is introduced to estimate its stability region and qualitative analysis is utilized to investigate the effect of various driving conditions on the stability region. An approximate dynamic stability boundary is established for different steering angle inputs. A new Lyapunov-function-based vehicle dynamic stability (LFVDS) controller is then designed to improve vehicle stability and dynamics performance based on the hierarchical structure. A test on a Hardware-In-the-Loop platform is formulated to validate the vehicle state response under the traditional and the proposed stability controllers. The results indicate that, compared with the traditional stability controller, the LFVDS controller can effectively reduce longitudinal velocity drop by 33% on a slippery road surface with ensured vehicle stability.

Keywords: Automated Vehicles, Situation Analysis and Planning, Vehicle Environment Perception
Abstract: Safe driving requires autonomous vehicles to anticipate potential hidden traffic participants and other unseen objects, such as a cyclist hidden behind a large vehicle, or an object on the road hidden behind a building. Existing methods are usually unable to consider all possible shapes and orientations of such obstacles. They also typically do not reason about observations of hidden obstacles over time, leading to conservative anticipations. We overcome these limitations by (1) modeling possible hidden obstacles as a set of states of a point mass model and (2) sequential reasoning based on reachability analysis and previous observations. Based on (1), our method is safer, since we anticipate obstacles of arbitrary unknown shapes and orientations. In addition, (2) increases the available drivable space when planning trajectories for autonomous vehicles. In our experiments, we demonstrate that our method, at no expense of safety, gives rise to significant reductions in time to traverse various intersection scenarios from the CommonRoad Benchmark Suite.

Keywords: Vulnerable Road-User Safety, Deep Learning, Smart Infrastructure
Abstract: This article presents a holistic approach for probabilistic cyclist intention detection. By combining probabilities for the current cyclist motion states with a probabilistic ensemble trajectory forecast, we are able to generate reliable estimates of cyclists' future positions. The probabilities are used as weights in the probabilistic ensemble trajectory forecast. The ensemble consists of specialized models, which produce individual forecasts in the form of Gaussian distributions under the assumption of a certain motion state of the cyclist (e.g. cyclist is starting). By weighting the specialized models, we create forecasts in the form of Gaussian mixtures that define regions within which the cyclists will reside with a certain probability. To evaluate our method, we rate the reliability, sharpness, and positional accuracy of our forecasted distributions. We compare our method to a widely used unimodal approach which produces forecasts in the form of Gaussian distributions and show that our method is able to produce more reliable and sharper outputs while retaining comparable positional accuracy. By comparing two different methods for basic movement detection, one based on the past cyclist trajectory and one using past image sequences, we demonstrate that the results can be further improved by incorporating video information. Our methods are evaluated using a dataset created at a public intersection. The code and the dataset are publicly available.

Keywords: Deep Learning
Abstract: Multi-agent trajectory prediction is a challenging task because of the uncertainty of agents’ behaviors, interactions between agents, complex road geometry in urban environments, and imperfect/noisy agent histories. Although accurate prediction results are critical for safe and reliable intelligent driving applications (e.g., decision making, motion planning), some other applications may prefer lightweight and computation-efficient trajectory prediction models to handle dynamically changed environments. In this work, we propose a multi-agent, multi-modal Graph Attention Isomorphism Network (GAIN) based trajectory prediction framework to effectively understand and aggregate long-term interactions across agents. We also take the model complexity and computation efficiency into consideration. Experiments on both pedestrian and vehicle datasets demonstrated the effectiveness of our proposed method.

Keywords: Autonomous / Intelligent Robotic Vehicles, Automated Vehicles, Situation Analysis and Planning
Abstract: In this work, we present a novel multi-modal multi-agent trajectory prediction architecture, focusing on map and interaction modeling using graph representation. For the purposes of map modeling, we capture rich topological structure into vector-based star graphs, which enable an agent to directly attend to relevant regions along polylines that are used to represent the map. We denote this architecture StarNet, and integrate it into a single-agent prediction setting. As the main result, we extend this architecture to joint scene-level prediction, which produces multiple agents' predictions simultaneously. The key idea in joint-StarNet is integrating the awareness of one agent in its own reference frame with how it is perceived from the points of view of other agents. We achieve this via masked self-attention. Both proposed architectures are built on top of the action-space prediction framework introduced in our previous work, which ensures kinematically feasible trajectory predictions. We evaluate the methods on the interaction-rich inD and INTERACTION datasets, with both StarNet and joint-StarNet achieving improvements over state of the art.

Keywords: Deep Learning, Image, Radar, Lidar Signal Processing, Self-Driving Vehicles
Abstract: Adverse weather conditions can negatively affect LiDAR-based object detectors. In this work, we focus on the phenomenon of vehicle gas exhaust condensation in cold weather conditions. This everyday effect can influence the estimation of object sizes, orientations and introduce ghost object detections, compromising the reliability of the state of the art object detectors. We propose to solve this problem by using data augmentation and a novel training loss term. To effectively train deep neural networks, a large set of labeled data is needed. In case of adverse weather conditions, this process can be extremely laborious and expensive. We address this issue in two steps: First, we present a gas exhaust data generation method based on 3D surface reconstruction and sampling which allows us to generate large sets of gas exhaust clouds from a small pool of labeled data. Second, we introduce a point cloud augmentation process that can be used to add gas exhaust to datasets recorded in good weather conditions. Finally, we formulate a new training loss term that leverages the augmented point cloud to increase object detection robustness by penalizing predictions that include noise. In contrast to other works, our method can be used with both grid-based and point-based detectors. Moreover, since our approach does not require any network architecture changes, inference times remain unchanged. Experimental results on real data show that our proposed method greatly increases robustness to gas exhaust and noisy data.

Keywords: Automated Vehicles, Vehicle Environment Perception, Deep Learning
Abstract: Processing camera information to perceive their 3D surrounding is essential for building scalable autonomous driving vehicles. For this task, deep learning networks provide effective real-time solutions. However, to compensate for missing depth information in cameras compared to LiDARs, a large amount of labeled data is required for training. Active learning is a training framework where the network actively participates in the data selection process to improve data efficiency and performance. In this work, we propose an active learning pipeline for 3D object detection from monocular images. The main components of our approach are (1) two training-efficient uncertainty estimation strategies, (2) a diversity-based selection strategy to select images that contain the most diverse set of objects, (3) a novel active learning strategy more suitable for training autonomous driving perception networks. Experiments show that combining our proposed uncertainty estimation methods provides a better data saving rate and reaches a higher final performance than baselines. Furthermore, we empirically show performance gains of the presented diversity-based selection strategy and the efficiency of the proposed active learning strategy.

Keywords: Radar Sensing and Perception, Vehicle Environment Perception, Deep Learning
Abstract: Recent research explored advantages of applying a learning-based method to the radar target detection problem. A single point target case was mainly considered, though. This work extends those studies to complex automotive scenarios. We propose a Convolutional Neural Networks-based model able to detect and locate targets in multi-dimensional space of range, velocity, azimuth, and elevation. Due to the lack of publicly available datasets containing raw radar data (after analog-to-digital converter), we simulated a dataset comprising more than 17,000 frames of automotive scenarios and various road objects including (but not limited to) cars, pedestrians, cyclists, trees, and guardrails. The proposed model was trained exclusively on simulated data and its performance was compared to that of conventional radar detection and angle estimation pipeline. In unseen simulated scenarios, our model outperformed the conventional CFAR-based methods, improving by 14.5% the dice score in range-Doppler domain. Our model was also qualitatively evaluated on unseen real-world radar recordings, achieving more detection points per object than the conventional processing.

Keywords: Radar Sensing and Perception, Deep Learning, Vulnerable Road-User Safety
Abstract: Frequency modulated continuous wave radars are an important component of modern driver assistance systems and enable safer automated driving. To achieve real time detection and classification of multiple road users in the range-Doppler map, the usage of neural target detection networks is proposed. Since the amount of labelled radar measurements available limits the training process, a new radar simulation framework is presented which generates arbitrary traffic scenarios with reflection models for pedestrians, bicyclists and vehicles. With an adaptive FMCW setup, sequences of dynamic urban multi-target radar measurements are simulated, maintaining minimum computational complexity. Solely trained on simulated measurement data, the neural network achieves a detection rate around 88% on bicyclists and vehicles in real measurement data which is comparable to the performance of neural networks trained on real measurement datasets.

Keywords: Vehicle Environment Perception, Lidar Sensing and Perception, Recurrent Networks
Abstract: A common approach for modeling the environment of an autonomous vehicle are dynamic occupancy grid maps, in which the surrounding is divided into cells, each containing the occupancy and velocity state of its location. Despite the advantage of modeling arbitrary shaped objects, the used algorithms rely on hand-designed inverse sensor models and semantic information is missing. Therefore, we introduce a multi-task recurrent neural network to predict grid maps providing occupancies, velocity estimates, semantic information and the driveable area. During training, our network architecture, which is a combination of convolutional and recurrent layers, processes sequences of raw lidar data, that is represented as bird’s eye view images with several height channels. The multi-task network is trained in an end-to-end fashion to predict occupancy grid maps without the usual preprocessing steps consisting of removing ground points and applying an inverse sensor model. In our evaluations, we show that our learned inverse sensor model is able to overcome some limitations of a geometric inverse sensor model in terms of representing object shapes and modeling freespace. Moreover, we report a better runtime performance and more accurate semantic predictions for our end-to-end approach, compared to our network relying on measurement grid maps as input data.

Keywords: Situation Analysis and Planning, Self-Driving Vehicles, Automated Vehicles
Abstract: Driving autonomous vehicles safely through a complex urban environment remains a difficult task. The sensor limitations, as well as the various occlusions in the urban environment caused by static and dynamic objects, make the decision-making task even more complex. To improve the autonomous vehicle’s ability to handle various occlusion driving scenarios, we propose a behavior planner with traffic mirror awareness based on the partially observable Markov decision process (POMDP). Our approach is based on the concept of phantom road users, which allows us to reason about the potentially occluded traffic participants and estimate the appearance probability in risky areas based on contextual information. A confidence modifier is introduced to either increase or decrease the appearance probability by utilizing the uncertain road users tracking results from available traffic mirror detections. Furthermore, we present an active traffic mirror perceiving method for encouraging the ego vehicle to explore the environment and plan driving policies that support perception. Finally, in the POMDP model, the detected real road users and inferred phantom traffic participants are represented in the state space. The driving policies are obtained by using the anytime Monte Carlo tree search (MCTS) algorithm to solve the POMDP model online. In various simulation scenarios with static and dynamic obstacles in an urban environment, the proposed approach is compared to the baseline approach. Our planner successfully uses the uncertain objects tracking information from traffic mirrors and provides safer and more efficient driving policies.

Keywords: Reinforcement Learning, Collision Avoidance, Vulnerable Road-User Safety
Abstract: Reliable pedestrian crash avoidance mitigation (PCAM) systems are crucial components of safe autonomous vehicles (AVs). The nature of the vehicle-pedestrian interaction where decisions of one agent directly affect the other agent's optimal behavior, and vice versa, is a challenging yet often neglected aspect of such systems. We address this issue by modeling a Markov decision process (MDP) for a simulated AV-pedestrian interaction at an unmarked crosswalk. The AV's PCAM decision policy is learned through deep reinforcement learning (DRL). Since modeling pedestrians realistically is challenging, we compare two levels of intelligent pedestrian behavior. While the baseline model follows a predefined strategy, our advanced pedestrian model is defined as a second DRL agent. This model captures continuous learning and the uncertainty inherent in human behavior, making the AV-pedestrian interaction a deep multi-agent reinforcement learning (DMARL) problem. We benchmark the developed PCAM systems according to the collision rate and the resulting traffic flow efficiency with a focus on the influence of observation uncertainty on the decision-making of the agents. The results show that the AV is able to completely mitigate collisions under the majority of the investigated conditions and that the DRL pedestrian model learns an intelligent crossing behavior.

Keywords: Automated Vehicles, Self-Driving Vehicles, Intelligent Vehicle Software Infrastructure
Abstract: One big challenge regarding the development of highly automated driving (HAD) functions is validation and, in particular, providing proof of the desired functionality in any given scenario. Especially, corner cases, representing atypical, rare scenarios such as unexpected object movements are of high interest and thus must be detected to treat them with special attention. First, this paper presents a taxonomy for corner cases (CC) with focus on HAD. Specifically, so-called systemic corner cases (SCC) are introduced. Next, a feasibility study is presented on how these SCCs can be detected using different Machine Learning (ML) approaches for anomaly detection. We propose to use a hybrid ensemble of a One-Class Support Vector Machine (OCSVM) and a Clustering-Based Local Outlier Factor (CBLOF) incorporating domain knowledge to account for the nature of corner cases in timely correlated scenarios. The underlying data are unlabeled multivariate time series of HAD-system internal variables. Our experiments on both, synthetically generated and representative real-world CC, show that the hybrid ensemble can detect a variety of real corner cases, which allows for promising validation support of HAD functions.

Keywords: Image, Radar, Lidar Signal Processing, Recurrent Networks, V2X Communication
Abstract: Storing and transmitting LiDAR point cloud data is essential for many AV applications, such as training data collection, remote control, cloud services or SLAM. However, due to the sparsity and unordered structure of the data, it is difficult to compress point cloud data to a low volume. Transforming the raw point cloud data into a dense 2D matrix structure is a promising way for applying compression algorithms. We propose a new lossless and calibrated 3D-to-2D transformation which allows compression algorithms to efficiently exploit spatial correlations within the 2D representation. To compress the structured representation, we use common image compression methods and also a self-supervised deep compression approach using a recurrent neural network. We also rearrange the LiDAR's intensity measurements to a dense 2D representation and propose a new metric to evaluate the compression performance of the intensity. Compared to approaches that are based on generic octree point cloud compression or based on raw point cloud data compression, our approach achieves the best quantitative and visual performance. Source code and dataset are available at https://github.com/ika-rwth-aachen/Point-Cloud-Compression.

Keywords: Radar Sensing and Perception, Deep Learning, Vehicle Environment Perception
Abstract: This paper presents a method to learn the Cartesian velocity of objects using an object detection network on automotive radar data. The proposed method is self-supervised in terms of generating its own training signal for the velocities. Labels are only required for single-frame, oriented bounding boxes (OBBs). Labels for the Cartesian velocities or contiguous sequences, which are expensive to obtain, are not required. The general idea is to pre-train an object detection network without velocities using single-frame OBB labels, and then exploit the network's OBB predictions on unlabelled data for velocity training. In detail, the network's OBB predictions of the unlabelled frames are updated to the timestamp of a labelled frame using the predicted velocities and the distances between the updated OBBs of the unlabelled frame and the OBB predictions of the labelled frame are used to generate a self-supervised training signal for the velocities. The detection network architecture is extended by a module to account for the temporal relation of multiple scans and a module to represent the radars' radial velocity measurements explicitly. A two-step approach of first training only OBB detection, followed by training OBB detection and velocities is used. Further, a pre-training with pseudo-labels generated from radar radial velocity measurements bootstraps the self-supervised method of this paper. Experiments on the publicly available nuScenes dataset show that the proposed method almost reaches the velocity estimation performance of a fully supervised training, but does not require expensive velocity labels. Furthermore, we outperform a baseline method which uses only radial velocity measurements as labels.

Keywords: Automated Vehicles, Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: There is consensus across the automotive industry that Automated Driving Systems and automated vehicles challenge the way how quality assurance and, particularly, testing must be performed. However, there is a lack of up-to-date empirical studies that substantiate this concern. We conducted interviews with several experts from industry and research to systematically identify challenges as well as improvement opportunities in methods and tools. We report in this paper on 31 challenges that we identified in the areas of scenario- and simulation-based testing, test automation, and test execution. One recurrent challenge expressed by many experts is the problem how to translate a desired condition to be tested into an executable scenario model. This is not alone a question of scripting the scenario, but also of considering a vehicle under test that might try to evade the desired test condition.

Keywords: Vision Sensing and Perception, Sensor and Data Fusion, Convolutional Neural Networks
Abstract: The RGB complementary metal-oxide-semiconductor (CMOS) sensor works within the visible light spectrum. Therefore it is very sensitive to environmental light conditions. On the contrary, a long-wave infrared (LWIR) sensor operating in 8-14 micrometer spectral band, functions independent of visible light.

In this paper, we exploit both visual and thermal perception units for robust object detection purposes. After delicate synchronization and (cross-) labeling of the FLIR dataset, this multi-modal perception data passes through a convolutional neural network (CNN) to detect three critical objects on the road, namely pedestrians, bicycles, and cars. After evaluation of RGB and infrared (thermal and infrared are often used interchangeably) sensors separately, various network structures are compared to fuse the data at the feature level effectively. Our RGB-thermal (RGBT) fusion network, which takes advantage of a novel entropy-based attention module (EBAM), outperforms the state-of-the-art network by 10% with 82.9% mAP.

Keywords: Convolutional Neural Networks, Sensor and Data Fusion, Lidar Sensing and Perception
Abstract: Robust environment perception for autonomous vehicles is a tremendous challenge, which makes a diverse sensor set with e.g. camera, lidar and radar crucial. In the process of understanding the recorded sensor data, 3D semantic segmentation plays an important role. Therefore, this work presents a pyramid-based deep fusion architecture for lidar and camera to improve 3D semantic segmentation of traffic scenes. Individual sensor backbones extract feature maps of camera images and lidar point clouds. A novel Pyramid Fusion Backbone fuses these feature maps at different scales and combines the multimodal features in a feature pyramid to compute valuable multimodal, multi-scale features. The Pyramid Fusion Head aggregates these pyramid features and further refines them in a late fusion step, incorporating the final features of the sensor backbones. The approach is evaluated on two challenging outdoor datasets and different fusion strategies and setups are investigated. It outperforms recent range view based lidar approaches as well as all so far proposed fusion strategies and architectures.

Keywords: Driver State and Intent Recognition, Deep Learning, Automated Vehicles
Abstract: Methods to estimate a driver's visual attention from video images have received increased research interest. Such methods are especially important for detecting inattentive drivers in partially automated vehicles. The current study compares different driver gaze region estimation techniques, which may serve as a basis for detecting inattentive drivers. The accuracy of these techniques was evaluated on data from automated drives in a driving simulator. The examined techniques include a classical, state-of-the-art eye tracking approach, two data-driven approaches that rely on eye tracking data, a data-driven approach that only considers the driver's facial configuration, and an end-to-end approach based on a convolutional neural network. The results showcase the advantages of data-driven approaches over a classical geometric interpretation of the eye tracking data. The results also highlight challenges regarding generalization for purely data-driven approaches and the benefits of data-driven approaches that operate on eye tracking data rather than video image data alone.

Keywords: Automated Vehicles, Vehicle Control
Abstract: Motion comfort is the basis of many societal benefits promised by automated driving and motion planning is primarily responsible for this. By planning the spatial trajectory and the velocity profile, motion planners can significantly enhance motion comfort, ideally without sacrificing time efficiency. Active suspensions can push the boundary further by enabling additional degrees of freedom in the controllable vehicle motions. In this paper, we propose to integrate the planning of roll motion into an optimization-based motion planning algorithm called 3DOP (3 Degrees-of-Freedom Optimal Planning), where the conflicting objectives of comfort and time efficiency are optimized. The feasibility of the planned motion is verified in a realistic simulation environment, where feedforward-proportional control suffices to track the speed, path, and roll references. The proposed scheme achieves a significant reduction of motion discomfort, namely by up to 28.1% over the variant without controllable roll motion, or up to 34.2% over an acceleration-bounded driver model. The results suggest considerable potential for improving motion comfort by equipping automated vehicles with active suspensions.

Keywords: Lidar Sensing and Perception, Automated Vehicles, Sensor and Data Fusion
Abstract: 3D object detection is the task of locating and classifying objects in a 3D space. The task of 3d object detection is accompanied by the problem that objects are often detected while they are reversing, introducing a directional error of 180 degrees. This has a negative impact on following tasks of tracking and motion forecasting. One approach to solving this problem is by using a high definition (HD) map based on the information that cars drive along lanes; however, in its current form, this method does not fully utilize the given lane information and remains incapable of solving the problem of objects reversing. We propose a 3D object detection framework ("LaneFusion") employing LiDAR and HD map fusion, using a vector map. LaneFusion overcomes the problem that the vector map format is difficult to input into current mainstream convolutional neural networks (CNNs), through a two-step rasterization process that incorporates vector map features into existing LiDAR-based detection methods. Our experiments confirmed that the proposed method increased the 3D average precision (AP) and average orientation similarity (AOS) of the vehicle class by up to 6.56 and 10.65 points, respectively. In addition, we analyzed the performance degradation caused by map input errors due to self-localization estimation and deviations from real road conditions. The proposed method was found to be more sensitive to orientation errors than to translation errors in self-localization, yet robust to the unavailability of map information by dropout during training.

Keywords: Vehicle Environment Perception, Vision Sensing and Perception, Deep Learning
Abstract: Selective attention to specific areas and specific targets according to driving intention is of great significance for autonomous vehicles to efficiently obtain external environment information. In order to achieve efficient environmental perception, we propose an intention-driven visual attention selection model by simulating human active perception of the external environment. Meanwhile, in order to improve the integrity of the target category attention heatmap, a deep network training method with feature region enhancement is proposed. In this paper, FIMF Score-CAM which can fast integrate multiple features of local space is proposed. It generates intention-related target attention map by weighting the feature map extracted by forward convolution calculation, and combines spatial attention and feature attention to improve the ability of target category location. At the same time, the network is forced to pay more attention to the more comprehensive target-related region by using the guided random erasing in training process, which overcomes the deficiency that the model only pays attention to the most discriminative feature region, and achieves the purpose of feature region enhancement. Experiments on KITTI dataset show that the positioning integrity and accuracy of our model are significantly improved compared with other top-down attention models.

Keywords: Deep Learning, Self-Driving Vehicles, Vision Sensing and Perception
Abstract: Monocular 3D object detection is an essential task for infrastructure-less autonomous navigation and driving due to its low cost. Most previous state-of-the-art monocular 3D object detection methods depended on Convolutional Neural Networks (CNNs). We show that this reliance on CNNs is not necessary and a Transformer-based method can also perform very well. In this paper, we present the development of a new and general framework DST3D to predict a 3D bounding box for each object based on DLA-Swin Transformer in an end-to-end fashion without any pre-trained network for depth estimation. As a second contribution, we propose an object-scale adaptive Gaussian Kernel for generating ground truth keypoint heatmap, which associate the keypoint of the object with its size and help to enhance the network's performance. In addition, while regressing 3D variables, we introduce a double predictions dropout loss, which significantly improves both training consistence and detection accuracy. All of these make our framework simple yet efficient. Compared with all state-of-the-art CNNs-based methods, our proposed DST3D network achieves comparative performance on challenging KITTI benchmark with a faster speed, giving the top results on both 3D object detection and Bird’s Eye View evaluation.

Keywords: Vulnerable Road-User Safety, Driver State and Intent Recognition, Deep Learning
Abstract: Pedestrian crossing prediction has been a topic of active research, resulting in many new algorithmic solutions. While measuring the overall progress of those solutions over time tends to be more and more established due to the new publicly available benchmark and standardized evaluation procedures, knowing how well existing predictors react to unseen data remains an unanswered question. This evaluation is imperative as serviceable crossing behavior predictors should be set to work in various scenarios without compromising pedestrian safety due to misprediction. To this end, we conduct a study based on direct cross-dataset evaluation. Our experiments show that current state-of-the-art pedestrian behavior predictors generalize poorly in cross-dataset evaluation scenarios, regardless of their robustness during a direct training-test set evaluation setting. In the light of what we observe, we argue that the future of pedestrian crossing prediction, e.g. reliable and generalizable implementations, should not be about tailoring models, trained with very little available data, and tested in a classical train-test scenario with the will to infer anything about their behavior in real life. It should be about evaluating models in a cross-dataset setting while considering their uncertainty estimates under domain shift.

Keywords: Advanced Driver Assistance Systems, Information Fusion
Abstract: GNSS (Global Navigation Satellite System) is virtually becoming an autonomous train localization technology for the next-generation train control system. However, potential threats from the intentional interference may severely degrade the availability of GNSS due to its vulnerability. It is of great significance to detect and isolate the negative effects from GNSS interference for the Train Control System (TCS) in the railway field. For the protection against GNSS jamming, extra information from the Inertial Navigation System (INS) and odometer are involved, and an INS/odometer/trackmap-aided GNSS localization method for railway trains is raised in this paper. While the GNSS receiver cannot identify the real signals under a high-power jamming attack condition, a prediction deduced train position generation approach is proposed. In this strategy, velocity from the odometer and the geospatial constraint from the trackmap are involved to calibrate INS, with which continuous positioning is realized under a GNSS-denied situation. Furthermore, while the measurements degradation occurs caused by a relatively low power jamming, a residual-test-based detection solution based on the deviation between the predicted reference pseudo-ranges and the real ones is proposed to isolate degraded measurements. Results from an experiment under a GPS jamming condition demonstrate that the proposed solution outperforms the GPS Single Point Positioning (SPP) and the conventional GPS/INS method. The jamming protection and continuous positioning performance under specific jamming conditions enhance the capability of resilient train positioning.

Keywords: Cooperative Systems (V2X), Reinforcement Learning, Deep Learning
Abstract: In this paper, a cooperative adaptive cruise control (CACC) system is presented with integrated lidar and vehicle-to-vehicle (V2V) communication. Firstly, an adaptive cruise control system (ACC) is designed for the Q-Car electrical vehicle, an autonomous car. Secondly, a CACC system and V2V communication are designed based on a new algorithm to improve the ACC system performance. Lastly, the CACC agent was trained by Deep Q learning (DQN) and tested. The proposed CACC system improved the stability of the vehicle. Experimental results demonstrate that the CACC system can decrease the average inter-vehicular distance of ACC by 44.74%, with an additional 40.19% when DQN was utilized. The vehicles communicate with each other through a WiFi module to transmit information with 1ms latency.

Keywords: Deep Learning, Vision Sensing and Perception, Vulnerable Road-User Safety
Abstract: We present a multitask network that supports various deep neural network based pedestrian detection functions. Besides 2D and 3D human pose, it also supports body and head orientation estimation based on full body bounding box input. This eliminates the need for explicit face recognition. We show that the performance of 3D human pose estimation and orientation estimation is comparable to the state-of-the-art. Since very few data sets exist for 3D human pose and in particular body and head orientation estimation based on full body data, we further show the benefit of particular simulation data to train the network. The network architecture is relatively simple, yet powerful, and easily adaptable for further research and applications.

Keywords: Situation Analysis and Planning, Self-Driving Vehicles, Automated Vehicles
Abstract: Autonomous vehicles must comply with traffic rules. However, most motion planners do not explicitly consider all relevant traffic rules. Once traffic rule violations of an initially-planned trajectory are detected, there is often not enough time to replan the entire trajectory. To solve this problem, we propose to repair the initial trajectory by investigating the satisfiability modulo theories paradigm. This framework makes it efficient to reason whether and how the trajectory can be repaired and, at the same time, determine the part along the trajectory that can remain unchanged. Moreover, the robustness of traffic rule satisfaction is used to formulate a convex optimization problem for generating rule-compliant trajectories. We compare our approach with trajectory replanning and demonstrate its usefulness with traffic scenarios from the CommonRoad benchmark suite and recorded data. The evaluation result shows that rule-compliant trajectory repairing is computationally efficient and widely applicable.

Keywords: Autonomous / Intelligent Robotic Vehicles, Unsupervised Learning, Vehicle Environment Perception
Abstract: While supervised detection and classification frameworks in autonomous driving require large labelled datasets to converge, Unsupervised Domain Adaptation (UDA) approaches, facilitated by synthetic data generated from photo- real simulated environments, are considered low-cost and less time-consuming solutions. In this paper, we propose UDA schemes using adversarial discriminative and generative meth- ods for lane detection and classification applications in au- tonomous driving. We also present Simulanes dataset generator to create a synthetic dataset that is naturalistic utilizing CARLA’s vast traffic scenarios and weather conditions. The proposed UDA frameworks take the synthesized dataset with labels as the source domain, whereas the target domain is the unlabelled real-world data. Using adversarial generative and feature discriminators, the learnt models are tuned to predict the lane location and class in the target domain. The proposed techniques are evaluated using both real-world and our synthetic datasets. The results manifest that the proposed methods have shown superiority over other baseline schemes in terms of detection and classification accuracy and consistency. The ablation study reveals that the size of the simulation dataset plays important roles in the classification performance of the proposed methods. Our UDA frameworks are available at github.com/anita-hu/sim2real-lane-detection and our dataset generator is released at github.com/anita-hu/simulanes.

Keywords: Self-Driving Vehicles, V2X Communication, Driver Recognition
Abstract: Communication between vehicles with varying degrees of automation is increasingly challenging as highly automated vehicles are unable to interpret the non-verbal signs of other road users. The lack of understanding on roads leads to lower trust in automated vehicles and impairs traffic safety. To address these problems, we propose the Road User Communication Service, a software as a service platform, which provides information exchange and cloud computing services for vehicles with varying degrees of automation. To inspect the operability of the proposed solution, field tests were carried out on a test track, where the autonomous JKU-ITS research vehicle requested the state of a driver in a manually-controlled vehicle through the implemented service. The test results validated the approach showing its feasibility to be used as a communication platform. A link to the source code is available.

Keywords: Electric and Hybrid Technologies, Reinforcement Learning, Deep Learning
Abstract: The deep reinforcement learning-based energy management strategies (EMS) have become a promising solution for hybrid electric vehicles (HEVs). When driving cycles are changed, the neural network will be retrained, which is a time-consuming and laborious task. A more efficient way of choosing EMS is to combine deep reinforcement learning (DRL) with transfer learning, which can transfer knowledge of one domain to the other new domain, making the network of the new domain reach convergence values quickly. Different exploration methods of DRL, including adding action space noise and parameter space noise, are compared against each other in the transfer learning process in this work. Results indicate that the network added parameter space noise is more stable and faster convergent than the others. In conclusion, the best exploration method for transferable EMS is to add noise in the parameter space, while the combination of action space noise and parameter space noise generally performs poorly. Our code is available at https://github.com/BIT-XJY/RL-based-Transferable-EMS.git.

Keywords: Automated Vehicles, Reinforcement Learning
Abstract: In a multi-agent system (MAS), a social learning scheme allows independent agents to learn through interactions with agents randomly selected from a pool. Such a scheme is important for autonomous vehicles (AV) to navigate complex traffic environments consisting of many road users. In this paper, we apply the social learning scheme to Markov games and leverage deep reinforcement learning (DRL) to investigate how individual AVs learn policies and form social norms in traffic scenarios. To capture agents' different attitudes toward traffic environments, a heterogeneous agent pool with cooperative and defective AVs is introduced to the social learning scheme. To solve social norms formed by AVs, we propose a DRL algorithm, and apply them to traffic scenarios: unsignalized intersection and highway platoon. We find that compared to defective AVs, cooperative AVs can easily conform to expected social norms. In addition, cooperative AVs would lead to lower crash rates. We also find that prioritized roads/lanes can make AVs conform to expected social norms.

Keywords: Deep Learning, Self-Driving Vehicles, Unsupervised Learning
Abstract: Clustering traffic scenarios and detecting novel scenario types are required for scenario-based testing of autonomous vehicles. These tasks benefit from either good similarity measures or good representations for the traffic scenarios. In this work, an expert-knowledge aided representation learning for traffic scenarios is presented. The latent space so formed is used for successful clustering and novel scenario type detection. Expert-knowledge is used to define objectives that the latent representations of traffic scenarios shall fulfill. It is presented, how the network architecture and loss is designed from these objectives, thereby incorporating expert-knowledge. An automatic mining strategy for traffic scenarios is presented, such that no manual labeling is required. Results show the performance advantage compared to baseline methods. Additionally, extensive analysis of the latent space is performed.

Keywords: Deep Learning, Unsupervised Learning, Intelligent Vehicle Software Infrastructure
Abstract: It is necessary to calibrate the hydraulic pressure of the shift control to develop an automatic transmission (AT), and this calibration process entails a subjective shift quality assessment by experienced engineers. An objective shift quality assessment methodology has been explored for a long time to replace the engineer. The most recent data-based assessment model has attained a nearly human-like performance. However, preparing the large number of data labels required for supervised learning of the model has limitations. This study proposes an unsupervised anomaly detection model for objective shift quality assessment to address data label shortages and high data labeling costs. The proposed anomaly detection model is trained to classify a normal shift and an abnormal shift using just normal shift data. It is possible to easily obtain many train datasets from ordinary vehicles, and data labeling is not required. On the basis of real vehicle shift data, multiple anomaly detection models composed of various deep neural networks are developed and assessed. The evaluation results show that training exclusively on normal shift data can detect abnormal shifts; the best area under receiver operating characteristic curve is 0.902.

Keywords: Radar Sensing and Perception
Abstract: This paper explores object detection on radar range-Doppler map. Most of the radar processing algorithms are proposed for detecting objects without classifying. Meanwhile, these approaches neglect the useful information available in the temporal domain. To address these problems, we propose an online radar deep temporal detection framework by frame-to-frame prediction and association with low computation. The core idea is that once an object is detected, its location and class can be predicted in the future frame to improve detection results. The experiment results illustrate this method achieves better detection and classification performance, and shows the usability of radar data for traffic scenes.

Keywords: Active and Passive Vehicle Safety, Convolutional Neural Networks, Automated Vehicles
Abstract: Real-time safety systems are crucial components of intelligent vehicles. This paper introduces a prediction-based collision risk assessment approach on highways. Given a point mass vehicle dynamics system, a stochastic forward reachable set considering two-dimensional motion with vehicle state probability distributions is firstly established. We then develop an acceleration prediction model, which provides multimodal probabilistic acceleration distributions to propagate the vehicle state uncertainties. The collision probability is calculated by summing up the probabilities of the states where two vehicles spatially overlap. Simulation results show that the prediction model has superior performance in terms of vehicle motion position errors, and the proposed collision detection approach is agile and effective to identify the collision in cut-in crash events.

Keywords: Advanced Driver Assistance Systems, Collision Avoidance
Abstract: In order to be able to cope with the increasing amount of testing on the path to autonomous driving, the number of test kilometers required with the real vehicle must be drastically reduced. A crucial tool for achieving this challenging goal is simulation. For an effective test of a driving function in virtual environments, realistic scenarios are necessary. In this paper a method is presented how relevant test cases can be generated. Furthermore, an approach is proposed how the parameterization of a driving can be performed in virtual driving tests. First, scenarios are selected from a traffic flow simulation in which a collision occurs or a vehicle brakes strongly. These traffic situations are used to test the driving function with an empirically selected parameterization. Based on the performance that the driving function shows in these scenarios, characteristic scenarios are chosen. This reduced set of test scenarios is used to determine the optimal parameterization. The procedure is shown exemplarily for an emergency brake function. Finally, the result of the optimization process is evaluated based on the frequency distribution of the performance values.

Keywords: Advanced Driver Assistance Systems, Driver State and Intent Recognition, Driver Recognition
Abstract: Drowsy drivers cause many deadly crashes. As a result, researchers focus on using driver drowsiness classifiers to predict this condition in advance. However, they only consider constraint situations. Under highly unrestricted scenarios, this categorization remains extremely difficult. For example, several studies consider the driver's mouth closure crucial for detecting drowsiness. However, the mouth closure cannot be seen when the driver wears a mask, which is a potential failure for these classifiers. Moreover, these works do not make experiments under unconstrained situations as environments with considerable light variation or a driver with eyeglasses reflections. As a result, this paper proposes a video-based novel pipeline that employs new parameters, computer vision and deep-learning techniques to identify drowsiness in drivers under unconstrained situations. First, we alter the Lab color space of the frame to ease strong light changes. Then, we achieve a robust recognition of the face, eyes and body-joints landmarks using dense landmark detection that includes optical flow estimation methods for 3D eyelid and facial expression movement tracking and an online optimization framework to build the association of cross-frame poses. After this, we consider three important landmarks: eyes, lower-face contour, and chest. We performed several pre-processing and combinations using these landmarks to compare the efficiency of three alternative feature vectors. Finally, we fuse spatio-temporal features using a Gated Recurrent Units (GRU) model. Results over a dataset with highly unconstrained driving conditions demonstrate that our method outperforms classifying the driver's drowsiness correctly in various challenging situations, all under mask-wearing scenarios.

Keywords: Autonomous / Intelligent Robotic Vehicles, Vehicle Control
Abstract: In navigation activities, kinematic parameters of a mobile vehicle play a significant role. Odometry is most commonly used for dead reckoning. However, the unrestricted accumulation of errors is a disadvantage using this method. As a result, it is necessary to calibrate odometry parameters to minimize the error accumulation. This paper presents a pipeline based on sequential least square programming to minimize the relative position displacement of an arbitrary landmark in consecutive time steps of a kinematic vehicle model by calibrating the parameters of applied model. Results showed that the developed pipeline produced accurate results with small datasets.

Keywords: Vulnerable Road-User Safety
Abstract: Modelling pedestrian behavior is crucial in the development and testing of autonomous vehicles. In this work, we present a hierarchical pedestrian behavior model that generates high-level decisions through the use of behavior trees, in order to produce maneuvers executed by a low-level motion planner using an adapted Social Force model. A full implementation of our work is integrated into GeoScenario Server, a scenario definition and execution engine, extending its vehicle simulation capabilities with pedestrian simulation. The extended environment allows simulating test scenarios involving both vehicles and pedestrians to assist in the scenario-based testing process of autonomous vehicles. The presented hierarchical model is evaluated on two real-world data sets collected at separate locations with different road structures. Our model is shown to replicate the real-world pedestrians' trajectories with a high degree of fidelity and a decision-making accuracy of 98% or better, given only high-level routing information for each pedestrian.

Keywords: Autonomous / Intelligent Robotic Vehicles, Collision Avoidance, Self-Driving Vehicles
Abstract: With the increasing emphasis on the safe autonomy for robots, model-based safe control approaches such as Control Barrier Functions have been extensively studied to ensure guaranteed safety during inter-robot interactions. In this paper, we introduce the Parametric Control Barrier Function (Parametric-CBF), a novel variant of the traditional Control Barrier Function to extend its expressivity in describing different safe behaviors among heterogeneous robots. Instead of assuming cooperative and homogeneous robots using the same safe controllers, the ego robot is able to model the neighboring robots' underlying safe controllers through different Parametric-CBFs with observed data. Given learned parametric-CBF and proved forward invariance, it provides greater flexibility for the ego robot to better coordinate with other heterogeneous robots with improved efficiency while enjoying formally provable safety guarantees. We demonstrate the usage of Parametric-CBF in behavior prediction and adaptive safe control in the ramp merging scenario from the applications of autonomous driving. Compared to traditional CBF, Parametric-CBF has the advantage of capturing varying drivers' characteristics given richer description of robot behavior in the context of safe control. Numerical simulations are given to validate the effectiveness of the proposed method.

Keywords: Cooperative Systems (V2X), Self-Driving Vehicles, Automated Vehicles
Abstract: The move towards connected and autonomous vehicles (CAVs) has gained a strong focus in recent years due to the many benefits they provide. While the autonomous aspect has seen substantial advancement in both the development and testing methodologies, the connected aspect has lagged behind, especially in the verification and validation (V&V) discussions. Integrating connectivity into the development and testing framework for CAVs is a necessity for ensuring the early deployment of cooperative driving systems. A key element within such a framework is a test scenario, which represents a set of scenery, environmental conditions, and dynamic conditions, that a system needs to be tested in. However, the connectivity element is not present in any of the current state of the art scenario description languages (SDLs) that are publicly available. This leaves a gap within the CAV development ecosystem. To accommodate for, and accelerate the development of, connected vehicle systems and their verification and validation methods, this paper proposes a novel V2X extension to the previously published two-level abstraction SDL. The extension enables communications between vehicles, infrastructures, and further additional entities to be specified as part of the scenario and be subsequently tested in virtual testing or real-world testing. Eight new V2X attributes have been added to the SDL. An example set of syntax and semantic definitions are presented in this paper targeting two different abstraction levels – level 1 aims at the abstract scenario level for non-technical end-users such as regulators, and level 2 aims at the logical and concrete scenario level for end-users such as simulation test engineers.

Keywords: Automated Vehicles, Hand-off/Take-Over, Active and Passive Vehicle Safety
Abstract: When developing truly driverless mobility for the future, one key index used to measure the matureness of a particular self-driving technology is the driver intervention rate. One method which has proven to be effective for decreasing intervention rates is the use of personalized driving models that can mimic the driving style and preferences of a targeted user, so that autonomous driving feels safer and more natural to them. To create such models, quantitative data should be collected from users in order to determine the style of driving that a particular user, or type of user, prefers. In this paper, we introduce the Subjective Risk Lane Change (SRLC) Dataset, which includes ego vehicle driving behavior data, surrounding vehicle location information, and the subjective risk scores of users, collected during both safe and risky lane change scenarios encountered in CARLA simulators, as well as demographic information for our 30 participants. Furthermore, user intervention data for all of our participants was collected from Personalized Model Predictive Controllers during the generated lane change maneuvers. As far as the authors are able to determine, no other public dataset provides driving behavior signal and intervention timing information collected during driver interventions. Our dataset can be used to gain insights into a variety of personal driving styles, allowing the improvement of adaptive autonomous driving systems, and leading to safer and more widely accepted driverless technology.

Keywords: Human-Machine Interface
Abstract: Although partially autonomous driving (AD) systems are already available in production vehicles, drivers are still required to maintain a sufficient level of situational awareness (SA) during driving. Previous studies have shown that providing information about the AD's capability using user interfaces can improve the driver's SA. However, displaying too much information increases the driver's workload and can distract or overwhelm the driver. Therefore, to design an efficient user interface (UI), it is necessary to understand its effect under different circumstances. In this paper, we focus on a UI based on augmented reality (AR), which can highlight potential hazards on the road. To understand the effect of highlighting on drivers' SA for objects with different types and locations under various traffic densities, we conducted an in-person experiment with 20 participants on a driving simulator. Our study results show that the effects of highlighting on drivers' SA varied by traffic densities, object locations and object types. We believe our study can provide guidance in selecting which object to highlight for the AR-based driver-assistance interface to optimize SA for drivers driving and monitoring partially autonomous vehicles.

Keywords: Traffic Flow and Management, Cooperative ITS, Smart Infrastructure
Abstract: Intelligent intersection management aims to schedule vehicles so that vehicles can pass through an intersection efficiently and safely. However, inaccurate control, imperfect communication, and malicious information or behavior lead to robustness issues of intelligent intersection management. In this work, we focus on improving robustness against deadlocks by changing the trajectories of vehicles. To guarantee the resolvability of deadlocks, we limit the number of vehicles in an intersection to be smaller than or equal to an intersection-specific value called the maximal deadlock-free load. We develop an algorithm to compute the maximal deadlock-free load. We further reduce the computation time by computing the loads which are pessimistic (smaller) but still deadlock-free. Since the maximal deadlock-free load only depends on the given intersection, it can be integrated with different scheduling algorithms. Experimental results demonstrate that, by changing the trajectories of vehicles and limiting the number of vehicles under maximal deadlock-free loads, our approach can guarantee deadlock-freeness and maintain good traffic efficiency.

Keywords: Automated Vehicles, Security
Abstract: One essential goal for Highly Automated Vehicles (HAVs) safety test is to assess their risk rate in naturalistic driving environment, and to compare their performance with human drivers. The probability of exposure to risk events is generally low, making the test process extremely time-consuming. To address this, we proposed a surrogate-based method in scenario-based simulation test to expediate the assessment of the risk rate of HAVs. HighD data were used to fit the naturalistic distribution and to estimate the probability of each concrete scenario. Machine learning model-based surrogates were proposed to quickly approximate the test result of each concrete scenario. Considering the different capabilities and domains of various surrogate models, we applied six surrogate models to search for two types of targeted scenarios with different risk levels and rarity levels. We proved that the performances of different surrogate models greatly distinguish from each other when the target scenarios are extremely rare. Inverse Distance Weighted (IDW) was the most efficient surrogate model, which could achieve risk rate assessment with only 2.5% test resources. The required CPU runtime of IDW was 2% of that required by Kriging. The proposed method has great potential in accelerating the risk assessment of HAVs.

Keywords: Automated Vehicles, Vision Sensing and Perception, Vehicle Environment Perception
Abstract: Visual-based perception systems are widely used in autonomous vehicles (AVs). In severe weather conditions, hazardous events of AVs may be induced by the performance limitations of perception system. We propose a staged analyzing method to quantitatively evaluate the performance limitations of visual-based perception system under severe weather conditions and explore the influence mechanism. In our method, the working process of visual-based perception systems is divided into two stages of image obtaining by camera and target recognition by recognition algorithm. Firstly, in image obtaining stage, the quality of images obtained in scenarios with different weather types and intensity is evaluated using monofactor analysis method. The relationship between different weather and metrics of image quality is analyzed. Secondly, in target recognition stage, metrics values of image quality and recognition results are fitted with (weighted) multiple linear regression model, and a regression model representing the influence relationship is acquired. Finally, the importance of indicators in image quality metrics is verified with BP neural network, and the performance of the regression model is analyzed with the results acquired in two example scenarios. With the obtained monofactor analysis results and the regression model, the influence mechanisms of high luminance and fog conditions are analyzed and compared, which shows the effectiveness of the method in performance limitation and its influence mechanism analysis.

Keywords: Lidar Sensing and Perception, Deep Learning, Vehicle Environment Perception
Abstract: To train a well performing neural network for semantic segmentation, it is crucial to have a large dataset with available ground truth for the network to generalize on unseen data. In this paper we present novel point cloud augmentation methods to artificially diversify a dataset. Our sensor-centric methods keep the data structure consistent with the lidar sensor capabilities. Due to these new methods, we are able to enrich low-value data with high-value instances, as well as create entirely new scenes. We validate our methods on multiple neural networks with the public SemanticKITTI dataset and demonstrate that all networks improve compared to their respective baseline. In addition, we show that our methods enable the use of very small datasets, saving annotation time, training time and the associated costs.

Keywords: Mapping and Localization, Vehicle Environment Perception, Active and Passive Vehicle Safety
Abstract: We present a procedure to create high definition maps of lanes based on detected and tracked vehicles from perception sensor data as well as the ego vehicle using multiple observations of the same location. The procedure consists of two parts. First, an aggregation part in which the detected and tracked road users as well as the driving path of the ego vehicle are aggregated into a map representation. Second, lanes are extracted from those maps as lane center lines in a structured data format. The final lane centers are represented in a directed graph representation including the driving direction. They are accurate up to a few centimeters. Our procedure is not restricted to any environment and does not rely on any prior map information. In our experiments with real world data and available ground truth, we study the performance of different map aggregations e.g., based on the ego vehicle only or based on other road users. Furthermore, we study the dependence on the number of data recording repetitions.

Keywords: Automated Vehicles, Situation Analysis and Planning, Self-Driving Vehicles
Abstract: Ensuring safety is arguably one of the largest remaining challenges before wide-spread market adoption of Automated Driving Systems (ADSs). One central aspect is how to provide evidence for the fulfilment of the safety claims and, in particular, how to produce a predictive and reliable safety case considering both the absence and the presence of faults in the system. In order to provide such evidence, there is a need for describing and modelling the different elements of the ADS and its operational context: models of event exposure, sensing and perception models, as well as actuation and closed-loop behaviour representations. This paper explores how estimates from such statistical models can impact the performance and operation of an ADS and, in particular, how such models can be continuously improved by incorporating more field data retrieved during the operation of (previous versions of) the ADS. Focusing on the safe driving velocity, this results in the ability to update the driving policy so to maximise the allowed safe velocity, for which the safety claim still holds. For illustration purposes, an example considering statistical models of the exposure to an adverse event, as well as failures related to the system's perception system, is analysed. Estimations from these models, using statistical confidence limits, are used to derive a safe driving policy of the ADS. The results highlight the importance of leveraging field data in order to improve the system's abilities and performance, while remaining safe. The proposed methodology, leveraging a data-driven approach, also shows how the system's safety can be monitored and maintained, while allowing for incremental expansion and improvements of the ADS.

Keywords: Advanced Driver Assistance Systems, Telematics, Novel Interfaces and Displays
Abstract: Remote driving of vehicles using in-vehicle camera videos is essential for the uncrewed operation of automated vehicles. A remote operator views the in-vehicle camera videos to understand the situation around the vehicle and sometimes performs remote driving. Therefore, it is essential to satisfy three in-vehicle camera video transmission requirements: high image quality, low latency, and high reliability. Sub-6 and mm-Wave 5G networks are capable of high throughput and low latency video transmission and attract attention as a means of in-vehicle camera video transmission. However, 5G networks with high frequencies have narrower coverage than the existing 4G networks and are easily affected by object blockage. Therefore, cases of a sudden deterioration in network quality are likely to occur. Therefore, using 5G networks to transmit in-vehicle camera videos requires coping with sudden network quality deterioration. This paper proposes an in-vehicle camera image transmission system using 5G and 4G networks. In the proposed system, when the system detects a disturbance in the video transmitted via the 5G networks, it switches the display to the video transmitted via the 4G networks. When the 5G networks are available, the system displays the video with high quality and low latency. When unavailable, the system switches to the video transmitted via the 4G networks. We have confirmed on the network simulator that the proposed system has a shorter latency than the video transmission system using a jitter buffer to ensure reliability. In a video transmission experiment using commercial Sub-6 5G networks, we confirmed that the proposed system could detect all video disturbances and avoid the displayed video's disturbance by switching the video.

Keywords: Cooperative Systems (V2X)
Abstract: Connected vehicle (CV) applications are an emerging technology that promises to revolutionize our transportation system. They can improve safety and efficiency of transportation systems while reducing their environmental footprints. A large number of CV applications have been proposed towards these goals, with the US Department of Transportation (USDOT) recently initiating three deployment sites. However, the resilience of these protocols has not been considered carefully. Due to the fact that these protocols may affect the driver behavior or even control of the vehicle, vulnerabilities in terms of cyber security can lead to breakdowns in safety (causing accidents), efficiency (causing congestion and reducing capacity), sustainability (inducing excessive energy consumption and pollutant emissions), or even social equity (gaining priorities for passage at intersections by cheating). To avoid or mitigate such consequences, it is essential to develop attack detection and mitigation techniques for CVs. In this paper, we present a novel attack detection framework to identify application layer attacks against CVs. We also carry out analysis of the impact due to the attacks, showing that an individual attacker can have substantial effects on the safety and efficiency of traffic flow even in the presence of message security standards developed by USDOT. The paper explores the evaluation efficiency of the defense framework to mitigate these classes of attacks in different CV applications.

Keywords: Automated Vehicles, Collision Avoidance
Abstract: This paper proposes a convolution-based method for reconfiguring highly constrained environments, which considers the contour and heading of an autonomous vehicle. The vehicle with possible different heading angles is taken as the kernels. The multiple convolutions between the kernels and the environment are performed to generate a three-dimensional grid map, which significantly improves the computational efficiency of the collision detection algorithm. Moreover, a hierarchical and multistage trajectory planning method based on the reconfigured grid map is proposed. The superiority of the proposed method is verified by comparative simulations and real-time experiments.