Keywords: Automated Vehicles, Reinforcement Learning, Recurrent Networks
Abstract: In this paper, we present a safe deep reinforcement learning system for automated driving. The proposed framework leverages merits of both rule-based and learning-based approaches for safety assurance. Our safety system consists of two modules namely handcrafted safety and dynamically-learned safety. The handcrafted safety module is a heuristic safety rule based on common driving practice that ensure a minimum relative gap to a traffic vehicle. On the other hand, the dynamically-learned safety module is a data-driven safety rule that learns safety patterns from driving data. Specifically, the dynamically-leaned safety module incorporates a model lookahead beyond the immediate reward of reinforcement learning to predict safety longer into the future. If one of the future states leads to a near-miss or collision, then a negative reward will be assigned to the reward function to avoid collision and accelerate the learning process. We demonstrate the capability of the proposed framework in a simulation environment with varying traffic density. Our results show the superior capabilities of the policy enhanced with dynamically-learned safety module.

Keywords: Reinforcement Learning, Deep Learning, Vehicle Control
Abstract: In recent years, deep reinforcement learning (DRL) algorithms have been widely studied and utilized in the area of Intelligent Transportation Systems (ITS). DRL agents are mostly trained with transition pairs and interaction trajectories generated from simulation, and they can achieve satisfying or near optimal performances under familiar input states. However, for relative rare visited or even unvisited regions in the state space, there is no guarantee that the agent could perform well. Unfortunately, novel conditions are inevitable in real-world problems and there is always a gap between the real data and simulated data. Therefore, to implement DRL algorithms in real-world transportation systems, we should not only train the agent learn a policy that maps states to actions, but also the model uncertainty associated with each action. In this study, we adapt the method of Bayesian ensemble to train a group of agents with imposed diversity for an energy management system of a delivery vehicle. The agents in the ensemble agree well on familiar states but show diverse results on unfamiliar or novel states. This uncertainty estimation facilitates the implementation of interpretable postprocessing modules which can ensure robust and safe operations under high uncertainty conditions.

Keywords: Reinforcement Learning, Self-Driving Vehicles, Automated Vehicles
Abstract: Reinforcement learning (RL) can be used to create a tactical decision-making agent for autonomous driving. However, previous approaches only output decisions and do not provide information about the agent's confidence in the recommended actions. This paper investigates how a Bayesian RL technique, based on an ensemble of neural networks with additional randomized prior functions (RPF), can be used to estimate the uncertainty of decisions in autonomous driving. A method for classifying whether or not an action should be considered safe is also introduced. The performance of the ensemble RPF method is evaluated by training an agent on a highway driving scenario. It is shown that the trained agent can estimate the uncertainty of its decisions and indicate an unacceptable level when the agent faces a situation that is far from the training distribution. Furthermore, within the training distribution, the ensemble RPF agent outperforms a standard Deep Q-Network agent. In this study, the estimated uncertainty is used to choose safe actions in unknown situations. However, the uncertainty information could also be used to identify situations that should be added to the training process.

Keywords: Reinforcement Learning, Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: Deep Reinforcement Learning has proved to be able to solve many control tasks in different fields, but the behavior of these systems is not always as expected when deployed in real-world scenarios. This is mainly due to the lack of domain adaptation between simulated and real-world data and that of a train-test distinction that increase the risk of overfitting on the training scenarios. In this work, we investigate these problems in the autonomous driving field, especially for a maneuver planning module for roundabout insertions. In particular, we develop a system based on multiple environments in which agents are trained simultaneously, evaluating the behavior of the model in different scenarios. Finally, we develop techniques in order to reduce the gap between simulated and real-world data showing that this increases the generalization capabilities of the system both on unseen and real-world scenarios.

Keywords: Self-Driving Vehicles, Deep Learning, Reinforcement Learning

Keywords: Reinforcement Learning, Self-Driving Vehicles, Deep Learning
Abstract: Navigating through unsignalized intersections is one of the most challenging problems in urban environments for autonomous vehicles. Existing methods need to train specific policy models to deal with different tasks including going straight, turning left and turning right. In this paper we formulate intersection navigation as a multi-task reinforcement learning problem and propose a unified learning framework for all three navigation tasks at the intersections. We propose to represent multiple tasks with a unified four-dimensional vector, which elements mean common sub-task and three specified target sub-tasks respectively. Meanwhile, we design a vectorized reward function combining with deep Q-networks (DQN) to learn to handle multiple intersection navigation tasks concurrently. We train the agent to navigate through intersections by adjusting the speed of the ego vehicle under given route. Experimental results in both simulation and real-world vehicle test demonstrate that the proposed multi-task DQN algorithm outperforms baselines for all three navigation tasks in several different intersection scenarios.

Keywords: Reinforcement Learning, Automated Vehicles, Deep Learning
Abstract: Most reinforcement learning approaches used in behavior generation utilize vectorial information as input. However, this requires the network input to have a pre-defined input-size -- in semantic environments assuming a maximum number of vehicles. Additionally, this vectorial representation is not invariant to the order and number of vehicles. To mitigate the above-stated disadvantages, we propose combining graph neural networks with Actor-Critic reinforcement learning. % third point As graph neural networks apply the same network to every vehicle and aggregate incoming edge information, they are invariant to the number and order of vehicles. This makes them ideal candidates to be used as networks in semantic environments -- environments consisting of objects lists. Graph neural networks use relational information contained in the graph and do not have to implicitly infer these. Moreover, as graph neural networks have similar characteristics to traditional convolutional layers, they are not only able to propagate first-order, but also higher-degree information. We demonstrate our approach using a highway lane-change scenario and compare the performance of graph neural networks to traditional ones. Moreover, we show that graph neural networks are capable of handling a varying number and order of vehicles during training and application. To demonstrate the convolutional properties, we conduct ablation studies by varying the position of a single-vehicle.

Keywords: Reinforcement Learning, Self-Driving Vehicles, Deep Learning
Abstract: Several key scenarios, such as intersection navigation, lane changing, and ramp merging, are active areas of research in autonomous driving. In order to properly navigate these scenarios, autonomous vehicles must implicitly negotiate with human drivers. Prior work in driving behaviors presents reinforcement learning as a promising technique, as it can leverage data as well as the underlying decision-making structure of driving with interaction. We apply a hierarchical approach to decision-making, where we train a high-level policy using reinforcement learning, and execute the policy’s output on a low-level controller. This hierarchical structure helps increase the policy’s overall safety, and allows the learning component to be agnostic to the low-level control scheme. We validate our approach on a simulation using real-world highway data and find improved results compared to prior work in ramp merging.

Keywords: Unsupervised Learning, Sensor and Data Fusion, Lidar Sensing and Perception
Abstract: LiDAR sensors are crucial in automotive perception for accurate object detection. However, LiDAR data is hard to interpret for humans and consequently time-consuming to label. Whereas camera data is easy interpretable and thus comparably simpler to label. Within this work we present a transductive transfer learning approach to transfer the knowledge for the object detection task from images to point cloud data. We propose a multi-modal adversarial Auto Encoder architecture which disentangles uni-modal features into two groups: common (transferable) features, and complementary (modality-specific) features. This disentanglement is based on the hypothesis that a set of common features exist. An important point of our framework is that the disentanglement is learned in an unsupervised manner. Furthermore, the results show that only a small amount of multi-modal data is needed to learn the disentanglement, and thus to transfer the knowledge between modalities. As a result we our experiments show that training with 75% less data of the KITTI objects, the classification accuracy achieved is of 71.75%, only 3.12% less than when using the full data set. The implications of these findings can have great impact in perception pipelines based on LIDAR data.

Keywords: Smart Infrastructure, Unsupervised Learning, Sensor and Data Fusion
Abstract: The paper presents a method for audio-based vehicle counting (VC) in low-to-moderate traffic using one-channel sound. We formulate VC as a regression problem, i.e., we predict the distance between a vehicle and the microphone. Minima of the proposed distance function correspond to vehicles passing by the microphone. VC is carried out via local minima detection in the predicted distance. We propose to set the minima detection threshold at a point where the probabilities of false positives and false negatives coincide so they statistically cancel each other in total vehicle number. The method is trained and tested on a traffic-monitoring dataset comprising 422 short, 20-second one-channel sound files with a total of 1421 vehicles passing by the microphone. Relative VC error in a traffic location not used in the training is below 2% within a wide range of detection threshold values. Experimental results show that the regression accuracy in noisy environments is improved by introducing a novel high-frequency power feature.

Keywords: Sensor and Data Fusion, Advanced Driver Assistance Systems
Abstract: This work presents a Virtual Sensor for vehicle planar velocities and tire forces. The Virtual Sensor is based on an Extended Kalman Filter utilizing a vehicle model and onboard sensors. The Virtual Sensor is evaluated at different stages, starting from a model with lower complexity, i.e. a 3 degrees of freedom bicycle model, and evolving to higher complexity, i.e. a 10 degrees of freedom vehicle model with wheel suspension. The estimation results from the different stages are compared qualitatively as well as with experimental data. The evaluation brings forth the gains, limitations and drawbacks when increasing model complexity accompanied by sensor additions. This gives insight when a trade-off is required between accuracy and model complexity, e.g. in Advanced Driver Assistance Systems. The conducted analysis also clarifies performance dependencies on the sensors as well as sensor redundancies, which is of added value, e.g. for robustifying against sensor failures.

Keywords: Sensor and Data Fusion, Deep Learning, Image, Radar, Lidar Signal Processing
Abstract: Fusing the raw data from different automotive sensors for real-world environment perception is still challenging due to their different representations and data formats. In this work, we propose a novel method termed High Dimensional Frustum PointNet for 3D object detection in the context of autonomous driving.

Motivated by the goals data diversity and lossless processing of the data, our deep learning approach directly and jointly uses the raw data from camera, LiDAR, and radar.

In more detail, given 2D region proposals and classification from camera images, a high dimensional convolution operator captures local features from a point cloud enhanced with color and temporal information.

Radars are used as adaptive plug-in sensors to refine object detection performance.

As shown by an extensive evaluation on the nuScenes 3D detection benchmark, our network outperforms most of the previous methods.

Keywords: Sensor and Data Fusion, Automated Vehicles, Lidar Sensing and Perception
Abstract: Occupancy grids are widely used in robotics and autonomous systems to create a representation of the environment. Originally designed to map a static environment, recently also dynamic occupancy grids are emerging for the handling of dynamic scenes. However, a major drawback of (dynamic) occupancy grids is the high computational cost (memory and compute) to store and process the information. This is due to the fact, that occupancy grids usually divide the environment in cells of the same size (i.e. uniform grids). As a result, the computational cost increases quadratically with decreasing cell size. Therefore, for many use cases, a trade-off between accuracy (high resolution grid) and distance covered by the grid is required to keep the computational cost in an acceptable range. To overcome this issue we propose in this paper a novel approach for dynamic occupancy grids using non-uniform cell sizes. Our results show, that these non-uniform occupancy grids reduce the numbers of required cells and therefore the computational cost significantly without compromising on the quality of the results.

Keywords: Sensor and Data Fusion, Vehicle Environment Perception, Information Fusion
Abstract: The information of the road grade plays an important role in improving the ride comfort and fuel consumption. This paper proposes a Quaternion Unscented Kalman Filter (QUKF) to estimate the road grade accurately, which needs only measurements from low-cost Inertial Measurement Unit (IMU). The model is built based on the data from accelerometer and gyroscope. The quaternion, which represents orientations and rotations, is chosen to be the state variables, while the three-axle acceleration is set as measurement vector. The proposed observer is tested and verified using the simulation software CarSim and MATLAB Simulink under several scenarios. To compare the performance of the algorithm, the Kalman filter and complementary filter are also implemented under the same simulation conditions. The results illustrate that the presented observer improves the accuracy and stability. Finally, the results of experiments are delivered and the performance of the filter is assessed against the output of a complete GPS/INS available in the same real-world dataset.

Keywords: Sensor and Data Fusion, V2X Communication, Vision Sensing and Perception
Abstract: Cooperative perception with many sensors involved greatly improves the performance of perceptual systems in autonomous vehicles. However, the increasing amount of sensor data leads to a bottleneck due to limited capacity of vehicle-to-X (V2X) communication channels. We leverage lossy learned image compression by means of an autoencoder with adversarial loss function to reduce the overall bitrate. Our key contribution is to focus image compression on regions of interest (ROIs) governed by a binary mask. A transmitter-sided semantic segmentation network extracts semantically important classes being the basis for the generation of a ROI. A second key contribution is that the mask is not transmitted as side information, only the quantized bottleneck data is transmitted. To train the network, we use a loss function operating only on the pixels in the ROI. We report peak-signal-to-noise ratio (PSNR) both in the entire image and only in the ROI, evaluating various fusion architectures and fusion operations involving input image and mask. Showing the high generalizability of our approach, we achieve consistent improvements in the ROI in all experiments on the Cityscapes dataset.

Keywords: Sensor and Data Fusion, Advanced Driver Assistance Systems, Vehicle Environment Perception
Abstract: Lane detection plays a crucial role in any autonomous driving system. Currently commercialized vehicles offer lane keep assist and lane departure warning via integrated smart cameras, deployed for road markings detection. These sensors alone, however, do not generally ensure adequate performance for higher autonomy levels. In this paper, a multi-sensor tracking approach for generic lane boundaries is proposed. This solution is based on well-established filtering techniques and supports a flexible clothoid spline representation. It relies on fine-tuned measurement models, tailored on collected data from both off-the-shelf and prototype smart sensors. The implementation takes into account real-time constraints and ADAS ECUs scarcity of resources. The result is finally validated against lane-level ground truth and experimental data acquisitions.

Keywords: Self-Driving Vehicles, Convolutional Neural Networks, Vulnerable Road-User Safety
Abstract: Following detection and tracking of traffic actors, prediction of their future motion is the next critical component of a self-driving vehicle (SDV) technology, allowing the SDV to operate safely and efficiently in its environment. This is particularly important when it comes to vulnerable road users (VRUs), such as pedestrians and bicyclists. These actors need to be handled with special care due to an increased risk of injury, as well as the fact that their behavior is less predictable than that of motorized actors. To address this issue, in the current study we present a deep learning-based method for predicting VRU movement, where we rasterize high-definition maps and actor's surroundings into a bird's-eye view image used as an input to deep convolutional networks. In addition, we propose a fast architecture suitable for real-time inference, and perform an ablation study of various rasterization approaches to find the optimal choice for accurate prediction. The results strongly indicate benefits of using the proposed approach for motion prediction of VRUs, both in terms of accuracy and latency.

Keywords: Vulnerable Road-User Safety, Sensor and Data Fusion, Intelligent Ground, Air and Space Vehicles
Abstract: This paper summarizes the progress in developing a rugged, low-cost, automated ground cone robot network capable of traffic delineation at lane-level precision. A holonomic omnidirectional base with a traffic delineator was developed to allow flexibility in initialization. RTK GPS was utilized to reduce minimum position error to 2 centimeters. Due to recent developments, the cost of the platform is now less than 1,600. To minimize the effects of GPS-denied environments, wheel encoders and an Extended Kalman Filter were implemented to maintain lane-level accuracy during operation and a maximum error of 1.97 meters through 50 meters with little to no GPS signal. Future work includes increasing the operational speed of the platforms, incorporating lanelet information for path planning, and cross-platform estimation.

Keywords: Vulnerable Road-User Safety
Abstract: The breakthrough of intelligent vehicles will also be determined by the safety gain they provide. In order to perform accident avoiding reactions at the earliest point in time possible, predictions about the future behavior of other traffic participants are needed. The most exposed share of traffic participants regarding this issue are single-track two-wheelers (1T2W): they share the road with cars and trucks but are not as agile due to their kinematics. Furthermore, they are faster than pedestrians but comparably vulnerable as those.

In order to guarantee their safety, we make use of their movement restricting kinematics. We simulate three typical classes of 1T2W under conservative assumptions about their agility in order to generate a spatial region in which they have to be due to physics after a fixed prediction horizon of up to 1.5 seconds. The proposed approach was verified in experiments with real high-dynamic driving maneuvers.

Keywords: Vision Sensing and Perception, Deep Learning, Automated Vehicles
Abstract: Big data leads to a great success of deep learning in computer vision. Unfortunately, big datasets are often not balanced in all dimensions and rare cases are often under-represented. On-board data collection by a moving vehicle can capture thousands of normal pedestrians and vehicles, but what about special persons like police officers, road workers, and school guards? Not only that those types of classes are hard to get, they are crucial to be recognized and classified as such for the task of automated driving. Future self-driving cars need to interact with their environment and need to also understand and follow the signals and instructions of those special persons. In this paper, we show how to classify special person types using Convolutional Neural Networks. The big data imbalance is handled by data augmentation using Generative Models, showing a clear advantage over classical data augmentation. The classification performance of special persons can be significantly improved using our Generative Model based Data Augmentation.

Keywords: Automated Vehicles, Vulnerable Road-User Safety, Recurrent Networks
Abstract: Traffic violation and the flexible and changeable nature of pedestrians make it more difficult to predict pedestrian behavior or intention, which might be a potential safety hazard on the road. Pedestrian motion state (such as walking and standing) directly affects or reflects its intention. In combination with pedestrian motion state and other influencing factors, pedestrian intention can be predicted to avoid unnecessary accidents. In this paper, pedestrian is treated as non-rigid object, which can be represented by a set of two-dimensional key points, and the movement of key point relative to the torso is introduced as micro motion. Static and dynamic micro motion features, such as position, angle and distance, and their differential calculations in time domain, are used to describe its motion pattern. Gated recurrent neural network based seq2seq model is used to learn the dependence of motion state transition on previous information, finally the pedestrian motion state is estimated via a softmax classifier. The proposed method only needs the previous hidden state of GRU and current feature to evaluate the probability of current motion state, and it is computation efficient to deploy on vehicles. This paper verifies the proposed algorithm on the JAAD public dataset, and the accuracy is improved by 11.6% compared with the existing method.

Keywords: Vulnerable Road-User Safety, Recurrent Networks, Automated Vehicles
Abstract: Driving in urban traffic requires making quick and safe decisions while interacting with multiple pedestrians and other road users. Early anticipation of others' intentions is especially important in predicting their future behavior.

In this work, we explore the human ability to estimate intentions of pedestrians in typical urban traffic conditions. Towards this goal, we analyze the results of our large-scale experiment that involved over 700 subjects to establish a human reference point for the task of pedestrian intention estimation. We determine what visual features correlate with human decisions and the relative difficulty of scenarios and validate our conclusion using a linear logistic model. Furthermore, we propose two models to demonstrate the benefits of using intention for pedestrian trajectory and future crossing action prediction. Our experiments show that an improvement of up to 5% can be achieved on both tasks.

Keywords: Convolutional Neural Networks, Deep Learning, Sensor and Data Fusion
Abstract: 3D localization of persons from a single image is a challenging problem, where advances are largely data-driven. In this paper, we enhance the recently released EuroCity Persons detection dataset, a large and diverse automotive dataset covering pedestrians and riders. Previously, only 2D annotations and image data were provided. We introduce an automatic 3D lifting procedure by using additional LiDAR distance measurements, to augment a large part of the reasonable subset of 2D box annotations with their corresponding 3D point positions (136K persons in 46K frames of day- and night-time). The resulting dataset (coined ECP2.5D), now including LiDAR data as well as the generated annotations, is made publicly available for (non-commercial) benchmarking of camera-based and/or LiDAR 3D object detection methods. We provide baseline results for 3D localization from single images by extending the YOLOv3 2D object detector with a distance regression including uncertainty estimation.

Keywords: Vision Sensing and Perception, Vulnerable Road-User Safety, Reinforcement Learning
Abstract: Achieving excellent generalization on perceiving real traffic scenarios with diversity is the long-term goal for building robust autonomous driving systems. In this paper, we propose to discover potential shortness of certain perception module by analyzing its worst-scenario performance. However, with the benchmark datasets growing huge and tremendous, exhaustive searching for the worst perception scenario (WPS) seems to be time consuming and unnecessary. To address, we present an automatic searching scheme empowered by reinforcement learning. In this case, worst scenario mining is formulated as a discrete search problem. A single layer recurrent neural network with LSTM neurons is employed to predict WPS according to the searching reward, which is optimized by a vanilla policy gradient method. Moreover, to deal with the imbalanced distribution of real traffic scenarios, a KNN-like retrieval is utilized for searching closest scenario samples. Effective yet efficient, the proposed method has been validated by finding the most challenging scenarios for various vehicle detectors on KITTI, BDD100k and our own benchmark set EVB. Further experiments reveal that detection networks with structural similarity share the similar WPS.

Keywords: Unsupervised Learning, Deep Learning, Image, Radar, Lidar Signal Processing
Abstract: In the past few years, the performance of road defect detection has been remarkably improved thanks to advancements in various studies on computer vision and deep learning. Although large-scale and well-annotated datasets enhance the performance of detecting road defects to some extent, it is still challengeable to derive a model which can perform reliably for various road conditions in practice, because it is intractable to construct a dataset considering diverse road conditions and defect patterns. To end this, we propose an unsupervised approach to detect road defects, using Adversarial Image-to-Frequency Transform (AIFT). AIFT adopts the unsupervised manner and adversarial learning in deriving the defect detection model, so AIFT does not require annotations for road defects. We evaluate the efficiency of AIFT using GAPs384 dataset, Cracktree200 dataset, CRACK500 dataset, and CFD dataset. The experimental results demonstrate that the proposed approach detects various road detects, and it outperforms existing state-of-the-art approaches.

Keywords: Advanced Driver Assistance Systems, Deep Learning, Recurrent Networks
Abstract: For the path planning of autonomous vehicles, it is important to predict the future trajectory of the surrounding vehicles. However, predicting future trajectory is difficult because it needs to consider the invisible interaction between the vehicles in a dynamic driving environment. In this paper, a new approach, which considers the interaction between surrounding vehicles, is proposed for accurate prediction of the future trajectory. The proposed method provides continuous predicted trajectories over time in the longitudinal and lateral directions, respectively. The deep ensemble technique is also used to predict the uncertainty of the estimated trajectory. This paper performs the training and verification of the algorithm using NGSIM dataset, which is the vehicle driving data obtained through actual vehicle driving.

Keywords: Autonomous / Intelligent Robotic Vehicles, Deep Learning, Self-Driving Vehicles
Abstract: This paper presents online-capable deep learning model for probabilistic vehicle trajectory prediction. We propose a simple encoder-decoder architecture based on multi-head attention. The proposed model generates the distribution of the predicted trajectories for multiple vehicles in parallel. Our approach to model the interactions can learn to attend to a few influential vehicles in an unsupervised manner, which can improve the interpretability of the network. The experiments using naturalistic trajectories at highway show the clear improvement in terms of positional error on both longitudinal and lateral direction.

Keywords: Deep Learning, Cooperative ITS, Automated Vehicles
Abstract: Efficient driving in urban traffic scenarios requires foresight. The observation of other traffic participants and the inference of their possible next actions depending on the own action is considered cooperative prediction and planning. Humans are well equipped with the capability to predict the actions of multiple interacting traffic participants and plan accordingly, without the need to directly communicate with others. Prior work has shown that it is possible to achieve effective cooperative planning without the need for explicit communication. However, the search space for cooperative plans is so large that most of the computational budget is spent on exploring the search space in unpromising regions that are far away from the solution. To accelerate the planning process, we combined learned heuristics with a cooperative planning method to guide the search towards regions with promising actions, yielding better solutions at lower computational costs.

Keywords: Self-Driving Vehicles, Advanced Driver Assistance Systems, Deep Learning
Abstract: Traffic light detection by camera is a challenging task for autonomous driving mainly due to the small size of traffic lights in the road scene especially for early detection. The limited resolution in the corresponding area of traffic lights reduces their contrast to the background, as well as the effectiveness of the visual cues from the traffic light itself. We believe understanding the scene semantics between traffic lights and their surroundings can play a vital role in tackling this challenge. Towards this goal, we build a generative adversarial network (GAN) model to predict the existence of traffic lights from the road scene image where existing traffic lights are removed with image inpainting. %and anchor center sampling. Using Cityscape dataset, we verify that the proposed GAN model indeed captures the desired semantics by showing effective predictions of existence of traffic lights that are consistent with real images. Moreover, we leverage this model to augment the training data where traffic lights are inserted to the road scene images based on the prediction of the GAN model. While the augmented images may not be realistic looking, results show that such data augmentation can improve the traffic light detector performance that is comparable to using additional real data collection, and better than other data augmentation with various randomization schemes. These results verify the importance of semantic consistency for data augmentation to improve the traffic light detection.

Keywords: Autonomous / Intelligent Robotic Vehicles, Vehicle Control, Intelligent Ground, Air and Space Vehicles
Abstract: In this work we evaluate Iterative Linear Quadratic Regulator(ILQR) for trajectory tracking of two different kinds of wheeled mobile robots namely Warthog , an off-road holonomic robot with skid-steering and Polaris GEM e6, a non-holonomic six seater vehicle. We use multilayer neural network to learn the discrete dynamic model of these robots which is used in ILQR controller to compute the control law. We use model predictive control (MPC) to deal with model imperfections and perform extensive experiments to evaluate the performance of the controller on human driven reference trajectories with vehicle speeds of 3m/s- 4m/s for warthog and 7m/s-10m/s for the Polaris GEM

Keywords: Advanced Driver Assistance Systems, Reinforcement Learning, Automated Vehicles
Abstract: Lane-change maneuvers are commonly executed by drivers to follow a certain routing plan, overtake a slower vehicle, adapt to a merging lane ahead, etc. However, improper lane change behaviors can be a major cause of traffic flow disruptions and even crashes. While many rule-based methods have been proposed to solve lane change problems for autonomous driving, they tend to exhibit limited performance due to the uncertainty and complexity of the driving environment. Machine learning-based methods offer an alternative approach, as Deep reinforcement learning (DRL) has shown promising success in many application domains including robotic manipulation, navigation, and playing video games. However, applying DRL for autonomous driving still faces many practical challenges in terms of slow learning rates, sample inefficiency, and non-stationary trajectories. In this study, we propose an automated lane change strategy using proximal policy optimization-based deep reinforcement learning, which shows great advantage in learning efficiency while maintaining stable performance. The trained agent is able to learn a smooth, safe, and efficient driving policy to determine lane-change decisions (i.e. when and how) even in dense traffic scenarios. The effectiveness of the proposed policy is validated using task success rate and collision rate, which demonstrates the lane change maneuvers can be efficiently learned and executed in a safe, smooth and efficient manner.

Keywords: Deep Learning, Sensor and Data Fusion
Abstract: Recent work has shown that it is possible to learn neural networks with provable guarantees on the output of the model when subject to input perturbations, however these works have focused primarily on defending against adversarial examples for image classifiers. In this paper, we study how these provable guarantees can be naturally applied to other real world settings, namely getting performance specifications for robust virtual sensors measuring fuel injection quantities within an engine. We first demonstrate that, in this setting, even simple neural network models are highly susceptible to reasonable levels of adversarial sensor noise, which are capable of increasing the mean relative error of a standard neural network from 6.6% to 43.8%. We then leverage methods for learning provably robust networks and verifying robustness properties, resulting in a robust model which we can provably guarantee has at most 16.5% mean relative error under any sensor noise. Additionally, we show how specific intervals of fuel injection quantities can be targeted to maximize robustness for certain ranges, allowing us to train a virtual sensor for fuel injection which is provably guaranteed to have at most 10.69% relative error under noise while maintaining 3% relative error on non-adversarial data within normalized fuel injection ranges of 0.6 to 1.0.

Keywords: Advanced Driver Assistance Systems, Vehicle Control, Information Fusion
Abstract: Automated driving is one of the main drivers in the automotive industry. On the way to full automation current Advanced Driver Assistant Systems (ADAS) and Automated Driving Systems (ADS) backed by new and enhanced sensor systems take over more and more driving tasks. Developers and engineers are challenged with the increasing Operational Design Domain (ODD) of their systems. The application of these systems has become a daunting task, as a manifold of new situations has to be covered. The number of application parameters has skyrocketed and their scopes are intertwined and not always visible to the vehicle’s driver. Virtual, simulation based approaches are on the rise to give developers another tool for the application of their systems. In order to retrieve valid evaluations from the simulation, current research focuses on objectifying the subjective passenger assessments that so far were only conceivable in real world driving tests. Furthermore, with objective and comparable simulation results on statistically significant data samples, application parameters whose effects are not clearly visible in real world driving tests can now be applied systematically and justified. We propose a process reference model for the virtual application of predictive control features with a clear focus on the quality and representativity of the virtual application.

Keywords: Lidar Sensing and Perception, Vision Sensing and Perception, Sensor and Data Fusion
Abstract: This work presents an extrinsic parameter estimation algorithm between a 3D LIDAR and a Projective Camera using a marker-less planar target, by exploiting Planar Surface Point to Plane and Planar Edge Point to back-projected Plane geometric constraints. The proposed method uses the data collected by placing the planar board at different poses in the common field of view of the LIDAR and the Camera. The steps include, detection of the target and the edges of the target in LIDAR and Camera frames, matching the detected planes and lines across both the sensing modalities and finally solving a cost function formed by the aforementioned geometric constraints that link the features detected in both the LIDAR and the Camera using non-linear least squares. We have extensively validated our algorithm using two Basler Cameras, Velodyne VLP-32 and Ouster OS1 LIDARs.

Keywords: Information Fusion, Vehicle Control, Deep Learning
Abstract: This paper proposes a road friction coefficient estimation method based on the fusion of machine vision and vehicle dynamics. A vehicle-mounted camera is used to obtain the front image. Based on the deep learning method, the road surface in the image is segmented and identified to obtain the road type. Besides, a road friction estimator with the tire longitudinal force estimation is designed. With the visual estimation results, a fusion estimation method is designed based on the structural parameter optimization. The results of simulation experiments show that the fusion estimator has the characteristics of fast convergence and high accuracy.

Keywords: Autonomous / Intelligent Robotic Vehicles, Unsupervised Learning, Information Fusion
Abstract: Road friction estimation concerns inference of the coefficient between the tire and road surface to facilitate active safety features. Current state-of-the-art methods lack generalization capability to cope with different tire character- istics and models are restricted when using Bayesian inference in estimation while recent supervised learning methods lack uncertainty prediction on estimates. This paper introduces variational inference to approximate intractable posterior of friction estimates and learns an amortized variational inference model from tire measurement data to facilitate probabilistic estimation while sustaining the flexibility of tire models. As a by-product, a probabilistic tire model can be learned jointly with friction estimator model. Experiments on simulated and field test data show that the learned friction estimator provides accurate estimates with robust uncertainty measures in a wide range of tire excitation levels. Meanwhile, the learned tire model reflects well-studied tire characteristics from field test data.

Keywords: Information Fusion, Lidar Sensing and Perception, Deep Learning
Abstract: In object detection for autonomous driving and robotic applications, conventional RGB cameras often fail to sense objects under extreme illumination conditions and on texture-less surfaces, while LIDAR sensors often fail to sense small or thin objects located far from the sensor. For these reasons, an intuitive and obvious choice for perception system designers is to install multiple sensors of different modalities to increase (in theory) the detection robustness. In this paper we focus on the analysis of an object detector that performs early fusion of RGB images and LIDAR 3D points. Our goal is to go beyond the intuition of simply adding more sensor modalities to improve performance, and instead analyze, quantify, and understand the performance differences, strengths and weaknesses of the object detector under three different modalities: 1) RGBonly, 2) LIDAR-only, and 3) Early fusion (RGB and LIDAR), and under two key scene variables: 1) Distance of objects from the sensor (density), and 2) Illumination (Darkness). We also propose methodologies to generate 2D weak semantic training masks, and a methodology to evaluate the object detection performance separately at different distance ranges, which provides a more reliable detection performance measure and correlates well with object LIDAR point density.

Keywords: Autonomous / Intelligent Robotic Vehicles, Deep Learning, Information Fusion
Abstract: Forecasting the motion of surrounding obstacles (vehicles, bicycles, pedestrians and etc.) benefits the on-road motion planning for intelligent and autonomous vehicles. Complex scenes always yield great challenges in modeling the patterns of surrounding traffic. For example, one main challenge comes from the intractable interaction effects in a complex traffic system. In this paper, we propose a multi-layer architecture Interaction-aware Kalman Neural Networks (IaKNN) which involves an interaction layer for resolving high-dimensional traffic environmental observations as interaction-aware accelerations, a motion layer for transforming the accelerations to interaction-aware trajectories, and a filter layer for estimating future trajectories with a Kalman filter network. Attributed to the multiple traffic data sources, our end-to-end trainable approach technically fuses dynamic and interaction-aware trajectories boosting the prediction performance. Experiments on the NGSIM dataset demonstrate that IaKNN outperforms the state-of-the-art methods in terms of effectiveness for traffic trajectory prediction.

Keywords: Vulnerable Road-User Safety, Recurrent Networks, Convolutional Neural Networks
Abstract: Pedestrian crossing action prediction is a crucial task for autonomous driving. Numerous studies show that an early estimation of the pedestrian's intention can decrease or even avoid a high percentage of accidents. In this paper, different variations of a deep learning system are proposed to attempt to solve this problem. The proposed models are composed of two parts: a CNN-based feature extractor and an RNN module. All the models were trained and tested on the JAAD dataset. The results obtained indicate that the choice of the features extraction method, the inclusion of contextual variables and the chosen RNN type have a great impact on the final performance

Keywords: Vulnerable Road-User Safety, Advanced Driver Assistance Systems, Collision Avoidance
Abstract: Vehicle-pedestrian interaction (VPI) is one of the most challenging tasks for automated driving systems. The design of driving strategies for such systems usually starts with verifying VPI in simulation. This work proposed an improved framework for the study of VPI in uncontrolled pedestrian crossing scenarios. The framework admits the mutual effect between the pedestrian and the vehicle. A multi-state social force based pedestrian motion model was designed to describe the microscopic motion of the pedestrian crossing behavior. The pedestrian model considers major interaction factors such as the accepted gap of the pedestrian's decision on when to start crossing, the desired speed of the pedestrian, and the effect of the vehicle on the pedestrian while the pedestrian is crossing the road. Vehicle driving strategies focus on the longitudinal motion control, for which the feedback obstacle avoidance control and the model predictive control were tested and compared in the framework. The simulation results verified that the proposed framework can generate a variety of VPI scenarios, consisting of either the pedestrian yielding to the vehicle or the vehicle yielding to the pedestrian. The framework can be easily extended to apply different approaches to the VPI problems.

Keywords: Lidar Sensing and Perception, Vehicle Environment Perception
Abstract: This paper presents an experimental study on 3D person localization (i.e. pedestrians, cyclists) in traffic scenes, using monocular vision and LiDAR data. We first analyze the detection performance of two top-ranking methods (PointPillars and AVOD) on the KITTI benchmark, with respect to varying Intersection over Union (IoU) settings and the underlying parameters of 3D bounding box location, extent and orientation. Given that the KITTI dataset contains relatively few 3D person instances, we also consider the new EuroCity Persons 2.5D (ECP2.5D) dataset, which is one order of magnitude larger. We perform domain transfer experiments between the KITTI and ECP2.5D datasets, to examine how these datasets generalize with respect to each other.

Keywords: Human-Machine Interface, Novel Interfaces and Displays, Self-Driving Vehicles
Abstract: Autonomous Vehicles (AV) will transform transportation, but also the interaction between vehicles and pedestrians. In the absence of a driver, it is not clear how an AV can communicate its intention to pedestrians. One option is to use visual signals. To advance their design, we conduct four human-participant experiments and evaluate six representative AV visual signals for visibility, intuitiveness, persuasiveness, and usability at pedestrian crossings. Based on the results, we distill twelve practical design recommendations for AV visual signals, with focus on signal pattern design and placement. Moreover, the paper advances the methodology for experimental evaluation of visual signals, including lab, closed-course, and public road tests using an autonomous vehicle. In addition, the paper also reports insights on pedestrian crosswalk behaviours and the impacts of pedestrian trust towards AVs on the behaviors. We hope that this work will constitute valuable input to the ongoing development of international standards for AV lamps, and thus help mature automated driving in general.

Keywords: Human-Machine Interface, Advanced Driver Assistance Systems, Cooperative ITS
Abstract: As vulnerable road users, cyclists were often killed or injured because they were not perceived by drivers on rural roads on time. Nowadays, many sensor technology and assistance systems are used to improve traffic safety and efficiency. However, few studies focus on the interaction between drivers and cyclists, especially enhancing drivers’ perception of cyclists. Following a cooperative support framework between drivers and vehicles, we designed an interface to improve drivers’ perception of cyclists using Augmented Reality (AR) and then evaluated it in the driving simulator. The results show that driving behavior regarding the distance to the cyclist shows a significant change while the driver is supported by the developed AR interface. Consequently, the results implicate that the safety of vulnerable traffic participants can be increased, using a cooperation strategy between the driver and the vehicle. The behaviour while driving manually can be persuaded by the system without any distraction of the driver.

Keywords: Vision Sensing and Perception, Human-Machine Interface, Vulnerable Road-User Safety
Abstract: Gestures from traffic police give the authorized information, especially in some urgent situation. Thus, understanding of traffic police instruction accurately and promptly is particularly crucial for the automated driving system. However, this task is a great challenge not only because of the dynamic and diversity characteristics of the human gesture, but also the high requirement for real-time performance in each frame. We propose an online activity recognition method based on pose estimation and Graph Convolutional Networks (GCN) to recognize the traffic police gesture in frame level. The main contribution in this work is the development of an online framework based on graph convolutional networks for traffic police recognition. Our approach obtained the state-of-the-art results on Traffic Police Gesture Recognition (TPGR) dataset.

Keywords: Vehicle Environment Perception, Vulnerable Road-User Safety, Automated Vehicles
Abstract: Risk mitigation is an important element to consider in risk evaluation. It is known that safety features have helped to decrease the death ratio over the years. However, to date, each driver assistance system works on a single domain of operation. The problem still remains in how to use perception to contextualize the scene in order to fully minimize the severity of a collision in a complex emergency scenario. Up to now, works on cost maps have consider simple contextualized object in mitigation scenarios. For instance, the use of binary allowed/forbidden zones or, a fixed weight to each type of object in the scene. In this work, the risk of injury issued by accidentology is employed to each class of object present in the scene. Each class of object presents then a probability of injury with respect to the impact speed and ethical/economical/political factors. The method generates a cost map containing the probability of collision associated to the risk of injury. It dynamically contextualizes the objects, since the risk of injury depends on the characteristics of the scene. Simulation and dataset results validate that changing the referred parameters alters the context and evaluation of the scene. Then, the proposed methodology allows a better assessment of the surroundings by creating a dynamic navigation cost map for complex scenarios.

Keywords: Vulnerable Road-User Safety, Mapping and Localization, Sensor and Data Fusion
Abstract: Over the last decade, the number of accidents involving Vulnerable Road Users (VRU), i.e. pedestrians, cyclists and motorbike drivers, has not decreased in the same way as accidents between passenger cars have. Cooperative systems based on Vehicle-to-X (V2X) communication make it possible to directly exchange information between VRUs and vehicles and to increase the overall situational awareness beyond the capabilities of on-board ranging sensors. To detect and avoid collisions, vehicles require up-to-date and precise information on the location and trajectory of VRUs. In this paper, we propose a VRU localization system based on Global Navigation Satellite System (GNSS), inertial sensors and ultra-wideband (UWB) round-trip-delay ranging technology. We present an exhaustive measurement campaign comprising pedestrians, cyclists and vehicles performed in an urban setting and show first results on the localization performance for a pedestrian crossing an intersection. In the experiments, the pedestrian inertial system supported by GNSS and UWB ranges is able to achieve 0.65m 1-sigma-position accuracy.

Keywords: Mapping and Localization, Sensor and Data Fusion, Self-Driving Vehicles
Abstract: Augmented reality visualizes map content or sensor observations by projections that fit naturally into the surroundings, thereby providing relevant information in a convenient and safe way for the customer. A stable projection of map data into the driver’s field of view requires precise knowledge of the vehicle’s pose in six degrees of freedom (dof) with respect to a map. We developed an efficient 6dof localization based on odometry, camera recognized landmarks and a high definition (HD) map using an extended Kalman filter (EKF). As a result, we compared the accuracy of two fusion methods on simulated driving scenarios based on real HD map data: (i) fusing 2D landmarks from the image plane directly, and (ii) fusing 3D landmarks assuming prior 3D reconstruction. Using five types of traffic signs and pole-like landmarks, we found that the accuracy difference between the approaches reduces with increasing number of landmarks per cycle. In presence of five or more landmarks per cycle, the 2D measurement approach reaches the accuracy of the 3D measurement approach. Thus, with many landmark types, the 2D approach becomes a beneficial alternative: While achieving comparable accuracy, it circumvents 3D reconstruction, thereby reducing the computational effort, which is highly desirable in autonomous driving.

Keywords: Mapping and Localization, Lidar Sensing and Perception, Vehicle Environment Perception
Abstract: This paper presents a LiDAR-based 3D Monte Carlo localization (MCL) with an efficient distance field (DF) representation method. To implement 3D MCL, high computing capacity is required because the likelihood of many pose candidates, i.e., particles, must be calculated in real time by comparing sensor measurements and a map. Additionally, a large-scale map is needed for allocation to embedded computers since autonomous vehicles are required to navigate wide areas. These make it difficult for 3D MCL implementation. This paper first presents an efficient DF representation method while considering the 3D LiDAR-based localization characteristics. Because each DF voxel has the closest distance from occupied voxels, swift comparison of the sensor measurements and map can be achieved. Consequently, 3D MCL using the likelihood field model (LFM) can be executed in real time. Furthermore, this paper presents a method for improving the localization robustness to environmental changes without increasing memory and computational cost from that of the LFM-based MCL. Through experiments using the SemanticKITTI dataset, we show that the presented method can efficiently and robustly work in dynamic environments.

Keywords: Mapping and Localization, Self-Driving Vehicles, Lidar Sensing and Perception
Abstract: In this paper, we present SROM, a novel real-time Simultaneous Localization and Mapping (SLAM) system for autonomous vehicles. The keynote of the paper showcases SROM's ability to maintain localization at low sampling rates or at high linear or angular velocities where most popular LiDAR based localization approaches get degraded fast. We also demonstrate SROM to be computationally efficient and capable of handling high-speed maneuvers. It also achieves low drifts without the need for any other sensors like IMU and/or GPS. Our method has a two-layer structure wherein first, an approximate estimate of the rotation angle and translation parameters are calculated using a Phase Only Correlation (POC) method. Next, we use this estimate as an initialization for a point-to-plane ICP algorithm to obtain fine matching and registration. Another key feature of the proposed algorithm is the removal of dynamic objects before matching the scans. This improves the performance of our system as the dynamic objects can corrupt the matching scheme and derail localization. Our SLAM system can build reliable maps at the same time generating high-quality odometry. We exhaustively evaluated the proposed method in many challenging highways/country/urban sequences from the KITTI dataset and the results demonstrate better accuracy in comparisons to other state-of-the-art methods with reduced computational expense aiding in real-time realizations. We have also integrated our SROM system with our in-house autonomous vehicle and compared it with the state-of-the-art methods like LOAM and LeGO-LOAM.

Keywords: Self-Driving Vehicles, Lidar Sensing and Perception, Image, Radar, Lidar Signal Processing
Abstract: In the paper a fast and consistent method to manage uncertainties on detected traffic agents providing reliable results is presented. The information provided by a LiDAR- based object detector is combined with a high-definition map to identify the drivable space of the carriageway. Because the use of a HD map requires the use of a localization system, the uncertainty of the estimated pose shall be handled carefully. A novel approach taking into account the localization uncertainty in the perception task by direct propagation of it onto the LiDAR points is proposed. It is compared with a classical propagation that relies on linearized approximation. The good performances of this approach in terms of integrity are demonstrated by the use of real data acquired at the entrance of a roundabout being a particularly complex situation.

Keywords: Lidar Sensing and Perception, Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: LIDAR sensors are bound to become one of the core sensors in achieving full autonomy for self driving cars. LIDARs are able to produce rich, dense and precise spatial data, which can tremendously help in localizing and tracking a moving vehicle. In this paper, we review the latest finding in 3D LIDAR localization for autonomous driving cars, and analyze the results obtained by each method, in an effort to guide the research community towards the path that seems to be the most promising.

Keywords: Automated Vehicles, Vehicle Environment Perception, Deep Learning
Abstract: Visual localization in vehicle navigation remains a crucial image retrieval task to determine the best matched image. Developing an efficient algorithm to address the localization issues of vehicle is highly difficult, for severe appearance variations with vehicles moving around can bring about significant challenges and big obstacles. In this paper, we propose the CFVL framework which takes panoramas into use in the localizer and the system processes from coarse to fine, in order to attain more robust and stable descriptors. NetVALD descriptors based on explicit panorama construction, which are regarded robust to appearance changes, are extracted in the coarse stage, while Geodesc keypoint descriptors, which are believed to detect more detailed information, are utilized in the fine stage, so as to perceive the accurate localization. A comprehensive set of experiments is carried on several datasets with different appearances across seasonal cycling, illumination variations, diverse traversals, and so on, to verify the effectiveness of the coarse stage and fine stage in our system. Brute Force (BF) matching and Fundamental Matrix mapping are utilized to match and locate correct locations after coarse stage and after fine stage. The accuracy of the coarse matching and fine matching are verified separately. Our system is demonstrated to be with high location recall, generalization capacity across different environments.

Keywords: Lidar Sensing and Perception, Deep Learning, Vehicle Environment Perception
Abstract: In this paper, we consider the transformation of laser range measurements into a top-view grid map representation to approach the task of LiDAR-only semantic segmentation. Since the recent publication of the SemanticKITTI data set, researchers are now able to study semantic segmentation of urban LiDAR sequences based on a reasonable amount of data. While other approaches propose to directly learn on the 3D point clouds, we are exploiting a grid map framework to extract relevant information and represent them by using multi-layer grid maps. This representation allows us to use well-studied deep learning architectures from the image domain to predict a dense semantic grid map using only the sparse input data of a single LiDAR scan. We compare single-layer and multi-layer approaches and demonstrate the benefit of a multi-layer grid map input. Since the grid map representation allows us to predict a dense, 360° semantic environment representation, we further develop a method to combine the semantic information from multiple scans and create dense ground truth grids. This method allows us to evaluate and compare the performance of our models not only based on grid cells with a detection, but on the full visible measurement range.

Keywords: Automated Vehicles, Situation Analysis and Planning, Active and Passive Vehicle Safety
Abstract: Vehicles with Automated Driving Systems (ADS) operate in a high-dimensional continuous system with multi-agent interactions. This continuous system features various types of traffic agents (non-homogeneous) governed by continuous-motion ordinary differential equations (differential-drive). Each agent makes decisions independently that may lead to conflicts with the subject vehicle (SV), as well as other participants (non-cooperative). A typical vehicle safety evaluation procedure that uses various safety-critical scenarios and observes resultant collisions (or near collisions), is not sufficient enough to evaluate the performance of the ADS in terms of operational safety status maintenance. In this paper, we introduce a emph{Model Predictive Instantaneous Safety Metric} (MPrISM), which determines the safety status of the SV, considering the worst-case safety scenario for a given traffic snapshot. The method then analyzes the SV's closeness to a potential collision within a certain evaluation time period. The described metric induces theoretical guarantees of safety in terms of the time to collision under standard assumptions. Through formulating the solution as a series of minimax quadratic optimization problems of a specific structure, the method is tractable for real-time safety evaluation applications. Its capabilities are demonstrated with synthesized examples and cases derived from real-world tests.

Keywords: Self-Driving Vehicles, Situation Analysis and Planning, Recurrent Networks
Abstract: Failures in autonomous driving caused by complex traffic situations or model inaccuracies remain inevitable in the near future. While much research is focused on how to prevent such failures, comparatively little research has been done on predicting them. An early failure prediction would allow for more time to take actions to resolve challenging situations. In this work, we propose an introspective approach to predict future disengagements of the car by learning from previous disengagement sequences. Our method is designed to detect failures as early as possible by using sensor data from up to ten seconds before each disengagement. The car itself is treated as a black box, with only its state data and the number of detected objects being required. Since no model-specific knowledge is needed, our method is applicable to any self-driving system. Currently, no public data of real-life disengagements is available. To test our approach, we therefore use autonomous driving data provided by BMW that was collected with BMW research vehicles over multiple months. We show that an LSTM classifier trained with sequences of state data can predict failures up to seven seconds in advance with an accuracy of more than 80%. This is two seconds earlier than comparable approaches from the literature.

Keywords: Situation Analysis and Planning, Autonomous / Intelligent Robotic Vehicles, Self-Driving Vehicles
Abstract: Mixed-Integer Quadratic Programming (MIQP) has been identified as a suitable approach for finding an optimal solution to the behavior planning problem with low runtimes. Logical constraints and continuous equations are optimized alongside. However, it has only been formulated for a straight road, omitting common situations such as taking turns at intersections. This has prevented the model from being used in reality so far. Based on a triple integrator model formulation, we compute the orientation of the vehicle and model it in a disjunctive manner. That allows us to formulate linear constraints to account for the non-holonomy and collision avoidance. These constraints are approximations, for which we introduce the theory. We show the applicability in two benchmark scenarios and prove the feasibility by solving the same models using nonlinear optimization. This new model will allow researchers to leverage the benefits of MIQP, such as logical constraints, or global optimality.

Keywords: Automated Vehicles, Situation Analysis and Planning, Active and Passive Vehicle Safety
Abstract: On the path towards mass deployment of automated vehicles (AVs) several challenges have still to be resolved. One of these is the development of approaches that allow the safe operation of an AV within uncertain environments, while not imposing excessive safety margins. Existing proposals, such as the Responsibility Sensitive Safety (RSS) approach from Intel/Mobileye, are a good first step towards this goal. At the same time, these approaches are often too conservative, and hence further improvements are required. However, this behavior is not due to their underlying semantic rules, rather it is a problem of proper situation assessment within the models. Therefore, we propose to consider the risk of a driving situation inside RSS. The result is a novel risk-aware RSS approach, which allows to significantly reduce the safety margins (i.e. increased traffic density) in a situation-dependent manner, while risk limits are maintained.

Keywords: Automated Vehicles, Deep Learning, Situation Analysis and Planning
Abstract: Automated vehicles rely heavily on data-driven methods, especially for complex urban environments. Large datasets of real world measurement data in the form of road user trajectories are crucial for several tasks like road user prediction models or scenario-based safety validation. So far, though, this demand is unmet as no public dataset of urban road user trajectories is available in an appropriate size, quality and variety. By contrast, the highway drone dataset (highD) has recently shown that drones are an efficient method for acquiring naturalistic road user trajectories. Compared to driving studies or ground-level infrastructure sensors, one major advantage of using a drone is the possibility to record naturalistic behavior, as road users do not notice measurements taking place. Due to the ideal viewing angle, an entire intersection scenario can be measured with significantly less occlusion than with sensors at ground level. Therefore, we created a comprehensive, large-scale urban intersection dataset with naturalistic road user behavior using camera-equipped drones as successor of the highD dataset. The resulting dataset contains more than 13 500 road users including vehicles, bicyclists and pedestrians at intersections in Germany and is called inD. The dataset consists of 10 hours of measurement data from four intersections and is available online for non-commercial research at: https://www.inD-dataset.com

Keywords: Vehicle Environment Perception, Situation Analysis and Planning, Automated Vehicles
Abstract: Reliable representation of multi-vehicle interactions in urban traffic is pivotal but challenging for autonomous vehicles due to the volatility of the traffic environment, such as roundabouts and intersections. This paper describes a semi-stochastic potential field approach to represent multi-vehicle interactions by integrating a deterministic field approach with a stochastic one. First, we conduct a comprehensive evaluation of potential fields for representing multi-agent intersections from the deterministic and stochastic perspectives. For the former, the estimates at each location in the region of interest (ROI) are deterministic, which is usually built using a family of parameterized exponential functions directly. For the latter, the estimates are stochastic and specified by a random variable, which is usually built based on stochastic processes such as the Gaussian process. Our proposed semi-stochastic potential field, combining the best of both, is validated based on the INTERACTION dataset collected in complicated real-world urban settings, including intersections and roundabout. Results demonstrate that our approach can capture more valuable information than either the deterministic or stochastic ones alone. This work sheds light on the development of algorithms in decision-making, path/motion planning, and navigation for autonomous vehicles in the cluttered urban settings.

Keywords: Autonomous / Intelligent Robotic Vehicles, Self-Driving Vehicles, Situation Analysis and Planning
Abstract: This paper characterizes safe following distances for on-road driving when vehicles can avoid collisions by either braking or by swerving into an adjacent lane. In particular, we focus on safety as defined in the Responsibility-Sensitive Safety (RSS) framework. We extend RSS by introducing swerve maneuvers as a valid response in addition to the already present brake maneuver. These swerve maneuvers use the more realistic kinematic bicycle model rather than the double integrator model of RSS. We show that these swerve maneuvers allow a vehicle to safely follow a lead vehicle more closely than the RSS braking maneuvers do. The use of the kinematic bicycle model is then validated by comparing these swerve maneuvers to swerves of a dynamic single-track model. The analysis in this paper can be used to inform both offline safety validation as well as safe control and planning.

Keywords: Autonomous / Intelligent Robotic Vehicles, Advanced Driver Assistance Systems, Situation Analysis and Planning
Abstract: Autonomously following a leading vehicle is a major step towards fully autonomous vehicles. The contribution of this work consists in the development, implementation, and validation of two following modes: 1) Exact Following: accurate compliance with the reference path. 2) Flexible Following: tolerate deviation from the reference path in order to avoid obstacles. The proposed method can easily be integrated into existing frameworks for autonomous vehicles. Therefore our approach is flexible enough to be applied to a large variety of different vehicles. To demonstrate the feasibility of our approach an experimental validation is carried out on two autonomous vehicles with major differences in kinematics, weight, and size: A cross-country wheelchair and an off-road truck. Both exact and flexible following have been successfully demonstrated in unstructured outdoor environments.

Keywords: Hand-off/Take-Over, Human-Machine Interface, Active and Passive Vehicle Safety
Abstract: Autonomous vehicle systems and their users need to collaborate to navigate the driving environment, particularly during an unstructured transition of control from automation, when the system releases control and expects the human to immediately assume driving responsibility. We investigated how such transitions affect the user’s trust in the system and subsequent performance. In a full-vehicle driving simulator, participants encountered two system failures: the first varied in severity (mild or severe failure), and the second required a transition of control that was either detected and alerted (loud failure) or not (silent failure). We observed (i) significant changes in user trust in the system over time and between events, and (ii) that the first failure’s severity level did not affect user performance in the subsequent failure; rather, the system’s detection and alert both times was sufficient to successfully complete the transition of control.

Keywords: Human-Machine Interface, Automated Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: The automated vehicles are experiencing a rapid development in worldwide recently. It is commonly believed that before the achievement of fully autonomous driving, the driver will always need to be remained within the vehicle control loop. Hence, intelligent interaction and collaboration between the human driver and the automation will be an efficient solution for the improvement of road safety, traffic efficiency, and social acceptance to the automated vehicles. As a popular collaboration method, shared control has been widely studied in the past two decades. While, it is still a challenging task to involve rich human factors into the shared control system to increase the driving experience and acceptance of the automation. In this study, a literature review on human-centered shared control is proposed towards a solid research on driver-vehicle collaboration. First, the basic background and literature surveys on the human-machine collaboration (HMC) is proposed, and the important factors for efficient multi-agent collaboration and teaming are discussed. Then, different driver behavior and states modeling methods are reviewed. Based on the HMC schemes and driver behavior recognition techniques, literature surveys on human-centered shared control are proposed. Finally, challenges and future works on human-centered shared control are analyzed.

Keywords: Autonomous / Intelligent Robotic Vehicles, Hand-off/Take-Over, Human-Machine Interface
Abstract: Take-over process refers to the situation where drivers are required to take over control of vehicles under certain conditions, which currently lacks safety analysis. To address this problem, Systems-Theoretic Process Analysis (STPA) is extended according to ISO 21448 for the safety analysis of a take-over system, which consists of a driver and a human-machine interface (HMI). First, system-level hazards are analyzed to develop safety constraints. Then, a control structure of the take-over process is modeled to identify unsafe control actions considering human factors. Finally, causal factors of the unsafe control actions are analyzed, and safety requirements regarding HMI are put forward to guide the design of the take-over system. This method was applied to analyze the take-over system of an autonomous vehicle operating in a closed area, and corresponding safety requirements were determined. Based on the results of a questionnaire survey and a take-over test, the analysis method is proved to be applicable to improving the safety of the take-over system.

Keywords: Human-Machine Interface, Advanced Driver Assistance Systems, Hand-off/Take-Over
Abstract: Interaction research has been initially focusing on partially and conditionally automated vehicles. Augmented Reality (AR) may provide a promising way to enhance drivers' experience when using autonomous driving (AD) systems. This study sought to gain insights on drivers’ subjective assessment of a simulated driving automation system with AR-based support. A driving simulator study was conducted and participants’ ratings of the AD system in terms of information imparting, nervousness and trust was collected. Cumulative Link Models (CLMs) were developed to investigate the impacts of AR cues, traffic density and intersection complexity on drivers' attitudes towards the presented AD system. Random effects were incorporated in the CLMs to account for the heterogeneity among participants. Results indicated that AR graphic cues can significantly improve drivers’ experience by providing advice for their decision-making and mitigating their anxiety and stress. However, the magnitude of AR’s effect was impacted by traffic conditions (i.e. diminished at more complex intersections). The study also revealed a strong correlation between elf-rated trust and takeovers, suggesting takeover and other driving behavior could be further examined in future studies.

Keywords: Cooperative ITS, Automated Vehicles, Human-Machine Interface
Abstract: The interactions between driver, vehicle and environment generate vehicle’s behaviors, and the interactions of vehicle groups shape the traffic modes. Therefore, the conditionally or fully automated driving technologies should be developed and tested in the driver-vehicle-environment (DVE) system rather than being developed and tested individually. To build DVE system dynamics, firstly, we propose an architecture to cope with the complicated interactions in automated vehicles (AVs) and in mixed traffic situations. Then we summarize the driver behavior models and compare the differences of intelligence between human driver and automated vehicle. Finally, we summarize the feasible modeling approaches of DVE system into five categories. The primary distinctions are the modeling methods of human drivers’ roles, which are realized by human driver per se, human driver’s cognitive architecture, psychological motivation model, mechanism imitation, and specific mechanism transfer respectively. Taking the applications of human machine interface and AD strategy developments as examples, we analyze the benefits and drawbacks of these approaches.

Keywords: Advanced Driver Assistance Systems, Vehicle Control, Human-Machine Interface
Abstract: This paper presents an approach of utilizing Driving Envelope (DE) restrictions to assist the driver in lateral maneuvers. Two fundamental issues are addressed in this work. First, DE protection. Second, how to implement such functionality on a conventional car configuration, where the driver still needs to be a part of the control loop. The proposed functionality is based on a Model Predictive Controller (MPC). Vehicle states are constrained to avoid car critical spin situations (DE protection). The power-assisted steering system is used to guide the driver inside boundaries defined by the DE. The proposed architecture is compared with the standard car, without such an Advanced Driver-Assistance System (ADAS), by means of virtual ride tests performed using a high-fidelity vehicle model.

Keywords: Automated Vehicles, Human-Machine Interface, Privacy
Abstract: Autonomous vehicles (AVs) are expected to be an integral part of the next generation of transportation systems, where they will share the transportation network with human-driven vehicles during the transition period. In this work, we model the interactions between vehicles (two AVs or an AV and a human-driven vehicle) in a lane changing process by leveraging the Stackelberg game. We explicitly model driving attitudes for both vehicles involved in lane changing. We design five cases, in which the two vehicles have different levels of knowledge, and make different assumptions, about the driving attitude of the rival. We conduct theoretical analysis and simulations for different cases in two lane changing scenarios, namely changing lanes from a higher speed lane to a lower speed lane, and from a lower speed lane to a higher speed lane. We use four metrics (fuel consumption, discomfort, minimum distance gap and lane change success rate) to investigate how the performance of a single vehicle and that of the system will be influenced by the level of information sharing, and whether a vehicle trajectory optimized based on selfish criteria can provide system-level benefits.

Keywords: Situation Analysis and Planning, Automated Vehicles, Societal Impacts
Abstract: We address ethical dilemma situations that may arise during autonomous driving. To evaluate how this delib- eration could work, we propose a decision-making algorithm based on a Markov Decision Process (MDP) which controls the vehicle in normal conditions. When a dilemma situation is detected, the collision severity is determined by an evaluation of the harm incurred by different types of road users. Then, to illustrate different moral approaches, three different policies are proposed: one based on ralwsian contractarianism, another on utilitarianism, and finally on egalitarianism. Each policy supports a different view of the concept of fairness, potentially producing different behaviors for the same dilemma situation.

Keywords: Mapping and Localization, Information Fusion
Abstract: In this paper we propose an iterative Graph SLAM based approach to create HD maps from series production vehicles fleet data. Only high level sensor measurements provided by advanced driver assistance systems are used. This reduces the required bandwidth and makes this approach scalable to vehicle fleet data. Creating HD maps from fleet data enables up to date HD maps since no dedicated mapping vehicles are required. At first, the data is aligned based on odometry, GNSS and traffic sign measurements. Next, road boundary measurements are included, which results in an optimized lateral alignment. Finally, lane boundary association can be carried out. This results in an HD map containing high accuracy data of traffic signs, road and lane boundaries. The approach was evaluated using series production vehicle fleet data recorded on US highways and on a German autobahn covering a distance of 35.4km. The final HD map was evaluated with a groundtruth HD map and achieved an average error of 0.59m.

Keywords: Mapping and Localization, Vision Sensing and Perception, Active and Passive Vehicle Safety
Abstract: Calibration of relative poses between cameras is a challenging problem, known as extrinsic calibration, for non-overlapping cameras that do not share the field of view. We propose a method for calibrating non-overlapping in-vehicle cameras placed at front, back, left and right positions by using visual SLAM(vSLAM). Our proposal is to calibrate the cameras during the motion of 90-degree backing-up parking on the fly, without using any dedicated calibration equipment. With this motion, the adjacent cameras are able to have the close field of view at different moments. The relative poses can be computed if the maps computed with vSLAM on each camera are merged by using the common structures. Therefore, we propose an efficient calibration framework with this feature. The proposed method is divided into three steps: map reconstruction with vSLAM on each camera, map merging for all the cameras, and extrinsic calibration. Especially, we propose to separately utilize the frames for vSLAM and the ones for the calibration so that the accuracy of vSLAM can be maximized for the calibration. In the evaluation, the calibration was performed in a practical environment to investigate the performance in comparison with the ground truth acquired by using a calibration equipment.

Keywords: Mapping and Localization, Self-Driving Vehicles, Advanced Driver Assistance Systems
Abstract: This paper surveys a number of recent developments in modern Global Navigation Satellite Systems (GNSS) and investigates the possible impact on autonomous driving architectures. Modern GNSS now consist of four independent global satellite constellations delivering modernized signals at multiple civil frequencies. New ground monitoring infrastructure, mathematical models, and internet services correct for errors in the GNSS signals at continent scale. Mass-market automotive-grade receiver chipsets are available at low Cost, Size, Weight, and Power (CSWaP). The result is that GNSS in 2020 delivers better than lane-level accurate localization with 99.99999% integrity guarantees at over 95% availability. In autonomous driving, SAE Level 2 partially autonomous vehicles are now available to consumers, capable of autonomously following lanes and performing basic maneuvers under human supervision. Furthermore, the first pilot programs of SAE Level 4 driverless vehicles are being demonstrated on public roads.

However, autonomous driving is not a solved problem. GNSS can help. Specifically, incorporating high-integrity GNSS lane determination into vision-based architectures can unlock lane-level maneuvers and provide oversight to guarantee safety. Incorporating precision GNSS into LiDAR-based systems can unlock robustness and additional fallbacks for safety and utility. Lastly, GNSS provides interoperability through consistent timing and reference frames for future V2X scenarios.

Keywords: Automated Vehicles, Self-Driving Vehicles, Vision Sensing and Perception
Abstract: Autonomous vehicle navigation requires the desired trajectory and the current localization to be able to calculate the command that must be sent to the actuators. The localization of the vehicle (usually defined by a position vector and an orientation vector), can be provided by external systems. GPS localization is the most accurate solution but when it is no longer available or precise, there is a need for on-board localization estimation based on proprioceptive and exteroceptive sensors. Visual odometry is a well known approach to estimate the vehicle motion from a camera. Unfortunately, visual localization is subject to errors that increase over time (drift). In this paper, we provide a study of the impact of localization errors on the control of an autonomous vehicle. In order to validate visual odometry algorithms in simulation, a drift model is proposed. Real navigation experiments with errors on the localization are presented in order to characterize the drift model and the propagation of localization errors in the controller module and the associated command signal.

Keywords: Self-Driving Vehicles, Convolutional Neural Networks, Deep Learning
Abstract: Teaching autonomous vehicles to imitate human driving in complex, urban traffic scenarios is a difficult task. “End-to-end” autonomous driving systems, based on “imitation learning”, are a expecting approach. A model learns the relationships between sensing input and vehicle control signal outputs. These methods can successfully achieve driving in simple scenarios such as lane keeping. In contrast, the “mid-to-mid” autonomous driving methods now being proposed. In such framework, the model learns the relationships between pre-processed feature maps from the model-based system as input and the future position of the ego vehicle as the output. Mid-to-mid driving methods can direct vehicles more robustly than end-to-end driving methods in some complex driving environments. However, mid-to-mid driving methods use the results of the object detection module to create the feature map. If object detection fails, or detection performance is poor due to changes in the driving environment, prediction performance may also be degraded. Our proposed method uses a prediction module that outputs point grid maps directly, without the use of an object detection module, which are then incorporated into the feature map. Point grid maps represent the locations of surrounding vehicles and obstacles directly, based on LiDAR point cloud data. Since the results of object detection are not used by the prediction module, detection performance does not affect prediction performance. In this study we conduct two experiments, an off-line evaluation using a Lyft dataset, and an on-line evaluation using the CARLA simulator. The results show that our model can achieve the same level of ego-vehicle position prediction performance as a model using annotated object location information.

Keywords: Smart Infrastructure, Advanced Driver Assistance Systems, Intelligent Vehicle Software Infrastructure
Abstract: Self-driving vehicle systems have become mainstream as major international research and development companies compete for road detection and mapping projects such as localization. The latter plays an important role in self-driving systems because an accurate determination of the vehicle’s position improves trajectory and offset results. And with a good localization function, a high-precision map can be established. This paper proposes using pole maps to enhance vehicle localization while correcting previous vehicle localization trajectory offsets. We have established a pole feature map with pole-shaped characteristic point clouds along the road that are marked with spatial coordinate information. We use three-dimensional light detection and ranging (3D Lidar) as a sensor for environmental scanning, dynamic segment thresholds, and point cloud projections to the image to extract poles, which avoids identification errors typically caused by the scattering effect of 3D Lidar. The resulting pole map uses the occupancy grid map to filter out noise. In terms of localization, we continue to detect pole features while the vehicle is moving, and compare these with the map. When the results match, we update the current localized information. The localization point corrects the information of the trajectory of the undetected pole, which improves vehicle localization accuracy.

Keywords: Self-Driving Vehicles, Mapping and Localization, Sensor and Data Fusion
Abstract: This paper proposes a robust and reliable motion-based calibration algorithm of multiple LiDARs and INS (inertial navigation system). The conventional methods utilize appearance-based matching results or only motions of sensors. Therefore, this paper proposes a graph structure-based optimization method with three special constraints. The first constraint is matching-based constraint. This constraint provides reliable transformations between LiDARs. Second, motion-based constraint allows to calibrate between LiDARs and INS. Last one is rigid body constraint from an assumption of rigid body platform. This constraint significantly improves position errors from ambiguities of monotonous motions, and it makes LiDAR sensors possible to fully calibrate with 6-DoF (degree-of-freedom) motions of INS. The proposed method was analyzed with KITTI data and applied to real autonomous vehicle to show its superiority.

Keywords: Automated Vehicles, Situation Analysis and Planning, Advanced Driver Assistance Systems
Abstract: Virtual testing plays an important role in the validation and verification of automated vehicles. State-of-the-art approaches first generate a huge amount of test scenarios through simulations or test drives, which are later filtered to obtain relevant scenarios for a given set of specifications. However, only few works exist on synthesizing scenarios directly from specifications. In this work, we present an optimization-based approach to synthesize scenarios only from formal specifications and a given map. To concretize the specifications, we formulate predicates, which are subsequently converted to a mixed-integer quadratic optimization problem. We demonstrate how our method can generate scenarios for maps featuring merging lanes and intersections given a variety of specifications.

Keywords: Automated Vehicles, Situation Analysis and Planning, Collision Avoidance
Abstract: Navigation in congested environments is a challenge for autonomous vehicles and they should consider collision risk metric into their driving behavior. In this paper, we propose a novel two-fold indicator: On the one hand, single-shot risk works in space domain, considering geometries, locations and the velocities of the obstacles in the current scene. On the other hand, long-term risk considers the evolution of the current scene and provides risk values in time domain. The map information and different prediction models (e.g. reachable sets, probabilistic) are considered in the long-term risk, which can then be used in trajectory planning or decision making approaches. Our method can be applied to scenarios with arbitrary road topologies (intersections, roundabouts, highway, etc.) and it is suitable regardless of the scene prediction method. We formulate the single-shot (or short-term) risk with one single function fitted using Monte Carlo (MC) Simulations. The results are evaluated in real scenarios using HighD dataset and compared with other risk indicators such as THW and TTC. In addition, it is applied to a simple trajectory planner in order to demonstrate that the proposed approach imitates human driving style.

Keywords: Vision Sensing and Perception, Image, Radar, Lidar Signal Processing, Situation Analysis and Planning
Abstract: Due to their capability of acquiring aerial imagery, camera-equipped Unmanned Aerial Vehicles (UAVs) are very cost-effective tools for acquiring traffic information. However, not enough attention has been given to the validation of the accuracy of these systems. In this paper, an analysis of the most significant sources of error is done. This includes three key components. First, a vehicle state estimation by means of statistical filtering. Second, a quantification of the most significant sources of error. Third, a benchmark of the estimated state compared with state-of-the-art reference sensors. This work presents ways to minimize the errors of the most relevant sources. With these error reductions, camera-equipped UAVs are very attractive tools for traffic data acquisition. The test data and the source code are made publicly available.

Keywords: Situation Analysis and Planning, Vision Sensing and Perception, Traffic Flow and Management
Abstract: The availability of real-world data is a key element for novel developments in the fields of automotive and traffic research. Aerial imagery has the major advantage of recording multiple objects simultaneously and overcomes limitations such as occlusions. However, there are only few data sets available. This work describes a process to estimate a precise vehicle position from aerial imagery. A robust object detection is crucial for reliable results, hence the state-of-the-art deep neural network Mask-RCNN is applied for that purpose. Two training data sets are employed: The first one is optimized for detecting the test vehicle, while the second one consists of randomly selected images recorded on public roads. To reduce errors, several aspects are accounted for, such as the drone movement and the perspective projection from a photograph. The estimated position is comapared with a reference system installed in the test vehicle. It is shown, that a mean accuracy of 20 cm can be achieved with flight altitudes up to 100 m, Full-HD resolution and a frame-by-frame detection. A reliable position estimation is the basis for further data processing, such as obtaining additional vehicle state variables. The source code, training weights, labeled data and example videos are made publicly available. This supports researchers to create new traffic data sets with specific local conditions.

Keywords: Self-Driving Vehicles, Automated Vehicles, Situation Analysis and Planning
Abstract: The deployment of autonomous vehicles requires safety assurance and performance guarantees of the developed system. However, this is complex due to the number of scenario variations and uncertainty associated with the operating environment.To alleviate this challenge, we propose a model-based validation methodology that relies on a functional hierarchy for the breakdown and simplification of the system navigation functions, and the Backtracking Process Algorithm to identify, trace, and probabilistically quantify risk significant event sequences (scenarios) that lead to Top Events of interest (such as requirement violations). This methodology is demonstrated on a scenario with an occluded pedestrian crossing the road. We are able to identify risks associated with the actor classification problem and sudden changes in behavior of the pedestrian.

Keywords: Automated Vehicles, Situation Analysis and Planning
Abstract: In this paper, we propose an improved sampling-based hybrid A* (SBA*) algorithm for path planning of intelligent vehicles, which works efficiently in complex urban environments. Two main modifications are introduced into the traditional hybrid A* algorithm to improve its adaptivity in both structured and unstructured traffic scenarios. Firstly, a hybrid potential field (HPF) model considering both traffic regulation and obstacle configuration is proposed to represent the vehicle's workspace, which is utilized as a heuristic function. Secondly, a set of directional motion primitives is generated by taking the prior topological structure of the workspace into account. The path planner using SBA* not only obeys traffic regulations in structured scenarios but also is capable of exploring complex unstructured scenarios rapidly. Finally, a post-optimization step is adopted to increase the feasibility of the path. The efficacy of the proposed algorithm is extensively validated and tested with an autonomous vehicle in real traffic scenarios. The experimental results show that SBA* works well in complex urban environments.

Keywords: Situation Analysis and Planning, Deep Learning, Automated Vehicles
Abstract: Motion planning for autonomous vehicles is a challenging task, especially in dynamic environments. The motion of the vehicle itself needs to be considered while the vehicle needs to react to its surroundings at the same time. Sampling-based algorithms proved to be suitable to cope with these challenges. However, the performance of these algorithms is highly dependent on the sampling heuristics, which in turn are often hand crafted and thus need a large amount of tuning. Therefore, we developed two approaches based on deep learning to learn these heuristics for sampling-based motion planning in dynamic environments. The first approach predicts a discrete probability distribution for each point in time of the future trajectory, whereas the second approach directly predicts a variety of trajectories by using dropout sampling. Both approaches are based on an environment representation encoded as a grid-based tensor. The learned heuristics are integrated into an existing planning framework based on particle swarm optimization and are evaluated in several situations. This shows how to combine the strengths of machine learning based approaches and the traceability of rule- or model-based approaches. The evaluation demonstrates that, in total, we were able to improve on our current heuristics. However, none of the approaches performed consistently better in all scenarios evaluated.

Keywords: Human-Machine Interface, Deep Learning, Autonomous / Intelligent Robotic Vehicles
Abstract: Generating realistic motion in a motion-based (dynamic) driving simulator is challenging due to the limited workspace of the motion system of the simulator compared to the motion range of the simulated vehicle. MCA render accelerations by controlling the motion system of the simulators to provide the driver with a realistic driving experience. Commonly used methods such as CW-MCA typically achieves suboptimal results due to scaling and filtering, which results in an inefficient usage of the workspace. The Model Predictive Control-based MCA (MPC-MCA) has been shown to achieve superior results and more efficient workspace use. However, it's performance is in practice constrained due to the computationally expensive operations and the requirement of an accurate prediction of future vehicle states. Finally, the OC has been shown to provide optimal cueing in an open-loop setup wherein the precalculated control signals are re-played to the driver. However, OC cannot be used in real-time with the driver-in-the-loop. Our work introduces a novel NN-MCA, which is trained to imitate the behavior of the OC. After training, the NN-MCA provides an approximated model of the OC, which can run in real-time with the driver in-the-loop, while achieving similar quality. The experiments demonstrate the potential of this approach through objective evaluations of the generated motion-cues on the simulator model and the real simulator. A demonstration video for the performance comparison of the CW-MCA, OC-MCA and our proposed method is available at http://go.tum.de/708350.

Keywords: Advanced Driver Assistance Systems, Convolutional Neural Networks, Human-Machine Interface
Abstract: Image-to-image translation attempts at mapping an image from a given domain into a different domain. In this paper we introduce a novel image-to-image translation paradigm, where the translation itself is performed by a system of generative adversarial networks (GANs), guided by a set of pretrained networks, called task networks. The underlying hypothesis is that these task networks can guide the translation process of the GAN generator to focus on those aspects of the image which matter the most for the tasks in question. Moreover, our work is motivated by a real-life situation, created by the need to adapt existing, duly trained networks to work on images acquired with a different sensor than the one used to acquire the images used to train the networks. In order to optimize the image domain translation from the new sensor to the old one, we use the output loss of the trained neural networks to guide the training process of the image-to-image translation engine, thus forcing it to create images which improve the performances of the trained, guiding networks. Experimental results presented therein confirm the usefulness of our approach for an automotive application within the context of driver monitoring systems. More specifically we show that guiding the learning of the image-to-image translation module with the loss of a fixed neural network trained to estimate the driver eyelid opening, improves the performances of the latter when being fed with images output by the translation module.

Keywords: Automated Vehicles, Advanced Driver Assistance Systems, Human-Machine Interface
Abstract: Autonomous vehicles will find an infinite number of possible scenarios while driving in urban environments and need to react in a proper manner. For that reason, it is important to have algorithms that can propose driving alternatives for different type of scenarios in a global and unified way instead of using rule-based algorithms which depend on the driving scene. This paper presents a reachability estimation algorithm designed to obtain a safe and comfort-optimized trajectory set for different driving scenarios. First, a finite number of path candidates are created using B´ezier curves. Then, all valid path candidates are combined with the reachable sets of dynamic obstacles to generate speed profiles consistent with safety and comfort requirements. The output of this algorithm would allow a decision-making strategy to select the optimum candidate depending on different criteria.

Keywords: Eco-driving and Energy-efficient Vehicles, Advanced Driver Assistance Systems, Electric and Hybrid Technologies
Abstract: This paper explains a novel adaptive cruise control (ACC) driving with coasting to improve fuel economy. The purpose is to reduce the energy loss with predictive control when the preceding vehicle decelerates, while acceptable driving feeling is guaranteed. To achieve this goal, prediction of the preceding vehicle behavior is introduced to determine the ego vehicle behavior realized by using inverse reinforcement learning (IRL). In addition, the evaluation function is designed to determine the best coasting timing by balancing longer coasting time and acceptable driving feeling, while the ego vehicle speed is controlled with a rule-based control at a non-coasting period. The performance of this control strategy has been validated with simulation, showing 9.7% fuel economy improvement on average for hybrid electric vehicles in the case of following the preceding vehicle before an intersection. It has also been verified with an actual test vehicle, where a high level balance between efficiency and acceptable feeling is realized.

Keywords: Collision Avoidance, Self-Driving Vehicles, Unsupervised Learning
Abstract: Imitation learning provides a way to automatically construct a controller by mimicking human behavior from data. For safety-critical systems such as autonomous vehicles, it can be problematic to use controllers learned from data because they cannot be guaranteed to be collision-free. Recently, a method has been proposed for learning a multi-mode hybrid automaton cruise controller (MOHA). Besides being accurate, the logical nature of this model makes it suitable for formal verification. In this paper, we demonstrate this capability using the SpaceEx hybrid model checker as follows. After learning, we translate the automaton model into constraints and equations required by SpaceEx. We then verify that a pure MOHA controller is not collision-free. By adding a safety state based on headway in time, a rule that human drivers should follow anyway, we do obtain a provably safe cruise control. Moreover, the safe controller remains more human-like than existing cruise controllers.

Keywords: Advanced Driver Assistance Systems
Abstract: Highly automated vehicles control the lateral and longitudinal movement of the vehicle in defined use cases, without being monitored by the human driver. According to the high and full automation level defined by SAE J3016 the driver may request the deactivation of the automated driving system at any time. However, interventions by the driver can lead to critical situations, especially in highly dynamic driving situations. Therefore, this paper investigates at which point driver interventions during highly automated driving can be critical and how the driver can be assisted in order to avoid dangerous driver inputs. To answer these questions, this paper presents concepts for intervention assistance systems that are implemented, tested and evaluated in a driving simulator.

Keywords: Human-Machine Interface, Automated Vehicles, Driver Recognition
Abstract: Automated driving systems are envisioned as the future mode of transportation owing to their projected ability to reduce human error and achieve more efficient and comfortable transportation. Accordingly, designing an interface that ensures the situational awareness of the human operator to reduce confusion, false expectations, and over-reliance on the automated system is important. When a human operator is in control, the automated system is expected to handle troublesome situations that the human is unable to manage. Thus, an interface is required to provide the appropriate information when necessary so that the human operator can easily perceive the reason for the sudden automated intervention. In this study, such a scenario is highlighted, in which a simulated automated intervention avoided a potential collision with a pedestrian who suddenly appeared on the roadside. To convey the reason for the automated intervention, an augmented reality-based head-up display (AR-HUD) cue that targets the pedestrian is developed. To understand the effects of the AR-HUD cue on the speed at which a human operator can recognize a pedestrian and the contribution of this visual cue to the perception of acceptability and credibility of the automated intervention, we compared AR-HUD with a static head-up display (S-HUD) that displays a pedestrian symbol at the bottom portion of the windshield. The results showed that the AR-HUD cue yielded faster recognition of the targeted pedestrian and provided a relatively more acceptable perception of the automated intervention.