Keywords: Vulnerable Road-User Safety, Image, Radar, Lidar Signal Processing, Deep Learning
Abstract: Interaction detection between vehicles and vulnerable road users (e.g. pedestrians and cyclists) is important for e.g. safety control and autonomous driving. However, there are many challenges for automatically detecting interactions, such as the ambiguity of defining when interaction is required in dynamic traffic activities among different road users and the lack of labeled data for training a machine learning detector. To overcome the challenges, we introduce a way to define whether or not interaction is required in various traffic scenes and create a large real-world dataset from a very challenging intersection. A sequence-to-sequence method that uses the object information and motion information of the traffic scenes extracted by a state-of-the-art object detector and from optical flow, respectively, is proposed for automatic interaction detection. The proposed method generates a probability of interaction at each short interval (<0.1 s) that represents the changing of interaction along a sequence. We obtain a baseline model that differentiates no interaction from interaction on the basis of the location and road user type from the detected object information. Compared with the baseline model, the empirical results of the proposed method demonstrate very accurate predictions for vehicle turning sequences with varying length.

Keywords: Lidar Sensing and Perception, Image, Radar, Lidar Signal Processing, Vision Sensing and Perception
Abstract: In recent years, topics like autonomous driving increased the demand on robust environmental sensors. Depth sensors are most commonly used. Solid state Light Detection And Ranging (LiDAR) sensors are well suited for these applications. The measurement principle is based on measuring the phase and consequently the delay of emitted and reflected light. Problems arise if strong reflectors, like street-signs, impair the measurement. In this paper, we present a novel algorithm for depth calculation, based on indirect Time-of-Flight (ToF) data. With this approach it is possible to separate multiple reflectors in the scenery. This allows the generation of multiple depth images. In our approach an arbitrary number of different code sequences are applied as modulation signal. With these code sequences we generate a so called ToF-matrix. With this ToF-matrix, the measured environmental response can be mapped to a distance. As our evaluation shows, our method is able to achieve results with more information compared to conventional ToF-imaging. We demonstrate the separation of the reflection of a street-sign, from a target. This algorithm enables the usage of indirect ToF in automotive areas. We believe that this versatile calculation approach can increase the benefit of indirect LiDAR application for autonomous driving.

Keywords: Lidar Sensing and Perception, Deep Learning, Self-Driving Vehicles
Abstract: In this paper, we introduce a deep encoder-decoder network, named SalsaNet, for efficient semantic segmentation of 3D LiDAR point clouds. SalsaNet segments the road, i.e. drivable free-space, and vehicles in the scene by employing the Bird-Eye-View (BEV) image projection of the point cloud. To overcome the lack of annotated point cloud data, in particular for the road segments, we introduce an autolabeling process which transfers automatically generated labels from the camera to LiDAR. We also explore the role of imagelike projection of LiDAR data in semantic segmentation by comparing BEV with spherical-front-view projection and show that SalsaNet is projection-agnostic. We perform quantitative and qualitative evaluations on the KITTI dataset, which demonstrate that the proposed SalsaNet outperforms other state-ofthe-art semantic segmentation networks in terms of accuracy and computation time. Our code and data are publicly available at https://gitlab.com/aksoyeren/salsanet.git.

Keywords: Sensor and Data Fusion, Vehicle Environment Perception, Deep Learning
Abstract: The rapid development of embedded hardware in autonomous vehicles broadens their computational capabilities, thus bringing the possibility to mount more complete sensor setups able to handle driving scenarios of higher complexity. As a result, new challenges such as multiple detections of the same object have to be addressed. In this work, a siamese network is integrated into the pipeline of a well-known 3D object detector approach to suppress duplicate proposals coming from different cameras via re-identification. Additionally, associations are exploited to enhance the 3D box regression of the object by aggregating their corresponding LiDAR frustums. The experimental evaluation on the nuScenes dataset shows that the proposed method outperforms traditional NMS approaches.

Keywords: Lidar Sensing and Perception, Unsupervised Learning, Vehicle Environment Perception
Abstract: We present our approach to unsupervised domain adaptation for single-stage object detectors on top-view grid maps in automated driving scenarios. Our goal is to train a robust object detector on grid maps generated from custom sensor data and setups. We first introduce a single-stage object detector for grid maps based on RetinaNet. We then extend our model by image- and instance-level domain classifiers at different feature pyramid levels which are trained in an adversarial manner. This allows us to train robust object detectors for unlabeled domains. We evaluate our approach quantitatively on the nuScenes and KITTI benchmarks and present qualitative domain adaptation results for unlabeled measurements recorded by our experimental vehicle. Our results demonstrate that object detection accuracy for unlabeled domains can be improved by applying our domain adaptation strategy.

Keywords: Lidar Sensing and Perception, Self-Driving Vehicles, Vehicle Environment Perception
Abstract: Estimating the bounding box from an object point cloud is an essential task in autonomous driving with LiDAR/laser sensors. We present an efficient bounding box estimation method that can be applied on 2D bird’s-eye view (BEV) LiDAR points to generate the bounding box geometry including length, width and orientation. Given a set of 2D points, the method utilizes their convex-hull points to calculate a small set of candidate directions of the box yaw orientation, and therefore reduces the searching space – usually a fine partition of an angle range (e.g. [0, PI/2)) as in the previous solutions – to find the optimal angle. To further improve the efficiency, we investigate the techniques of controlling the number of convex-hull points, by both applying approximate collinearity condition and downsampling the raw point cloud to a smaller size. We provide comprehensive analysis on both accuracy and efficiency of the proposed method on the KITTI 3D object dataset. The results show that without obviously sacrificing the accuracy, the method, especially when using approximate convex-hull points, can significantly improve the time of estimating the bounding box orientation by almost one order of magnitude.

Keywords: Deep Learning, Lidar Sensing and Perception
Abstract: 3D object detection has become a hot topic in intelligent vehicle applications in recent years. Generally, deep learning has been the primary framework used in 3D object detection and regression of the object location and classification of the objectness are the two indispensable components. In the process of training, the L-n and the focal loss are considered as the frequent solutions to minimize the regression and classification loss, respectively. However, there are two problems to be solved in the existing methods. For regression component, there is a gap between evaluation metrics, e.g., 3D Intersection over Union (IoU), and the traditional regression loss. As for the classification component, confidence score exists ambiguous due to the binary label assignment of target. To solve these problems, we propose a loss by jointing 3D IoU and other geometric attributes (named as jointed attributeoriented 3D loss), which can be directly used in optimizing the regression component. In addition, the jointed attributeoriented 3D loss can assign a soft label for supervising the training of the classification. By incorporating the proposed loss function into several state-of-the-art 3D object detection methods, the significant performance improvement has been achieved on the KITTI benchmark

Keywords: Unsupervised Learning, Automated Vehicles, Lidar Sensing and Perception
Abstract: The comprehension of the behavior of objects in the environment of an automated vehicle is essential information to be extracted out of traffic measurement data. It can be made accessible by applying a data description process consisting of a cluster analysis and subsequent classification step. In this paper, two different algorithms, TRACLUS and Self-Organizing Maps are utilized for a clustering of laser scanner measurement data, recorded by experimental vehicles in real traffic. The results are evaluated and discussed. A remaining open issue of a cluster analysis is the necessity for a classification of clusters, which is missing or solved through manual solutions in related work. We address the matter with a novel approach for an automated classification of clusters using reference trajectories, which is successfully validated for a first exemplary maneuver on highway measurement data.

Keywords: Autonomous / Intelligent Robotic Vehicles, Self-Driving Vehicles, Situation Analysis and Planning
Abstract: The complexity to handle complex situations in automated driving requires increasing computational resources. In this work, we propose a machine learning approach for motion planning aiming at optimizing the set of path candidates to be evaluated in accordance with the driving context. Thus, the computation cost of the whole motion planning strategy can be reduced while generating safe and comfortable trajectories when required. The proposed strategy has been implemented in a real experimental platform and validated in different operating environments, successfully providing high quality trajectories in a small time frame.

Keywords: Situation Analysis and Planning, Autonomous / Intelligent Robotic Vehicles, Reinforcement Learning
Abstract: Terrain traversability analysis is a fundamental issue to achieve the autonomy of a robot at off-road environments. Geometry-based and appearance-based methods have been studied in decades, while behavior-based methods exploiting learning from demonstration (LfD) are new trends. Behavior-based methods learn cost functions that guide trajectory planning in compliance with experts' demonstrations, which can be more scalable to various scenes and driving behaviors. This research proposes a method of off-road traversability analysis and trajectory planning using Deep Maximum Entropy Inverse Reinforcement Learning. To incorporate the vehicle's kinematics while solving the problem of exponential increase of state-space complexity, two convolutional neural networks, i.e., RL ConvNet and Svf ConvNet, are developed to encode kinematics into convolution kernels and achieve efficient forward reinforcement learning. We conduct experiments in off-road environments. Scene maps are generated using 3D LiDAR data, and expert demonstrations are either the vehicle's real driving trajectories at the scene or synthesized ones to represent specific behaviors such as crossing negative obstacles. Different cost functions of traversability analysis are learned and tested at various scenes of capability in guiding the trajectory planning of different behaviors. We also demonstrate the performance and computation efficiency of the proposed method.

Keywords: Self-Driving Vehicles, Automated Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: Path planning is a key component in motion planning for autonomous vehicles. A path specifies the geometrical shape that the vehicle will travel, thus, it is critical to safe and comfortable vehicle motions. For urban driving scenarios, autonomous vehicles need the ability to navigate in cluttered environment, e.g., roads partially blocked by a number of vehicles/obstacles on the sides. How to generate a kinematically feasible and smooth path, that can avoid collision in complex environment, makes path planning a challenging problem. In this paper, we present a novel quadratic programming approach that generates optimal paths with resolution-complete collision avoidance capability.

Keywords: Automated Vehicles, Collision Avoidance, Vehicle Control
Abstract: Vehicle collision avoidance system (CAS) is a control system that can guide the vehicle into a collision-free safe region in the presence of other objects on road. Common CAS functions, such as forward-collision warning and automatic emergency braking, have recently been developed and equipped on production vehicles. However, these CASs focus on mitigating or avoiding potential crashes with the preceding cars and objects. They are not effective for crash scenarios with vehicles from the rear-end or lateral directions. This paper proposes a novel collision avoidance system that will provide the vehicle with all-around (360-degree) collision avoidance capability. A risk evaluation model is developed to calculate potential risk levels by considering surrounding vehicles (according to their relative positions, velocities, and accelerations) and using a predictive occupancy map (POM). By using the POM, the safest path with the minimum risk values is chosen from 12 acceleration-based trajectory directions. The global optimal trajectory is then planned using the optimal rapidly exploring random tree (RRT*) algorithm. The planned vehicle motion profile is generated as the reference for future control. Simulation results show that the developed POM-based CAS demonstrates effective operations to mitigate the potential crashes in both lateral and rear-end crash scenarios.

Keywords: Automated Vehicles, Advanced Driver Assistance Systems, Situation Analysis and Planning
Abstract: Making long-term trajectory prediction accurately for surrounding vehicles is the crucial prerequisite for intelligent vehicles to accomplish superb decision making and motion planning. In this paper, to achieve high-quality prediction accuracy both in the short and long term, we propose an integrated probabilistic framework with the combination of driver awareness model and Gaussian process model. The former model can obtain high-level semantic information using low-level two-dimensional motion elements. And the latter incorporates the vehicle physical model to reach good prediction performance with strengthened historical input sequence. Furthermore, experiments on the public naturalistic driving dataset in lane-changing scenarios are conducted to verify our novel approach. Compared with another advanced method, the superiorities of our proposed approach are demonstrated with higher estimation and prediction accuracy, as well as more reasonable uncertainty description in terms of the whole prediction process.

Keywords: Situation Analysis and Planning, Self-Driving Vehicles, Automated Vehicles
Abstract: Driving heavy-duty vehicles, such as buses and tractor-trailer vehicles, is a difficult task in comparison to passenger cars. Most research on motion planning for autonomous vehicles has focused on passenger vehicles, and many unique challenges associated with heavy-duty vehicles remain open. However, recent works have started to tackle the particular difficulties related to on-road motion planning for buses and tractor-trailer vehicles using numerical optimization approaches. In this work, we propose a framework to design an optimization objective to be used in motion planners. Based on geometric derivations, the method finds the optimal trade-off between the conflicting objectives of centering different axles of the vehicle in the lane. For the buses, we consider the front and rear axles trade-off, whereas for articulated vehicles, we consider the tractor and trailer rear axles trade-off. Our results show that the proposed design strategy produces planned paths that considerably improve the behavior of heavy-duty vehicles by keeping the whole vehicle body in the center of the lane.

Keywords: Automated Vehicles, Unsupervised Learning, Self-Driving Vehicles
Abstract: Test scenario generation for testing automated and autonomous driving systems requires knowledge about the recurring traffic cases, known as scenario types. The most common approach in industry is to have experts create lists of scenario types. This poses the risk both that certain types are overlooked; and that the mental model that underlies the manual process is inadequate. We propose to extract scenario types from real driving data by clustering recorded scenario instances, which are composed of timeseries. Existing works in the domain of traffic data either cannot cope with multivariate timeseries; are limited to one or two vehicles per scenario instance; or they use handcrafted features that are based on the mental model of the data scientist. The latter suffers from similar shortcomings as manual scenario type derivation. Our approach clusters scenario instances relying as little as possible on a mental model. As such, we consider the approach an important complement to manual scenario type derivation. It may yield scenario types overlooked by the experts, and it may provide a different segmentation of a whole set of scenarios instances into scenario types, thus overall increasing confidence in the handcrafted scenario types. We present the application of the approach to a real driving dataset.

Keywords: Autonomous / Intelligent Robotic Vehicles, Collision Avoidance, Vehicle Control
Abstract: Collision avoidance, kinematic feasibility, and road-compliance must be validated to ensure the drivability of planned motions for autonomous vehicles. Although these tasks are highly repetitive, computationally efficient toolboxes are still unavailable. The CommonRoad Drivability Checker---an open-source toolbox---unifies these mentioned checks. It is compatible with the CommonRoad benchmark suite, which additionally facilitates the development of motion planners. Our toolbox drastically reduces the effort of developing and validating motion planning algorithms. Numerical experiments show that our toolbox is real-time capable and can be used in real test vehicles.

Keywords: Driver State and Intent Recognition, Advanced Driver Assistance Systems, Human-Machine Interface
Abstract: Older drivers experience a high rate of crashes due to road intersections, unseen objects, and failure to find the traffic signals. These characteristics would be recognized if a system could watch and monitor the behaviors of the drivers inside their car. Therefore, in this study, current advances in driver monitoring cameras are used for measuring the head movements of older drivers to evaluate the visual intent of the driver. Several quantitative metrics that compute the temporal and spatial aspects of head movements allow for assessing the behaviors of older and middle-aged drivers. A driving simulator study using urban road scenarios shows that at high vehicle speed, on average, older drivers move their heads more slowly within a narrower range and glance too quickly to recognize surrounding traffic correctly. Correlation analysis validated the high prediction capabilities of head-movement measures towards the future evaluation of driving risks.

Keywords: Driver Recognition, Human-Machine Interface, Advanced Driver Assistance Systems
Abstract: Japan has become a more aged society and there are more drivers, 65 years of age and above. Cars represent an important mode of transportation for the elderly; however, in recent years, the number of traffic accidents caused by elderly drivers has been on the rise, and this has become a social issue. Thus, to ensure driving safety, we study a driver agent system that provides driving and feedback support to the elderly drivers for encouraging them to improve their driving. In this paper, we present a summary of the proposed agent and report on a set of experiments using our agent for the elderly and non-elderly drivers in an actual environment with car on public roads. From the analysis of driving operations and fixation points during driving, the results revealed that the acceptability of the agent was high, the agent in the actual car environment did not distract the driver, and the agent could improve driving behavior.

Keywords: Driver Recognition, Human-Machine Interface, Driver State and Intent Recognition
Abstract: Eye-tracking techniques have the potential for estimating driver awareness of road hazards. However, tradi- tional eye-movement measures based on static areas of interest may not capture the unique characteristics of driver eye- glance behavior and challenge the real-time application of the technology on the road. This article proposes a novel method to operationalize driver eye-movement data analysis based on moving objects of interest. A human-subject experiment conducted in a driving simulator demonstrated the potential of the proposed method. Correlation and regression analyses between indirect (i.e., eye-tracking) and direct measures (i.e., SAGAT) of driver awareness identified some promising variables that feature both spatial and temporal aspects of driver eye-glance behavior relative to objects of interest. Results also suggest that eye-glance behavior might be a promising but insufficient predictor of driver awareness. This work is a preliminary step toward real-time, on-road estimation of driver awareness of road hazards. The proposed method could be further combined with computer-vision techniques such as object recognition to fully automate eye-movement data processing as well as machine learning approaches to improve the accuracy of driver awareness estimation.

Keywords: Vision Sensing and Perception, Driver Recognition, Deep Learning
Abstract: We encounter a wide range of obstacles when integrating computer vision algorithms into applications inside the vehicle cabin, e.g. variations in illumination, sensor-type and -placement. Thus, designing domain-invariant representations is crucial for employing such models in practice. Still, the vast majority of driver activity recognition algorithms are developed under the assumption of a static domain, i.e. an identical distribution of training- and test data. In this work, we aim to bring driver monitoring to a setting, where domain shifts can occur at any time and explore generative models which learn a shared representation space of the source and target domain. First, we formulate the problem of unsupervised domain adaptation for driver activity recognition, where a model trained on labeled examples from the source domain (i.e. color images) is intended to adjust to a different target domain (i.e. infrared images) where only unlabeled data is available during training. To address this problem, we leverage current progress in image-to-image translation and adopt multiple strategies for learning a joint latent space of the source and target distribution and a mapping function to the domain of interest. As our long-term goal is a robust cross-domain classification, we enhance a Variational Auto-Encoder (VAE) for image translation with a classification-driven optimization strategy. Our model for classification-driven domain transfer leads to the best cross-domain recognition results and outperforms a conventional classification approach in color-to-infrared recognition by 13.75%.

Keywords: Driver State and Intent Recognition, Vision Sensing and Perception, Driver Recognition
Abstract: A common obstacle for applying computer vision models inside the vehicle cabin is the dynamic nature of the surrounding environment, as unforeseen situations may occur at any time. Driver monitoring has been widely researched in the context of closed set recognition i.e. under the premise that all categories are known a priori. Such restrictions represent a significant bottleneck in real-life, as the driver observation models are intended to handle the uncertainty of an open world. In this work, we aim to introduce the concept of open sets to the area of driver observation, where methods have been evaluated only on a static set of classes in the past. First, we formulate the problem of open set recognition for driver monitoring, where a model is intended to identify behaviors previously unseen by the classifier and present a novel Open-Drive&Act benchmark. We combine current closed set models with multiple strategies for novelty detection adopted from general action classification in a generic open set driver behavior recognition framework. In addition to conventional approaches, we employ the prominent I3D architecture extended with modules for assessing its uncertainty via Monte-Carlo dropout. Our experiments demonstrate clear benefits of uncertainty-sensitive models, while leveraging the uncertainty of all the output neurons in a voting-like fashion leads to the best recognition results. To create an avenue for future work, we make Open-Drive&Act public at www.github.com/aroitberg/open-set-driver-activity-recognition.

Keywords: Driver State and Intent Recognition, Advanced Driver Assistance Systems, Deep Learning
Abstract: A driver's gaze is critical for determining the driver's attention level, state, situational awareness, and readiness to take over control from partially and fully automated vehicles. Tracking both the head and eyes (pupils) can provide a reliable estimation of a driver's gaze using face images under ideal conditions. However, the vehicular environment introduces a variety of challenges that are usually unaccounted for - harsh illumination, nighttime conditions, and reflective/dark eyeglasses. Unfortunately, relying on head pose alone under such conditions can prove to be unreliable owing to significant eye movements. In this study, we offer solutions to address these problems encountered in the real world. To solve issues with lighting, we demonstrate that using an infrared camera with suitable equalization and normalization usually suffices. To handle eyeglasses and their corresponding artifacts, we adopt the idea of image-to-image translation using generative adversarial networks (GANs) to pre-process images prior to gaze estimation. To this end, we propose the Gaze Preserving CycleGAN (GPCycleGAN). This network preserves the driver's gaze while removing potential eyeglasses from infrared face images. Our approach exhibits improved performance and robustness on challenging real-world data spanning 13 subjects and a variety of driving conditions.

Keywords: Driver State and Intent Recognition, Deep Learning, Convolutional Neural Networks
Abstract: Deep learning methods have proven useful for head pose estimation, but the effect of their depth, type and input resolution based on infrared (IR) images still need to be explored. In this paper, we present a study on in-car head pose estimation on the IR images of the AutoPOSE dataset, where we extract 64  64 and 128  128 pixel cropped head images. We propose the novel networks Head Orientation Network (HON) and ResNetHG and compare them with state-of-the-art methods like the HPN model from DriveAHead on different input resolutions. In addition, we evaluate multiple depths within our HON and ResNetHG networks and their effect on the accuracy. Our experiments show that higher resolution images lead to lower estimation errors. Furthermore, we show that deep learning methods with fewer layers perform better on head orientation regression based on IR images. Our HON and ResNetHG18 architectures outperform the state-of-the-art on IR images on four different metrics, where we achieve a reduction of the residual error of up to 74%.

Keywords: Driver State and Intent Recognition, Convolutional Neural Networks, Deep Learning
Abstract: This paper proposes a novel real-time pose estimation system on embedded devices for a driver and a front passenger. The main goal of the proposed system is to operate in real time with limited hardware resources while preserving the high accuracy. The proposed system is divided into an object detection and a pose estimation. In the object detection, we eliminate the redundant and inaccurate bounding boxes by considering the characteristics of the target image domain. In the pose estimation, a single-person pose estimation with a lightweight deep learning model has been proposed and knowledge distillation has been adopted to maximize the performance while maintaining the high speed. In the experimental results, the proposed pose estimation has up to 92% of the accuracy and the 9 times less computation compared to the previous methods. The operation speed is 195 frame per second on NVIDIA Jetson TX2.

Keywords: Self-Driving Vehicles, Deep Learning, Lidar Sensing and Perception
Abstract: In this paper, we propose a point cloud based 3D object detection framework that accounts for both contextual and local information by leveraging multi-receptive field pillars, named as MuRF-Net. Recently, common pipelines can be divided into a voxel-based feature encoder and an object detector. During the feature encoding steps, contextual information is neglected, which is critical for the 3D object detection task. Thus the encoded features are not suitable to input to the subsequent object detector. To address this challenge, we propose the MuRF-Net with a multi-receptive field voxelization mechanism to capture both contextual and local information. After the voxelization, the voxelized points are processed by a feature encoder, and a channel-wise feature reconfiguration module is proposed to combine the features with different receptive fields using a lateral enhanced fusion network. In addition, to handle the increase of memory and computational cost brought by multi-receptive field voxelization, a dynamic voxel encoder is applied taking advantage of the sparseness of point cloud. Experiments on the KITTI benchmark for both 3D object and Bird’s Eye View (BEV) detection tasks on car class are conducted and MuRF-Net outperforms other voxel-based methods these detection tasks by a large margin. Besides, the MuRF-Net can achieve nearly real-time speed with 20Hz.

Keywords: Automated Vehicles, Lidar Sensing and Perception, Advanced Driver Assistance Systems
Abstract: In intelligent vehicles it is indispensable to have reliable Advanced Driver-Assistance Systems (ADAS) on board. These ADAS require various types of sensors, like Light Detection and Ranging (LiDAR). Nowadays, drivers delegate some responsibilities to their highly automated vehicles, however, it is not legally secured. Nevertheless, the legislator will in future deal with automated vehicles. The fundamentals will be laid to ensure that the transfer of responsibilities will be permitted under certain conditions. Car manufacturers, on the other hand, must ensure that components are safe and reliable. With LiDAR, this could be achieved with Micro-Electro-Mechanical System (MEMS) technology. As with humans as drivers, it is also advantageous for intelligent systems if obstacles in the environment are detected promptly. Especially when the obstacles are moving, it helps to initiate appropriate measures, such as braking. Therefore, it is attempted to extend the Field-of-View (FoV) of the various sensors. By synchronising multiple MEMS mirrors, it is able to extend the FoV of the LiDAR part in an environmental perception system. In this publication, an architecture is proposed for MEMS-based Micro-Scanning LiDAR Systems to achieve synchronisation of multiple independently controlled MEMS mirrors. The architecture was implemented in an FPGA prototyping platform to show its feasibility and evaluate its performance.

Keywords: Lidar Sensing and Perception, Deep Learning, Vehicle Environment Perception
Abstract: This work proposes a spatially-conditioned neural network to perform semantic segmentation and geometric scene completion in 3D on real-world LiDAR data. Spatially-conditioned scene segmentation (SCSSnet) is a representation suitable to encode properties of large 3D scenes at high resolution. A novel sampling strategy encodes free space information from LiDAR scans explicitly and is both simple and effective. We avoid the need for synthetically generated or volumetric ground truth data and are able to train and evaluate our method on semantically annotated LiDAR scans from the Semantic KITTI dataset. Ultimately, our method is able to predict scene geometry as well as a diverse set of semantic classes over a large spatial extent at arbitrary output resolution instead of a fixed discretization of space.

Our experiments confirm that the learned scene representation is versatile and powerful and can be used for multiple downstream tasks. We perform point-wise semantic segmentation, point-of-view depth completion and ground plane segmentation. The semantic segmentation performance of our method surpasses the state of the art by a significant margin of 7 % mIoU.

Keywords: Lidar Sensing and Perception, Autonomous / Intelligent Robotic Vehicles
Abstract: In this work, we present LIBRE: LiDAR Benchmarking and Reference, a first-of-its-kind dataset featuring 10 different LiDAR sensors, covering a range of manufacturers, models, and laser configurations. Data captured independently from each sensor includes three different environments and configurations: static targets, where objects were placed at known distances and measured from a fixed position within a controlled environment; adverse weather, where static obstacles were measured from a moving vehicle, captured in a weather chamber where LiDARs were exposed to different conditions (fog, rain, strong light); and finally, dynamic traffic, where dynamic objects were captured from a vehicle driven on public urban roads, multiple times at different times of the day, and including supporting sensors such as cameras, infrared imaging, and odometry devices. LIBRE will contribute to the research community to (1) provide a means for a fair comparison of currently available LiDARs, and (2) facilitate the improvement of existing self-driving vehicles and robotics-related software, in terms of development and tuning of LiDAR-based perception algorithms.

Keywords: Lidar Sensing and Perception, Image, Radar, Lidar Signal Processing, Vision Sensing and Perception
Abstract: We present an endpoint box regression module(epBRM), which is designed for predicting precise 3D bounding boxes using raw LiDAR 3D point clouds. The proposed epBRM is built with sequence of small networks and is computationally lightweight. Our approach can improve a 3D object detection performance by predicting more precise 3D bounding box coordinates. The proposed approach requires 40 minutes of training to improve the detection performance. Moreover, epBRM imposes less than 12ms to network inference time for up-to 20 objects.

The proposed approach utilizes a spatial transformation mechanism to simplify the box regression task. Adopting spatial transformation mechanism into epBRM makes it possible to improve the quality of detection with a small sized network.

We conduct in-depth analysis of the effect of various spatial transformation mechanisms applied on raw LiDAR 3D point clouds. We also evaluate the proposed epBRM by applying it to several state-of-the-art 3D object detection systems.

We evaluate our approach on KITTI dataset, a standard 3D object detection benchmark for autonomous vehicles. The proposed epBRM enhances the overlaps between ground truth bounding boxes and detected bounding boxes, and improves 3D object detection. Our proposed method evaluated in KITTI test server outperforms current state-of-the-art approaches.

Keywords: Vehicle Environment Perception, Lidar Sensing and Perception, Image, Radar, Lidar Signal Processing
Abstract: The major challenges of developing 3D point cloud annotation systems for autonomous driving datasets include convenient user-data interfaces, efficient operations on geometric data units, and scalable annotation tools. This paper presents a {Portable pOint-cloud Interactive aNnotation plaTform System} (i.e. SUSTech POINTS), which contains a set of user-friendly interfaces and efficient annotation tools to help achieve high-quality data annotations with high efficiency. The novelty of this work is threefold: (1) developing a set of visualization modules for fast annotation error localization and convenient annotator-data interactions; (2) developing a set of interactive tools for annotators labeling 3D point clouds and 2D images in high speed; (3) developing an annotation transfer method to label the same objects in different data frames. The developed POINTS system is tested with public datasets such as KITTI and a private dataset (SUSTech SCAPES). The experimental results show that the developed platform can help improve the annotation accuracy and efficiency compared with using other open-source annotation platforms.

Keywords: Lidar Sensing and Perception, Convolutional Neural Networks, Vehicle Environment Perception
Abstract: Autonomous vehicles need to have a semantic understanding of the three-dimensional world around them in order to reason about their environment. State of the art methods use deep neural networks to predict semantic classes for each point in a LiDAR scan. A powerful and efficient way to process LiDAR measurements is to use two-dimensional, image-like projections. In this work, we perform a comprehensive experimental study of image-based semantic segmentation architectures for LiDAR point clouds. We demonstrate various techniques to boost the performance and to improve runtime as well as memory constraints.

First, we examine the effect of network size and suggest that much faster inference times can be achieved at a very low cost to accuracy. Next, we introduce an improved point cloud projection technique that does not suffer from systematic occlusions. We use a cyclic padding mechanism that provides context at the horizontal field-of-view boundaries. In a third part, we perform experiments with a soft Dice loss function that directly optimizes for the intersection-over-union metric. Finally, we propose a new kind of convolution layer with a reduced amount of weight-sharing along one of the two spatial dimensions, addressing the large difference in appearance along the vertical axis of a LiDAR scan.

We propose a final set of the above methods with which the model achieves an increase of 3.2% in mIoU segmentation performance over the baseline while requiring only 42% of the original inference time.

Keywords: Situation Analysis and Planning, Convolutional Neural Networks, Recurrent Networks
Abstract: Predicting the future trajectories for all vehicles relevant to the ego vehicle is a crucial, yet unsolved challenge to master automated driving. This paper proposes a combination of two lines of research for predicting all the trajectories of a group of vehicles of arbitrary size, considering the mutual interactions possible. Treating the prediction of other vehicles as a planning task for themselves enables the application of the artificial potential field approach. Modeling the driving situation as a potential field turns the trajectory prediction problem back to its original domain – utility space. Humans generate trajectories (that should be predicted) during driving by balancing costs and rewards, which lead to a total utility. The main difficulty inherent to the potential field approach is the hard problem of parameter tuning. Therefore, it is not directly used for prediction. Instead, the potential field representation is used as input for a neural network, which predicts a distribution over trajectories based on distinct maneuvers. This allows a multi-modal prediction for each vehicle and reflects the pattern recognition character.

Keywords: Situation Analysis and Planning, Automated Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: Recent trajectory or maneuver planning approaches in automated driving show the tendency to get more complex or even become a black box. Thus, an algorithm‘s decisions become less transparent, especially when uncertain input parameters have a large influence. To identify those input parameters whose uncertainty is more relevant than others’, Morris’ method of elementary effects is used here. It is a quantitative sensitivity analysis that classifies the inputs into relevant and irrelevant, depending on how sensitive the algorithm reacts to changes in the input. The method is adapted to analyze the behavior of a car-following and lane-changing model during an overtaking maneuver with two vehicles. The results show that Morris’ method is capable of determining important parameters for each situation. It is even possible to identify boundaries for the necessary accuracy of each input. With this, we are able to determine input parameter ranges for which the planning algorithm is able to produce reliable output.

Keywords: Autonomous / Intelligent Robotic Vehicles
Abstract: This paper presents the application of optimal control problem formulation for motion planning in order to minimise motion sickness in autonomous vehicles. The optimum velocity profile is sought for a predefined road path from a specific starting point to a final one within specific and given boundaries and constrains in order to minimise the motion sickness without compromising the journey time. For the representation of the motion sickness as a cost function to our optimal control problem, the illness rating is used. The influence of road width flexibility on illness rating and for a set of fixed journey times are investigated. According to the results, the road with flexible lateral manoeuvrability is found to provide lower sickness compared to the fixed road path. Moreover, the correlation between sickness and journey time can be represented as a Pareto front.

Keywords: Situation Analysis and Planning, Deep Learning, Self-Driving Vehicles
Abstract: Humans navigate complex environments in an organized yet flexible manner, adapting to the context and implicit social rules. Understanding these naturally learned patterns of behavior is essential for applications such as autonomous vehicles. However, algorithmically defining these implicit rules of human behavior remains difficult. This work proposes a novel self-supervised method for training a probabilistic network model to estimate the regions humans are most likely to drive in as well as a multimodal representation of the inferred direction of travel at each point. The model is trained on individual hu- man trajectories conditioned on a representation of the driving environment. The model is shown to successfully generalize to new road scenes, demonstrating the model’s potential for real- world application as a prior over socially acceptable driving behavior in challenging or ambiguous scenarios which are poorly handled by explicit traffic rules.

Keywords: Assistive Mobility Systems, Situation Analysis and Planning, Recurrent Networks
Abstract: Two-thirds of the people who buy a new car prefer to use a substitute instead of the built-in navigation system. However, for many applications, knowledge about a user's intended destination and route is crucial. For example, suggestions for available parking spots close to the destination can be made or ride-sharing opportunities along the route are facilitated. Our approach predicts probable destinations and routes of a vehicle, based on the most recent partial trajectory and additional contextual data. The approach follows a three-step procedure: First, a k-d tree-based space discretization is performed, mapping GPS locations to discrete regions. Secondly, a recurrent neural network is trained to predict the destination based on partial sequences of trajectories. The neural network produces destination scores, signifying the probability of each region being the destination. Finally, the routes to the most probable destinations are calculated. To evaluate the method, we compare multiple neural architectures and present the experimental results of the destination prediction. The experiments are based on two public datasets of non-personalized, timestamped GPS locations of taxi trips. The best performing models were able to predict the destination of a vehicle with a mean error of 1.3km and 1.43km respectively.

Keywords: Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles, Automated Vehicles
Abstract: Automated lane change is one of the most challenging task to be solved of highly automated vehicles due to its safety-critical, uncertain and multi-agent nature. This paper presents the novel deployment of the state of art Q learning method, namely Rainbow DQN, that uses a new safety driven rewarding scheme to tackle the issues in an dynamic and uncertain simulation environment. We present various comparative results to show that our novel approach of having reward feedback from the safety layer dramatically increases both the agent's performance and sample efficiency. Furthermore, through the novel deployment of Rainbow DQN, it is shown that more intuition about the agent's actions is extracted by examining the distributions of generated Q values of the agents. The proposed algorithm shows superior performance to the baseline algorithm in the challenging scenarios with only 200000 training steps (i.e. equivalent to 55 hours driving).

Keywords: Autonomous / Intelligent Robotic Vehicles, Vehicle Control, Self-Driving Vehicles
Abstract: Model-based approaches have become increasingly popular in the domain of automated driving. This includes runtime algorithms, such as Model Predictive Control, as well as formal and simulative approaches for the verification of automated vehicle functions. With this trend, the quality of models becomes a key factor for automated vehicle safety. Established tools from model theory which can be applied to assure the quality of models are uncertainty and sensitivity analysis.

In this paper, we conduct sensitivity analyses for a single and double track vehicle dynamics model to gain insights about the models' behavior under different operating conditions. We compare the models, point out the most important findings regarding the obtained parameters sensitivities, and provide examples of possible applications of the gained insights.

Keywords: Driver State and Intent Recognition, Automated Vehicles, Deep Learning
Abstract: We investigate the problem of predicting driver behavior in parking lots, an environment which is less structured than typical road networks and features complex, interactive maneuvers in a compact space. Using the CARLA simulator, we develop a parking lot environment and collect a dataset of human parking maneuvers. We then study the impact of model complexity and feature information by comparing a multi-modal Long Short-Term Memory (LSTM) prediction model and a Convolution Neural Network LSTM (CNN-LSTM) to a physics-based Extended Kalman Filter (EKF) baseline. Our results show that 1) intent can be estimated well (roughly 85% top-1 accuracy and nearly 100% top-3 accuracy with the LSTM and CNN-LSTM model); 2) knowledge of the human driver’s intended parking spot has a major impact on predicting parking trajectory; and 3) the semantic representation of the environment improves long term predictions.

Keywords: Driver State and Intent Recognition, Advanced Driver Assistance Systems, Deep Learning
Abstract: Head pose estimation has been a key task in computer vision since a broad range of applications often require accurate information about the orientation of the head. Achieving this goal with regular RGB cameras faces challenges for automotive applications due to occlusions, extreme head poses and sudden changes in illumination. Most of these challenges can be attenuated with algorithms relying on depth cameras. This paper proposes a novel point-cloud based deep learning approach to estimate the driver head pose from depth camera data, addressing these challenges. The proposed algorithms is inspired by the PointNet++ framework, where points are sampled and grouped before extracting discriminative features. We demonstrate the effectiveness of our algorithm by evaluating our approach on a naturalistic driving database from 22 drivers, where the benchmark for the orientation of the driver's head is obtained with the Fi-Cap device. The experimental evaluation demonstrates that our proposed approach relying on point-cloud data achieves predictions that are almost always more reliable than state-of-the-art head pose estimation methods based on regular cameras. Furthermore, our approach provides predictions even for extreme rotations, which is not the case for the baseline methods. To the best of our knowledge, this is the first study to propose head pose estimation using deep learning on point cloud data.

Keywords: Driver State and Intent Recognition, Driver Recognition, Advanced Driver Assistance Systems
Abstract: In this paper, a driver's intention prediction near a road intersection is proposed. Our approach uses a deep bidirectional Long Short-Term Memory (LSTM) with an attention mechanism model based on a hybrid-state system (HSS) framework. As intersection is considered to be as one of the major source of road accidents, predicting a driver's intention at an intersection is very crucial. Our method uses a sequence to sequence modeling with an attention mechanism to effectively exploit temporal information out of the time-series vehicular data including velocity and yaw-rate. The model then predicts ahead of time whether the target vehicle/driver will go straight, stop, or take right or left turn. The performance of the proposed approach is evaluated on a naturalistic driving dataset and results show that our method achieves high accuracy as well as outperforms other methods. The proposed solution is promising to be applied in advanced driver assistance systems (ADAS) and as part of active safety system of autonomous vehicles.

Keywords: Driver State and Intent Recognition, Human-Machine Interface, Automated Vehicles
Abstract: Many producers of automated vehicle systems have begun testing autonomous vehicles on the road. In order to ensure safety and prevent crashes, human drivers are enlisted to monitor autonomous vehicles. However, operators of autonomous systems exhibit negative behavior adaptations in response to prolonged supervision of automation. To pre-vent the onset of undesirable behaviors in safety drivers, we must investigate driver state and behavior changes during the operation of highly automated vehicles. In the study presented here, we examine the effects of theoretical and practical training on the drivers’ response to potentially critical situations in a longitudinal driving simulator study. We also present the effects of encountering a failure of the automated vehicle on driver state and behavior. We conducted a two-part panel driving simulator study (N=28), with an interval of 25-days between the training and testing sessions. We found that while participants with training are better prepared for a potential failure of the automation, participants in both conditions show a rise in sleepy or drowsy behavior before a potential failure of automation

Keywords: Automated Vehicles, Driver State and Intent Recognition, Unsupervised Learning
Abstract: Generating multi-vehicle interaction scenarios can benefit the motion planning and decision making of autonomous vehicles when on-road data is insufficient. This paper presents an efficient approach to generate varied multi-vehicle interaction scenarios that can both adapt to different road geometries and inherit the key interaction patterns in real-world driving. Towards this end, the available multi-vehicle interaction scenarios are temporally segmented into several interpretable fundamental building blocks, called traffic primitives, via the Bayesian nonparametric learning. Then, the changepoints of traffic primitives are transformed into the desired road to generate collision-free interaction trajectories through a sampling-based path planning algorithm. The Gaussian process regression is finally introduced to control the variance and smoothness of the generated multi-vehicle interaction trajectories. Experiments with simulation results of three multi-vehicle trajectories at different road conditions are carried out. The experimental results demonstrate that our proposed method can generate a bunch of human-like multi-vehicle interaction trajectories that can fit different road conditions remaining the key interaction patterns of agents in the provided scenarios, which is import to the development of autonomous vehicles.

Keywords: Reinforcement Learning, Automated Vehicles, Driver State and Intent Recognition
Abstract: Reinforcement learning is nowadays a popular framework for solving different decision making problems in automated driving. However, there are still some remaining crucial challenges that need to be addressed for providing more reliable policies. In this paper, we propose a generic risk-aware DQN approach in order to learn high level actions for driving through unsignalized occluded intersections. The proposed state representation provides lane based information which allows to be used for multi-lane scenarios. Moreover, we propose a risk based reward function which punishes risky situations instead of only collision failures. Such rewarding approach helps to incorporate risk prediction into our deep Q network and learn more reliable policies which are safer in challenging situations. The efficiency of the proposed approach is compared with a DQN learned with conventional collision based rewarding scheme and also with a rule-based intersection navigation policy. Evaluation results show that the proposed approach outperforms both of these methods. It provides safer actions than collision-aware DQN approach and is less overcautious than the rule-based policy.

Keywords: Driver Recognition, Electric and Hybrid Technologies, Advanced Driver Assistance Systems
Abstract: Accurate recognition of driver braking intensity is of great importance for advanced control of intelligent braking system. In this paper, the braking intensity is classified into four clusters based on an unsupervised Gaussian mixture model (GMM). Then, the architecture of an adaptive-network-based fuzzy inference system (ANFIS) is proposed for braking intensity prediction. A batch learning rule that combines the recursive least squares and gradient descent method used for training ANFIS is adopted to improve the generalization capability. The training data are collected from a hybrid vehicle under real driving conditions. In addition, co-simulation with the software of MATLAB/Simulink and Hardware-in-the-Loop (HiL) tests for an Electronic-Hydraulic Brake (EHB) system are carried out. In comparison to other typical learning methods such as the feedback neural network and the recurrent neural network, the simulation and experimental results demonstrate the effectiveness and accuracy of the proposed hybrid learning approach for braking intensity recognition in different braking scenarios.

Keywords: Advanced Driver Assistance Systems, Vision Sensing and Perception, Vehicle Environment Perception
Abstract: The concept of a visually-assisted anti-lock braking system (ABS) is presented. The road conditions in front of the vehicle are assessed in real time based on camera data. The surface type classification (tarmac, gravel, ice, etc.) and related road friction properties are provided to the braking control algorithm in order to adjust the vehicle response accordingly. The system was implemented and tested in simulations as well as on an instrumented sub-scale vehicle. Simulations and experimental results quantitatively demonstrate the benefits of the proposed system in critical maneuvers, such as emergency braking and collision avoidance.

Keywords: Vision Sensing and Perception, Automated Vehicles, Advanced Driver Assistance Systems
Abstract: For safe driving in parking lots, differentiating the few moving vehicles from the many parked is essential but also challenging. Vehicles tend to move at slow speeds in parking areas and, therefore, low-cost sensors including cameras and radars cannot detect their motion using conventional object-localization and speed-measurement methods. This paper presents a novel method of detecting motion of a target vehicle by equating it to rotation of the vehicle’s wheels. Using a monocular 2MP camera, the algorithm is demonstrated to detect motion as slow as 0.1 mph at distances of up to 30 meters away while the host vehicle itself moves at speeds up to 15 mph. Test results also promise an easy and cost-effective path to further increasing the range of the algorithm.

Keywords: Vision Sensing and Perception, Image, Radar, Lidar Signal Processing, Vehicle Environment Perception
Abstract: Environment perception for driving applications requires very accurate sensors especially when dealing with depth measurements. LiDAR is the most trustworthy sensor in this domain, but it suffers from disadvantages in terms of number of scene points and their temporal alignment. These issues are especially relevant when dealing with long-range measurements, where each 3D point is crucial. As an alternative, in this work we focus on camera-based depth perception for objects at large distance by using stereo reconstruction and monocular depth estimation. Towards improving the capabilities of camera-based, we initially introduce a taxonomy to categorize all types of camera-based depth perception methods with respect to their long-range capabilities. We then present a correction method that works for both stereo and monocular depth perception algorithms that output a depth in discrete setting (most suitable for real-time applications). We show that our method improves the precision for such algorithms for objects at large distances without affecting the near-range accuracy. The method requires only several additional operations, preserving the real-time capabilities of the underlying algorithms.

Keywords: Vision Sensing and Perception, Automated Vehicles, Advanced Driver Assistance Systems
Abstract: Although the number of camera-based sensors mounted on vehicles has recently increased dramatically, robust and accurate object velocity detection is difficult. Additionally, it is still common to use radar as a fusion system. We have developed a method to accurately detect the velocity of object using a camera, based on a large-scale dataset collected over 20 years by the automotive manufacturer, SUBARU. The proposed method consists of three methods: a High Dynamic Range (HDR) detection method that fuses multiple stereo disparity images, a fusion method that combines the results of monocular and stereo recognitions, and a new velocity calculation method. The evaluation was carried out using measurement devices and a test course that can quantitatively reproduce severe environment by mounting the developed stereo camera on an actual vehicle.

Keywords: Vision Sensing and Perception, Vehicle Environment Perception, Advanced Driver Assistance Systems
Abstract: The validation of automated driving requires billions of kilometers of test drives to be performed so that safety-in-use is assured. It is difficult to validate it only making use of data acquired on field tests in public roads due to the lack of controllability, e.g. over environment conditions. Therefore, the automotive industry relies on test drives to be executed on a proving ground under controlled conditions or in environment simulation software. The first is realistic, but costly in terms of time and effort. The latter provides a high level of reproducibility, but it is still uncertain how valid the delivered test results are. In this paper, a test method for measuring the simulation-to-reality gap is proposed. For this purpose, a test scenario is defined, built on a proving ground and reproduced in two environment simulation software. Four different environment conditions are considered: day, night, fog and rain. The video data of the real and simulated test drives are recorded and fed into a seriesproduced multi-class object detection algorithm for automated driving. Performance metrics are calculated across the real and virtual domains. Finally, the test results are compared so that the simulation-to-reality gap concerning object detection is measured.

Keywords: Vision Sensing and Perception, Vehicle Environment Perception, Automated Vehicles
Abstract: One major task in automated driving is the development of robust and safe visual perception modules. It is of utmost importance that visual perception reacts adequately to so-called corner cases, which range from overexposure of the image sensor to unexpected and potentially dangerous traffic situations. Their detection thus has high significance both as an online system in the intelligent vehicle, but also in the extraction of relevant training and test data for perception modules. In this paper, we provide a systematization of corner cases for visual perception in automated driving, with the categories being structured by detection complexity. Furthermore, we discuss existing metrics and datasets which can be used for the evaluation of corner case detection methods depending on their suitability to provide beneficial information for the various categories.

Keywords: Vision Sensing and Perception
Abstract: In this paper, a method for estimating the position of obstacles existing outside the line of sight of the vehicle is proposed. By using a stereo vision camera, it estimates the position of obstacles reflected on the side of another vehicle near the subject vehicle or the glass surface of the building next to the road. First, to clarify the feasibility of such an application, the frequency of the reflection of the virtual image of the vehicle traveling in front of the preceding vehicle of the ego-vehicle on the side of the vehicle traveling in the adjacent lane is measured. Next, since those reflecting surfaces are oriented in various directions, proposed method takes its orientation into account for estimating the position of obstacles. As a result of experiments on the 1/5 scale, we could estimate the position within 20 cm of the error for obstacles existing at distances of 1.4 m to 4 m from the camera in the longitude distance direction. This result indicates that it is possible to estimate the position of the obstacle existing outside the line of sight that is present at a distance of about 20 m from the 5 m in the actual road situation with a precision of about 1 to 2 m.

Keywords: Vision Sensing and Perception, Image, Radar, Lidar Signal Processing, Vehicle Environment Perception
Abstract: In this paper, an image captioning network is proposed for traffic scene modeling, which incorporates element attention into the encoder-decoder mechanism to generate more reasonable scene captions. Firstly, the traffic scene elements are detected and segmented according to their clustered locations. Then, the image captioning network is applied to generate the corresponding caption of each subregion. The static and dynamic traffic elements are appropriately organized to construct a 3D corridor scene model. The semantic relationships between the traffic elements are specified according to the captions. The constructed 3D scene model can be utilized for the offline test of unmanned vehicles. The evaluations and comparisons based on the TSD-max and MSCOCO datasets prove the effectiveness of the proposed framework.

Keywords: Automated Vehicles, Self-Driving Vehicles
Abstract: Automated Driving System (ADS) safety assessment is a crucial step before deployment on public roads. Despite the importance of ADS safety assurance to test ADS reliability, most of the existing work is strongly attached to a single testing data source (i.e. on-road collected testing data, simulation or test track). Each source has different fidelity levels and capabilities, therefore there is a lack of a solution that allows for all data sources to complement each other to enable agnostic end to end evaluation and contributes towards different testing goals. Evaluation of ADSs is considered as a mandatory step in the autonomous vehicle development life cycle, demanding a reliable and comprehensive method. Here, we propose a source-agnostic framework, which can perform ADS evaluation compatible with different testing sources. Our findings show that this comprehensive solution can save time, effort and money consumed in ADS evaluation.

Keywords: Self-Driving Vehicles, Intelligent Vehicle Software Infrastructure, Collision Avoidance
Abstract: The current test of autonomous driving technology requires extensive experimental verification, whether in simulation or on real roads, but how to test autonomous vehicles thoroughly in a safe and comprehensive manner remains a major challenge. The purpose of this paper is to propose a novel mixed test environment-based validation method with vehicle in the loop(ViL) for safer and more effective autonomous driving testing: (1) our method supports more realistic drive safety tests in mixed scenarios which integrate the synthetic and the real-world scenarios. Synthetic scenarios offer complex traffic simulation with diverse road conditions and the real-world scenarios introduce the real autonomous driving vehicle, the real sensor suite as well as the test field to the test loop, having further bridged the gap between Hardware-in-the-Loop(HiL) testing and real road tests than ViL; (2) virtual perceptional results are simulated directly and delivered to the real vehicle in Unified Fusion Data Format(UFDF) without rendering virtual detection data for reduced resource consumption; (3) diverse test scenarios are configurable and reproducible with OSM-based High Definition(HD) map, enabling the simulation to be decoupled from a specific test filed or traffic facilities. A series of experiments on the application of our method have been demonstrated, and our approach is proved to be a promising drive safety testing technique before actual road testing.

Keywords: Self-Driving Vehicles, Collision Avoidance, Situation Analysis and Planning
Abstract: Abstract--- Autonomous vehicles navigation in complex scenarios is still an open issue. One of the major challenges is the safe autonomous vehicles navigation among regular vehicles. In facts, human-driven vehicles behavior and intentions are hard to predict and understand. In this mixed traffic environment, it is considered a hard task to take decisions for an autonomous vehicle. In this work, we propose a strategy to make an autonomous vehicle able to cross safely a roundabout. Our approach relies on High-Definition maps with lane level description which allows to predict the future situation thanks to the concept of virtual vehicles. In particular, this method handles safely collision avoidance and guarantees that no priority constraint is violated during the insertion maneuver without being overly cautious. The performance is evaluated with the SUMO simulator framework. An highly interactive vehicles flow has been generated according to the real data of roundabout scenarios from the INTERACTION dataset. We also propose strategies to extend our algorithm to multi-lane roundabouts and report how these extensions behave in terms of safety and traffic flow.

Keywords: Self-Driving Vehicles, Active and Passive Vehicle Safety
Abstract: The complex functional structure of driverless vehicles induces a multitude of potential malfunctions. Established approaches for a systematic hazard identification generate individual potentially hazardous scenarios for each identified malfunction. This leads to inefficiencies in a purely expert-based hazard analysis process, as each of the many scenarios has to be examined individually. In this contribution, we propose an adaptation of the strategy for hazard identification for the development of automated vehicles. Instead of focusing on malfunctions, we base our process on deviations from desired vehicle behavior in selected operational scenarios analyzed in the concept phase. By evaluating externally observable deviations from a desired behavior, we encapsulate individual malfunctions and reduce the amount of generated potentially hazardous scenarios. After introducing our hazard identification strategy, we illustrate its application on one of the operational scenarios used in the research project UNICARagil.

Keywords: Automated Vehicles, Self-Driving Vehicles, Advanced Driver Assistance Systems
Abstract: Many approaches for testing automated and autonomous driving systems in dynamic traffic scenarios rely on the reuse of test cases, e.g., recording test scenarios during real test drives or creating “test catalogs”. Both are widely used in industry and in literature. By counterexample, we show that the quality of test cases is system-dependent and that faulty system behavior may stay unrevealed during testing if test cases are naively re-used. We argue that, in general, system-specific “good” test cases need to be generated. Thus, recorded scenarios in general cannot simply be used for testing, and regression testing strategies needs to be rethought for automated and autonomous driving systems. The counterexample involves a system built according to state-of-the-art literature, which is tested in a traffic scenario using a high-fidelity physical simulation tool. Test scenarios are generated using standard techniques from the literature and state-of-the-art methodologies. By comparing the quality of test cases, we argue against a naive re-use of test cases.

Keywords: Reinforcement Learning, Automated Vehicles, Vehicle Control
Abstract: Automated driving in urban settings is challenging. Human participant behavior is difficult to model, and conventional, rule-based Automated Driving Systems (ADSs) tend to fail when they face unmodeled dynamics. On the other hand, the more recent, end-to-end Deep Reinforcement Learning (DRL) based model-free ADSs have shown promising results. However, pure learning-based approaches lack the hard-coded safety measures of model-based controllers. Here we propose a hybrid approach for integrating a path planning pipe into a vision based DRL framework to alleviate the shortcomings of both worlds. In summary, the DRL agent is trained to follow the path planner's waypoints as close as possible. The agent learns this policy by interacting with the environment. The reward function contains two major terms: the penalty of straying away from the path planner and the penalty of having a collision. The latter has precedence in the form of having a significantly greater numerical value. Experimental results show that the proposed method can plan its path and navigate between randomly chosen origin-destination points in CARLA, a dynamic urban simulation environment. Our code is open-source and available online.

Keywords: Self-Driving Vehicles, Advanced Driver Assistance Systems, Intelligent Vehicle Software Infrastructure
Abstract: Assuring the safety of autonomous vehicles is one of the most significant challenges in the automotive industry. Tech companies and automotive manufacturers use the idea of Operational Design Domain (ODD) to indicate where their Automated Driving Systems (ADS) can operate safely. By definition from SAE J3016, an ODD defines where the ADS is designed to operate. However, it is loosely defined in no particular format, and it is unclear how exactly to formulate the ODD, which leaves it up to the ADS developer to determine. This paper proposes a methodology to identify an ODD for an ADS with statistical data and risk tolerance, where the identified ODD is constituted of a geographical map where the risk of ADS operation is lower than the pre-determined risk threshold for a given set of environmental conditions. Two different ADSs are run through this method as an example to showcase the methodology and link the identified ODD directly to the calculated performance of the ADSs. This systematically generated ODD can mitigate potential safety issues by informing the limitations of the ADS to safety drivers, through geographic and environmental boundaries.

Keywords: Automated Vehicles, Collision Avoidance, Vehicle Control
Abstract: It is necessary to evaluate the safety of highly automated vehicles (HAVs) rigorously before their deployment on public roads. This paper describes a procedure to conduct behavioral competence tests for HAVs using the unprotected left-turn as the illustrating scenario. We first describe a model-based method for test case generation. Subsequently, we propose two methods for synchronizing the motions of the primary other vehicle (POV) and the vehicle under test (VUT), one using fixed speed profile, the other using model predictive control (MPC). Finally, we implement the POV algorithm for the left-turn scenario on an experimental vehicle, and conduct field tests using both virtual and real VUT in the Mcity test facility.

Keywords: Hand-off/Take-Over, Human-Machine Interface, Active and Passive Vehicle Safety
Abstract: Autonomous vehicle systems and their users need to collaborate to navigate the driving environment, particularly during an unstructured transition of control from automation, when the system releases control and expects the human to immediately assume driving responsibility. We investigated how such transitions affect the user’s trust in the system and subsequent performance. In a full-vehicle driving simulator, participants encountered two system failures: the first varied in severity (mild or severe failure), and the second required a transition of control that was either detected and alerted (loud failure) or not (silent failure). We observed (i) significant changes in user trust in the system over time and between events, and (ii) that the first failure’s severity level did not affect user performance in the subsequent failure; rather, the system’s detection and alert both times was sufficient to successfully complete the transition of control.

Keywords: Human-Machine Interface, Automated Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: The automated vehicles are experiencing a rapid development in worldwide recently. It is commonly believed that before the achievement of fully autonomous driving, the driver will always need to be remained within the vehicle control loop. Hence, intelligent interaction and collaboration between the human driver and the automation will be an efficient solution for the improvement of road safety, traffic efficiency, and social acceptance to the automated vehicles. As a popular collaboration method, shared control has been widely studied in the past two decades. While, it is still a challenging task to involve rich human factors into the shared control system to increase the driving experience and acceptance of the automation. In this study, a literature review on human-centered shared control is proposed towards a solid research on driver-vehicle collaboration. First, the basic background and literature surveys on the human-machine collaboration (HMC) is proposed, and the important factors for efficient multi-agent collaboration and teaming are discussed. Then, different driver behavior and states modeling methods are reviewed. Based on the HMC schemes and driver behavior recognition techniques, literature surveys on human-centered shared control are proposed. Finally, challenges and future works on human-centered shared control are analyzed.

Keywords: Autonomous / Intelligent Robotic Vehicles, Hand-off/Take-Over, Human-Machine Interface
Abstract: Take-over process refers to the situation where drivers are required to take over control of vehicles under certain conditions, which currently lacks safety analysis. To address this problem, Systems-Theoretic Process Analysis (STPA) is extended according to ISO 21448 for the safety analysis of a take-over system, which consists of a driver and a human-machine interface (HMI). First, system-level hazards are analyzed to develop safety constraints. Then, a control structure of the take-over process is modeled to identify unsafe control actions considering human factors. Finally, causal factors of the unsafe control actions are analyzed, and safety requirements regarding HMI are put forward to guide the design of the take-over system. This method was applied to analyze the take-over system of an autonomous vehicle operating in a closed area, and corresponding safety requirements were determined. Based on the results of a questionnaire survey and a take-over test, the analysis method is proved to be applicable to improving the safety of the take-over system.

Keywords: Human-Machine Interface, Advanced Driver Assistance Systems, Hand-off/Take-Over
Abstract: Interaction research has been initially focusing on partially and conditionally automated vehicles. Augmented Reality (AR) may provide a promising way to enhance drivers' experience when using autonomous driving (AD) systems. This study sought to gain insights on drivers’ subjective assessment of a simulated driving automation system with AR-based support. A driving simulator study was conducted and participants’ ratings of the AD system in terms of information imparting, nervousness and trust was collected. Cumulative Link Models (CLMs) were developed to investigate the impacts of AR cues, traffic density and intersection complexity on drivers' attitudes towards the presented AD system. Random effects were incorporated in the CLMs to account for the heterogeneity among participants. Results indicated that AR graphic cues can significantly improve drivers’ experience by providing advice for their decision-making and mitigating their anxiety and stress. However, the magnitude of AR’s effect was impacted by traffic conditions (i.e. diminished at more complex intersections). The study also revealed a strong correlation between elf-rated trust and takeovers, suggesting takeover and other driving behavior could be further examined in future studies.

Keywords: Cooperative ITS, Automated Vehicles, Human-Machine Interface
Abstract: The interactions between driver, vehicle and environment generate vehicle’s behaviors, and the interactions of vehicle groups shape the traffic modes. Therefore, the conditionally or fully automated driving technologies should be developed and tested in the driver-vehicle-environment (DVE) system rather than being developed and tested individually. To build DVE system dynamics, firstly, we propose an architecture to cope with the complicated interactions in automated vehicles (AVs) and in mixed traffic situations. Then we summarize the driver behavior models and compare the differences of intelligence between human driver and automated vehicle. Finally, we summarize the feasible modeling approaches of DVE system into five categories. The primary distinctions are the modeling methods of human drivers’ roles, which are realized by human driver per se, human driver’s cognitive architecture, psychological motivation model, mechanism imitation, and specific mechanism transfer respectively. Taking the applications of human machine interface and AD strategy developments as examples, we analyze the benefits and drawbacks of these approaches.

Keywords: Advanced Driver Assistance Systems, Vehicle Control, Human-Machine Interface
Abstract: This paper presents an approach of utilizing Driving Envelope (DE) restrictions to assist the driver in lateral maneuvers. Two fundamental issues are addressed in this work. First, DE protection. Second, how to implement such functionality on a conventional car configuration, where the driver still needs to be a part of the control loop. The proposed functionality is based on a Model Predictive Controller (MPC). Vehicle states are constrained to avoid car critical spin situations (DE protection). The power-assisted steering system is used to guide the driver inside boundaries defined by the DE. The proposed architecture is compared with the standard car, without such an Advanced Driver-Assistance System (ADAS), by means of virtual ride tests performed using a high-fidelity vehicle model.

Keywords: Automated Vehicles, Human-Machine Interface, Privacy
Abstract: Autonomous vehicles (AVs) are expected to be an integral part of the next generation of transportation systems, where they will share the transportation network with human-driven vehicles during the transition period. In this work, we model the interactions between vehicles (two AVs or an AV and a human-driven vehicle) in a lane changing process by leveraging the Stackelberg game. We explicitly model driving attitudes for both vehicles involved in lane changing. We design five cases, in which the two vehicles have different levels of knowledge, and make different assumptions, about the driving attitude of the rival. We conduct theoretical analysis and simulations for different cases in two lane changing scenarios, namely changing lanes from a higher speed lane to a lower speed lane, and from a lower speed lane to a higher speed lane. We use four metrics (fuel consumption, discomfort, minimum distance gap and lane change success rate) to investigate how the performance of a single vehicle and that of the system will be influenced by the level of information sharing, and whether a vehicle trajectory optimized based on selfish criteria can provide system-level benefits.

Keywords: Situation Analysis and Planning, Automated Vehicles, Societal Impacts
Abstract: We address ethical dilemma situations that may arise during autonomous driving. To evaluate how this delib- eration could work, we propose a decision-making algorithm based on a Markov Decision Process (MDP) which controls the vehicle in normal conditions. When a dilemma situation is detected, the collision severity is determined by an evaluation of the harm incurred by different types of road users. Then, to illustrate different moral approaches, three different policies are proposed: one based on ralwsian contractarianism, another on utilitarianism, and finally on egalitarianism. Each policy supports a different view of the concept of fairness, potentially producing different behaviors for the same dilemma situation.

Keywords: Vehicle Environment Perception, Deep Learning, Vision Sensing and Perception
Abstract: Semantic segmentation renders a unified way of surrounding perception, where most of driving scene detection tasks can be covered by running a single efficient ConvNet through a forward pass. However, current frameworks posit the closed-world paradigm expressed as a single source of distribution over a predetermined set of visual classes, forgetting that a deep model must be deployed in the wild facing unseen domains and unforeseen hazards. In spite of being accurate in its comfort zone, the segmentation model may not generalize well to a new domain. In addition, a model trained with single dataset is heavily limited in terms of recognizable classes. In this paper, we propose an omni-supervised learning framework for semantic segmentation which is able to leverage heterogeneous data sources. Our omni-supervised training framework incorporates all available labeled and unlabeled data, meanwhile bridges multiple training sets to be capable of recognizing more classes that are needed for autonomous navigation application at hand in the new domain. A comprehensive variety of experiments shows that with the proposed multi-source omni-supervised learning solution, an efficient ConvNet like our ERF-PSPNet attains significant robustness gains in open domains that are of critical relevance to real deployment of vision algorithms. Our approach surpasses the state of the art on the highly unconstrained PASS and IDD20K datasets.

Keywords: Vision Sensing and Perception, Automated Vehicles, Deep Learning
Abstract: Nowadays, deep learning techniques are widely used for lane detection, but application in low-light conditions remains a challenge until this day. Although multi-task learning and contextual-information-based methods have been proposed to solve the problem, they either require additional manual annotations or introduce extra inference overhead respectively. In this paper, we propose a style-transfer-based data enhancement method, which uses Generative Adversarial Networks (GANs) to generate images in low-light conditions, that increases the environmental adaptability of the lane detector. Our solution consists of three parts: the proposed SIM-CycleGAN, light conditions style transfer and lane detection network. It does not require additional manual annotations nor extra inference overhead. We validated our methods on the lane detection benchmark CULane using ERFNet. Empirically, lane detection model trained using our method demonstrated adaptability in low-light conditions and robustness in complex scenarios. Our code for this paper will be publicly available.

Keywords: Vision Sensing and Perception, Vehicle Environment Perception, Automated Vehicles
Abstract: In this paper we propose a fast fully convolutional neural network for panoptic segmentation that can provide an accurate semantic and instance-level representation of the environment in the 2D space. We tackle panoptic segmentation as a dense classification problem and generate masks for stuff classes as well as for each instance things classes. Our network employs a shared backbone and Feature Pyramid Network for multi-scale feature extraction which we extend with dual-decoders that learn background and foreground specific masks. Guided by object proposals, the panoptic head assembles location-sensitive prototype masks using a learned weighting scheme. Our solution runs in real-time, in 82 ms on high resolution images, making it suitable for robotic applications and automated driving. Extensive experiments on the Cityscapes dataset demonstrate that our panoptic segmentation network is robust and accurate, with 57.3% PQ and 76.9% mIoU.

Keywords: Deep Learning, Vision Sensing and Perception, Advanced Driver Assistance Systems
Abstract: Inexpensive sensing and computation, as well as insurance innovations, have made smart dashboard cameras ubiquitous. Increasingly, simple model-driven computer vision algorithms focused on lane departures or safe following distances are finding their way into these devices. Unfortunately, the long-tailed distribution of road hazards means that these hand-crafted pipelines are inadequate for driver safety systems. We propose to apply data-driven anomaly detection ideas from deep learning to dashcam videos, which hold the promise of bridging this gap. However, there exists almost no literature applying anomaly understanding to moving cameras, and correspondingly there is also a lack of relevant datasets. To counter this issue, we present a large and diverse dataset of truck dashcam videos, namely RetroTrucks, that includes normal and anomalous driving scenes. We apply: (i) one-class classification loss, and (ii) reconstruction-based loss for anomaly detection on RetroTrucks as well as on existing static-camera datasets. We introduce formulations for modeling object interactions in this context as priors. Our experiments indicate that our dataset is indeed more challenging than standard anomaly datasets, and previous anomaly detection methods do not perform well here out-of-the-box. In addition, we share insights into the behavior of these two important families of anomaly detection approaches on dashcam data.

Keywords: Vehicle Control, Vision Sensing and Perception, Convolutional Neural Networks
Abstract: The ability of autonomous vehicles to maintain an accurate trajectory within their road lane is crucial for safe operation. This requires detecting the road lines and estimating the car relative pose within its lane. Lateral lines are usually retrieved from camera images. Still, most of the works on line detection are limited to image mask retrieval and do not provide a usable representation in world coordinates. What we propose in this paper is a complete perception pipeline based on monocular vision and able to retrieve all the information required by a vehicle lateral control system: road lines equation, centerline, vehicle heading and lateral displacement. We evaluate our system by acquiring data with accurate geometric ground truth. To act as a benchmark for further research, we make this new dataset publicly available at http://airlab.deib.polimi.it/datasets/ .

Keywords: Convolutional Neural Networks, Vision Sensing and Perception, Deep Learning
Abstract: Current state-of-the-art object detection algorithms still suffer the problem of imbalanced distribution of training data over object classes and background. Recent work introduced a new loss function called focal loss to mitigate this problem, but at the cost of an additional hyperparameter. Manually tuning this hyperparameter for each training task is highly time-consuming.

With automated focal loss we introduce a new loss function which substitutes this hyperparameter by a parameter that is automatically adapted during the training progress and controls the amount of focusing on hard training examples. We show on the COCO benchmark that this leads to an up to 30% faster training convergence. We further introduced a focal regression loss which on the more challenging task of 3D vehicle detection outperforms other loss functions by up to 1.8 AOS and can be used as a value range independent metric for regression.

Keywords: Deep Learning, Convolutional Neural Networks
Abstract: Deep Neural Networks trained in a fully supervised fashion are the dominant technology in perception-based autonomous driving systems. While collecting large amounts of unlabeled data is already a major undertaking, only a subset of it can be labeled by humans due to the effort needed for high-quality annotation. Therefore, finding the right data to label has become a key challenge. Active Learning is a powerful technique to improve data efficiency for supervised learning methods, as it aims at selecting the smallest possible training set to reach a required performance. We have built a scalable production system for Active Learning in the domain of autonomous driving. In this paper, we describe the resulting high-level design, sketch some of the challenges and their solutions, our current results at scale and briefly describe the open problems and future directions.

Keywords: Unsupervised Learning, Automated Vehicles, Self-Driving Vehicles
Abstract: A novel unsupervised outlier score, which can be embedded into graph based dimensionality reduction techniques, is presented in this work. The score uses the directed nearest neighbor graphs of those techniques. Hence, the same measure of similarity that is used to project the data into lower dimensions, is also utilized to determine the outlier score. The outlier score is realized through a weighted normalized entropy of the similarities. This score is applied to road infrastructure images. The aim is to identify newly observed infrastructures given a pre-collected base dataset. Detecting unknown scenarios is a key for accelerated validation of autonomous vehicles. The results show the high potential of the proposed technique. To validate the generalization capabilities of the outlier score, it is additionally applied to various real world datasets. The overall average performance in identifying outliers using the proposed methods is higher compared to state-of-the-art methods. In order to generate the infrastructure images, an openDRIVE parsing and plotting tool for Matlab is developed as part of this work. This tool and the implementation of the entropy based outlier score in combination with Uniform Manifold Approximation and Projection are made publicly available.

Keywords: Cooperative ITS, Vehicle Control, Self-Driving Vehicles
Abstract: We consider the problem of incentivization and optimal control of autonomous vehicles for improving traffic congestion. In our scenario, autonomous vehicles must be incentivized in order to participate in traffic improvement. Using the theory and methods of optimal transport, we propose a constrained optimization framework over dynamics governed by partial differential equations, so that we can optimally select a portion of vehicles to be incentivized and controlled.

The goal of the optimization is to obtain a uniform distribution of vehicles over the spatial domain. To achieve this, we consider two types of penalties on vehicle density, one is the L2 cost and the other is a multiscale-norm cost, commonly used in fluid-mixing problems. To solve this non-convex optimization problem, we introduce a novel algorithm, which iterates between solving a convex optimization problem and propagating the flow of uncontrolled vehicles according to the Lighthill-Whitham-Richards model. We perform numerical simulations, which suggest that the optimization of the L2 cost is ineffective while optimization of the multiscale norm is effective. The results also suggest the use of a dedicated lane for this type of control in practice.

Keywords: Intelligent Vehicle Software Infrastructure, Autonomous / Intelligent Robotic Vehicles, Reinforcement Learning
Abstract: Autonomous driving (AV) has been intensively researched over the last decade. In this paper, we introduce an open autonomous driving software stack to enable faster design, development, and testing of algorithms on simulated or experimental vehicles, which we hope will become a useful tool for AV researchers.

Keywords: Automated Vehicles, Active and Passive Vehicle Safety
Abstract: The determination of safety assurances for automated driving vehicles is one of the most critical challenges in the industry today. Several behavioral safety models for automated driving have been proposed recently and standards discussions are on the way. In this paper we present a method to automatically explore the performance of automated vehicle (AV) safety models utilizing robustness of Metric Temporal Logic (MTL) specifications as a continuous metric of safety. We present a case study of the Responsibility Sensitive Safety model (RSS), introducing a safety evaluation pipeline based on the CARLA driving simulator, RSS and a set of safety critical driving scenarios. Our method automatically extracts safety relevant profiles for these scenarios providing practical parametric boundaries for implementation. Furthermore, we evaluate the trade-offs between safety and utility within the safe RSS parameter space through a proposed naturalistic benchmark challenge that we open-sourced. We analyze different RSS parameter configurations including assertive and more conservative settings, extracted by our specification driven framework. Our results show that while maintaining the safety boundaries, the extracted RSS configuration for assertive driving behavior achieves the highest utility.

Keywords: Automated Vehicles, Advanced Driver Assistance Systems, Impact on Traffic Flows
Abstract: This paper introduces a novel vehicle-in-the-loop testing methodology called “Hybrid Testing”, which enables the evaluation of a real vehicle in a virtual traffic scenario in an enclosed proving ground with simulated traffic components and sensor signals. Among the other benefits, the introduced methodology is particularly suited to test and verify ADAS functions in virtual scenarios, which would otherwise be very difficult to create in real world. Development of ADAS functions require extensive testing and validation prior to deployment and generally, exhaustive real-life scenario evaluation of such systems is not feasible, let alone possible. With Hybrid Testing we can combine the benefits of simulation and real-life testing. We show how this methodology, as was developed in the EU-funded project INFRAMIX, can be used to evaluate ADAS functions on an example of a trajectory planning algorithm to demonstrate its working principles and benefits.

Keywords: Self-Driving Vehicles, Vehicle Control, Automated Vehicles
Abstract: In the future, the presence of highly automated vehicles is expected to become more and more wide spread. In such systems, the whole driving task will be performed by the vehicle autonomously, thus, vehicles must be able to control their motion in various circumstances, even at stability limits. In this paper, the authors consider the control of a steady-state drifting maneuver, which means saturated rear tire forces. In a previous article, a MIMO linear quadratic regulator (LQR) controller was designed, and it showed good performance in simulation environment. The test results of a real vehicle implementation are presented here, which was the logical next step of the work. For the vehicle platform, a series production sports car was chosen. Modifications were made in order to enable by-wire control. After identifying the vehicle model parameters through measurements, the control algorithm was implemented on a rapid prototyping unit. Vehicle states were measured with a high precision dual antenna GNSS module with RTK correction. Additionally, other dynamic parameters from the vehicle CAN bus signals were also used. The main goal was to stabilize different drifting equilibria, which showed satisfying performance of the proposed controller in a real vehicle setup as well.

Keywords: Automated Vehicles, Collision Avoidance
Abstract: High automated vehicles rely on the computing system in the car to understand the environment and make driving decisions. Therefore, computing system design is essential for ensuring the driving safety. However, to our knowledge, no clear guideline exists so far regarding how to guide the safety-aware autonomous vehicle (AV) computing system design. To understand the safety requirement of AV computing system, we performed a field study by operating industrial Level-4 AV fleets in multiple locations for three months. The field study indicates that traditional computing system performance metrics, such as tail latency, average latency, maximum latency, and timeout, cannot fully satisfy the safety requirement for AV computing system design. To address this issue, we propose the ``safety score'' as a primary metric for measuring the level of safety in AV computing system design.

Keywords: Automated Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: This paper discusses a platform both in simulation and experimentation for testing autonomous heavy trucking. In simulation, we present a novel use of the video game American Truck Simulator (ATS) as the simulation platform, only costing a fraction of commercial simulator software. In experimentation, we present a modified ProStar 122+ using the PACMod system from AutonomouStuff, a popular by-wire kit. Discussion and review of the by-wire kit and sensors is provided. A proof-of-concept of the platform is shown by performing lane keeping at 65 mph using the Stanley lateral controller and MobilEye detection system. Further, we introduce a rapidly developed longitudinal control algorithm using a pedal actuation map, and 3D lookup tables created from braking and acceleration data. Introductory results are presented to aid the research community for single vehicle autonomous trucking.

Keywords: Vehicle Control, Automated Vehicles, Active and Passive Vehicle Safety
Abstract: Fail-operational lateral control is a necessary aspect of autonomous driving systems to reach SAE level 4 of automated driving. This paper focuses on fault tolerant control reconfiguration techniques for two fault types: a yawrate sensor fault and an on-board electric motor fault. The yaw-rate fault reconfiguration combines an estimation of the yaw-rate signal with a smooth switch to a yaw-rate independent controller. The actuator fault reconfiguration is based on control allocation techniques. These algorithms, including Fault Detection and Isolation are implemented in a four-wheel driven electric demonstration vehicle. Both types of reconfiguration strategies are experimentally evaluated in an evasive manoeuvre and the results show that the fault reconfiguration techniques succeed to provide fail-operational behavior.

Keywords: Human-Machine Interface, Deep Learning, Autonomous / Intelligent Robotic Vehicles
Abstract: Generating realistic motion in a motion-based (dynamic) driving simulator is challenging due to the limited workspace of the motion system of the simulator compared to the motion range of the simulated vehicle. MCA render accelerations by controlling the motion system of the simulators to provide the driver with a realistic driving experience. Commonly used methods such as CW-MCA typically achieves suboptimal results due to scaling and filtering, which results in an inefficient usage of the workspace. The Model Predictive Control-based MCA (MPC-MCA) has been shown to achieve superior results and more efficient workspace use. However, it's performance is in practice constrained due to the computationally expensive operations and the requirement of an accurate prediction of future vehicle states. Finally, the OC has been shown to provide optimal cueing in an open-loop setup wherein the precalculated control signals are re-played to the driver. However, OC cannot be used in real-time with the driver-in-the-loop. Our work introduces a novel NN-MCA, which is trained to imitate the behavior of the OC. After training, the NN-MCA provides an approximated model of the OC, which can run in real-time with the driver in-the-loop, while achieving similar quality. The experiments demonstrate the potential of this approach through objective evaluations of the generated motion-cues on the simulator model and the real simulator. A demonstration video for the performance comparison of the CW-MCA, OC-MCA and our proposed method is available at http://go.tum.de/708350.

Keywords: Advanced Driver Assistance Systems, Convolutional Neural Networks, Human-Machine Interface
Abstract: Image-to-image translation attempts at mapping an image from a given domain into a different domain. In this paper we introduce a novel image-to-image translation paradigm, where the translation itself is performed by a system of generative adversarial networks (GANs), guided by a set of pretrained networks, called task networks. The underlying hypothesis is that these task networks can guide the translation process of the GAN generator to focus on those aspects of the image which matter the most for the tasks in question. Moreover, our work is motivated by a real-life situation, created by the need to adapt existing, duly trained networks to work on images acquired with a different sensor than the one used to acquire the images used to train the networks. In order to optimize the image domain translation from the new sensor to the old one, we use the output loss of the trained neural networks to guide the training process of the image-to-image translation engine, thus forcing it to create images which improve the performances of the trained, guiding networks. Experimental results presented therein confirm the usefulness of our approach for an automotive application within the context of driver monitoring systems. More specifically we show that guiding the learning of the image-to-image translation module with the loss of a fixed neural network trained to estimate the driver eyelid opening, improves the performances of the latter when being fed with images output by the translation module.

Keywords: Automated Vehicles, Advanced Driver Assistance Systems, Human-Machine Interface
Abstract: Autonomous vehicles will find an infinite number of possible scenarios while driving in urban environments and need to react in a proper manner. For that reason, it is important to have algorithms that can propose driving alternatives for different type of scenarios in a global and unified way instead of using rule-based algorithms which depend on the driving scene. This paper presents a reachability estimation algorithm designed to obtain a safe and comfort-optimized trajectory set for different driving scenarios. First, a finite number of path candidates are created using B´ezier curves. Then, all valid path candidates are combined with the reachable sets of dynamic obstacles to generate speed profiles consistent with safety and comfort requirements. The output of this algorithm would allow a decision-making strategy to select the optimum candidate depending on different criteria.

Keywords: Eco-driving and Energy-efficient Vehicles, Advanced Driver Assistance Systems, Electric and Hybrid Technologies
Abstract: This paper explains a novel adaptive cruise control (ACC) driving with coasting to improve fuel economy. The purpose is to reduce the energy loss with predictive control when the preceding vehicle decelerates, while acceptable driving feeling is guaranteed. To achieve this goal, prediction of the preceding vehicle behavior is introduced to determine the ego vehicle behavior realized by using inverse reinforcement learning (IRL). In addition, the evaluation function is designed to determine the best coasting timing by balancing longer coasting time and acceptable driving feeling, while the ego vehicle speed is controlled with a rule-based control at a non-coasting period. The performance of this control strategy has been validated with simulation, showing 9.7% fuel economy improvement on average for hybrid electric vehicles in the case of following the preceding vehicle before an intersection. It has also been verified with an actual test vehicle, where a high level balance between efficiency and acceptable feeling is realized.

Keywords: Collision Avoidance, Self-Driving Vehicles, Unsupervised Learning
Abstract: Imitation learning provides a way to automatically construct a controller by mimicking human behavior from data. For safety-critical systems such as autonomous vehicles, it can be problematic to use controllers learned from data because they cannot be guaranteed to be collision-free. Recently, a method has been proposed for learning a multi-mode hybrid automaton cruise controller (MOHA). Besides being accurate, the logical nature of this model makes it suitable for formal verification. In this paper, we demonstrate this capability using the SpaceEx hybrid model checker as follows. After learning, we translate the automaton model into constraints and equations required by SpaceEx. We then verify that a pure MOHA controller is not collision-free. By adding a safety state based on headway in time, a rule that human drivers should follow anyway, we do obtain a provably safe cruise control. Moreover, the safe controller remains more human-like than existing cruise controllers.

Keywords: Advanced Driver Assistance Systems
Abstract: Highly automated vehicles control the lateral and longitudinal movement of the vehicle in defined use cases, without being monitored by the human driver. According to the high and full automation level defined by SAE J3016 the driver may request the deactivation of the automated driving system at any time. However, interventions by the driver can lead to critical situations, especially in highly dynamic driving situations. Therefore, this paper investigates at which point driver interventions during highly automated driving can be critical and how the driver can be assisted in order to avoid dangerous driver inputs. To answer these questions, this paper presents concepts for intervention assistance systems that are implemented, tested and evaluated in a driving simulator.

Keywords: Human-Machine Interface, Automated Vehicles, Driver Recognition
Abstract: Automated driving systems are envisioned as the future mode of transportation owing to their projected ability to reduce human error and achieve more efficient and comfortable transportation. Accordingly, designing an interface that ensures the situational awareness of the human operator to reduce confusion, false expectations, and over-reliance on the automated system is important. When a human operator is in control, the automated system is expected to handle troublesome situations that the human is unable to manage. Thus, an interface is required to provide the appropriate information when necessary so that the human operator can easily perceive the reason for the sudden automated intervention. In this study, such a scenario is highlighted, in which a simulated automated intervention avoided a potential collision with a pedestrian who suddenly appeared on the roadside. To convey the reason for the automated intervention, an augmented reality-based head-up display (AR-HUD) cue that targets the pedestrian is developed. To understand the effects of the AR-HUD cue on the speed at which a human operator can recognize a pedestrian and the contribution of this visual cue to the perception of acceptability and credibility of the automated intervention, we compared AR-HUD with a static head-up display (S-HUD) that displays a pedestrian symbol at the bottom portion of the windshield. The results showed that the AR-HUD cue yielded faster recognition of the targeted pedestrian and provided a relatively more acceptable perception of the automated intervention.