Keywords: Driver Recognition, Driver State and Intent Recognition, Convolutional Neural Networks
Abstract: In order to ensure safe autonomous driving, precise information about the conditions in and around the vehicle must be available. Accordingly, the monitoring of occupants and objects inside the vehicle is crucial. In the state-of-the-art, single or multiple deep neural networks are used for either object recognition, semantic segmentation, or human pose estimation. In contrast, we propose our Multitask Detection, Segmentation and Pose Estimation Network (MDSP) -- the first multitask network solving all these three tasks jointly in the area of occupancy monitoring. Due to the shared architecture, memory and computing costs can be saved while achieving higher accuracy. Furthermore, our architecture allows a flexible combination of the three mentioned tasks during a simple end-to-end training. We perform comprehensive evaluations on the public datasets SVIRO and TiCaM in order to demonstrate the superior performance.

Keywords: Vehicle Environment Perception, Convolutional Neural Networks, Active and Passive Vehicle Safety
Abstract: Recent adversarial attacks with real world applications are capable of deceiving deep neural networks~(DNN), which often appear as printed stickers applied to objects in physical world. Though achieving high success rate in lab tests and limited field tests, such attacks have not been tested on multiple DNN architectures with a standard setup to unveil the common robustness and weakness points of both the DNNs and the attacks. Furthermore, realistic looking stickers applied by normal people as acts of vandalism are not studied to discover their potential risks as well the risk of optimizing the location of such realistic stickers to achieve the maximum performance drop. In this paper, (a) we study the case of realistic looking sticker application effects on traffic sign detectors performance; (b) we use traffic sign image classification as our use case and train and attack 11 of the modern architectures for our analysis; (c) by considering different factors like brightness, blurriness and contrast of the train images in our sticker application procedure, we show that simple image processing techniques can help realistic looking stickers fit into their background to mimic real world tests; (d) by performing structured synthetic and real-world evaluations, we study the difference of various traffic sign classes in terms of their crucial distinctive features among the tested DNNs.

Keywords: Mapping and Localization, Sensor and Data Fusion, Unsupervised Learning
Abstract: Visual-inertial Odometry (VIO) provides cars with position information by fusing data from a camera and inertial measurement unit (IMU) which are both widely equipped on intelligent vehicles. Recently, unsupervised VIO has made great progress. However, existing VIOs mainly concatenate features extracted from different domains (visual and inertial), leading to inconsistency during integration. These methods are also difficult to scale to longer sequences because absolute velocity is not available. Hence, we propose a novel network based on attention mechanism to fuse sensors in a self-motivated and meaningful manner. We design spatial and temporal branches that focus on pairwise images and a sequence of images respectively. Meanwhile, a tiny but effective module (referred to as “warm start”) is introduced to produce velocity-related information for the IMU encoder. The proposed attention branches and warm start are shown to improve the robustness of the model in dynamic scenarios and in the case of rapid changes in vehicle velocity. Evaluation on KITTI and Malaga datasets shows that our method outperforms other recent state-of-the-art VO/VIO methods.

Keywords: Autonomous / Intelligent Robotic Vehicles, Collision Avoidance, Automated Vehicles
Abstract: While substantial research on trajectory planning for large scale vessels on maritime environment has been conducted, the research on inland waterways has been limited to small vessels. For large scale vessels in ship canals, the narrow canal width in comparison to the large geometries of inland vessels require a dynamically feasible trajectory planning to avoid collisions with static and dynamic obstacles. Thus, the use of an accurate vessel model for motion planning to ensure the dynamical feasibility is desirable. In this regard, the Abkowitz model is one of the classic models used in the study of ship maneuvring, especially in calm water environments. However, the nonlinear model structure makes the model-based trajectory planning challenging, as most trajectory planning methods either rely on specific model structures or require fully-actuated vessels. In this paper, we introduce an incremental sampling method for the trajectory planning of rudder-steered and therefore underactuated surface vessels when sailing in ship canals. The trajectory planning is based on an heuristic tree search along the tubular geometry of ship canals. The discretization of input value and the prediction horizon transforms the trajectory planning into a minimum-cost graph search on a tree structure, which is partially explored using weighted A*-search. Efficient collision detection is added by applying computer graphic strategies. An evaluation for a large scale vessel is shown in a simulation study on the Dortmund-Ems-Canal. The proposed method finds a dynamic feasible local trajectory with a horizon of several hundred meter, avoiding both dynamic and static obstacles. This opens the possibility to extend autonomous driving to commercial vessel operations in inland waterways.

Keywords: Autonomous / Intelligent Robotic Vehicles, Vision Sensing and Perception, Collision Avoidance
Abstract: 3D object detection is a well-known problem for autonomous systems. Most of the existing methods use sensor fusion techniques with Radar, LiDAR, and Cameras. However, one of the challenges is to estimate the 3D shape and location of the adjoining vehicles from a single monocular image without other 3D sensors; such as Radar or LiDAR. To solve the lack of depth information, a novel method for 3D vehicle detection is presented. In this work, instead of using the whole depth map and the viewing angle (allocentric angle), only the depth mask of each object is used to refine the projected centroid and estimate its egocentric angle directly.

The performance of the proposed method is tested and validated using the KITTI dataset, obtaining similar results to other state-of-the-art methods for Monocular 3D Object Detection.

Keywords: Advanced Driver Assistance Systems, Active and Passive Vehicle Safety, Sensor and Data Fusion
Abstract: This paper presents a novel methodology to auto-tune an Unscented Kalman Filter (UKF). It involves using a Two-Stage Bayesian Optimisation (TSBO), based on a t-Student Process to optimise the process noise parameters of a UKF for vehicle sideslip angle estimation. Our method minimises performance metrics, given by the average sum of the states' and measurement' estimation error for various vehicle manoeuvres covering the wide range of vehicle behaviour. The predefined cost function is minimised through a TSBO which aims to find a location in the feasible region that maximises the probability of improving the current best solution. Results on an experimental dataset show the capability to tune the UKF in 79.9 % less time than using a genetic algorithm (GA) and the overall capacity to improve the estimation performance in an experimental test dataset of 9.9 % to the current state-of-the-art GA.

Keywords: Intelligent Vehicle Software Infrastructure, Automated Vehicles, Eco-driving and Energy-efficient Vehicles
Abstract: This paper presents an approach for allocation of resources in an automotive service-oriented software architecture. Using mathematical optimization, we assign computational resources of an automotive compute cluster to a set of software services. Additionally, scheduling parameters of services are optimized under the consideration of dependencies between data flows and computations within services. The optimization minimizes power consumption and the maximum execution times of critical effect chains in a multi-objective optimization problem. The evaluation investigates the achievable reduction in power consumption using an exemplary system. Furthermore, we demonstrate a sharp reduction in maximum execution times of effect chains that span multiple services and ECUs.

Keywords: Impact on Traffic Flows, Traffic Flow and Management, Situation Analysis and Planning
Abstract: Car-following (CF) behavior is the most abstract form of driving action and, CF behavior modeling has been one of the core aspects of traffic engineering studies for several decades. The literature about CF behavior modeling is vibrant and still evolving. Furthermore, the effect of CF models on the traffic flow performance through case studies on different traffic facilities is still being investigated. To shed light on this matter, this study presents a microsimulation-based case study considering a freeway stretch in Istanbul, Turkey, employing two different CF models, i.e., Intelligent Driver Model (IDM) and Wiedemann 99 through scenarios. Simulation of Urban Mobility (SUMO) is utilized as the microsimulation environment. Both CF models are calibrated according to the measurements. Scenarios for the comparative evaluation are setup based on the questions "What if German drivers used this freeway stretch? How much would the traffic flow performance change?" Using different case studies conducted in German Freeways on the literature, simulation model parameters are obtained for both models and, simulation analyses are performed. Traffic flow performances are evaluated based on the selected performance measures, such as throughput and total travel time. According to the findings, it is seen that results differ significantly between scenarios. We elaborate on the differences obtained and discuss the implications on different scenarios which are handled through different CF models.

Keywords: Legal Impacts, Automated Vehicles
Abstract: This study enhances automated driving scenario-based safety assessment methods previously developed for highways, and enables their application to urban areas. First, we propose a methodology for matching open source map data with naturalistic driving data recorded with test vehicles. The methodology proposed proved feasible detecting various geometry-related scenarios and can contribute to overcome the difficulties to create representative real driving urban scenario databases that cover such geometries. Second, a search-based test case generation methodology previously developed to fulfill requirements of severity, exposure and realism with a focus on highways, is further developed and adapted to active urban scenarios. Active scenarios require an active maneuver decision of the Vehicle under Test and have not been considered in related work so far. To show the feasibility of the methodologies proposed, we apply them to a set of Left Turn Across Path / Opposite Direction scenarios, extracted from an existing urban driving database. The map matching and the search-based test case generation methodology succeeded in deriving test cases which equally account for exposure and coverage criteria for normal driving situations in urban settings.

Keywords: Assistive Mobility Systems, Societal Impacts, Automated Vehicles
Abstract: The term “Mobility” is gaining new perspectives. Due to the paradigm shift driven by information technologies and autonomous vehicles, on-demand mobility services have experienced significant growth. Operating such a service efficiently is a challenging task. Especially, assigning vehicles to customers plays a vital role in this regard. To meet a satisfactory level of service while keeping the operational costs to a minimum requires efficient assignment strategies. Work summarized in this paper utilizes several shared and non-shared assignment algorithms in order to propose a methodology to assess the effects on the overall system performance for an Autonomous Mobility on Demand system. Selected algorithms are tested in a theoretical network with real-world taxi data with the help of microscopic traffic simulation software Simulation of Urban Mobility. Simulation scenarios are generated for both varying demand levels and increasing fleet sizes. Results suggest that for high demand levels and small fleet sizes, shared algorithms outperform non-shared algorithms for every performance measure chosen: total vehicle kilometers traveled, the ratio of empty fleet kilometers, average passenger waiting time for pick up, and the number of customers served in a period.

Keywords: Automated Vehicles, Situation Analysis and Planning, Vehicle Control
Abstract: Autonomous or semi-autonomous driving constitutes an important business case to the trucking industry. To realize its full potential it is mandatory to enable autonomous or semi-autonomous driving especially in challenging traffic situations such as roundabouts and sharp bends. As a step towards this goal, the main objective of this paper is to investigate the optimal trajectory planning of heavy-duty long-haul trucks for real-world applications with coupled longitudinal and lateral dynamics of the truck-trailer system. We present a detailed dynamical model and propose objectives that optimize with respect to safety, and to find a suitable compromise between efficiency and driving performances. Using these ingredients, we generate trajectories for maneuvering an articulated vehicle, subject to realistic constraints on narrow roads. We use Bock's Direct Multiple Shooting method in combination with structure-exploiting iterative optimization methods, efficiently implemented in the software package MUSCOD-II, to generate fast, reliable, and feasible solutions for the complex and highly nonlinear problem. We analyze the solutions with differently weighted objectives for a realistic scenario, maintaining safety, and building compromises between energy efficiency, driving quality, and speed maintenance that reflects time efficiency.

Keywords: Autonomous / Intelligent Robotic Vehicles, Situation Analysis and Planning, Cooperative Systems (V2X)
Abstract: This paper discusses main factors which establish responsible driving and the consequences for technical provisions, which are needed to support evidence of responsible behavior in autonomous driving. It relates these arguments to the concept of Tactical Safety: act early and proactively in traffic situations, in order to avoid non-controllable situations with possibly high accident severity. A continuous safety score, which serves to measure danger as the distance to a collision, is an essential utility for vigilant monitoring and gentle intervention previous to criticality. As a second prerequisite, a dependable communication community for traffic, road and environmental conditions enables early recognition of conditions for possible dangers beyond the accessible range of onboard sensors. Finally, the combination of those aspects for the safety assurance of autonomous vehicles is discussed.

Keywords: Vision Sensing and Perception, Convolutional Neural Networks, Advanced Driver Assistance Systems
Abstract: Recent work done on traffic sign and traffic light detection focus on improving detection accuracy in complex scenarios, yet many fail to deliver real-time performance, specifically with limited computational resources. In this work, we propose a simple deep learning based end-to-end detection framework, which effectively tackles challenges inherent to traffic sign and traffic light detection such as small size, large number of classes and complex road scenarios. We optimize the detection models using TensorRT and integrate with Robot Operating System to deploy on an Nvidia Jetson AGX Xavier as our embedded device. The overall system achieves a high inference speed of 63 frames per second, demonstrating the capability of our system to perform in real-time. Furthermore, we introduce CeyRo, which is the first ever large-scale traffic sign and traffic light detection dataset for the Sri Lankan context. Our dataset consists of 7984 total images with 10176 traffic sign and traffic light instances covering 70 traffic sign and 5 traffic light classes. The images have a high resolution of 1920 x 1080 and capture a wide range of challenging road scenarios with different weather and lighting conditions. Our work is publicly available at https://github.com/oshadajay/CeyRo.

Keywords: Automated Vehicles, Human-Machine Interface, Vehicle Environment Perception
Abstract: Automated driving (AD) has the potential to change our mobility by giving us back the riding time for our free disposal. Since this aims at a broad base of customers, the people using such technology can strongly differ. Differences can comprise for instance prior AD experience, driving frequency, or physical impairments. In this paper, we investigate which effects these user characteristics have on the preferences regarding displayed surrounding traffic information. We do so by using a moving-base simulator study with 58 participants. We found that drivers with experience with automatic cruise control perceive information regarding surrounding traffic as significantly less important, while users with higher driving frequency or sight impair found them significantly more important. For theory this implicates that different user interfaces for different users are ideal and per-sonalization can foster user experience.

Keywords: Vision Sensing and Perception, Recurrent Networks, Convolutional Neural Networks
Abstract: Prediction of the movement of all traffic participants is a very important task in autonomous driving. Well-predicted behavior of other cars and actors is crucial for safety. A~sequence of bird’s-eye view artificially rasterized frames are used as input to neural networks which are trained to predict the future behavior of the participants. The~emph{Lyft Motion Prediction for Autonomous Vehicles} dataset is explored and adapted for this task. We developed and applied a novel approach where the prediction problem is viewed as a problem of spatiotemporal prediction and we use methods based on convolutional recurrent neural networks.

Keywords: Reinforcement Learning, Intelligent Vehicle Software Infrastructure, Advanced Driver Assistance Systems
Abstract: Automotive is becoming more and more sensor-equipped. Collision avoidance, lane departure warning, and self-parking are examples of applications becoming possible with the adoption of more sensors in the automotive industry. Moreover, the driver is now equipped with sensory systems like wearables and mobile phones. This rich sensory environment and the real-time streaming of contextual data from the vehicle make the human factor integral in the loop of computation. By integrating the human's behavior and reaction into the advanced driver-assistance systems (ADAS), the vehicles become a more context-aware entity. Hence, we propose MAConAuto, a framework that helps design human-in-the-loop automotive systems by providing a common platform to engage the rich sensory systems in wearables and mobile to have context-aware applications. By personalizing the context adaptation in automotive applications, MAConAuto learns the behavior and reactions of the human to adapt to the personalized preference where interventions are continuously tuned using Reinforcement Learning. Our general framework satisfies three main design properties, adaptability, generalizability, and conflict resolution. We show how MAConAuto can be used as a framework to design two applications as human-centric applications, forward collision warning, and vehicle HVAC system with negligible time overhead to the average human response time.

Keywords: Unsupervised Learning, Cooperative Systems (V2X), V2X Communication
Abstract: While privacy concerns entice connected and automated vehicles to incorporate on-board federated learning (FL) solutions, an integrated vehicle-to-everything communication with heterogeneous computation power aware learning platform is urgently necessary to make it a reality. Motivated by this, we propose a novel mobility, communication and computation aware online FL platform that uses on-road vehicles as learning agents. Thanks to the advanced features of modern vehicles, the on-board sensors can collect data on causal fashion as vehicles move along their trajectories, while the on-board processors can train machine learning models using collected data. To take the high mobility of vehicles into account, we consider the delay as a learning parameter and restrict it to be less than a tolerable threshold. To satisfy this threshold, the learning server accepts partially trained models, the distributed roadside units (a) perform downlink multicast beamforming to minimize global model distribution delay and (b) allocate optimal uplink radio resources to minimize local model offloading delay, and the vehicle agents conduct heterogeneous local model training. Using real-world vehicle trace datasets, we validate our FL solutions. Simulation shows that the proposed integrated FL platform is robust and outperforms baseline models. With reasonable local training episodes, it can effectively satisfy all constraints and deliver near ground truth multi-horizon velocity and vehicle-specific power predictions

Keywords: Automated Vehicles, Self-Driving Vehicles, Advanced Driver Assistance Systems
Abstract: In this paper, we propose and analyze a method for trajectory prediction in longitudinal car-following scenarios. Hereby, the prediction is realized by a longitudinal car-following model (Intelligent Driver Model, IDM) with online estimated parameters. Previous work has shown that IDM online parameter adaptation is possible but difficult and slow, while providing only small improvement of prediction quality over e.g. constant velocity or constant acceleration baseline models.

In our approach (Online IDM, OIDM), we use the difference between the parameter-specific trajectory and the real past trajectory as objective function of the optimization. Instead of optimizing the model parameters “directly”, we gain them based on a weighted sum of a set of prototype parameters, optimizing these weights. To show the benefits of the method, we compare the properties of our approach against state-of-the-art prediction methods for longitudinal driving, such as Constant Velocity (CV), Constant Acceleration (CA) and particle filter approaches on an open freeway driving dataset. The evaluation shows significant improvements in several aspects: (I) The prediction accuracy is significantly increased, (II) the obtained parameters exhibit a fast convergence and increased temporal stability and (III) the computational effort is reduced so that an online parameter adaptation becomes feasible.

Keywords: Convolutional Neural Networks, Automated Vehicles, Deep Learning
Abstract: Retrieving images with objects that are semantically similar to objects of interest (OOI) in a query image has many practical use cases. A few examples include fixing failures like false negatives/positives of a learned model or mitigating class imbalance in a dataset. The targeted selection task requires finding the relevant data from a large-scale pool of unlabeled data. Manual mining at this scale is infeasible. Further, the OOI are often small and occupy less than 1% of image area, are occluded, and co-exist with many semantically different objects in cluttered scenes. Existing semantic image retrieval methods often focus on mining for larger sized geographical landmarks, and/or require extra labeled data, such as images/image-pairs with similar objects, for mining images with generic objects. We propose a fast and robust template matching algorithm in the DNN feature space, that retrieves semantically similar images at the object-level from a large unlabeled pool of data. We project the region(s) around the OOI in the query image to the DNN feature space for use as the template. This enables our method to focus on the semantics of the OOI without requiring extra labeled data. In the context of autonomous driving, we evaluate our system for targeted selection by using failure cases of object detectors as OOI. We demonstrate its efficacy on a large unlabeled dataset with 2.2M images and show high recall in mining for images with small-sized OOI. We compare our method against a well-known semantic image retrieval method, which also does not require extra labeled data. Lastly, we show that our method is flexible and retrieves images with one or more semantically different co-occurring OOI seamlessly.

Keywords: Convolutional Neural Networks, Deep Learning
Abstract: Transportation mode detection from smartphone data is investigated as a relevant problem in the multi-modal transportation systems context. Neural networks are chosen as a timely and viable solution. The goal of this paper is to solve such a problem with a combination model of Convolutional Neural Network (CNN) and Bidirectional-Long short-term memory (BiLSTM) only processing accelerometer and magnetometer data. The performance in terms of accuracy and F1 score on the Sussex-Huawei Locomotion-Transportation (SHL) challenge 2018 dataset is comparable to methods that require the processing of a wider range of sensors. The uniqueness of our work is the light architecture requiring less computational resources for training and consequently a shorter inference time.

Keywords: Vehicle Control, Automated Vehicles
Abstract: This paper proposed a road adaptive semi-active suspension control method, where a trade-off between driving comfort and vehicle stability/road-holding is accomplishable in order to achieve desirable performance results at different road irregularities and velocities by modifying the scheduling variable that is designed by Fuzzy Logic Control. The proposed semi-active controller is founded on the Linear Parameter-Varying framework. Hungarian highway route data has been implemented into the TruckSim simulation environment based on real geographical data having road irregularities in order to compare the proposed adaptive method with a non-adaptive scenario. Simulation results show that all performances have been improved with the proposed method in different road irregularities and velocities.

Keywords: Hand-off/Take-Over, Vulnerable Road-User Safety, Human-Machine Interface
Abstract: In order to facilitate safe interactions between automated vehicles (AVs) and vulnerable road users (VRUs) such as bicyclists, we present a communication application for mobile devices that allows an AV or its passenger and a bicyclist to interact in certain traffic scenarios. At the intersection, the AV or its passenger can change the existing right-of-way rules to prioritise the ego-vehicle or the bicyclist. In a coupled driving simulator in which these two road users can interact, 16 proof-of-concept experiments are conducted. It is found that the perceived safety at conflict points can be increased through the use of the application. An investigation of the user data provides insights into the AV passengers' decision types and duration in the scenarios studied. Moreover, the simulation results are used to revise and further develop the application concept.

Keywords: Automated Vehicles, Reinforcement Learning, Vehicle Control
Abstract: This paper presents the design and implementation of an intelligent switched control for lateral control of autonomous vehicles. The switched control is designed based on Linear Parameter-Varying (LPV) and Youla-Kucera (YK) parameterization. The proposed intelligent system aims to optimize the control switching performance using a Reinforcement Learning (RL) model. The present approach studies the critical problem of initial or sudden large lateral errors in lane-tracking or lane-changing. It ensures stable and smooth switching performance to provide a smooth vehicle response regardless of the lateral error. The proposed RL-based switching strategy is validated using a RENAULT simulator on MATLAB, and compared to another modeled switching strategy with encouraging results.

Keywords: Active and Passive Vehicle Safety, Automated Vehicles
Abstract: To achieve safety of high level automated driving, not only functional failures like E/E system malfunctions and software crashes should be excluded, but also functional insufficiencies and performance limitations such as sensor resolution should be thoroughly investigated and considered. The former problem is known as functional safety (FuSa), which is coped with by ISO 26262. The latter focuses on safe vehicle behavior and is summarized as safety of the intended functionality (SOTIF) within the under development standard ISO 21448. For realizing this safety level, it is crucial to understand the system and the triggering conditions that activate its existing functional insufficiencies. However, the concept of triggering condition is new and still lacks relevant research. In this paper, we interpret triggering condition and other SOTIF-relevant terms in the scope of ISO 21448. We summarize the formal formulations of triggering conditions based on several key principles and provide possible categories for facilitating the systematization. We contribute a novel method for the identification of triggering conditions and offer a comparison with two other proposed methods regarding diverse aspects. Furthermore, we show that our method requires less insight into the system and fewer brainstorm efforts and provides well-structured and distinctly formulated triggering conditions.

Keywords: Autonomous / Intelligent Robotic Vehicles, Intelligent Vehicle Software Infrastructure, Situation Analysis and Planning
Abstract: High-performance autonomy often must operate at the boundaries of safety. When external agents are present in a system, the process of ensuring safety without sacrificing performance becomes extremely difficult. In this paper we present an approach to stress test such systems based on the rapidly exploring random tree (RRT) algorithm. We propose to find faults in such systems through adversarial agent perturbations, where the behaviors of other agents in an otherwise fixed scenario are modified. This creates a large search space of possibilities, which we explore both randomly and with a focused strategy that runs RRT in a bounded projection of the observable states that we call the objective space. The approach is applied to generate tests for evaluating overtaking logic and path planning algorithms in autonomous racing, where the vehicles are driving at high speed in an adversarial environment. We evaluate several autonomous racing path planners, finding numerous collisions during overtake maneuvers in all planners. The focused RRT search finds several times more crashes than the random strategy, and, for certain planners, tens to hundreds of times more crashes in the second half of the track.

Keywords: Lidar Sensing and Perception, Deep Learning, Automated Vehicles
Abstract: The estimation of uncertainty in robotic vision, such as 3D object detection, is an essential component in developing safe autonomous systems aware of their own performance. However, the deployment of current uncertainty estimation methods in 3D object detection remains challenging due to timing and computational constraints. To tackle this issue, we propose LiDAR-MIMO, an adaptation of the multi-input multi-output (MIMO) uncertainty estimation method to the LiDAR-based 3D object detection task. Our method modifies the original MIMO by performing multi-input at the feature level to ensure the detection, uncertainty estimation, and runtime performance benefits are retained despite the limited capacity of the underlying detector and the large computational costs of point cloud processing. We compare LiDAR-MIMO with MC dropout and ensembles as baselines and show comparable uncertainty estimation results with only a small number of output heads. Further, LiDAR-MIMO can be configured to be twice as fast as MC dropout and ensembles, while achieving higher mAP than MC dropout and approaching that of ensembles.

Keywords: Self-Driving Vehicles, Vehicle Environment Perception, Situation Analysis and Planning
Abstract: Detecting other agents and forecasting their behavior is an integral part of the modern robotic autonomy stack, especially in safety-critical scenarios entailing human-robot interaction such as autonomous driving. Due to the importance of these components, there has been a significant amount of interest and research in perception and trajectory forecasting, resulting in a wide variety of approaches. Common to most works, however, is the use of the same few accuracy-based evaluation metrics, e.g., intersection-over-union, displacement error, log-likelihood, etc. While these metrics are informative, they are task-agnostic and outputs that are evaluated as equal can lead to vastly different outcomes in downstream planning and decision making. In this work, we take a step back and critically assess current evaluation metrics, proposing task-aware metrics as a better measure of performance in systems where they are deployed. Experiments on an illustrative simulation as well as real-world autonomous driving data validate that our proposed task-aware metrics are able to account for outcome asymmetry and provide a better estimate of a model's closed-loop performance.

Keywords: Automated Vehicles, Intelligent Vehicle Software Infrastructure, Self-Driving Vehicles
Abstract: Current autonomous vehicle (AV) simulators are built to provide large-scale testing required to prove capabilities under varied conditions in controlled, repeatable fashion. However, they have certain failings including the need for user expertise and complex inconvenient tutorials for customized scenario creation. Simulation of Urban Mobility (SUMO) simulator, which has been presented as an open-source traffic simulation platform, has found use as an AV simulator but suffers from similar issues which makes it difficult for entry-level practitioners to utilize the simulator without significant time investment. In that regard, we provide two enhancements to SUMO simulator geared towards massively improving user experience and providing real-life like variability for surrounding traffic. Firstly, we calibrate a car-following model, Intelligent Driver Model (IDM), for highway and urban naturalistic driving data and sample randomly from the parameter distributions to create realistic background vehicle driving behavior. Secondly, we combine SUMO with OpenAI gym, creating a Python package placed in a docker container which can run simulations based on real world highway and urban layouts with generic output observations and input actions that can be processed via any AV pipeline. For the calibration, we provide results using simulated and real-life data. For the Sumo-Gym package, we showcase a simple AV platform which runs IDM and lane change throughout the highway loop and provide some qualitative results. Our aim through these enhancements is to provide an easy-to-use simulation environment which can be installed in any operating platform and can be readily used for AV testing and validation.

Keywords: Advanced Driver Assistance Systems, Collision Avoidance
Abstract: Testing in virtual road environments has become a widespread approach to validate advanced driver assistance systems (ADAS). A number of automated strategies have been proposed to explore dangerous scenarios, like search-based strategies guided by fitness functions. However, such strategies are likely to produce many uninteresting scenarios, representing so extreme driving situations that fatal accidents are unavoidable irrespective of the action of the ADAS. We propose leveraging datasets from real drives to better align the virtual scenarios to reasonable ones. The alignment is based on a simple distance metric that relate the virtual scenario parameters to the real data. We demonstrate the use of this metric for testing an autonomous emergency braking (AEB) system, taking the highD dataset as a reference for normal situations. We show how search-based testing quickly converges toward very distant scenarios that do not bring much insight into the AEB performance. We then provide an example of a distance-aware strategy that searches for less extreme scenarios that the AEB cannot overcome.

Keywords: Vision Sensing and Perception, Unsupervised Learning, Deep Learning
Abstract: Aiming for higher-level scene understanding, this work presents a neural network approach that takes a road-layout map in bird's-eye-view as input, and predicts a human-interpretable graph that represents the road's topological layout. Our approach elevates the understanding of road layouts from pixel level to the level of graphs. To achieve this goal, an image-graph-image auto-encoder is utilized. The network is designed to learn to regress the graph representation at its auto-encoder bottleneck. This learning is self-supervised by an image reconstruction loss, without needing any external manual annotations. We create a synthetic dataset containing common road layout patterns and use it for training of the auto-encoder in addition to the real-world Argoverse dataset. By using this additional synthetic dataset, which conceptually captures human knowledge of road layouts and makes this available to the network for training, we are able to stabilize and further improve the performance of topological road layout understanding on the real-world Argoverse dataset. The evaluation shows that our approach exhibits comparable performance to a strong fully-supervised baseline.

Keywords: Vehicle Environment Perception, Deep Learning, Self-Driving Vehicles
Abstract: We present TransMOT, a novel transformer-based end-to-end trainable online tracker and detector for point cloud data. The model utilizes a cross- and a self-attention mechanism and is applicable to lidar data in an automotive context, as well as other data types, such as radar. Both track management and the detection of new tracks are performed by the same transformer decoder module and the tracker state is encoded in feature space. With this approach, we make use of the rich latent space of the detector for tracking rather than relying on low-dimensional bounding boxes. Still, we are able to retain some of the desirable properties of traditional Kalman-filter based approaches, such as an ability to handle sensor input at arbitrary timesteps or to compensate frame skips. This is possible due to a novel module that transforms the track information from one frame to the next on feature-level and thereby fulfills a similar task as the prediction step of a Kalman filter. Results are presented on the challenging real-world dataset nuScenes, where the proposed model outperforms its Kalman filter-based tracking baseline.

Keywords: Automated Vehicles, Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: One of the main challenges for the introduction of Automated Driving Systems (ADSs) is their verification and validation (V&V). Simulation based testing has been widely accepted as an essential aspect of the ADS V&V processes.Simulations are especially useful when exposing the ADS to challenging driving scenarios, as they offer a safe and efficient alternative to real world testing. It is thus suggested that evidence for the safety case for an ADS will include results from both simulation and real-world testing. However, for simulation results to be trusted as part of the safety case of an ADS for its safety assurance, it is essential to prove that the simulation results are representative of the real world, thus validating the simulation platform itself.

In this paper, we propose a novel methodology for validating the simulation environments focusing on comparing point cloud data from real LiDAR sensor and a simulated LiDAR sensor model. A 3D object dissimilarity metric is proposed to compare between the two maps (real and simulated), to quantify how accurate the simulation is. This metric is tested on collected LiDAR point cloud data and the simulated point cloud generated in the simulated environment.

Keywords: Deep Learning
Abstract: Several research works have been proposed to evaluate the robustness of deep-learning-based semantic segmentation models for automated driving under adversarial attacks. However, the types of tested adversarial examples and evaluated segmentation models are limited in the previous empirical studies, which imposes the restrictions on the understanding of the robustness of semantic segmentation models. To alleviate these problems, we would promote the research on the robustness of semantic segmentation models systematically from two aspects of influence factors: internal factors of model structures and external factors of environmental perturbations at the data level. In this paper, we provide a comprehensive study using these typical models with different internal structures: Fully Convolutional Networks (FCN), Pyramid Scene Parsing Network (PSPNet), DeepLabV3+, and SegNet with different backbones. These models would be evaluated on the metrics of robustness under both white-box attacks and black-box attacks. Based on our experiment, we make qualitative and quantitative analyses of the robustness of diverse models under different influence factors. With more empirical study cases, our work gives inspiration to the robustness study of semantic segmentation for automated driving which is meaningful and beneficial to the safety of the intended functionality (SOTIF).

Keywords: Vulnerable Road-User Safety, Advanced Driver Assistance Systems, Deep Learning
Abstract: For safe and comfortable navigation of autonomous vehicles, it is crucial to know the pedestrian's intention of crossing the street. Generally, human drivers are aware of the traffic objects (e.g., crosswalks and traffic lights) in the environment while driving; likewise, these objects would play a crucial role for autonomous vehicles. In this research, we propose a novel pedestrian intention estimation method that not only takes into account the influence of traffic objects but also learns their contribution levels on the intention of the pedestrian. Our proposed method, referred to as Scene Spatio-Temporal Graph Convolutional Network (Scene-STGCN), takes benefits from the strength of Graph Convolutional Networks and efficiently encodes the relationships between the pedestrian and the scene objects both spatially and temporally. We conduct several experiments on the Pedestrian Intention Estimation (PIE) dataset and illustrate the importance of scene objects and their contribution levels in the task of pedestrian intention estimation. Furthermore, we perform statistical analysis on the relevance of different traffic objects in the PIE dataset and carry out an ablation study on the effect of various information sources in the scene. Finally, we demonstrate the significance of the proposed Scene-STGCN through experimental comparisons with several baselines. The results indicate that our proposed Scene-STGCN outperforms the current state-of-the-art method by 0.03 in terms of ROC-AUC metric.

Keywords: Image, Radar, Lidar Signal Processing, Deep Learning, Intelligent Vehicle Software Infrastructure
Abstract: To reduce sideswipes and collision accidents involving two- and four-wheeled vehicles under mixed traffic flow conditions, we previously created a smartphone application (app) that predicts acceleration and driving lane behaviors of two-wheeled vehicles. In this system, vehicles are detected from road images taken by a smartphone camera, and vehicles positions on the road are estimated by our projection conversion algorithm. However, regarding that app, it is necessary to improve the accuracy of the vanishing point calculations in the camera images. Accordingly, in order to reduce calculation costs, we created a method that integrates road segmentation and vehicle detection to create a new scheme for detecting road edges and the vanishing point, even on roads without lane lines. These improvements will help maintain the accuracy of vanishing point calculations while facilitating their high real-time characteristics.

Keywords: Vehicle Environment Perception, Autonomous / Intelligent Robotic Vehicles
Abstract: Environmental context detection is a topic of interest for the navigation community since it enables to build a context-adaptive solution by choosing the proper data processing algorithm or by selecting the sensors to be used to dynamically adapt the navigation solution design itself. This paper proposes to build a supervised machine learning model which can robustly classify multiple contexts such as urban canyons, urban, trees and open-sky areas using GNSS data only. A training and test database have been built with four datasets acquired at different times in order to prove the relevance of the solution. These datasets are made available to the public for research purpose. The choices of features and classifier are also discussed and compared to others papers. The classifier achieved an average 82.40% of classification accuracy.

Keywords: Automated Vehicles, Vulnerable Road-User Safety, Deep Learning
Abstract: The human driver is no longer the only one concerned with the complexity of the driving scenarios. Autonomous vehicles (AV) are similarly becoming involved in the process. Nowadays, the development of AV in urban places raises essential safety concerns for vulnerable road users (VRUs) such as pedestrians. Therefore, to make the roads safer, it is critical to classify and predict the pedestrians’ future behavior. In this paper, we present a framework based on multiple variations of the Transformer models able to predict the pedestrian street-crossing decision-making based on the dynamics of its initiated trajectory. We showed that using solely bounding boxes as input features can outperform the previous state-of-the-art results by reaching a prediction accuracy of 91% and an F1-score of 0.83 on the PIE dataset. In addition, we introduced a large-size simulated dataset (CP2A) using CARLA for action prediction. Our model has similarly reached high accuracy (91%) and F1-score (0.91) on this dataset. Interestingly, we showed that pre-training our Transformer model on the CP2A dataset and then fine-tuning it on the PIE dataset is beneficial for the action prediction task. Finally, our model’s results are successfully supported by the "human attention to bounding boxes" experiment which we created to test humans ability for pedestrian action prediction without the need for environmental context. The code for the dataset and the models is available at: https://github.com/linaashaji/Action_Anticipation

Keywords: Automated Vehicles, Advanced Driver Assistance Systems, Vehicle Environment Perception
Abstract: Advanced Driver Assistance Systems have been integrated in vehicles with the aim of reducing traffic accidents caused by human error. Due to the safety-critical application of these systems, a reliable implementation requires an extensive testing process of hardware and software units as well as the entire integrated system. The increasing complexity of these systems, which incrementally have more control over the vehicle, makes purely real-world safeguarding impracticable. As a solution, complementary virtual tests are used to support the validation process in a safe and accelerated way. However, a method for distributing test cases across real and virtual domains according to the relevance of the test method has yet to be defined. In this paper, we investigate different levels of reality in testing ADAS/automated driving functions and derive advantages, disadvantages, and system limitations for each. An emergency braking system serves as a use case, which is evaluated using Hardware-in-the-Loop, Sensor-in-the-Loop, Vehicle-in-the-Loop test methods, as well as vehicle tests on a proving ground. The obtained results show that simulation-based testing is a useful complement to real-world testing. However, various phenomena such as actuator delays or camera lenses have a large impact on the obtained results and need to be taken into account (in models) to ensure the realism of the simulation-based approaches. In addition, the inclusion of real components increases the deviations between test repetitions, being necessary to conduct real tests to evaluate the final performance of the integrated system.

Keywords: Situation Analysis and Planning, Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: Path planning algorithms for autonomous vehicles need to account for safety and comfort, more so, in scenarios where the possibility of casualties are higher due to increased traffic frequency and limited visibility. In this paper, we discuss the idea of a virtual obstacle deployed at occluded scenarios to avoid a potential collision or severe deceleration of the ego-vehicle. Urban scenarios like intersections, roundabout and merging are experimented. Results of simulating the integration of virtual obstacle with the trajectory planning algorithm, are analyzed in detail comparing speed and acceleration profiles.

Keywords: Traffic Flow and Management, Societal Impacts, Automated Vehicles
Abstract: We propose a new model for fair division and vehicle routing, where drivers have monotone profit preferences, and their vehicles have feasibility constraints, for customer requests. For this model, we design two new axiomatic notions for fairness for drivers: FEQ1 and FEF1. FEQ1 encodes driver pairwise bounded equitability. FEF1 encodes driver pairwise bounded envy freeness. We compare FEQ1 and FEF1 with popular fairness notions such as EQ1 and EF1. We also give algorithms for guaranteeing FEQ1 and FEF1, respectively.

Keywords: Automated Vehicles, Reinforcement Learning, Deep Learning
Abstract: Learning-based approaches, such as reinforcement learning (RL) and imitation learning (IL), have indicated superiority over rule-based approaches in complex urban autonomous driving environments, showing great potential to make intelligent decisions. However, current RL and IL approaches still have their own drawbacks, such as low data efficiency for RL and poor generalization capability for IL. In light of this, this paper proposes a novel learning-based method that combines deep reinforcement learning and imitation learning from expert demonstrations, which is applied to longitudinal vehicle motion control in autonomous driving scenarios. Our proposed method employs the soft actor-critic structure and modifies the learning process of the policy network to incorporate both the goals of maximizing reward and imitating the expert. Moreover, an adaptive prioritized experience replay is designed to sample experience from both the agent's self-exploration and expert demonstration, in order to improve sample efficiency. The proposed method is validated in a simulated urban roundabout scenario and compared with various prevailing RL and IL baseline approaches. The results manifest that the proposed method has a faster training speed, as well as better performance in navigating safely and time-efficiently.

Keywords: Mapping and Localization, Lidar Sensing and Perception, Autonomous / Intelligent Robotic Vehicles
Abstract: The following work presents the development of a vehicle positioning function using vehicle mounted lidar sensors of the type Ouster OS1-32 and retroreflective landmarks. The function is developed for the use case of a wheeled mobile driving simulator, which is a mobile robot performing driving maneuvers within a virtually limited circular workspace. Nevertheless, the function is transferable to other applications where a vehicle’s dynamic position on a limitable area is to be determined with high dependability and independently of random environmental features. Based on the specific requirements for the simulator operation, a suitable architecture of landmarks, consisting of retroreflective cylinders, is derived. Then, the software architecture is presented, which mainly relies on a map matching algorithm. In comparison with a DGPS reference system and under artificial perturbation of the lidar-landmark-interaction, the performance and robustness of the function is evaluated on a real prototype. The results show high potential of the developed function for a safety relevant positioning of the vehicle.

Keywords: Automated Vehicles, Self-Driving Vehicles, Situation Analysis and Planning
Abstract: One way to increase user acceptance in automated vehicles is to explain their driving decisions, but current methods still involve human interpretations and are thus prone to errors. Therefore, the presented method formulates summaries that clarify the automated vehicle’s driving decision by extracting all necessary information automatically from the planning algorithm. This paper shows the generation of three exemplary statement types and their validation with an online survey that investigated users' preferences. The results suggest that participants favor statements describing information that affect the driving decision as well as applicable traffic rules. Additionally, individual information needs should be considered when constructing modular explanations. Although this analysis does not consider sophisticated human machine interfaces nor real traffic scenarios, it does show, for the first time, how satisfying statements can be generated using a planning algorithm without any human-induced bias. This is an important step towards self-contained transparency of automated driving functions and can therefore lay the basis for future human machine interfaces.

Keywords: Collision Avoidance, Vision Sensing and Perception, Deep Learning
Abstract: We propose an image-based collision risk prediction model and a training strategy that allows training on simulated video data and successfully generalizes to real data. By doing so, we solve the data scarcity problem of collecting and labeling real (near) collisions, which are exceptionally rare events. Domain generalization from simulated to real data is taken into account by design by decoupling the learning strategy, and using task-specific, domain-resilient intermediate representations. Specifically, we use optical flow and vehicle bounding boxes, since they are instinctively related to the task of collision risk prediction and because their simulated-to-real domain gap is significantly lower than that of camera video data, i.e., they are more domain resilient. To demonstrate our approach, we present RiskNet, a novel neural network for image-based collision risk prediction, which classifies individual frames of a video sequence of a front-facing camera as safe or unsafe. Additionally, we present two novel datasets: the simulated Prescan dataset (which we intend to make publicly available) for training and the YouTube Driving Incidents Database (YDID) for real-world testing. The performance of RiskNet, trained solely on simulated data and tested on the real-world YDID, is comparable to that of a human driver, both in accuracy (91.8% vs. 93.6%) and F1-score (0.92 vs 0.94).

Keywords: Autonomous / Intelligent Robotic Vehicles, Collision Avoidance, Self-Driving Vehicles
Abstract: With the increasing need for safe control in the domain of autonomous driving, model-based safety-critical control approaches are widely used, especially Control Barrier Function (CBF) based approaches. Among them, Exponential CBF (eCBF) is particularly popular due to its realistic applicability to high-relative-degree systems. However, for most of the optimization-based controllers utilizing CBF-based constraints, solution feasibility is a common issue raised from potential conflict among different constraints. Moreover, how to incorporate uncertainty into the eCBF-based constraints in high-relative-degree systems to account for safety remains an open challenge. In this paper, we present a novel approach to extend eCBF-based safe critical controller to a probabilistic setting to handle potential motion uncertainty from system dynamics. More importantly, we leverage an optimization-based technique to provide a solution feasibility guarantee in run time, while ensuring probabilistic safety. Lane changing and intersection handling are demonstrated as two use cases, and experiment results are provided to show the effectiveness of the proposed approach.

Keywords: Automated Vehicles, Situation Analysis and Planning, Self-Driving Vehicles
Abstract: Development and safety validation of highly automated vehicles increasingly relies on data and data-driven methods. In processing sensor datasets for environment perception, it is common to use public and commercial datasets for training and evaluating machine learning based systems. For system-level evaluation and safety validation of an automated driving system, real-world trajectory datasets are of great value for several tasks in the process, i. a. for testing in simulation, scenario extraction or training of road user agent models. Ground-based recording methods such as sensor-equipped vehicles or infrastructure sensors are sometimes limited, for instance, due to their field of view. Camera-equipped drones, however, offer the ability to record road users without vehicle-to-vehicle occlusion and without influencing traffic. The highway drone dataset (highD) has shown that the recording method is efficient in terms of cumulative kilometers and has become a benchmark dataset for many research questions. It contains many vehicle interactions due to dense traffic, but lacks merging scenarios, which are challenging for highly automated vehicles. Therefore, we propose this highway drone dataset called exiD, recorded using camera-equipped drones at entries and exits on the German Autobahn. The dataset contains 69172 road users classified as car, truck and vans and a total amount of more than 16 hours of measurement data. For non-commercial public research, the exiD dataset is available free of charge at https://www.exid-dataset.com.

Keywords: Automated Vehicles, Smart Infrastructure, Deep Learning
Abstract: Data-intensive machine learning based techniques increasingly play a prominent role in the development of future mobility solutions - from driver assistance and automation functions in vehicles, to real-time traffic management systems realized through dedicated infrastructure. The availability of high quality real-world data is often an important prerequisite for the development and reliable deployment of such systems in large scale. Towards this endeavour, we present the A9-Dataset based on roadside sensor infrastructure from the 3 km long Providentia++ test field near Munich in Germany. The dataset includes anonymized and precision-timestamped multi-modal sensor and object data in high resolution, covering a variety of traffic situations. As part of the first set of data, which we describe in this paper, we provide camera and LiDAR frames from two overhead gantry bridges on the A9 autobahn with the corresponding objects labeled with 3D bounding boxes. The first set includes in total more than 1000 sensor frames and 14000 traffic objects. The dataset is available for download at https://a9-dataset.com.

Keywords: Automated Vehicles, Self-Driving Vehicles, Vehicle Environment Perception
Abstract: Testing and validating advanced automotive software is of paramount importance to guarantee safety and quality. While real-world testing is highly demanding and simulation testing is not reliable, we propose a new augmented reality framework that takes advantage of both environments. This new testing methodology is intended to be a bridge between Vehicle-in-the-Loop and real-world testing. It enables to easily and safely place the whole vehicle and all its software, from perception to control, in realistic test conditions. This framework provides a flexible way to introduce any virtual element in the outputs of the sensors of the vehicle under test. For each modality of sensing, the framework requires a real time augmentation function that preserves real sensor data and enhances them with virtual data. The LiDAR data augmentation function is presented together with its implementation details. Relying on both qualitative and quantitative analysis of experimental results, the representability of tests scenes generated by the augmented reality framework is finally proven.

Keywords: Automated Vehicles, Cooperative ITS, Collision Avoidance
Abstract: This paper considers a coordination problem for Connected and Automated Vehicles (CAVs) in mixed traffic at unsignalized intersections. In such a setting, the behavior of the Human-Driven Vehicles (HDVs) is difficult to predict, thus challenging the formulation and the solution of the coordination problem. To solve this problem, we propose a coordination strategy, where CAVs are used as both sensors and actuators in mixed platoons. A timeslot-based approach is used to coordinate the occupancy of the intersection and to compensate for the HDVs behavior. The proposed approach has a bi-level optimization structure built upon the Model Predictive Control (MPC) framework that decides the crossing order and computes the vehicles' commands. In simulations, we show that the choice of the HDV prediction model heavily affects the coordination by evaluating the performance of two different HDV models: car-following and constant velocity, where the latter demonstrates more consistent results in the presence of deviation of the HDVs' behavior from a nominal model.

Keywords: Lidar Sensing and Perception, Autonomous / Intelligent Robotic Vehicles, Intelligent Ground, Air and Space Vehicles
Abstract: Localization in the environment is an essential navigational capability for animals and indoor robotic vehicles. In indoor environments, it is still challenging to perfectly solve the global localization problem using probabilistic methods. However, animals are able to instinctively localize themselves with much less effort. Therefore, an intriguing and promising approach is to seek biological inspiration from animals. In this paper, we present a biologically-inspired global localization system using a LiDAR sensor that utilizes a hippocampal model and a landmark-based relocalization approach. The experiment results show that the proposed method is competitive with Monte Carlo Localization, and the results demonstrate the high accuracy, applicability, and reliability of the proposed biologically-inspired localization system in various localization scenarios.

Keywords: Advanced Driver Assistance Systems, Automated Vehicles, Self-Driving Vehicles
Abstract: This research work focuses on electrical investigations and characterization of the Automotive Ethernet channel for 25Gbps (25GBASE-T1). This characterization is performed with the aid of insertion loss (SDD12/SDD21) mixed-mode scattering parameters (S-parameters) which describe the transmitted signal electrical behavior within the Ethernet channel considering it’s coupled transmission line characteristics. This paper commences with an introductory background of this research topic. This is then followed with an overview of the Automotive Ethernet channel and components. A succeeding section addresses the various channel electrical characteristic parameters. With the aid of implemented multi-gigabit Ethernet test boards, to emulate an ECU-ECU communication system setup, the fourth section investigates and discusses insertion loss test bench measurements and simulations on channel segments (PCB, link segment) and complete single 25Gbps (25GBASE-T1) Ethernet channel. The investigations in this study deploy Shielded Twisted Pair (STP) cables of varying length and cable topologies as a physical transmission medium. Last section addresses the key takeaways of this paper and recommendations on subsequent analysis.

Keywords: Deep Learning, Vision Sensing and Perception, Legal Impacts
Abstract: For the development of machine learning-based driver assistance systems and highly automated driving functions, training data play a significant role in ensuring machine learning algorithms generalize well on real driving scenarios. However, data protection regulations in Europe require that individuals' data should be processed in such a way that the individual cannot be identified from the collected data. Therefore, before camera images taken from test vehicles save on a server, license plates and faces of individuals should be anonymized first. Nevertheless, the impact of using anonymized data on the performance of machine learning algorithms remains unclear. Our work aims to evaluate the impact of anonymization on the task of semantic segmentation using diverse neural network architectures, a range of input image resolutions, and different anonymization patterns. We observe statistically significant effects of anonymizing image data on model performance and investigate methods for mitigating segmentation precision loss.

Keywords: Electric and Hybrid Technologies, Deep Learning
Abstract: The usability of vehicles is highly dependent on their energy consumption. In particular, one of the main factors hindering the mass adoption of electric (EV), hybrid (HEV), and plug-in hybrid (PHEV) vehicles is textit{range anxiety}, which occurs when a driver is uncertain about the availability of energy for a given trip. To tackle this problem, we propose a machine learning approach for modeling the battery energy consumption. By reducing predictive uncertainty, this method can help increase trust in the vehicle's performance and thus boost its usability. Most related work focuses on physical and/or chemical models of the battery that affect the energy consumption. We propose a data-driven approach which relies on real-world datasets including battery related attributes. Our approach shows an improvement in terms of predictive uncertainty as well as in accuracy compared to traditional methods.

Keywords: Deep Learning, Security, Self-Driving Vehicles
Abstract: Visual detection is a key task in autonomous driving, and it serves as a crucial foundation for self-driving planning and control. Deep neural networks have achieved promising results in various visual tasks, but they are known to be vulnerable to adversarial attacks. A comprehensive understanding of deep visual detectors' vulnerability is required before people can improve their robustness. However, only a few adversarial attack/defense works have focused on object detection, and most of them employed only classification and/or localization losses, ignoring the objectness aspect. In this paper, we identify a serious objectness-related adversarial vulnerability in YOLO detectors and present an effective attack strategy targeting the objectness aspect of visual detection in autonomous vehicles. Furthermore, to address such vulnerability, we propose a new objectness-aware adversarial training approach for visual detection. Experiments show that the proposed attack targeting the objectness aspect is 45.17% and 43.50% more effective than those generated from classification and/or localization losses on the KITTI and COCO_traffic datasets, respectively. Also, the proposed adversarial defense approach can improve the detectors' robustness against objectness-oriented attacks by up to 21% and 12% mAP on KITTI and COCO_traffic, respectively.

Keywords: Vision Sensing and Perception, Image, Radar, Lidar Signal Processing, Vehicle Environment Perception
Abstract: Amodal perception terms the ability of humans to imagine the entire shapes of occluded objects. This gives humans an advantage to keep track of everything that is going on, especially in crowded situations. Typical perception functions, however, lack amodal perception abilities and are therefore at a disadvantage in situations with occlusions. Complex urban driving scenarios often experience many different types of occlusions and, therefore, amodal perception for automated vehicles is an important task to investigate. In this paper, we consider the task of amodal semantic segmentation and propose a generic way to generate datasets to train amodal semantic segmentation methods. We use this approach to generate an amodal Cityscapes dataset. Moreover, we propose and evaluate a method as baseline on Amodal Cityscapes, showing its applicability for amodal semantic segmentation in automotive environment perception. Upon paper acceptance, we will provide the means to re-generate this dataset on github.

Keywords: Reinforcement Learning, Cooperative ITS, Cooperative Systems (V2X)
Abstract: Autonomous driving system controls a vehicle using path planning.Path planning for automated vehicles observes a vehicle and the surrounding information and plans a trajectory on the basis of rule-based approach.However, the rule-based path planning cannot generate an appropriate trajectory for complex scenes, such as two vehicles passes each other at an intersection without traffic lights.Such complex scene is called deadlock.For avoiding the deadlock, it is very costly to create rules manually.In this paper, we propose a multi-agent deep reinforcement learning method to generate appropriate trajectories at the deadlock scenes.The proposed method consists of a single feature extractor and actor-critic branches.Moreover, we introduce a mask-attention mechanism for visual explanation.By taking a look at the obtained attention maps, we can confirm the obtained agent and the reason of the behavior.For evaluating our method, we develop a simulator environment of autonomous driving that produces a certain deadlock scene.The experimental results with the developed environment show that the proposed method can generate trajectories avoiding deadlocks.

Keywords: Self-Driving Vehicles, Situation Analysis and Planning, Vehicle Control
Abstract: Motion planning is one of the key modules in autonomous driving systems to generate trajectories for self-driving vehicles to follow. A common motion planning approach is to generate trajectories within semantic safe corridors. The trajectories are generated by optimizing parametric curves (textit{e.g.} Bezier curves) according to an objective function. To guarantee safety, the curves are required to satisfy the convex hull property, and be contained within the safety corridors. The convex hull property however does not necessary hold for time-dependent corridors, and depends on the shape of corridors. The existing approaches only support simple shape corridors, which is restrictive in real-world, complex scenarios. In this paper, we provide a sufficient condition for general convex, spatio-temporal corridors with theoretical proof of guaranteed convex hull property. The theorem allows for using more complicated shapes to generate spatio-temporal corridors and minimizing the uncovered search space to O(frac{1}{n^2}) compared to O(1) of trapezoidal corridors, which can improve the optimality of the solution. Simulation results show that using general convex corridors yields less harsh brakes, hence improving the overall smoothness of the resulting trajectories.

Keywords: Human-Machine Interface, Automated Vehicles, Societal Impacts
Abstract: Commentary driving is a technique in which drivers verbalise their observations, assessments and intentions. By speaking out their thoughts, both learning and expert drivers are able to create a better understanding and awareness of their surroundings. In the intelligent vehicle context, automated driving commentary can provide intelligible explanations about driving actions, thereby assisting a driver or an end-user during driving operations in challenging and safety-critical scenarios. In this paper, we conducted a field study in which we deployed a research vehicle in an urban environment to obtain data. While collecting sensor data of the vehicle's surroundings, we obtained driving commentary from a driving instructor using the think-aloud protocol. We analysed the driving commentary and uncovered an explanation style; the driver first announces his observations, announces his plans, and then makes general remarks. He also makes counterfactual comments. We successfully demonstrated how factual and counterfactual natural language explanations that follow this style could be automatically generated using a transparent tree-based approach. Generated explanations for longitudinal actions (e.g., stop and move) were deemed more intelligible and plausible by human judges compared to lateral actions, such as lane changes. We discussed how our approach can be built on in the future to realise more robust and effective explainability for driver assistance as well as partial and conditional automation of driving functions.

Keywords: Automated Vehicles
Abstract: Due to the technical advances of automated vehicles (AVs), new uncertainties for human road users arise. To overcome these uncertainties, driving strategies of AVs might be aligned to human interaction styles. In vehicle-vehicle interactions, driving behavior is informed by remaining time gaps between vehicles. This video-based experiment investigated the influence of gap sizes and the measurement method on driving decisions. N = 32 participants experienced a highly automated drive in which their AV approached narrow passages. The time to arrival (TTA) of the oncoming traffic was varied. Participants had to decide to drive first or second, indicate their decision certainty, and the situation’s criticality. The videos were presented in ascending, descending, and random order. Moreover, participants adjusted the TTA at which they would drive first and second. The results indicated a higher probability of driving first and lower criticality with increasing TTA. Decision certainty was lowest around the 50% threshold, while longer and shorter TTAs resulted in higher certainty. Results differed between the methods. The findings provide guidance for the design of automated systems to mimic human driving behavior.

Keywords: Automated Vehicles, Vehicle Control
Abstract: One of the major challenges of testing and validation of automated vehicles is covering the enormous amount of possible driving situations. Efficient and reliable simulation tools are therefore required to speed up those phases. The SET Level project aims at providing an environment for simulation-based test and development of automated driving functions, focusing, as one of its main objectives, on providing an open, flexible, and extendable simulation environment, compliant to current simulation standards as Functional Mock-up Interface (FMI) and Open Simulation Interface (OSI). Within this context, the authors proposed a vehicle simulation model chain including models of motion control, actuators (with actuator management) and vehicle dynamics with two different detail levels. The models were built in Matlab/Simulink, including a developed OSI wrapper for integration into existing simulation environments. In the paper, the simulation architecture including the OSI wrapper and the single models of the chain is presented, as well as simulation results, showing the potential of the presented model chain in carrying out analyses in the field of testing automated driving functions.

Keywords: Automated Vehicles, Deep Learning, Advanced Driver Assistance Systems
Abstract: This paper presents a hierarchical framework combining a machine learning (ML)-based approach with a model-based approach to predict the behavior and trajectory of surrounding target vehicles on a highway. First, the behavior predictor based on a recurrent neural network determines the behavior of the target vehicle by learning its complex interactions with surrounding vehicles and the traffic environment. Then, the trajectory predictor generates a predicted trajectory which follows the predicted behavior for each target vehicle. A curvature continuous spiral curve and model predicted control are used for the trajectory predictor to consider the dynamic constraints and the collision safety of the target vehicle. The hierarchical predictor composed of the ML-based approach and the model-based approach can predict the behavior and trajectory of a target vehicle, taking into account dynamic constraints and collisions as well as complex interactions with surrounding traffic. We evaluated the proposed predictor through NGSIM public dataset. The results showed that the predicted trajectories have lower errors over a long prediction time. We also showed that the proposed predictor could operate in real-time by efficiently utilizes computing resources.

Keywords: Human-Machine Interface, Deep Learning, Advanced Driver Assistance Systems
Abstract: Predicting driver’s attention in complex driving scenarios is becoming a hot topic due to it helps the design of some autonomous driving tasks, optimizing visual scene understanding and contributing knowledge to the decision making. We introduce ARAGAN, a driver attention estimation model based on a conditional Generative Adversarial Network (cGAN). This architecture uses some of the most challenging and novel deep learning techniques to develop this task. It fuses adversarial learning with Multi-Head Attention mechanisms. To the best of our knowledge, this combination has never been applied to predict driver’s attention. Adversarial mechanism learns to map an attention image from an RGB traffic image while mapping the loss function. Attention mechanism contributes to the deep learning paradigm finding the most interesting feature maps inside the tensors of the net. In this work, we have adapted this concept to find the saliency areas in a driving scene. An ablation study with different architectures has been carried out, obtained the results in terms of some saliency metrics. Besides, a comparison with other state-of-the-art models has been driven, outperforming results in accuracy and performance, and showing that our proposal is adequate to be used on real-time applications. ARAGAN has been trained in BDDA and tested in BDDA and DADA2000, which are two of the most complex driver attention datasets available for research.

Keywords: Human-Machine Interface, Advanced Driver Assistance Systems, Novel Interfaces and Displays
Abstract: A lack of sufficient situational awareness is a primary cause of traffic crashes due to human error. Redirecting a driver's attention to critical objects is essential, but alerting driver about all critical objects can lead to distraction. This paper develops and evaluates an adaptive support system that incorporates drivers' fixations as a proxy for their situational awareness. We implement an experimental system that detects a driver's gaze on important objects in the traffic scene and adapts a cueing strategy in an augmented reality-based driver awareness assistance interface. We collect and analyze data from 15 participants and show that our adaptive support system strategy is effective without increasing the drivers' cognitive workload. Finally, we show that our system can increase ratio of drivers‘ fixations on critical objects in their view without significantly increasing dwell time per object.

Keywords: Active and Passive Vehicle Safety, Automated Vehicles
Abstract: As automated vehicles are expected to significantly reduce the number of fatalities in road traffic, ensuring their safety is one of the most critical challenges in the industry today. The Responsibility-Sensitive Safety (RSS) concept is a step towards this goal. RSS formalizes reasonable boundaries for the foreseeable worst-case behavior of traffic participants by defining clear mathematically proven rules. All parameters used in RSS have a physical meaning and are thus well understandable, but the values of these parameters have a crucial effect on the applicability of this approach. Choosing too conservative parameter values may impact traffic flow, while the opposite could lead to uncomfortable or potentially unsafe driving behavior. While a majority of work concentrated on the longitudinal use case, in this work we focus on finding reasonable parameter values for lateral safety. We propose scopes and parameter sets for the RSS minimum lateral distance extracted from human driving behavior that allow for a comfortable driving behavior while not hindering traffic flow.

Keywords: Automated Vehicles
Abstract: Advances in the field of environment perception for automated agents have resulted in an ongoing increase in generated sensor data. The available computational resources to process these data are bound to become insufficient for real-time applications. Reducing the amount of data to be processed by identifying the most relevant data based on the agents' situation, often referred to as situation-awareness, has gained increasing research interest, and the importance of complementary approaches is expected to increase further in the near future. In this work, we extend the applicability range of our recently introduced concept for situation-aware environment perception to the decentralized automation architecture of the UNICARagil project. Considering the specific driving capabilities of the vehicle and using real-world data on target hardware in a post-processing manner, we provide an estimate for the daily reduction in power consumption that accumulates to 36.2%. While achieving these promising results, we additionally show the need to consider scalability in data processing in the design of software modules as well as in the design of functional systems if the benefits of situation awareness shall be leveraged optimally.

Keywords: Autonomous / Intelligent Robotic Vehicles, Reinforcement Learning, Impact on Traffic Flows
Abstract: Driving on roads is restricted by various traffic rules, aiming to ensure safety for all traffic participants. However, human road users usually do not adhere to these rules strictly, resulting in varying degrees of rule conformity. Such deviations from given rules are key components of today's road traffic. In autonomous driving, robotic agents can disturb traffic flow, when rule deviations are not taken into account. In this paper, we present an approach to derive the distribution of degrees of rule conformity from human driving data. We demonstrate our method with the Waymo Open Motion dataset and Safety Distance and Speed Limit rules.

Keywords: Automated Vehicles, Vulnerable Road-User Safety, Deep Learning
Abstract: Gesture recognition is essential for the interaction of autonomous vehicles with humans. While the current approaches focus on combining several modalities like image features, keypoints and bone vectors, we present neural network architecture that delivers state-of-the-art results only with body skeleton input data. We propose the spatio-temporal multilayer perceptron for gesture recognition in the context of autonomous vehicles. Given 3D body poses over time, we define temporal and spatial mixing operations to extract features in both domains. Additionally, the importance of each time step is re-weighted with Squeeze-and-Excitation layers. An extensive evaluation of the TCG and Drive&Act datasets is provided to showcase the promising performance of our approach. Furthermore, we deploy our model to our autonomous vehicle to show its real-time capability and stable execution.

Keywords: Smart Infrastructure, Mapping and Localization, Vision Sensing and Perception
Abstract: This paper demonstrates a system comprised of infrared beacons and a camera equipped with an optical band-pass filter. Our system can reliably detect and identify individual beacons at SI{100}{m} distance regardless of lighting conditions. We describe the camera and beacon design as well as the image processing pipeline in detail. In our experiments, we investigate and demonstrate the ability of the system to recognize our beacons in both daytime and nighttime conditions. High precision localization is a key enabler for automated vehicles but remains unsolved, despite strong recent improvements. Our low-cost, infrastructure-based approach is a potential step towards solving the localization problem. All datasets are made available here https://embedded.rwth-aachen.de/doku.php?id=forschung:mobility:infralocalization:concept.

Keywords: Autonomous / Intelligent Robotic Vehicles, Self-Driving Vehicles
Abstract: Autonomous vehicle navigation in shared pedestrian environments requires the ability to predict future crowd motion as well as understand human behaviour. However, most existing methods predict pedestrian future motion without considering potential collisions within the crowd. Furthermore, most current predictive models are tested on datasets that assume full observability of the crowd by relying on a top-down view, which does not reflect the real-world use case of autonomous vehicles due to the inherent limitations of on-board sensors such as visual occlusion. Inspired by prior works, we propose a pedestrian motion prediction model trained via contrastive learning, improving prediction accuracy as well as forecasting collision-free trajectories. Additionally, we propose a method for implementing a predictor using a multi-pedestrian probabilistic tracker, which fuses multiple on-board sensors to track pedestrians in 3D space. Through comprehensive experiments on both aerial view and driving datasets collected in a real-world urban environment, we show that our proposed method improves on state of art methods with better prediction accuracy and more socially acceptable prediction trajectories.

Keywords: Autonomous / Intelligent Robotic Vehicles, Vehicle Environment Perception, Recurrent Networks
Abstract: Reliably predicting future occupancy of highly dynamic urban environments is an important precursor for safe autonomous navigation. Common challenges in the prediction include forecasting the relative position of other vehicles, modelling the dynamics of vehicles subjected to different traffic conditions, and vanishing surrounding objects. To tackle these challenges, we propose a spatio-temporal prediction network pipeline that takes the past information from the environment and semantic labels separately for generating future occupancy predictions. Compared to the current SOTA, our approach predicts occupancy for a longer horizon of 3 seconds and in a relatively complex environment from the nuScenes dataset. Our experimental results demonstrate the ability of spatio-temporal networks to understand scene dynamics without the need for HD-Maps and explicit modeling dynamic objects. We publicly release our occupancy grid dataset based on nuScenes to support further research.

Keywords: Lidar Sensing and Perception, Vision Sensing and Perception, Cooperative ITS
Abstract: Improvements in sensor technologies as well as machine learning methods allow an efficient collection, pro- cessing and analysis of the dynamic environment, which can be used for detection and tracking of traffic participants. Current datasets in this domain mostly present a single view, making highly accurate pose estimation impossible due to occlusions. The integration of different, simultaneously acquired data allows to exploit and develop collaboration principles to increase the quality, reliability and integrity of the derived information. This work addresses this problem by providing a multi-view dataset, including 2D image information (videos) obtained by up to three cameras and 3D point clouds from up to five LiDAR sensors together with labels of the traffic participants in the scene. The measurements were conducted during different weather conditions on several days at a large junction in Hanover, Germany, resulting in a total duration of 145 minutes.

Keywords: Automated Vehicles, Self-Driving Vehicles, Legal Impacts
Abstract: Intersections are difficult to navigate for both human drivers and autonomous vehicles because several diverse traffic rules must be considered. In addition, current traffic rules are ambiguous and cannot be applied directly by autonomous vehicles. Therefore, national traffic rules must be concretized and formalized so that they are machine-interpretable. We present formalized intersection traffic rules in temporal logic and use the German traffic regulations as a concrete example. Our formalization considers different types of intersections, i.e., signalized, traffic-sign-regulated, and unregulated intersections. We also define predicates and functions that can be easily reused for other national traffic laws. We evaluate our formalized traffic rules on recorded real-world scenarios and manually-created test scenarios. Our evaluation validates the formalization from different legal sources.

Keywords: Driver Recognition, Vision Sensing and Perception, Automated Vehicles
Abstract: When developing autonomous or automated driving functions, having a sound understanding of the environment is critical. The earlier and the more detailed the environment is recognized, the more precise the vehicle can plan future actions. One major component in the field of autonomous driving, therefore, is visual perception. Cameras deliver consecutive images of the world, and computer vision algorithms attempt to extract the relevant information, for example, detecting other road users such as vehicles. However, current automated driving functions share the limitation of relying on the object to be directly visible to be detected. On the other side, humans intuitively use other visual effects to draw assumptions about objects that are not yet directly visible. Such effects can be shadows during the day or light reflections during the night. Recent work has already shown the potential of using this information when driving at night on highways. Highways at night thereby are easy ground: The traffic volume is considerably low, and the surrounding environment is mainly dark (e. g., there are no street lamps causing other light sources or reflections). This paper takes the research on this topic a step further and brings light to the question to which extent this effect of detecting oncoming vehicles at night can also be used in cities. We present a comprehensive analysis of human behavior when detecting other road users at night in urban areas while driving. The data was gathered throughout a test group study in two medium-sized German cities under various weather and traffic conditions. We prove the importance of light reflections when driving through urban areas at night and provide a solid human benchmark to compare future perception algorithms' performance.

Keywords: Situation Analysis and Planning, Self-Driving Vehicles, Advanced Driver Assistance Systems
Abstract: Motion planning and decision-making while considering uncertainty is critical for an autonomous vehicle to safely and efficiently drive on a highway. This paper presents a new combined two-step approach for this problem, where a partially observable Markov decision process (POMDP) is tightly coupled with a scenario model predictive control (SCMPC) step. To generate the scenarios in the SCMPC step, the solution to the POMDP is used together with a novel scenario-reduction procedure, which selects a small representative subset of all scenarios considered in the POMDP. The resulting planner is evaluated in a simulation study where the impact of the two-step approach and the scenario-reduction method is shown.

Keywords: Automated Vehicles, Unsupervised Learning, Collision Avoidance
Abstract: Model predictive control (MPC) is a commonly applied vehicle control technique, but its performance depends highly on how accurate the model captures the vehicle dynamics. It is disreputable hard to get a precise vehicle model in complex situations. The unmodeled dynamic will cause the uncertainty of the prediction which brings the risk while overtaking. To address this issue, Gaussian process (GP) regression is employed to acquire the unexplored discrepancy between the nominal vehicle model and the real vehicle dynamics which can result in a more accurate model. To achieve safe overtaking at highway curves, the constraint conditions are carefully designed. The implementation of GP-based MPC including approximate uncertainty propagation and safety constraints ensures that the ego vehicle overtakes the obstacles without collision. Simulation results show that GP-based MPC has a strong adaptability to different scenarios and outperforms MPC in overtaking control.

Keywords: Human-Machine Interface, Automated Vehicles
Abstract: With the advent of automation in road traffic, vehicle interaction design is undergoing a major shift and facing new challenges. In this paper, we adopted a user-centered design approach to identify suitable interface types for the interaction between automated light vehicles for urban lastmile deliveries and their human operator. In an exploratory co-creation workshop, we first identified possible interface types with laypeople and logistics employees (N = 8). Based on the workshop insights, we surveyed user acceptance of various interface options (e.g., app, voice, and gesture control), the situation- and user-dependency of interface acceptance, and the users’ motivations for preferring specific interface types (online survey study: N = 188). The analysis revealed that ease of use, road safety, and compatibility were the most mentioned reasons for preferring a particular interface type. Additionally, results showed that app and voice control were, on average, perceived as most desirable. However, none of the queried interface types was assessed as a perfect fit for each interaction situation and user.

Keywords: Cooperative Systems (V2X), Traffic Flow and Management, Smart Infrastructure
Abstract: Intersections are among the scenarios that are most crucial for efficiency and traffic flow on roads. Several approaches for traffic control at intersections exist, each with its own advantages and drawbacks. These days, wireless connections between road users, automated vehicles, and intelligent infrastructure enable new ways of coordinating traffic. However, the gradual deployment of those advanced technologies leads to a heterogeneous mixture of partially automated, connected, and legacy vehicles. Planning and coordinating maneuvers for this mixed traffic is a challenge and subject to current research, as it can achieve significant efficiency improvements in those scenarios. In this paper, we propose a new maneuver planning system for cooperative connected vehicles in mixed traffic at unsignalized intersections, which often occur in urban areas. Our system consists of a probabilistic multi-modal prediction based on a driver model and an efficient optimization algorithm to find the best maneuvers. We present the functionality of our approach and evaluate the impact on traffic efficiency using simulations of two different intersection layouts at various rates of cooperative vehicle penetration.

Keywords: Vehicle Environment Perception, Sensor and Data Fusion, Automated Vehicles
Abstract: For autonomous driving on rural or dirt roads - neither urban nor off-road - a large terrain area needs to be estimated at high spatial resolution. However, available computing time is very limited. Since different areas of the ground surface require different minimum resolution, we propose a dynamic resolution terrain estimation. Based on support points, accumulated measurements are spatially smoothed to a continuous terrain model using maximum a posteriori estimation. Splitting the terrain into tiles, we dynamically adjust the support point resolution of single tiles, depending on their accuracy in areas of interest. Areas of interest are determined by fusing information about probable road areas from LiDAR and vision preprocessing steps. As demonstrated in real-world examples, our approach can model the terrain almost as accurately as if all tiles had the highest resolution, but with much less computational effort.

Keywords: Advanced Driver Assistance Systems, Vehicle Control, Active and Passive Vehicle Safety
Abstract: Integrated automated safety systems in vehicles significantly reduced the number of car crashes. They help the driver in critical maneuvers when tires lose their grip on the driving surface. For instance, the technology of the anti-lock braking system and its augmentations (electronic stability control and traction control system) has already saved thousands of lives. Nevertheless, we still see room for improvement. This work defines boundaries in the vehicle state-space, excluding unstable vehicle maneuvers. Such boundaries form a so-called driving envelope. The resulting set includes all states where the vehicle's wheels are not locked, overspun, or skidding. For the definition of the driving envelope, we use the Pacejka tire model and nonlinear single-track model. This paper shows how each tire dynamic property results in vehicle dynamics. Also, it discusses the application of nonlinear and linearized driving envelope boundaries on a single-track model. Then it shows that the linearized driving envelope constraints form a close to control invariant set over the vehicle state-space. Thus, the driving envelope is almost a feasible set, and it could be used in the model predictive control approaches with soft constraints. Protecting the driving envelope, one can preserve each wheel from locking, wheelspin, and skidding.

Keywords: Sensor and Data Fusion, Vision Sensing and Perception, Deep Learning
Abstract: When detecting objects, depth sensors are not always available, requiring 3D object detection from monocular images. However, for many object classes, datasets with 3D annotations are missing. Recent monocular 3D object detection methods lack the semantic diversity needed for autonomous systems, because of missing 3D ground truth data for static classes such as poles and traffic lights. To overcome this gap we combine a large scale dataset for 2D object detection, with an unlabeled dataset containing depth measurements. We lift 2D object detections of the depth dataset into the 3D domain, associating detections with corresponding depth values. This leverages 2D annotated datasets to enable semantically rich 3D object detection, without extra labelling effort. We train an object detection model with mixed batches and evaluate it comparing the predicted depth with the projected centerpoint depth of cars manually annotated in 3D space. The result is a monocular object detector that can predict 3D positions of up to 37 static and dynamic object classes from camera only.

Keywords: Vehicle Environment Perception, Active and Passive Vehicle Safety, Self-Driving Vehicles
Abstract: To enable safe autonomous vehicle (AV) operations, it is critical that an AV's obstacle detection module can reliably detect obstacles that pose a safety threat (i.e., are safety-critical). It is therefore desirable that the evaluation metric for the perception system captures the safety-criticality of objects. Unfortunately, existing perception evaluation metrics tend to make strong assumptions about the objects and ignore the dynamic interactions between agents, and thus do not accurately capture the safety risks in reality. To address these shortcomings, we introduce an interaction-dynamics-aware obstacle detection evaluation metric by accounting for closed-loop dynamic interactions between an ego vehicle and obstacles in the scene. By borrowing existing theory from optimal control theory, namely Hamilton-Jacobi reachability, we present a computationally tractable method for constructing a "safety zone": a region in state space that defines where safety-critical obstacles lie for the purpose of defining safety metrics. Our proposed safety zone is mathematically complete, and can be easily computed to reflect a variety of safety requirements. Using an off-the-shelf detection algorithm from the nuScenes detection challenge leaderboard, we demonstrate that our approach is computationally lightweight, and can better capture safety-critical perception errors than a baseline approach.

Keywords: Mapping and Localization, Self-Driving Vehicles, Vehicle Environment Perception
Abstract: Autonomous vehicle (AV) is one of the most challenging engineering tasks of our era. High-Definition (HD) maps are a fundamental tool in the development of AVs, being considered as pseudo sensors that provide a trusted baseline that other sensors cannot. Our approach is focused on the use of OpenDRIVE standard based HD maps in order to conduct the different mapping and planning tasks involved in Autonomous Driving (AD). In this paper we present a method for exploiting the HD map potential for two specific purposes: i) Global Path Planning and ii) Monitoring the relevant lanes and regulatory elements around the ego-vehicle to support the perception module. Mapping and planning modules are connected to the other modules of the AV stack by using ROS (Robot Operating System). Our AD architecture has been validated both in local and CARLA Autonomous Driving Leaderboard cloud, where we can appreciate a considerable improvement in the metrics by incorporating information from the HD map, not only used to conduct the Global Path Planning task but also providing prior information to the Perception module. Code is available in https://github.com/AlejandroDiazD/opendrive-mapping-plannin g .

Keywords: Automated Vehicles, Self-Driving Vehicles
Abstract: There has been tremendous progress in the development of individual modules of the standard perception-prediction-planning robot autonomy stack. However, the principled integration of these modules has received less attention, particularly in terms of cascading errors. In this work, we both characterize and address the problem of cascading errors, focusing on the coupling between tracking and prediction. First, we comprehensively evaluate the impact of tracking errors on prediction performance with modern tracking and prediction methods on real-world data. We find that prediction methods experience a significant (even order of magnitude) drop in performance when consuming tracked trajectories as inputs (typical in practice), compared to the idealized setting where ground truth past trajectories are used as inputs. To address this issue, we propose a multi-hypothesis tracking and prediction framework. Rather than relying on a single set of tracking results for prediction, we simultaneously reason about multiple sets of tracking results, thereby increasing the likelihood of including accurate tracking results as inputs to prediction. We show that our framework improves overall prediction performance over the standard single-hypothesis tracking-prediction pipeline by up to 34.2% on the nuScenes dataset, with even more significant improvements (up to ~70%) when restricting evaluation to challenging scenarios involving identity switches and fragments, all with an acceptable computation overhead.

Keywords: Eco-driving and Energy-efficient Vehicles, Electric and Hybrid Technologies, Reinforcement Learning
Abstract: The eco-driving strategy that targets driving speed optimization is recognized as a promising technique to improve vehicle energy efficiency. However, it is difficult to achieve real-time eco-driving control of hybrid electric vehicle (HEV) since the speed optimization and powertrain energy management should be resolved simultaneously. This paper proposes a hierarchical control architecture consisting of learning-based velocity planner and real-time energy management system. In the upper stage, Proximal Policy Optimization (PPO) agent is trained to generate acceleration which meets multiple control objectives. The lower stage adopts Equivalent Consumption Minimization Strategy (ECMS) for real-time power split control considering powertrain dynamics. Finally, the eco-driving simulations of six signalized intersections in Nanjing are conducted. Compared with two different rule-based strategies, the proposed control architecture can achieve at least 7.39% of fuel economy saving and avoid a significant drop in the battery state of charge at the expense of higher than 5% of travel time. Simulation results also prove that the proposed strategy has an energy-saving potential in unseen scenarios.

Keywords: Driver State and Intent Recognition, Driver Recognition, Vehicle Environment Perception
Abstract: Risk perception is one of the most important driving skills for drivers to detect potential traffic accidents and make correct risk avoidance behaviors. An accurate model and evaluation of risk perception can effectively identify driver’s perception deficiencies and serve as an important human factor for the design of advanced driver assistance systems. Most traditional perception ability assessing methods are based on macroscopic statistical results and lack effective mathematical models, thus making it difficult to evaluate risk perception quantitatively. To this end, this paper first obtains a semantic understanding of traffic scenes through a semantic segmentation method based on deep learning. Then, the semantic understanding information is fused with the driver’s risk perception results to form the time series data reflecting the risk perception ability. Finally, we use a gate recurrent unit (GRU) network-based learning framework to learn the time series data features and thus obtain the behavioral model that responds to the driver’s risk perception ability. By verifying the classification performance of multiple drivers, the risk perception model based on traffic scene understanding can accurately predict the actual cognitive behavior of the individual driver. Besides, the proposed GRU-based method outperforms the traditional machine learning algorithm in modeling the driver’s risk perception behavior.

Keywords: Self-Driving Vehicles, Convolutional Neural Networks
Abstract: We show how to design a motion prediction algorithm that works with 3D object detections and map locations. In particular, we obtain object id's -- even though the training data does not contain any object id's -- across multiple time-steps into the future by propagating a Gaussian Mixture of likely object (e.g., vehicle) locations through time.

We demonstrate our approach on the nuScenes dataset. First, we find that a motion prediction algorithm without tracking id's performs as well as motion prediction algorithm with tracking id's. Second, the 3D labels of an on-board perception system are inferior (e.g., loss of detections, positional uncertainty) to those generated by offline labelling (automatic labelling pipeline, manual labelling). Even so, we find that a moderate increase in the size of the training data offsets the deterioration in prediction performance (with no additional offline labelling).

Keywords: Self-Driving Vehicles, Situation Analysis and Planning, Collision Avoidance
Abstract: Motion planning is one of the key modules in autonomous driving systems to generate trajectories for self-driving vehicles to follow. Spatio-temporal (SLT) motion planners are a trend to tackle more complicated scenarios. Such planners are prone to the limitation of optimizing in ST/LT (spatio-temporal) planes, but not in SL (spacial) plane, due to the difficulty of expressing spacial optimization with quadratic formulation, resulting in unstable paths. Existing works tend to go around this problem as the conversion from SL plane to ST/LT planes is non-linear. In this paper, we prove some theorems that allows for approximating the upper bounds of spacial cost terms with spatio-temporal cost terms. As the main benefit, this proposed approach enables optimizing for better quality trajectories in the SL plane and addressing planning objectives that can only be truly explained in the SL plane, e.g. closeness to the reference path and minimizing lateral acceleration and directional oscillation. The effectiveness of this approach is demonstrated with simulations in multiple scenarios.

Keywords: Autonomous / Intelligent Robotic Vehicles, Human-Machine Interface, Advanced Driver Assistance Systems
Abstract: Humans, interacting with automated machines, expect certain behaviors: either because they have experienced this behavior (e.g., driving) or because they build such expectations from the machine (e.g., a user would expect from an AI based personal assistant to recognize all the sentences they might tell in any accent). In reality, these advanced AI systems might not behave perfectly or their optimal decisions might also differ from the subjective optimal decisions a human user might expect. This becomes a challenging problem when considering AI decision making algorithms, controlling the complex behaviors of autonomous vehicles, affected by their uncertain environments and their own sensing suites. This paper presents results from two large, on-line user studies, run in simulated autonomous driving scenarios. Our goal was to assess users’ trust in the automated behaviors, presented with different explanations and HMI solutions. We found that specific explanations, considering the risk of a driving scenario and what the vehicle is planning to do can reduce discomfort and increase understanding of an automated driving maneuver. We also present a data-driven solution to infer an explanation automatically and probabilistically that is the most suitable for a driving context and user’s group according to the data analysis and trust measures examined.

Keywords: Self-Driving Vehicles, Situation Analysis and Planning, Active and Passive Vehicle Safety
Abstract: The evolution towards autonomous driving involves operating safely in open-world environments. For this, autonomous vehicles and their Autonomous Driving System (ADS) are designed and tested for specific, so-called Operational Design Domains (ODDs). When moving from prototypes to real-world mobility solutions, autonomous vehicles, however, will face changing scenarios and operational conditions that they must handle safely. Within this work, we propose a fuzzy-based approach to consider changing operational conditions of autonomous driving based on smaller ODD fragments, called μODDs. By this, an ADS is enabled to smoothly adapt its driving behavior for meeting safety during shifting operational conditions. We evaluate our solution in simulated vehicle following scenarios passing through different μODDs, modeled by weather changes. The results show that our approach is capable of considering operational domain changes without endangering safety and allowing improved utility optimization.

Keywords: Societal Impacts, Automated Vehicles
Abstract: Social acceptance is seen as an important prerequisite for a successful adoption and diffusion of automated driving technologies and services. The paper presents a proposal about how social acceptance could be conceptualized and argues that expectations and promises regarding the role of autonomous driving in future mobility systems should play an important role here. It then provides first results of a representative survey among German citizens, focusing on their individual expectations on the longer-term effects of the widespread use of autonomous road vehicles as well as on their opinions on changes of framework conditions and regulations associated with their diffusion and deployment.