Keywords: Telematics, Advanced Driver Assistance Systems, Novel Interfaces and Displays
Abstract: Teleoperation is a crucial technology for unmanned autonomous vehicles. When an autonomous vehicle does not work as desired, an operator teleoperates the vehicle and drives it to a safe place. Therefore, driving accuracy is essential in teleoperation. In teleoperation, a latency, consisting of transmission time and signal processing time, exists and impairs driving accuracy. A latency visualization method, which shows the gap due to latency between the vehicle's actual position and the position that the driver can see on the on-board camera videos, is considered a promising solution. Several studies have shown the latency visualization method's effectiveness through simulation-based evaluations, but it has not been evaluated in a real environment. In this paper, we proposed a latency visualization method that can be implemented in a real environment, where the latency can change frequently, and evaluated the method through a user test. The results show that the proposed latency visualization method improves driving accuracy in a real environment, and the improvement is more significant with longer latency.

Keywords: Reinforcement Learning, V2X Communication, Active and Passive Vehicle Safety
Abstract: Radar detection and communication are two of several sub-tasks essential for the operation of next-generation autonomous vehicles (AVs). The former is required for sensing and perception, more frequently so under various unfavorable environmental conditions such as heavy precipitation; the latter is needed to transmit time-critical data. Forthcoming proliferation of faster 5G networks utilizing mmWave is likely to lead to interference with automotive radar sensors, which has led to a body of research on the development of Joint Radar Communication (JRC) systems and solutions. This paper considers the problem of time-sharing for JRC, with the additional simultaneous objective of minimizing the average age of information (AoI) transmitted by a JRC-equipped AV. We formulate the problem as a Markov Decision Process (MDP) where the JRC agent determines in a real-time manner when radar detection is necessary, and how to manage a multi-class data queue where each class represents different urgency levels of data packets. Simulations are run with a range of environmental parameters to mimic variations in real-world operation. The results show that the deep reinforcement learning method allows the agent to obtain good results with minimal a priori knowledge about the environment.

Keywords: Advanced Driver Assistance Systems, Collision Avoidance, Vehicle Control
Abstract: A number of simulation platforms allow the validation of Advanced Driver Assistance Systems (ADAS) in virtual road environments. However, the development of the virtual tests on top of such platform may face many technical issues.

Some are related to the management of the modular and configurable architecture of the simulators. Others come from the physical aspects of the simulation. Also, time and concurrency issues may affect the control of dynamic scenarios.

This paper shares our experience with the virtual testing of ADAS during a period of time of more than one year. The technical issues yielded simulation crashes, ill-controlled test executions, incorrect verdict assignments, and caused a waste of time in the running and analysis of useless tests.

We discuss the issues and provide recommendations for the practitioners.

Keywords: Cooperative ITS, Information Fusion, Collision Avoidance
Abstract: This article presents a model for traffic incident prediction. Specifically, we address the fundamental problem of data scarcity in road traffic accident prediction by training our model on emergency braking events instead of accidents. Based on relevant risk factors for traffic accidents and corresponding data categories, we evaluate different options for preprocessing sparse data and different Machine Learning models. Furthermore, we present a prototype implementing a traffic incident prediction model for Germany based on emergency braking data from Mercedes-Benz vehicles as well as weather, traffic and road data, respectively. After model evaluation and optimisation, we found that a Random Forest model trained on artificially balanced (under-sampled) data provided the highest classification accuracy of 85% on the original imbalanced data. Finally, we present our conclusions and discuss further work; from gathering more data over a longer period of time to build stronger classification systems, to addition of internal factors such as the driver's visual and cognitive attention.

Keywords: Sensor and Data Fusion, Intelligent Ground, Air and Space Vehicles, Advanced Driver Assistance Systems
Abstract: This study presents an Extended Kalman Filter for estimation of vehicle planar velocities and tire forces. For this, a 10 Degrees of Freedom vehicle model is used together with onboard sensors. To gain robustness, the estimator considers unknown road angles. The method is validated using data gathered with a test vehicle driving over graded and banked roads and driving a demanding test provoking significant wheel slip. An analysis is done to evaluate the gains and drawbacks of including road angle estimation while driving over both flat and non-flat roads. In addition, the analysis highlights the effects of sensor performance on the estimation results and capabilities for a wide range of realistic driving scenarios.

Keywords: Situation Analysis and Planning, Self-Driving Vehicles, Advanced Driver Assistance Systems
Abstract: In this paper, we propose and analyze a Generalized Intelligent Driver Model (GIDM) as an extension of the Intelligent Driver Model (IDM) for its applicability to model merging scenarios. For this purpose, we extend the original longitudinal car-following IDM with several terms: (1) for anticipatory acceleration capabilities, we include the most nearby backward agent, (2) we consider cars that merge from other paths by virtual projection continuous blending, and (3) we introduce a longitudinal shift that increases the anticipatory capabilities of the model. Then we analyze simulations of the proposed motion model and measuring safety, comfort and utility. The result is a systematic improvements of the GIDM compared to a baseline IDM in all considered measures.

Keywords: Active and Passive Vehicle Safety, Security
Abstract: Anti-skid control of the vehicles is an important technology to ensure safe driving, and the tire-road adhesion stability must be detected and controlled at all times. The anti-skid control of traditional vehicles is achieved by distributd mechanical braking system, which consumes a lot of energy. In contrast, the energy of electric vehicles flows in both directions, and energy can be stored and saved during the braking process, which places higher demands on tire-road stability control. It is important to study an anti-skid control method for electric vehicles that takes into account both tire-road stability and energy recovery in the braking process. At present, the tire-road stability control methods of electric vehicles based on motor energy bidirectional are basically implemented on distributed electric vehicles. For centralized drive electric vehicles, further research is needed. In this paper, a new force transfer factor suitable for centralized drive electric vehicles is proposed to judge the tire-road stability, and an anti-skid control strategy based on motor torque adjustment for electric vehicles is proposed. Simulations are carried out under different driving conditions to verify the effectiveness of the anti-skid stability control strategy proposed in this paper.

Keywords: Human-Machine Interface, Situation Analysis and Planning, Driver State and Intent Recognition
Abstract: State of the art motion cueing algorithms aim at reproducing a simulated vehicle's motion at maximum accuracy, while respecting the motion constraints of a cueing platform. The consideration of human sensory characteristics for motion perception allows to artificially increase this envelope. Model predictive control based approaches penalize motion deviation for each perception channel and consequently minimize every error individually. However, no effort is made to balance motion cues across different degrees of freedom. In the motorsport environment it is essential to replicate vehicle characteristics precisely and consistently. The latter is of particular interest, as an inconsistent replication of cues could easily cause a perceived change of vehicle characteristics for a professional race car driver. In consequence, the motion cues should generally retain specific characteristics of the vehicle reference. A minimization of each tracking error individually does not meet this requirement which is demonstrated in this work. Furthermore, this paper introduces two cost functions for a model predictive control based motion cueing algorithm which reduce the deviation of visual-vestibular incongruences. Across three degrees of freedom a reduction of up to 33% is achieved while scaling errors and translational workspace utilization are retained at a similar level.

Keywords: Autonomous / Intelligent Robotic Vehicles, Cooperative Systems (V2X), Human-Machine Interface
Abstract: Human-robot collaboration and cooperation are critical for Autonomous Mobile Robots (AMRs) in order to use them in indoor environments, such as offices, hospitals, libraries, schools, factories, and warehouses. Since a long transition period might be required to fully automate such facilities, we have to deploy AMRs while improving safety in the mixed environments of human and mobile robots. In addition, human behaviors in such environments might be difficult to predict. In this paper, we present a Digital Twin for Autonomous Mobile Robots system named DigiMobot to support, manage, monitor, and validate AMRs in indoor environments. First, DigiMobot can simulate human behaviors and robot movements to verify and validate AMRs to improve safety in a virtual world. Secondly, DigiMobot can monitor and manage AMRs in the physical world by collecting sensor data from each robot in real-time. Since DigiMobot enables us to test the robot systems in the virtual world, we can deploy and implement AMRs in each facility without any modifications. To show the feasibility of DigiMobot, we develop a software framework and two different types of autonomous mobile robots. Finally, we conduct real-world experiments in a warehouse located in Saitama, Japan, in which more than 400,000 items are stored for commercial purposes.

Keywords: Situation Analysis and Planning, Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: Balancing safety and efficiency when planning in dense traffic is challenging. Interactive behavior planners incorporate prediction uncertainty and interactivity inherent to these traffic situations. Yet, their use of single-objective optimality impedes interpretability of the resulting safety goal. Safety envelopes which restrict the allowed planning region yield interpretable safety under the presence of behavior uncertainty, yet, they sacrifice efficiency in dense traffic due to conservative driving. Studies show that humans balance safety and efficiency in dense traffic by accepting a probabilistic risk of violating the safety envelope. In this work, we adopt this safety objective for interactive planning. Specifically, we formalize this safety objective, present the Risk-Constrained Robust Stochastic Bayesian Game modeling interactive decisions satisfying a maximum risk of violating a safety envelope under uncertainty of other traffic participants’ behavior and solve it using our variant of Multi-Agent Monte Carlo Tree Search. We demonstrate in simulation that our approach outperforms baselines approaches, and by reaching the specified violation risk level over driven simulation time, provides an interpretable and tunable safety objective for interactive planning.

Keywords: Automated Vehicles, Self-Driving Vehicles
Abstract: In the field of automated driving, scenarios pose a systematic and intuitive way to structure the complex development problem. Their usage was proposed for several domains, such as safety validation, verification, testing, and systems engineering.

While those domains are key for a successful development from the technical and systems perspective, the maturity of the technology continuously evolves. This generates the need for a suitable product management in order to realize a successful market entry.

We propose, to extend scenario-based development by a product scope description incorporating the business value and development costs for each scenario. The definition of a product on scenario level leads to the questions, how scenarios interdepend from an architectural and technological perspective and how to derive an efficient development order. We therefore introduce scenario dependency graphs, which we assess to be a valuable communication instrument. These graphs can additionally support project planning, monitoring, and control leading to an increased development efficiency and supporting project management towards market entry.

Keywords: Automated Vehicles, Societal Impacts, Cooperative ITS
Abstract: In the future, many new challenges will arise in the transportation sector, one of which is the increasing urbanisation. This often leads to a reduced availability of public rail transport or even its discontinuation in rural areas. This shortcoming could be addressed by introducing an on-demand rail service with small-sized trains. The focus on on-demand rail transport in rural areas can have strong positive scaling effects and is currently very prominent politically.

In order to assess the potential of such a transport service, a status survey of the current equipment and technical possibilities must first be carried out. With this, the opportunities and open challenges of the different components can be debated. These include, for example, the introduction of a smart traffic management system or the communication between the infrastructure and the vehicles or between the vehicles themselves.

It can be seen that the technical possibilities that are already available show considerable potential, despite some unresolved issues. It is imperative that the potentials be used, as they entail more attractive connections in rural areas, technological progress through pilot projects and the addressing of major challenges of the future decades.

Keywords: Automated Vehicles, Collision Avoidance, Legal Impacts
Abstract: The need for safety in automated driving is undisputed. Since automated vehicles are expected to reduce the number of fatalities in road traffic significantly, hundreds of millions of test kilometers would be required for statistical safety validation. Physics-based safety verification approaches are promising in order to reduce this validation effort. Towards this goal, Mobileye introduced the concept of Responsibility-Sensitive Safety (RSS). In RSS, bounds for the reasonable worst-case behavior of traffic participants are assumed to be given, such as the reaction time or the maximum deceleration. These parameters have a crucial effect on the applicability of the approach: choosing conservative parameters likely hinders traffic flow, while the opposite could lead to collisions, as the assumptions are violated. Thus, in this work, we focus on finding reasonable parameters of RSS. Based on the physical limits, legal requirements and human driving behavior, we propose scopes and parameter sets that allow for a sound safety verification while not hindering traffic flow. Furthermore, we present an approach that explains seemingly frequent human drivers' RSS violations on highways and may lead to a useful extension of RSS.

Keywords: Automated Vehicles, Vehicle Environment Perception, Convolutional Neural Networks
Abstract: Slip is a crucial parameter in the kinematic and dynamic models of tracked vehicles, which also exercises considerable influence over the track-ground interactions. The methods based on machine learning for slip estimation only employ proprioceptive sensor signals and obtain models by learning from numerous data, but they still have some limitations. This paper explores the applicability of such methods to autonomous tracked vehicles, which considers wider speed range, bends and difference between two tracks slip. First, a tremendous amount of data are collected with a tracked vehicle. Then road types identification is realized based on convolutional neural networks because the road types and slip are closely related. Lastly the piecewise regression method is employed to estimate slip. The performance of proposed approach is evaluated on test set. The results indicate that it can accurately identify the road(success rate, >96%) and estimate the slip ratio(root mean square error, <0.7%). The contrast experiment and analysis also show that classification before regression improves the accuracy and reduces the computational burden. Further, sparse Gaussian process regression is applied to return the distribution of slip, which can reflect the uncertainty of estimation.

Keywords: Active and Passive Vehicle Safety, Automated Vehicles
Abstract: Addressing safety for future autonomous vehicles (AVs) while maintaining a high practicability, i.e. not using excessive safety margins, getting stuck or not able to drive at all, is still an open research question. In this regard, the Responsibility Sensitive Safety (RSS) approach, which is a formal parametric safety model, is a promising concept and is currently gaining lots of attraction. With RSS it is possible to combine safety and practicability, if the model parameters are chosen in a reasonable manner, i.e. good enough to achieve a desirable level of safety, but not covering the physical worst case. However, as we will show in this work, this can lead to a not optimal behavior of the AV in extreme situations, when other traffic participants violate RSS assumptions. To overcome this, we propose an extension to RSS, called RSS+, which can mitigate many of these situations in a pro-active manner, without sacrificing practicability.

Keywords: Self-Driving Vehicles, Deep Learning
Abstract: Trajectory prediction has received much attention recently, especially in autonomous driving. Many Proposed models generate multi-modal trajectories using a wide variety of frameworks for context representation and dynamic interaction modeling. But they can not estimate the intention of the target vehicle, and the predicted trajectories are not explicable. Towards this end, we propose a model to estimate the vehicle intention and predict the possible trajectories corresponding to the intention. We separate intentions to long-term intention indicating the future path of vehicle and short-term intention indicating the behavior of vehicle. We use rasterized map to represent context information and differentiate the long-term intentions into different channels. A multi encoder-decoder module generates forecasting trajectories based on the context feature of specific intention and learns various behaviors considering the interaction of surrounding obstacles. We demonstrate the performance of our model on the nuScenes prediction dataset, which outperforms the state-of-the-art methods.

Keywords: Automated Vehicles, Convolutional Neural Networks, Deep Learning
Abstract: Moving object segmentation is a crucial task for autonomous vehicles as it can be used to segment objects in a class agnostic manner based on their motion cues. It enables the detection of unseen objects during training (e.g., moose or a construction truck) based on their motion and independent of their appearance. Although pixel-wise motion segmentation has been studied in autonomous driving literature, it has been rarely addressed at the instance level, which would help separate connected segments of moving objects leading to better trajectory planning. As the main issue is the lack of large public datasets, we create a new InstanceMotSeg dataset comprising of 12.9K samples improving upon our KITTIMoSeg dataset. In addition to providing instance level annotations, we have added 4 additional classes which is crucial for studying class agnostic motion segmentation. We adapt YOLACT and implement a motion-based class agnostic instance segmentation model which would act as a baseline for the dataset. We also extend it to an efficient multi-task model which additionally provides semantic instance segmentation sharing the encoder. The model then learns separate prototype coefficients within the class agnostic and semantic heads providing two independent paths of object detection for redundant safety. To obtain real-time performance, we study different efficient encoders and obtain 39 fps on a Titan Xp GPU using MobileNetV2 with an improvement of 10% mAP relative to the baseline. Our model improves the previous state of the art motion segmentation method by 3.3%. The dataset and qualitative results video are shared in our website at https://sites.google.com/view/instancemotseg.

Keywords: Automated Vehicles, Collision Avoidance, Vehicle Environment Perception
Abstract: A typical autonomous driving system usually relies on the detected objects from an environment perception module. Current research still cannot guarantee a perfect perception, and failure detections may cause collisions, leading to untrustworthy autonomous vehicles. This work proposes a trajectory planner to tolerate the detection failure of the LiDAR sensors. This method will plan the path relying on the detected objects as well as the raw sensor data. The overlapping and contradiction of both perception routes will be carefully addressed for safe and efficient driving. The object detector in this work uses a deep learning-based method, i.e., CNN-Segmentation neural network. The designed trajectory planner has multi-layers to handle the multi-resolution environment formed by different perception routes. The final system will dynamically adjust its attention to the detected objects or the point cloud to avoid collision due to detection failures. This method is implemented on a real autonomous vehicle to drive in an open urban area. The results show that when the autonomous vehicle fails to detect a surrounding object, e.g., vehicles or some undefined objects, the autonomous vehicles still can plan an efficient and safe trajectory. In the meantime, when the perception system works well, the AV will not be affected by the point clouds. This technology can make the autonomous vehicle trustworthy even with the black-box neural networks. The codes are open-source with our autonomous driving platform to help other researchers for AV development.

Keywords: Vehicle Control, Vehicle Environment Perception, Security
Abstract: In recent years, we have witnessed increasingly high performance in the field of autonomous end-to-end driving. In particular, more and more research is being done on driving in urban environments, where the car has to follow high level commands to navigate. However, few evaluations are made on the ability of these agents to react in an unexpected situation. Specifically, no evaluations are conducted on the robustness of driving agents in the event of a bad high-level command. We propose here an evaluation method, namely a benchmark that allows to assess the robustness of an agent, and to appreciate its understanding of the environment through its ability to keep a safe behavior, regardless of the instruction.

Keywords: Intelligent Vehicle Software Infrastructure, Self-Driving Vehicles, Advanced Driver Assistance Systems
Abstract: This paper presents a Model in the Loop (MiL) framework to validate embedded autonomous driving (AD) and advanced driver assistant systems (ADAS) algorithms development. Recently, it has been recognized in the autonomous driving industry that simulation based testing is an efficient method to validate ADAS/AD functionalities complementing with physical testing. Coupled with AD algorithms such as perception, planning and control, an MiL toolchain can be executed and analyzed with various traffic scenarios and different algorithm parameters. This saves time and cost before actual vehicle test. Our contribution is twofold. First, we demonstrate our developed MiL toolchain combining high fidelity simulation models of vehicle, traffic and physics-based sensors with ADAS/AD functionalities closed loop algorithms. The solutions are implemented with both realistic traffic scenarios and standard scenario for certification. Second, our framework focuses on embedded development for the complete stack i.e. both algorithms and communication between different AD components. Thus, high performance, low latency, service, and type safety are the key parameters under consideration. The advantage is an efficient development process when transforming from MiL to hardware-vehicle in the loop testing.

Keywords: Automated Vehicles, Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: Automated vehicles need to be aware of the capa- bilities they currently possess. Skill graphs are directed acylic graphs in which a vehicle’s capabilities and the dependencies between these capabilities are modeled. The skills a vehicle requires depend on the behaviors the vehicle has to perform and the operational design domain (ODD) of the vehicle. Skill graphs were originally proposed for online monitoring of the current capabilities of an automated vehicle. They have also been shown to be useful during other parts of the development process, e.g. system design, system verification. Skill graph construction is an iterative, expert-based, manual process with little to no guidelines. This process is, thus, prone to errors and inconsistencies especially regarding the propagation of changes in the vehicle’s intended ODD into the skill graphs. In order to circumnavigate this problem, we propose to formalize expert knowledge regarding skill graph construction into a knowledge base and automate the construction process. Thus, all changes in the vehicle’s ODD are reflected in the skill graphs automatically leading to a reduction in inconsistencies and errors in the constructed skill graphs.

Keywords: Automated Vehicles, Intelligent Vehicle Software Infrastructure, Situation Analysis and Planning
Abstract: Hazard analysis is a quite significant step to ensure vehicle safety in the early stage of vehicle development according to current standards. However, the complexity of the Advanced Driving Assistance System (ADAS) and Automated Driving Systems (ADS), which consist of various software and hardware components, makes it difficult to identify system hazards. Nowadays, System-Theoretic Process Analysis (STPA), a hazard analysis method for complex systems, is applied to ADAS, and simple ADS gradually and proved applicable. This paper introduced Finite State Machine (FSM) to complement the STPA for its weakness in analyzing high-level autonomous vehicles with multiple automated modes and functions. Firstly, previous applications of STPA to ADAS and ADS and their limitations are analyzed. Secondly, the hazardous event is defined. An extended method combining STPA and FSM is proposed to model the vehicle states and environmental conditions and analyze unexpected behaviors. Finally, a case study on an autonomous vehicle is given to compare the traditional STPA and the extended method. Comparing with the traditional STPA, the proposed method can identify more hazardous events and give more detailed information about hazardous events to generate testing scenarios.

Keywords: Autonomous / Intelligent Robotic Vehicles, Self-Driving Vehicles, Active and Passive Vehicle Safety
Abstract: Extracting interesting scenarios from real-world data as well as generating failure cases is important for the development and testing of autonomous systems. We propose efficient mechanisms to both characterize and generate testing scenarios using a state-of-the-art driving simulator. For any scenario, our method generates a set of possible driving paths and identifies all the possible safe driving trajectories that can be taken starting at different times,

to compute metrics that quantify the complexity of the scenario. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project, as well as adversarial scenarios generated in simulation. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident. We demonstrate a strong correlation between the proposed metrics and human intuition.

Keywords: Situation Analysis and Planning, Autonomous / Intelligent Robotic Vehicles, Recurrent Networks
Abstract: Predicting the trajectories of surrounding agents is an essential ability for autonomous vehicles navigating through complex traffic scenes. The future trajectories of agents can be inferred using two important cues: the locations and past motion of agents, and the static scene structure. Due to the high variability in scene structure and agent configurations, prior work has employed the attention mechanism, applied separately to the scene and agent configuration to learn the most salient parts of both cues. However, the two cues are tightly linked. The agent configuration can inform what part of the scene is most relevant to prediction. The static scene in turn can help determine the relative influence of agents on each other’s motion. Moreover, the distribution of future trajectories is multimodal, with modes corresponding to the agent's intent. The agent's intent also informs what part of the scene and agent configuration is relevant to prediction. We thus propose a novel approach applying multi-head attention by considering a joint representation of the static scene and surrounding agents. We use each attention head to generate a distinct future trajectory to address multimodality of future trajectories. Our model achieves state of the art results on the nuScenes prediction benchmark and generates diverse future trajectories compliant with scene structure and agent configuration.

Keywords: Reinforcement Learning, Situation Analysis and Planning, Advanced Driver Assistance Systems
Abstract: Safe and efficient behavior are the key guiding principles for autonomous vehicles. Manually designed rule-based systems need to act very conservatively to ensure a safe operation. This limits their applicability to real-world systems. On the other hand, more advanced behaviors, i.e., policies, learned through means of reinforcement learning (RL) suffer from non-interpretability as they are usually expressed by deep neural networks that are hard to explain. Even worse, there are no formal safety guarantees for their operation.

In this paper we introduce a novel pipeline that builds on recent advances in imitation learning and that can generate safe and efficient behavior policies. We combine a reinforcement learning step that solves for safe behavior through the introduction of safety distances with a subsequent innovative safe extraction of decision tree policies. The resulting decision tree is not only easy to interpret, it is also safer than the neural network policy trained for safety. Additionally, we formally prove the safety of trained RL agents for linearized system dynamics, showing that the learned and extracted policy successfully avoids all catastrophic events.

Keywords: Intelligent Vehicle Software Infrastructure, Self-Driving Vehicles, Vehicle Control
Abstract: Collision avoidance is one of the crucial functions of autonomous driving systems (ADSs) to guarantee the safety of autonomous vehicles (AVs). It requires extensive testing before an AV is deployed to public roads. Most of the current ADS testing methods generate test cases either from real traffic data or manually designed for some specific scenarios. There is little work on systematic methods to generate test cases from a complete map where an AV operates. Systematic testing on such a map is challenging due to the enormous scenarios. In this paper, we propose a collision-avoidance testing method for ADSs running on a map, which aims to reduce the scenario space while maintaining scenario diversity. The method consists of test case classification and test case generation. First, we build the topology structure of a map, based on which we classify possible scenarios into different classes. Second, we divide test cases into different classes using the topology-based scenario classification and fuzzy number-based motion evaluation. Third, we implement a bisection method to generate test cases that can efficiently expose ADSs' failures. We evaluate our method on one of the state-of-the-art ADSs, Baidu Apollo. The experiment results show that our method discovers Apollo's issues effectively while reducing the number of generated test cases by 77.36%, compared with the random method.

Keywords: Automated Vehicles, Situation Analysis and Planning, Legal Impacts
Abstract: Autonomous vessels have to adhere to marine traffic rules to ensure traffic safety and reduce the liability of manufacturers. However, autonomous systems can only evaluate rule compliance if rules are formulated in a precise and mathematical way. This paper formalizes marine traffic rules from the Convention on the International Regulations for Preventing Collisions at Sea (COLREGS) using temporal logic. In particular, the collision prevention rules between two power-driven vessels are delineated. The formulation is based on modular predicates and adjustable parameters. We evaluate the formalized rules in three US coastal areas for over 1,200 vessels using real marine traffic data.

Keywords: Reinforcement Learning, Autonomous / Intelligent Robotic Vehicles, Self-Driving Vehicles
Abstract: Safety is essential for reinforcement learning (RL) applied in real-world tasks like autonomous driving. Imposing chance constraints (or probabilistic constraints) is a suitable way to enhance RL safety under model uncertainty. Existing chance constrained RL methods like the penalty methods and the Lagrangian methods either exhibit periodic oscillations or learn an over-conservative or unsafe policy. In this paper, we address these shortcomings by elegantly combining these two methods and propose a separated proportional-integral Lagrangian (SPIL) algorithm. We first rewrite penalty methods as optimizing safe probability according to the proportional value of constraint violation, and Lagrangian methods as optimizing according to the integral value of the violation. Then we propose to add up both the integral and proportion values to optimize the policy, with an integral separation technique to limit the integral value within a reasonable range. Besides, the gradient of policy is computed in a model-based paradigm to accelerate training. The proposed method is proved to reduce oscillations and conservatism while ensuring safety by a car-following experiment.

Keywords: Autonomous / Intelligent Robotic Vehicles, Automated Vehicles, Situation Analysis and Planning
Abstract: Making informed driving decisions requires reliable prediction of other vehicles’ trajectories. In this paper, we present a novel learned multi-modal trajectory prediction architecture for automated driving. It achieves kinematically feasible predictions by casting the learning problem into the space of accelerations and steering angles – by performing action-space prediction, we can leverage valuable model knowledge. Additionally, the dimensionality of the action manifold is lower than that of the state manifold, whose intrinsically correlated states are more difficult to capture in a learned manner. For the purpose of action-space prediction, we present the simple Feed-Forward Action-Space Prediction (FFW-ASP) architecture. Then, we build on this notion and introduce the novel Self-Supervised Action-Space Prediction (SSP-ASP) architecture that outputs future environment context features in addition to trajectories. A key element in the self-supervised architecture is that, based on an observed action history and past context features, future context features are predicted prior to future trajectories. The proposed methods are evaluated on real-world datasets containing urban intersections and roundabouts, and show accurate predictions, outperforming state-of-the-art for kinematically feasible predictions in several prediction metrics.

Keywords: Automated Vehicles, Situation Analysis and Planning, Sensor and Data Fusion
Abstract: Automated driving functions have received a lot of attention from the scientific community and the general public in the recent years. However, safety assurance, verification and validation of automated driving systems remain as a huge challenge, among others, towards making automated vehicles available to a broader public. For tackling these challenges, scenario-based validation, as one building block, can be used to show the effect and the impact of automated vehicles in (safety-) relevant scenarios. As of now, this is mainly done without considering external factors that can have a major impact on the performance of humans and systems such as road characteristics and adverse weather conditions. In this paper we focus on the aspect of the road characteristics. We present a method to extract road characteristics from online map data and compare that data to the coverage of road characteristics in various datasets. In doing so we gain an overview which of these characteristics still need to be covered or recorded by a given dataset.

Keywords: Self-Driving Vehicles, Advanced Driver Assistance Systems
Abstract: Finding comprehensive and relevant scenarios is a major challenge for autonomous vehicles validation and SOTIF. A functional scenario, e.g. a cut-in, encloses many concrete variations. Formal methods help covering an intermediate level of scenario families, called logical, and capitalizing them in a scenario database. Families are generated from discrete and modular symbolic models through a new subsumption criterion which allows the identification of scenario suffixes which are redundant and eliminate them during the generation. The generation, including the implementation of the subsumption criterion, benefits from: i) the compact representation of models thanks to discretization and symbolic arithmetic, ii) dedicated symbolic execution techniques. Analysis is performed to verify how the generated scenarios cover real situations by confronting them to time series from the modeled system and identify potential gaps in the model. We formally define our approach, implement it in the symbolic execution tool DIVERSITY. Assessment is carried out on a real autopilot black box module from the project 3SA. Keywords: Self-Driving Vehicles; Advanced Driver Assistance Systems

Keywords: Automated Vehicles, Reinforcement Learning
Abstract: The field of autonomous driving has seen increasing proposed use of machine learning methodologies. However, there are still challenges in applying such methods since autonomous driving involves complex and dynamic interactions with the environment. Supervised learning algorithms such as imitation learning can work in environments represented in the training data set, however, it is impractical or cost-prohibitive to collect data for all possible environments. Reinforcement learning methods can train the agent through trial and error, but it may still fail in new environments. To overcome these shortcomings, we thus propose a meta reinforcement learning (MRL) method to improve the agent's generalization capabilities to new environments. The method is applied to automated lane-changing maneuvers at different traffic congestion levels. Specifically, we train the model at light to moderate traffic conditions under a reinforcement learning framework, and then test it at heavy traffic conditions that are never encountered during training. For performance evaluation, we use both collision rate and success rate of the lane-change maneuvers to quantify the safety and effectiveness of the proposed model. A pretrained model is established as a benchmark, which uses the same network structure and training tasks as our proposed model for fair comparison. Simulation results show that the proposed method achieves an overall success rate up to 20% higher than the benchmark model when it is generalized to the new environment. The collision rate is also reduced by up to 18% compared with the benchmark model. Furthermore, the proposed model shows more stable and efficient adaptation capabilities and can achieve 100% successful rate and 0% collision rate with only a few gradient update.

Keywords: Human-Machine Interface, Automated Vehicles, Societal Impacts
Abstract: The safe deployment of autonomous vehicles (AVs) in real world scenarios requires that AVs are accountable. One way of ensuring accountability is through the provision of explanations for what the vehicles have ‘seen’, done and might do in a given scenario. Intelligible explanations can help developers and regulators to assess AVs' behaviour, and in turn, uphold accountability. In this paper, we propose an interpretable (tree-based) and user-centric approach for explaining autonomous driving behaviours. In a user study, we examined different explanation types instigated by investigatory queries. We conducted an experiment to identify scenarios that require explanations and the corresponding appropriate explanation types for such scenarios. Our findings show that an explanation type matters mostly in emergency and collision driving conditions. Also, providing intelligible explanations (especially contrastive types) with causal attributions can improve accountability in autonomous driving. The proposed interpretable approach can help realise such intelligible explanations with causal attributions.

Keywords: Electric and Hybrid Technologies, Autonomous / Intelligent Robotic Vehicles
Abstract: We present a solution for intelligent planning of engine activations for series hybrid electric vehicles (HEVs). Beyond minimizing energy expenditure, other real-world objectives must be incorporated, such as minimizing the perceived engine noise and the frequency of mode transitions between activation and deactivation. We model this problem as a multi-objective stochastic shortest path (MOSSP) problem that takes a vehicle model and navigation map as input and outputs a engine activation policy. The vehicle model and navigation map are learned from GPS traces with metadata, and includes the topological road structure, traversal speeds/times, battery consumption/regeneration, and ambient noise. We analyze our results in simulation on different navigation maps generated from actual GPS traces learned from a real series HEV. Experiments in simulation demonstrate that our approach compared with the baseline system can reduce total energy expenditure (EE), namely on hills, by up to 3%; total additional noise (AN) generated by up to 15%; and total mode transition (MT) frequency by up to 12%. The approach is demonstrated on a real series hybrid vehicle, driving on real public roads.

Keywords: Human-Machine Interface, Automated Vehicles, Vehicle Control
Abstract: Automated driving can improve traffic safety as well as the comfort for drivers. The development of automated driving services brings up the need for sufficient sensor data from autonomous vehicles. If the sensor data fails to reach a specific quality limit, the driver needs to act as a fall-back operator to complete the driving task. A more efficient way of dealing with system limitations is the cooperative task-sharing approach, where the driver and the vehicle act cooperatively as long as the system cannot drive in fully automated mode. In this evaluation, we implemented a cooperative speed regulation, where the driver was asked to manually adjust the speed due to the system's recognition failure, while the lateral control was still managed by the vehicle. Three interfaces, a central touch screen, the pedals, and steering wheel buttons, were compared against each other as input modalities in a driving simulator study (N = 36. User experience, suitability for the task, usability, workload, and the conclusive overall rank were evaluated in a within-subjects design. Results indicated that the highly learned pedal interface was rated significantly better than the other two interfaces, although the touch screen interface was rated significantly higher in its hedonic quality. In conclusion, the best known and naturalistic pedal interface should be preferred to design the interaction in driving-related scenarios, while the touch-screen can act as the infotainment control interface.

Keywords: Automated Vehicles
Abstract: Autonomous vehicle (AV) technology is positioned to have a significant impact on various industries. Hence, artificial intelligence powered AVs and modern vehicles with advanced driver-assistance systems have been operated in street networks for real-life testing. As these tests become more frequent, accidents have been inevitable and there have been reported crashes. The data from these accidents are invaluable for generating edge case test scenarios and understanding accident-time behavior. In this paper, we use the existing AV accident data and identify the atomic blocks within each accident, which are modular and measurable scenario units. Our approach formulates each accident scenario using these atomic blocks and defines them in the Measurable Scenario Description Language (M-SDL). This approach produces modular scenario units with coverage analysis, provides a method to assist in the measurable analysis of accident-time AV behavior, identifies edge scenarios using AV assessment metrics.

Keywords: Self-Driving Vehicles, Intelligent Vehicle Software Infrastructure, Collision Avoidance
Abstract: This paper presents an approach to specifying a modularised safety system which comprehensively addresses the safety requirements for highly autonomous (SAE Level 3+) road vehicles featuring advanced sensing and automated navigation. As these requirements are often overlooked in similar autonomous driving system proposals, we present a method of hazard and risk analysis which investigates hardware failures, environmental perception limitations, human interaction and functional requirements for artificial intelligence. We then define a system design which implements the required safeguards and examines the application on two electric autonomous vehicle testbeds: a race car and a shuttle bus. The close-coupling of a safety-oriented architecture and multi-regime Hazard and Risk Assessment process was tested to measure the system’s ability to detect and react to pedestrian stimuli, resulting in accurate detections and reactions, thereby confirming its ability to design safety systems for autonomous research vehicles in a scalable and easily assured fashion.

Keywords: Advanced Driver Assistance Systems, Security, Situation Analysis and Planning
Abstract: In the development of advanced driver-assistance systems (ADAS) and autonomous vehicles, machine learning techniques that are based on deep neural networks (DNNs) have been widely used for vehicle perception. These techniques offer significant improvement on average perception accuracy over traditional methods, however have been shown to be susceptible to adversarial attacks, where small perturbations in the input may cause significant errors in the perception results and lead to system failure. Most prior works addressing such adversarial attacks focus only on the sensing and perception modules. In this work, we propose an end-to-end approach that addresses the impact of adversarial attacks throughout perception, planning, and control modules. In particular, we choose a target ADAS application, the automated lane centering system in OpenPilot, quantify the perception uncertainty under adversarial attacks, and design a robust planning and control module accordingly based on the uncertainty analysis. We evaluate our proposed approach using both public dataset and production-grade autonomous driving simulator. The experiment results demonstrate that our approach can effectively mitigate the impact of adversarial attack and can achieve 55% to 90% improvement over the original OpenPilot.

Keywords: Automated Vehicles, Vehicle Control, Collision Avoidance
Abstract: This paper introduces a multi-constraint predictive control algorithm along with a safety layer to guarantee path tracking and object avoidance in emergency situations. A controller switching mechanism is designed which switches the controller between a main and an emergency controller. As the main controller, a nonlinear multi-constraint model predictive controller (MPC) is designed. The MPC algorithm is compared with Stanley and PID methods in terms of their efficiency to validate the MPC as the main controller. However, in unexpected situations, the high computational time of the planner and MPC modules threatens the safety of the vehicle. In order to respond as quickly as possible, emergency braking and maneuver systems are added which could be triggered separately per different situations. Two different emergency scenarios have been implemented in CARLA simulator and Python environment to evaluate the proposed method.

Keywords: Automated Vehicles, Deep Learning, Reinforcement Learning
Abstract: Achieving the highest levels of automated driving will require effective solutions to the key challenging maneuver of highway on-ramp merging. This paper extends our previous work on a multi-agent reinforcement-learning (MARL) approach to study the problem of highway on-ramp merging, with particular emphasis on the study of the behavior of the vehicle that is on the on-ramp with approaching traffic. Our previous model was based on a discretized space of states and actions. Here, we present results on a more sophisticated model based on a continuous space of states and actions. We exploit recent advances on deep reinforcement learning (deep RL) to train controllers for this task in an idealized environment using an implementation of our MARL approach. We specifically employ artificial neural network architectures for policy and function approximation within our multi-agent Q-learning approach. We show the effectiveness of our trained controllers by demonstrating their collision-avoidance performance on interaction scenarios with different in-traffic behavior. We compare their performance to those obtained using a similar deep RL single-agent approach. We argue why the resulting MARL-based controllers are essentially optimal within the context, conditions, and parameters of the evaluation environment that we employ and our previously established fundamental performance limitations governing the highway on-ramp merging maneuver.

Keywords: Automated Vehicles, Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: Identification of test scenarios for Automated Driving Systems (ADSs) remains a key challenge for the Verification & Validation of ADSs. Various approaches including data-based approaches and knowledge-based approaches have been proposed for scenario generation. Identifying the conditions that lead to high severity traffic accidents can help us not only identify test scenarios for ADSs but also implement measures to save lives and infrastructure resources. Taking a data-based approach, in this paper, we introduce a novel accident data analysis method for generating test scenarios where we analyze UK’s Stats19 accident data to identify trends in high severity accidents for test scenario generation. This paper first focuses on the severity of the accidents with the goal of relating it to static and time-dependent internal and external factors in a comprehensive way taking into account Operational Design Domain (ODD) properties, e.g. road, environmental conditions, and vehicle properties and driver characteristics. For this purpose, the paper utilizes a data grouping strategy (coarse-graining) and builds a logistic regression approach, derived from conventional regression models, in which emerging features become more pronounced, while uninteresting features and noise weaken. The approach makes the relationship between the factors and outcome variable more visible and hence well suited for the severity analysis. The method shows superior performance as compared to ordinary logistic models measured by goodness of fit and accounting for model variance (R2=0.05 for the ordinary model R2=0.85 for the current model). The model is then used to solve the inverse problem of constructing high-risk pre-crash conditions as test scenarios for simulation-based testing of ADSs.

Keywords: V2X Communication, Cooperative Systems (V2X), Vehicle Control
Abstract: Wireless communication plays a significant role in the control of connected and automated vehicles (CAVs). In particular, poor communication would cause worse vehicle performances, and may even cause safety issues. This paper aims to establish a communication model for vehicular environment and deeply analyze the impact of communication characteristics on CAVs control. Firstly, the three-parameter Burr distribution delay model and the Nakagami distribution packet delivery rate (PDR) model are proposed to describe vehicular wireless networks' characteristics. Then, the platooning control is selected for a case study, and a vehicle platoon control system incorporating the proposed communication model is established. Furthermore, a simulation platform is built based on SUMO and Python, and the impact of communication characteristics on the platoon’s performance is studied. The simulation results show that the characteristics presented by the communication model are consistent with those in field tests, and the quantized relationships between communication model parameters and vehicle control performance are also provided.

Keywords: Cooperative Systems (V2X), Eco-driving and Energy-efficient Vehicles, Self-Driving Vehicles
Abstract: Connected and Automated Vehicles (CAVs) have real-time information from the surrounding environment by using local on-board sensors, V2X (Vehicle-to-Everything) communications, pre-loaded vehicle-specific lookup tables, and map database. CAVs are capable of improving energy efficiency by incorporating these information. In particular, Eco-Cruise and Eco-Lane Selection on highways and/or motorways have immense potential to save energy, because there are generally fewer traffic controllers and the vehicles keep moving in general. In this paper, we present a cooperative and energy-efficient lane-selection strategy named MultiCruise, where each CAV selects one among multiple candidate lanes that allows the most energy-efficient travel. MultiCruise incorporates an Eco-Cruise component to select the most energy-efficient lane. The Eco-Cruise component calculates the driving parameters and prospective energy consumption of the ego vehicle for each candidate lane, and the Eco-Lane Selection component uses these values. As a result, MultiCruise can account for multiple data sources, such as the road curvature and the surrounding vehicles' velocities and accelerations. The eco-autonomous driving strategy, MultiCruise, is tested, designed and verified by using a co-simulation test platform that includes autonomous driving software and realistic road networks to study the performance under realistic driving conditions. Our experimental evaluations show that our eco-autonomous MultiCruise saves up to 8.5 % fuel consumption.

Keywords: Eco-driving and Energy-efficient Vehicles, Automated Vehicles, Cooperative Systems (V2X)
Abstract: With the increasing energy consumption in urban rail transit systems, energy-efficient train operation has been paid significant attention. Considering the application of regenerative braking technology, many studies focus on off-line energy-saving train trajectory optimization, in which an accelerating regime is inserted into the train trajectory when there is regenerative braking energy (RBE) that can be utilized. However, in practical operation, train states are dynamic and scheduled trajectory might be useless. This paper proposes a real-time cooperative train control method based on Train-to-Train (T2T) communication technology. According to the train states transmitted through T2T communication, whether there is a train in braking regime is judged. Besides, an accelerating regime is inserted by changing train running modes when there is another train in the braking regime. A cooperative control algorithm is developed to achieve the energy-efficient cooperative control strategy. Besides, simulations based on a real-life metro line demonstrate that the proposed cooperative control method can reduce substation energy consumption and improve the utilization of RBE.

Keywords: V2X Communication, Cooperative Systems (V2X), Automated Vehicles
Abstract: This article focuses on reviewing the results of a series of trials conducted in Europe and China to benchmark 5G’s benefits for automated driving challenges. The measurements have been conducted for studying the influence of the current 5G/LTE-V2X connectivity and optimizing antenna height, driving speed, and performance variation due to landscape variation. The results have been aggregated in real-world testing conducted in Finland and China. The vehicles have been equipped with onboard units (OBUs) and the infrastructure with the latest available 5G or LTE technologies.

The outcome of this study indicates that LTE-V2X highly depends on antenna height. However, the latencies are quite stable, being 20–50 ms unless line-of-sight connection is lost. The communication range is increased by 5G, and also package size can be increased by up to 1 MB without increasing the package error rate, which in the LTE-V2X case starts increasing when 0.5 MB is exceeded. This is not a problem for traditional C-ITS messages, but if considering “see through” or “remote video operation,” then the package size demand is much higher and goes beyond LTE-V2X’s capacity.

Keywords: Security, Cooperative ITS, Autonomous / Intelligent Robotic Vehicles
Abstract: The newly proposed cooperative intelligent transportation system (cITS) is a big step towards completely autonomous driving. It is a key requirement for vehicles to exchange crucial information. Only with exchanged data, such as hazard warnings or route planning each vehicle will have enough information to find its way without a driver. However, this data has to be authentic and trustworthy, since it will directly influence the behavior of every vehicle inside such a network. For authentic messages, public key infrastructure (PKI)-based asymmetric cryptography mechanisms were already proposed by different organizations, such as the European Telecommunications Standards Institute (ETSI). The second crucial information of trustworthiness is still missing. In this paper, a new security concept is presented, which introduces a trust-level for each vehicle to enable an assessment, whether data is trustworthy or not. Besides, a Pretty Good Privacy (PGP)-inspired certificate administration is proposed to manage the certificates and their affiliated trust-level. The new concept mitigates sybil attacks and increases the speed of data processing inside vehicles.

Keywords: Cooperative Systems (V2X)
Abstract: Vehicle Ad Hoc Networks (VANETs) refers to an open mobile Ad hoc network composed of communication between vehicles, between vehicles and fixed access points, and between vehicles and pedestrians in a traffic environment. The key factor for its safe operation is real-time accurate location information. The information is often provided by the global navigation satellite system, but some defects of the system limit its applications in the urban area, such as low accuracy or failure under signal occlusion. This study proposes a positioning algorithm based on vehicle-to-everything (V2X) communication. It combines the information obtained from the connected vehicle-to-infrastructure(V2I) or vehicle-to-vehicle (V2V) node and the one that is provided by the on-board inertial navigation system and sensing system to estimate the positions of the target vehicle. The simulation results show that this algorithm performs well in some simulated urban driving scenarios. Its accuracy and stability can meet the strict requirements of most positioning applications.

Keywords: Automated Vehicles, Cooperative Systems (V2X), V2X Communication
Abstract: Vehicle-to-Everything (V2X) communication, specifically Dedicated Short-Range Communication (DSRC), was developed for wireless communication and information sharing between vehicles and their surrounding environment in an effort to create a safer road ecosystem. With connected vehicle technology becoming more prevalent in today’s market, this paper intends to empirically define the external factors that influence the radiation pattern of a commercially available DSRC-equipped vehicle. As DSRC can be influenced by different road and environmental conditions, the aim was to investigate the potential sources that cause a reduction in communication efficiency by looking at their effect on signal strength and packet loss. 728 test runs were performed and their results used to evaluate the DSRC communication under 12 environmental and obstruction conditions. Results indicate that environmental conditions, such as temperature, water particles and type of road surface covering have minimal impact on Vehicle-to-Vehicle (V2V) communication. In contrast, physical obstructions, whether from the surrounding environment or on the vehicle’s antenna, cause larger variations in the signal strength and in the packet loss. These results may be used to inform the public, industry and regulators on DSRC limitations, while similar test methods can be used to evaluate other forms of connected vehicles technology, such as Cellular Vehicle-to-Everything (C-V2X).

Keywords: Cooperative Systems (V2X), Automated Vehicles, V2X Communication
Abstract: In this paper, an event-triggered vehicle-following control scheme for connected and automated vehicles (CAVs) is proposed considering nonideal Vehicle-to-Vehicle communications such as communication delays and packet dropouts. An output-based event-triggered mechanism is employed for reducing computational burden. An Event-Triggered Model Predictive Control (ETMPC) is proposed by combining with a multi-target controller for the lateral and longitudinal vehicle-following control of CAVs. The simulation results demonstrate that the proposed ETMPC can avoid unnecessary optimization implementation, achieving a computational reduction by 61.5% while maintaining the tracking precision compared with a conventional Model Predictive Controller. The proposed control scheme is also capable of being employed in vehicle platoon control.

Keywords: Cooperative Systems (V2X), Situation Analysis and Planning, V2X Communication
Abstract: In this paper, we propose an optimization-based strategy that utilizes vehicle-to-everything (V2X) communication in order to resolve conflicts between vehicles of different automation levels. The strategy consists of a decision checking mechanism and a control law to adjust the decision of an ego vehicle in a certain maneuver based on status update messages received from a remote vehicle involved in that maneuver. Using numerical simulations with real highway data, we demonstrate the proposed opportunistic strategy and show how it improves safety and maximizes the time efficiency of the ego vehicle. We also highlight the benefits of the strategy by comparing the results with an existing conservative strategy.

Keywords: Eco-driving and Energy-efficient Vehicles, Cooperative ITS
Abstract: Electrical vehicle (EV) raises to promote an eco-sustainable society. Nevertheless, the ``range anxiety'' of EV hinders its wider acceptance among customers. This paper proposes a novel solution to range anxiety based on a federated-learning model, which is capable of estimating battery consumption and providing energy-efficient route planning for vehicle networks. Specifically, the new approach extends the federated-learning structure with two components: anomaly detection and sharing policy. The first component identifies preventing factors in model learning, while the second component offers guidelines for information sharing amongst vehicle networks when the sharing is necessary to preserve learning efficiency. The two components collaborate to enhance learning robustness against data heterogeneities in networks. Numerical experiments are conducted, and the results show that compared with considered solutions, the proposed approach could provide higher accuracy of battery-consumption estimation for vehicles under heterogeneous data distributions, without increasing the time complexity or transmitting raw data among vehicle networks.

Keywords: V2X Communication, Cooperative ITS, Cooperative Systems (V2X)
Abstract: While today only a small fraction of vehicles is equipped with communication technology, this will hopefully change in the near future. But when market penetration rates increase, so will the load in the allocated frequency spectrum. Therefore, Decentralised Congestion Control mechanisms have been developed to limit the channel load and are mandatory in Europe. When one vehicle hosts multiple services, each of which regularly needs to transmit messages, those mechanisms are not appropriate to distribute available resources among them as this can lead to the complete starvation of low priority services. Therefore, an additional entity located in the Facilities Layer is required. This paper introduces a Weighted Fair Queuing-based approach to such a resource allocator. An extensive simulation study analyses its performance and shows how it is able to fairly allocate resources according to the services’ weights and completely mitigates the starvation issue.

Keywords: Cooperative Systems (V2X), Security, Vision Sensing and Perception
Abstract: Recently, Collective Perception Messages (CPM) that carry additional information about the surrounding environment beyond Basic Safety Messages (BSM) or Cooperative Awareness Messages (CAM) have been proposed to increase the situational awareness for Connected and Automated Vehicles (CAV) in Intelligent Transportation Systems. However, blindly trusting perception information from neighbors that cannot be locally verified is dangerous given the safety impact that erroneous or malicious information might have. This paper addresses the data trust challenge of CPMs, proposing a misbehavior detection scheme called MISO-V (Multiple Independent Sources of Observations over V2X) that leverages the inherently overlapping nature of the perception observations from multiple vehicles to verify the semantic correctness of the V2X data and improve the data trust and robustness of V2X systems. CPM-enabled CAVs are implemented and MISO-V performance is evaluated in CARLA-based simulation tool, where falsified V2X packets presenting a ghost car are injected in a suburban T-junction scenario with other cars. The results show that MISO-V is very effective in detecting the ghost car attacks and removing the impact of such misbehavior from influencing the receiver and offers a conservative and sensible approach towards trustworthy Collective Perception Services for CAVs.

Keywords: Cooperative Systems (V2X), Security, Smart Infrastructure
Abstract: Many intelligent transportation systems are multi-agent systems, i.e., both the traffic participants and the subsystems within the transportation infrastructure can be modeled as interacting agents. The use of AI-based methods to achieve coordination among the different agents systems can provide greater safety over transportation systems containing only human-operated vehicles, and also improve the system efficiency in terms of traffic throughput, sensing range, and enabling collaborative tasks. However, increased autonomy makes the transportation infrastructure vulnerable to compromised vehicular agents or infrastructure. This paper proposes a new framework by embedding the trust authority into transportation infrastructure to systematically quantify the trustworthiness of agents using an epistemic logic known as subjective logic. In this paper, we make the following novel contributions: (i) We propose a framework for using the quantified trustworthiness of agents to enable trust-aware coordination and control. (ii) We demonstrate how to synthesize trust-aware controllers using an approach based on reinforcement learning. (iii) We comprehensively analyze an autonomous intersection management (AIM) case study and develop a trust-aware version called AIM-Trust that leads to lower accident rates in scenarios consisting of a mixture of trusted and untrusted agents.

Keywords: Automated Vehicles, Sensor and Data Fusion, V2X Communication
Abstract: One of the main challenges in developing autonomous transport systems based on connected and automated vehicles is the comprehension and understanding of the environment around each vehicle. In many situations, the understanding is limited to the information gathered by the sensors mounted on the ego-vehicle, and it might be severely affected by occlusion caused by other vehicles or fixed obstacles along the road. Situational awareness is the ability to perceive and comprehend a traffic situation and to predict the intent of vehicles and road users in the surrounding of the ego-vehicle. The main objective of this paper is to propose a framework for how to automatically increase the situational awareness for an automatic bus in a realistic scenario when a pedestrian behind a parked truck might decide to walk across the road. Depending on the ego-vehicle's ability to fuse information from sensors in other vehicles or in the infrastructure, shared situational awareness is developed using a set-based estimation technique that provides robust guarantees for the location of the pedestrian. A two-level information fusion architecture is adopted, where sensor measurements are fused locally, and then the corresponding estimates are shared between vehicles and units in the infrastructure. Thanks to the provided safety guarantees, it is possible to adjust the ego-vehicle speed appropriately to maintain a proper safety margin. Three scenarios of growing information complexity are considered throughout the study. Simulations show how the increased situational awareness allows the ego-vehicle to maintain a reasonable speed without sacrificing safety.

Keywords: V2X Communication, Security, Active and Passive Vehicle Safety
Abstract: While connected vehicle (CV) applications have the potential to revolutionize traditional transportation system, cyber and physical attacks on them may lead to disastrous consequences. In this work, we propose an efficient dual cyber-physical blockchain framework to build trust and secure communication for CV applications. Our approach incorporates blockchain technology and physical sensing capabilities of vehicles to quickly react to attacks in a large-scale vehicular network, with low resource overhead. We explore the application of our framework to three CV applications, i.e., highway merging, intelligent intersection management, and traffic network with route choices. Simulation results demonstrate the effectiveness of our blockchain-based framework in defending against spoofing attacks, bad mouthing attacks, and Sybil and voting attacks. We also provide analysis to show the timing and resource efficiency of our framework.

Keywords: Cooperative Systems (V2X)
Abstract: Connected vehicles offer a range of sophisticated services that benefit owners, manufacturers, transportation authorities, and other mobility service providers. Securing the complex sensing and networking protocols that enable these applications is an important and difficult problem. In this paper, we use blockchain which is traditionally used in applications from cryptocurrencies to smart contracts, as a potential solution to CV security. Specifically, we exploit the immutability of blockchain to ensure safety from falsified information and attacks. We demonstrate these properties by developing an algorithm that uses blockchain to maintain trusted communications between vehicles in the context of a cooperative ramp merging application.

Keywords: V2X Communication, Cooperative Systems (V2X), Collision Avoidance
Abstract: Vehicle-to-vehicle (V2V) communication is an emerging technology which enables vehicles to share data, facilitating functions such as traffic information service, collision warning or cooperative control to potentially increase throughput and enhance road safety. However, the benefits of V2V communication during the early stages of market penetration, where communicating and non-communicating vehicles are co-existing, are not yet clear. This paper aims at quantifying the relationship between the penetration ratio of communicating vehicles and the additional safety that can be achieved thereby. We design a rule-based method for early detection of aggressive lane change behaviors using communicated motion data and evaluate its effectiveness according to different penetration ratios and traffic scenarios. Furthermore, we propose a proactive countermeasure which can be applied to mitigate the disruptive impact of erratic lane changes and analyze its performance in case different number of vehicles implement the measure. The results from a microscopic traffic simulator indicate that in partially-connected traffic, anomalies such as abrupt lane changes even of non-connected cars can be detected by observing the response of the following vehicles. In addition, by applying pre-braking as a countermeasure, traffic safety can be improved in terms of surrogate safety measures (SSM).

Keywords: Autonomous / Intelligent Robotic Vehicles, Automated Vehicles, Convolutional Neural Networks
Abstract: Big data-driven learning models are created by training connectionist models. With the increase in computing power and memory size, these models are becoming practical solutions for predicting image classifications, driving trajectories and users’ behaviors. Although these models can be shown to perform with high accuracy, this success measure is not enough to understand why the network predicts certain outputs for certain inputs. These networks behave as black boxes, able of processing very large amounts of data, without being transparent about their inner workings. This paper extends the architecture of a convolutional neural network and trains only the new connections to output an explanation for every prediction of the original classifier. The explanations are taken from a semantic language that is either computed or annotated from available data. Our work includes (1) defining and computing a language relevant to the classifier domain and semantically understandable by humans (2) computing the explanatory layer of the original network (3) training the extended architecture without changing the original given weights and (4) formatting the explanations in a user understandable manner. We applied our algorithmic solution to two existing classifiers in the automated driving domain. We showed successful results explaining predictive classifications of driving comfort and driving trajectories.

Keywords: Lidar Sensing and Perception, Convolutional Neural Networks, Deep Learning
Abstract: Vehicle detection, tracking and motion forecasting are critical for intelligent vehicle sensing system. In this paper, we propose a single-stage deep neural network (DNN) called TAPNet, which combines consecutive frames of LiDAR scans and high-definition (HD) maps to jointly reason about Bird’s Eye View (BEV) detection and trajectory prediction of vehicles. In our proposed method, an auxiliary past learning task is developed which can guide the backbone to exploit temporal information from training data by learning the motion history of vehicles. Moreover, we introduce a novel incremental trajectory loss function that can reduce the regressing varience and generate smooth trajectories. Experimental results on large-scale public dataset show that our proposed TAPNet outperforms other state-of-the-art models that jointly reason about BEV detection and motion forecasting.

Keywords: Vehicle Environment Perception, Deep Learning
Abstract: Stixels have been successfully applied to a wide range of vision tasks in autonomous driving, recently including instance segmentation. However, due to their sparse occurrence in the image, until now Stixels seldomly served as input for Deep Learning algorithms, restricting their utility for such approaches. In this work we present StixelPointNet, a novel method to perform fast instance segmentation directly on Stixels. By regarding the Stixel representation as unstructured data similar to point clouds, architectures like PointNet are able to learn features from Stixels. We use a bounding box detector to propose candidate instances, for which the relevant Stixels are extracted from the input image. On these Stixels, a PointNet models learns binary segmentations, which we then unify throughout the whole image in a final selection step. StixelPointNet achieves state-of-the-art performance on Stixel-level, is considerably faster than pixel-based segmentation methods, and shows that with our approach the Stixel domain can be introduced to many new 3D Deep Learning tasks.

Keywords: Self-Driving Vehicles, Lidar Sensing and Perception, Deep Learning
Abstract: One of the critical pieces of the self-driving puzzle is understanding the surroundings of a self-driving vehicle (SDV) and predicting how these surroundings will change in the near future. To address this task we propose MultiXNet, an end-to-end approach for detection and motion prediction based directly on lidar sensor data. This approach builds on prior work by handling multiple classes of traffic actors, adding a jointly trained second-stage trajectory refinement step, and producing a multimodal probability distribution over future actor motion that includes both multiple discrete traffic behaviors and calibrated continuous position uncertainties. The method was evaluated on large-scale, real-world data collected by a fleet of SDVs in several cities, with the results indicating that it outperforms existing state-of-the-art approaches.

Keywords: Reinforcement Learning, Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: Navigating through intersections is one of the main challenging tasks for an autonomous vehicle. However, for the majority of intersections regulated by traffic lights, the problem could be solved by a simple rule-based method in which the autonomous vehicle behavior is closely related to the traffic light states. In this work, we focus on the implementation of a system able to navigate through intersections where only traffic signs are provided. We developed a multi-agent system using a continuous, model-free Deep Reinforcement Learning algorithm used to train a neural network for predicting both the acceleration and the steering angle at each time step. We demonstrate that agents learn both the basic rules needed to handle intersections by understanding the priorities of other learners inside the environment, and to drive safely along their paths. Moreover, a comparison between our system and a rule-based method proves that our model achieves better results especially with dense traffic conditions. Finally, we test our system on real world scenarios using real recorded traffic data, proving that our module is able to generalize both to unseen environments and to different traffic conditions.

Keywords: Cooperative ITS, Recurrent Networks, Sensor and Data Fusion
Abstract: A novel multi-task federated learning (FL) framework is proposed in this paper to optimize the traffic prediction models without sharing the collected data among traffic stations. In particular, a divisive hierarchical clustering is first introduced to partition the collected traffic data at each station into different clusters. The FL is then implemented to collaboratively train the learning model for each cluster of local data distributed across the stations. Using the multi-task FL framework, the route planning is studied where the road map is modeled as a time-dependent graph and a modified A* algorithm is used to determine the route with the shortest traveling time. Simulation results showcase the prediction accuracy improvement of the proposed multi-task FL framework over two baseline schemes. The simulation results also show that, when using the multi-task FL framework in the route planning, an accurate traveling time can be estimated and an effective route can be selected.

Keywords: V2X Communication, Deep Learning, Recurrent Networks
Abstract: We compare the potential of neural network (NN)-based channel estimation with classical linear minimum mean square error (LMMSE)-based estimators, also known as Wiener filtering. For this, we propose a low-complexity recurrent neural network (RNN)-based estimator that allows channel equalization of a sequence of channel observations based on independent time- and frequency-domain long short-term memory (LSTM) cells. Motivated by Vehicle-to-Everything (V2X) applications, we simulate time- and requency-selective channels with orthogonal frequency division multiplex (OFDM) and extend our channel models in such a way that a continuous degradation from line-of-sight (LoS) to non-line-of-sight (NLoS) conditions can be emulated. It turns out that the NN-based system cannot just compete with the LMMSE equalizer, but it also can be trained w.r.t. resilience against system parameter mismatch. We thereby showcase the conceptual simplicity of such a data-driven system design, as this not only enables more robustness against, e.g., signal-to-noise-ratio (SNR) or Doppler spread estimation mismatches, but also allows to use the same equalizer over a wider range of input parameters without the need of re-building (or re-estimating) the filter coefficients. Particular attention has been paid to ensure compatibility with the existing IEEE 802.11p piloting scheme for V2X communications. Finally, feeding the payload data symbols as additional equalizer input unleashes further performance gains. We show significant gains over the conventional LMMSE equalization for highly dynamic channel conditions if such a data-augmented equalization scheme is used.

Keywords: Security, Deep Learning, Vulnerable Road-User Safety
Abstract: Modern neural networks have found to be miscalibrated in terms of confidence calibration, i.e., their predicted confidence scores do not reflect the observed accuracy or precision. Recent work has introduced methods for post-hoc confidence calibration for classification as well as for object detection to address this issue. Especially in safety critical applications, it is crucial to obtain a reliable self-assessment of a model. But what if the calibration method itself is uncertain, e.g., due to an insufficient knowledge base?

We introduce Bayesian confidence calibration - a framework to obtain calibrated confidence estimates in conjunction with an uncertainty of the calibration method. Commonly, Bayesian neural networks (BNN) are used to indicate a network's uncertainty about a certain prediction. BNNs are interpreted as neural networks that use distributions instead of weights for inference. We transfer this idea of using distributions to confidence calibration. For this purpose, we use stochastic variational inference to build a calibration mapping that outputs a probability distribution rather than a single calibrated estimate. Using this approach, we achieve state-of-the-art calibration performance for object detection calibration. Finally, we show that this additional type of uncertainty can be used as a sufficient criterion for covariate shift detection.

Keywords: Deep Learning, Unsupervised Learning, Recurrent Networks
Abstract: The clustering of autonomous driving scenario data can substantially benefit the autonomous driving validation and simulation systems by improving the simulation tests' completeness and fidelity. This article proposes a comprehensive data clustering framework for a large set of vehicle driving data. Existing algorithms utilize handcrafted features whose quality relies on the judgments of human experts. Additionally, the related feature compression methods are not scalable for a large data-set. Our approach thoroughly considers the traffic elements, including both in-traffic agent objects and map information. Meanwhile, we proposed a self-supervised deep learning approach for spatial and temporal feature extraction to avoid biased data representation. With the newly designed driving data clustering evaluation metrics based on data-augmentation, the accuracy assessment does not require a human-labeled data-set, which is subject to human bias. Via such unprejudiced evaluation metrics, we have shown our approach surpasses the existing methods that rely on handcrafted feature extractions.

Keywords: Deep Learning, Advanced Driver Assistance Systems, Vehicle Environment Perception
Abstract: Deep Neural Networks (DNNs) are increasingly applied in the real world in safety critical applications like advanced driver assistance systems. An example for such use case is represented by traffic sign recognition systems. At the same time, it is known that current DNNs can be fooled by adversarial attacks, which raises safety concerns if those attacks can be applied under realistic conditions. In this work we apply different black-box attack methods to generate perturbations that are applied in the physical environment and can be used to fool systems under different environmental conditions. To the best of our knowledge we are the first to combine a general framework for physical attacks with different black-box attack methods and study the impact of the different methods on the success rate of the attack under the same setting. We show that reliable physical adversarial attacks can be performed with different methods and that it is also possible to reduce the perceptibility of the resulting perturbations. The findings highlight the need for viable defenses of a DNN even in the black-box case, but at the same time form the basis for securing a DNN with methods like adversarial training which utilizes adversarial attacks to augment the original training data.

Keywords: Reinforcement Learning, Collision Avoidance, Advanced Driver Assistance Systems
Abstract: Deep reinforcement learning has been increasingly discussed for solving continuous control tasks in the field of autonomous driving and driver assistance systems. However, trial-and-error learning and the black-box character of neural networks make it prone to accidental damage in safety-critical environments. We propose to learn a safe vehicle following controller with deep reinforcement learning by imposing state-specific safe sets as output constraints on the policy and call the approach ACC 4S. The main safety goal is the avoidance of rear-end collisions with the front vehicle. To achieve this, we build on the Responsibility-Sensitive Safety model and derive an upper bound for the demanded acceleration. Further limitations emerge from regulatory standards and system limits. We end up with state-specific intervals of safe actions, the safe sets. To impose these safe sets as hard output constraints on the policy, we leverage the recently proposed neural network architecture ConstraintNet. We compare ConstraintNet with an unconstrained neural network, additional clipping as post-processing, and clipping as part of the neural network. The results show, that the proposed safe sets ensure collision avoidance and ConstraintNet shows superior performance compared to the other approaches.

Keywords: Deep Learning, Eco-driving and Energy-efficient Vehicles, V2X Communication
Abstract: The state prediction of the preceding vehicle (PV) is essential for anticipatory driving that enables a vehicle to take early steps for efficient driving. Such driving results in energy efficiency and traffic flow improvement but also requires repeated rigorous computations. Here we propose a Deep Neural Network (DNN) based driving scheme (DDS) that is trained to predict the PV and decide the control input of a host vehicle (HV) using two DNN models. We use the Road-Speed Profile (RSP), which provides approximated traffic speeds along the road using information from limited connected vehicles, to predict the future speed of the PV. Next, with the predicted speed of the PV, the control input is instantly determined to drive the HV efficiently. For making the optimal control decisions, the DNN is trained using a Model Predictive Control (MPC) scheme that generates the control input of the HV by minimizing a typical objective function reflecting costs related to inefficient driving. In the simulation, it is found that the proposed DDS guides the HV to change acceleration early in the optimized way by predicting the PV, particularly in changing traffic conditions, which results in a significant reduction in fuel consumption of the HV. Most remarkably, the DDS can determine the decision almost in real-time, which removes the necessity of setting a long step in the discrete-time control framework as found in MPC to match with its computation time. The proposed DDS is evaluated in some extreme cases and compared with traditional driving.

Keywords: Vehicle Environment Perception, Automated Vehicles, Deep Learning
Abstract: Street scene change detection continues to capture researchers' interests in the computer vision community. It aims to identify the changed regions of the paired street-view images captured at different times. The state-of-the-art network based on the encoder-decoder architecture leverages the feature maps at the corresponding level between two channels to gain sufficient information of changes. Intuitively, it is important to efficiently exploit the feature maps at different times for increasing the performance of change detection. Still, the efficiency of feature extraction, feature correlation calculation, even the whole network requires further improvement. This paper proposes the temporal attention and explores the impact of the dependency-scope size of temporal attention on the performance of change detection. In addition, based on the Temporal Attention Module (TAM), we introduce a more efficient and light-weight version - Dynamic Receptive Temporal Attention Module (DRTAM) and propose the Concurrent Horizontal and Vertical Attention (CHVA) to improve the accuracy of the network on specific challenging entities. On street scene datasets 'GSV', 'TSUNAMI' and 'VL-CMU-CD', our approach gains excellent performance, establishing new state-of-the-art scores without bells and whistles, while maintaining high efficiency applicable in autonomous vehicles.

Keywords: Deep Learning, Convolutional Neural Networks, Vision Sensing and Perception
Abstract: The last technologies and advances in the Unmanned Aerial Vehicles industries led to their use as a powerful tool for Traffic Monitoring and minimizing traffic accidents. However, detecting and classifying objects from aerial imagery becomes more challenging, especially in urban cities; due to the variation in size and scale of the objects; in addition to the lack of the datasets that contain information about the object classes and their 3D positions.

This paper introduced a novel algorithm to refine the semantic images captured from CARLA simulator; refining the semantic images, creating a new dataset with more classes of the objects, and calculating its 3D bounding box positions. The proposed method and dataset have been validated by training a CNN-based 2D object detector and inferring with real images from VisDrone-2019. The obtained results illustrated the improvements of using the proposed dataset in the object detection process, with real images in complex environments.

Keywords: Situation Analysis and Planning, Convolutional Neural Networks, Deep Learning
Abstract: In this paper, we address the challenge of employing Machine Learning (ML) algorithms in safety critical driving functions. Despite ML algorithms demonstrating good performance in various driving tasks, e.g., detecting when other vehicles are going to change lanes, the challenge of validating these methods has been neglected. To this end, we introduce an interpretable Lane Change Detector (LCD) algorithm which takes advantage of the performance of modern ML-based anomaly detection methods. We independently train three Deep Autoencoders (DAEs) on different driving maneuvers: lane keeping, right lane changes and left lane changes. The lane changes are subsequently detected by observing the reconstruction errors at the output of each DAE. Since the detection is purely based on the reconstruction errors of independently trained DAEs, we show that the classification outputs are completely interpretable. We compare the introduced algorithm with black-box Recurrent Neural Network (RNN)-based classifiers, and train all methods on realistic highway driving data. We discuss both the costs and the benefits of an interpretable classification, and demonstrate the inherent interpretability of the algorithm.

Keywords: Security, Privacy, Deep Learning
Abstract: The emerging vehicular edge computing (VEC) technology has the potential to bring revolutionary development to vehicular ad hoc network (VANET). However, the edge computing servers (ECSs) are subjected to a variety of security threats. One of the most dangerous types of security attacks is the Sybil attack, which can create fabricated virtual vehicles (called Sybil vehicles) to significantly overload ECSs' limited computation resources and thus disrupt legitimate vehicles' edge computing applications. In this paper, we present a novel Sybil attack detection system on ECSs that is based on the design of a credibility enhanced temporal graph convolutional network. Our approach can identify the malicious vehicles in a dynamic traffic environment while preserving the legitimate vehicles' privacy, particularly their local position information. We evaluate our proposed approach in the SUMO simulator. The results demonstrate that our proposed detection system can accurately identify most Sybil vehicles while maintaining a low error rate.

Keywords: Deep Learning, Convolutional Neural Networks, Vision Sensing and Perception
Abstract: In order to handle the challenges of autonomous driving, deep learning has proven to be crucial in tackling increasingly complex tasks, such as 3D detection or instance segmentation. State-of-the-art approaches for image-based detection tasks tackle this complexity by operating in a cascaded fashion: they first extract a 2D bounding box based on which additional attributes, e.g. instance masks, are inferred. While these methods perform well, a key challenge remains the lack of accurate and cheap annotations for the growing variety of tasks. Synthetic data presents a promising solution but, despite the effort in domain adaptation research, the gap between synthetic and real data remains an open problem. In this work, we propose a weakly supervised domain adaptation setting which exploits the structure of cascaded detection tasks. In particular, we learn to infer the attributes solely from the source domain while leveraging 2D bounding boxes as weak labels in both domains to explain the domain shift. We further encourage domain-invariant features through class-wise feature alignment using ground-truth class information, which is not available in the unsupervised setting. As our experiments demonstrate, the approach is competitive with fully supervised settings while outperforming unsupervised adaptation approaches by a large margin.

Keywords: Deep Learning, Unsupervised Learning, Convolutional Neural Networks
Abstract: Deep learning perception models require a massive amount of labeled training data to achieve good performance. While unlabeled data is easy to acquire, the cost of labeling is prohibitive and could create a tremendous burden on companies or individuals. Recently, self-supervision has emerged as an alternative to leveraging unlabeled data. Here, we introduce a simple and flexible multi-task co-training framework that integrates a self-supervised task into any supervised task. Our approach exploits pretext tasks to incur minimum compute and parameter overheads and minimal disruption to existing training pipelines. We demonstrate the effectiveness of our framework by using two self-supervised tasks on different perception models. Our results show that both self-supervised tasks can improve the accuracy of the supervised task and, at the same time, demonstrates strong domain adaption capability when used with additional unlabeled data.

Keywords: Vehicle Environment Perception, Recurrent Networks, Deep Learning
Abstract: Behavioral and semantic relationship plays a vital role on intelligent self-driving and ADAS system. Here we define the relationship as pairwise semantic data between two objects. E.g. . Different from other research focusing on trajectory, position, and bounding boxes. Relationship data provides a human-kind description of the object's behavior. And it could describe an object's past and future status in an amazingly brief way. So it could be fundamental to tasks like risk detection, environment understanding, and decision making.

In this paper, we propose RSG-Net (Road Scene Graph Net). This is a graph convolutional network which is designed to predict potential semantic relationship from object proposals. And give a graph-structured result, called "Road Scene Graph". The experiment result indicates that this network, trained on Road Scene Graph dataset, could efficiently predict potential semantic relationships among objects around ego-vehicle.

Key Word: Relationship prediction, Environment understanding, Convolutional Graph Network

Keywords: Deep Learning, Automated Vehicles, Driver State and Intent Recognition
Abstract: In the foreseeable future, autonomous vehicles will have to drive alongside human drivers. In the absence of vehicle-to-vehicle communication, they will have to be able to predict the other road users' intentions. Moreover, they will also need to behave like a typical human driver so that other road users can infer their actions. It is critical to be able to learn a human driver's mental model and integrate it into the Planning & Control algorithm. In this paper, we present a robust method to predict lane changes as cooperative or adversarial. For that, we first introduce a method to extract and annotate lane changes as cooperative and adversarial based on the entire lane change trajectory. We then propose to train a specially designed neural network to predict the lane change label before the lane change has occurred and quantify the prediction uncertainty. The model will make lane change decisions following human drivers' driving habits and preferences, i.e., it will only change lanes when the surrounding traffic is considered to be appropriate for the majority of human drivers. It will also recognize unseen novel samples and output low prediction confidence correspondingly to alert the driver to take control in such cases. We published the lane change dataset and codes at https://github.com/huanghua1668/lc_csnn.

Keywords: Automated Vehicles, Driver Recognition, Driver State and Intent Recognition
Abstract: Integrating driver, in-cabin, and outside environment's contextual cues into the vehicle's decision making is the centerpiece of semi-automated vehicle safety. Multiple systems have been developed for providing context to the vehicle, which often rely on video streams capturing drivers' physical and environmental states. While video streams are a rich source of information, their ability in providing context can be challenging in certain situations, such as low illuminance environments (e.g., night driving), and they are highly privacy-intrusive. In this study, we leverage passive sensing through smartwatches for classifying elements of driving context. Specifically, through using the data collected from 15 participants in a naturalistic driving study, and by using multiple machine learning algorithms such as random forest, we classify driver's activities (e.g., using phone and eating), outside events (e.g., passing intersection and changing lane), and outside road attributes (e.g., driving in a city versus a highway) with an average F1 score of 94.55, 98.27, and 97.86 % respectively, through 10-fold cross-validation. Our results show the applicability of multimodal data retrieved through smart wearable devices in providing context in real-world driving scenarios and pave the way for a better shared autonomy and privacy-aware driving data-collection, analysis, and feedback for future autonomous vehicles.

Keywords: Driver State and Intent Recognition, Driver Recognition, Human-Machine Interface
Abstract: A rapidly growing amount of content posted online inherently holds knowledge about concepts of interest, i.e. driver actions. We leverage methods at the intersection of vision and language to surpass costly annotation and present the first automated framework for anticipating driver intention by learning from recorded driving exam conversations. We query YouTube and collect a dataset of posted mock road tests comprising student-teacher dialogs and video data, which we use for learning to foresee the next maneuver without any additional supervision. However, instructional conversations give us very loose labels, while casual chat results in a high amount of noise. To mitigate this effect, we propose a technique for automatic detection of smalltalk based on the likelihood of spoken words being present in everyday dialogs. While visually recognizing driver’s intention by learning from natural dialogs only is a challenging task, learning from less but better data via our smalltalk refinement consistently improves performance.

Keywords: Passive Safety, Advanced Driver Assistance Systems, Active and Passive Vehicle Safety
Abstract: Due to the recent progress on autonomous driving, some technologies have been gradually deployed to the public transport vehicles. The passenger safety under the unmanned operating environment has become an emerging issue which requires much more attention. This paper presents a method for passenger detection, counting and action recognition inside a minibus. A top-view camera system is mounted on the ceiling to have a full coverage of the interior. 2D and 3D convolutional neural networks are developed for pose recognition and action classification of the passengers. The experiments are carried out in a self-driving minibus, and the results have demonstrated the feasibility of the proposed technique.

Keywords: Driver Recognition, Driver State and Intent Recognition, Unsupervised Learning
Abstract: Although exists several drivers' gaze direction classifiers to prevent traffic accidents caused by inattentive driving, making this classification while the driver's face is temporarily or permanently occluded remains exceptionally challenging. For example, drivers using masks, sunglasses, or scarves and daily light variations are non-ideal conditions that recurrently appear in an everyday driving scenario and are frequently overlooked by the existing classifiers. This paper presents a single camera framework gaze zone classifier that operates robustly even during non-uniform lighting, non-frontal face pose, and faces undergo temporal or permanent occlusions. The usage of a normalized dense aligned face pose vector, the classification result of a pre-processed right eye area pixels, and the classification result of a pre-processed left eye area pixels is the cornerstone of the feature vector used in our model. The key of this paper is double-folded: firstly, the usage of a normalized dense alignment for a robust face, landmark, and head-pose direction detection and secondly, the processing of the right and left eye images using computer vision and deep learning techniques for refining, modifying, and finally labeling eyes information. Experiments on a challenging dataset involving non-uniform lighting, non-frontal face pose, and faces with temporal or permanent occlusions show each feature's importance towards making a robust gaze zone classifier under unconstrained driving situations.

Keywords: Reinforcement Learning, Situation Analysis and Planning, Passive Safety
Abstract: Driving behavior modeling (DBM) is widely used in the intelligent vehicle field to prevent accidents, which predicts actions that vehicles should take to optimize safe driving behaviors. According to some statistics, accidents easily happen at un-signalized intersections. Modeling driving behavior at such places is of great importance. However, current inverse reinforcement learning-based DBM methods fail to predict proper behaviors at the un-signalized intersections in the aspects of smoothness and stopping behavior by just using a single Markov decision process (MDP). We propose a novel sequential MDPs approach to model the driving behavior at the un-signalized intersections to solve the problems. Our approach decomposes the target behavior through the un-signalized intersections into three parts and models each decomposition's driving behaviors with appropriate time durations by a stopping-time-interval distribution through dynamic programming. Experiments on real driving data show that the proposed method achieved a better result and successfully improved the smoothness and stopping awareness of the planned driving path compared to the baselines.

Keywords: Driver Recognition, Driver State and Intent Recognition, Advanced Driver Assistance Systems
Abstract: Even though the safeguarding of automated driving functions is very present in automotive research, semi-automated and manually controlled systems will continue to be predominant in the next decade. Still, automation features such as driver assistance systems are becoming more prevalent and can operate more intuitively with the driver in a closed loop. Efficient interaction with the driver in partially automated systems can further improve road safety in the meantime. In this context, assessing the driver's perception of nearby traffic is crucial to making driver assistance safer and more collaborative. We propose a data-driven method to detect traffic-objects a driver is visually observing in the headway and side traffic using a multilabel neural-network as a classifier. This functionality is necessary to distinguish well controlled anticipatory driving from missing perception in partially automated driving. This could be an early reaction to decelerating vehicles ahead, even before a critical situation arises, or verification of the driver's perception of a cut-in. We validate the method using a data-driven ground truth from wearable eye trackers and automatically generated labels. Additional information comes from cameras installed in the vehicles to monitor the driver and traffic, as well as from measurement equipment that records CAN bus data. The proposed method achieves a Hamming accuracy of 69%, outperforming previous geometric models.

Keywords: Driver State and Intent Recognition, Advanced Driver Assistance Systems, Recurrent Networks
Abstract: In this paper, we present a novel approach for driving style classification based on time series data. Instead of automatically learning the embedding vector for temporal representation of the input data with RNNs, we propose a combination of HDC for data representation in high-dimensional vectors and much simpler feed-forward neural networks. This approach provides three key advantages: first, instead of having a black box of RNNs learning the temporal representation of the data, our approach allows to encode this temporal structure in high-dimensional vectors in a human-comprehensible way using the algebraic operations of HDC while only relying on feed-forward neural networks for the classification task. Second, we show that this combination is able to achieve at least similar and even slightly superior classification accuracy compared to state-of-the-art LSTM-based networks while significantly reducing training time and the necessary amount of data for successful learning. Third, our HDC-based data representation as well as the feed-forward neural network, allow implementation in the substrate of SNN. SNN show promise to be orders of magnitude more energy-efficient than their rate-based counterparts while maintaining comparable prediction accuracy when being deployed on dedicated neuromorphic computing hardware, which could be an energy-efficient addition in future intelligent vehicles with tight restrictions regarding on-board computing and energy resources. We present a thorough analysis of our approach on a publicly available data set including a comparison with state-of-the-art reference models.

Keywords: Deep Learning, Vision Sensing and Perception, Automated Vehicles
Abstract: The use of vehicle in-cabin monitoring has been increasing to fulfil the specifications of European safety regulations. These regulations present several requirements for detecting driver distraction, and more complex requirements are soon to be expected in higher automation levels. Today’s restraint systems provide optimal protection in standard frontal seat positions and deviations to this might cause severe airbag-induced injuries. This makes in-cabin monitoring critical to improve safety and mitigate dangerous situations in case of a crash, and especially in high levels of autonomous driving. Defining the best sensor positioning inside the vehicle’s cabin is a challenge due to its constraints and limitations. The main aim of this work was to verify if simulated 3D human models integrated into a 3D modelled vehicle interior environment can be used to run Deep Learning based human pose estimation models. To perform such task, we utilized the software MakeHuman combined with Blender, to build the virtual environment and create photorealistic scenes containing selected front occupants’ postures use cases, and then feed into Openspose and Mask R-CNN models. The results showed that using a 2D HPE (Human Pose Estimation) network pre-trained on real data, can detect successfully photorealistic synthetic data of humans under complex scenarios. It is also shown that complex and rare postures can cause failure on 2D HPE detections, as shown in the literature review. This work helps to define the most suitable camera positions which, in combination with specific camera lenses, can deliver quality images for a robust pose detection.

Keywords: Automated Vehicles, Driver Recognition, Unsupervised Learning
Abstract: Human drivers have complex and unique driving characteristics, even when driving in common, well-defined scenarios such as lane changes. In this study, we propose using probabilistic, deep generative models to predict personalized driving behavior including velocity, acceleration, and steering angle sequence. Probabilistic approaches are applied to model uncertainty in the driving behavior of individual drivers as a distribution, while surrounding vehicle and driver ID information are considered as given conditions in the distribution. We train individual driver models using real-world driving data, and use them to predict sequences of future driving behavior in dynamic environments, using historical data to take personal driving styles into account. Our results show that the proposed driver behavior modeling method is able to learn from a driver’s vehicle operation data and their interactions with surrounding vehicles to reproduce their specific driving style.

Keywords: Human-Machine Interface, Hand-off/Take-Over, Automated Vehicles
Abstract: As research and development related to the practical use of autonomous driving systems (ADS) continue to advance, one of the remaining challenges is achieving both safe and natural autonomous driving. In part, this is due to the difficulty of achieving flawless, automated perception and understanding of the driving environment. In our previous study, we proposed a recognition assistance interface to solve this problem by sharing ADS recognition information with the passenger, allowing them to assist in the recognition stage of the autonomous driving process. In this study, we incorporate our recognition assistance interface into Autoware and test it in a simulated driving environment, using scenarios in which the ADS must recognize the intent of pedestrians and respond to the presence of trash in the road. Natural driving was defined as driving that avoids significant, unnecessary deceleration when performing these challenging recognition tasks. The results of our experiment with 11 participants showed that sharing recognition information with passengers is effective for avoiding unnecessary deceleration and achieving only minor variations in speed when encountering obstacles flagged in error by the recognition system.

Keywords: Collision Avoidance, Deep Learning, Self-Driving Vehicles
Abstract: In this study, we present the development of a new method to detect unknown low-height obstacles using 3D point clouds captured by stereo cameras. Conventional semantic segmentation methods using a depth image by a deep neural network (DNN) can detect road surfaces with a high accuracy. However, it is difficult to detect unknown low-height obstacles that are not included in the training data. Methods that use 3D geometric information, such as the normal and height face difficulty in detecting objects with a surface that is parallel to the road surface and low objects, respectively. Therefore, the objective of this study is to address the difficult problem of detecting unknown low-height obstacles by focusing on the difference in the difficult obstacle detection between the DNN and 3D geometric methods. Based on the confidence from the output of the DNN, we accomplish difficult obstacle detection with the DNN by using 3D geometric information, and vice versa. When tested on a robot equipped with a stereo camera, the intersection over union (IoU), which indicates the detection accuracy of the unknown obstacles, was improved by 18.6 %age points compared to that with the DNN. Moreover, our method enabled the robot to safely avoid three types of unknown low-height obstacles.

Keywords: Intelligent Ground, Air and Space Vehicles, Active and Passive Vehicle Safety, Vehicle Environment Perception
Abstract: When Unmanned Aerial Vehicles (UAVs) autonomously operate in urban environments, it is especially important for these systems to behave safely and not harm anybody or anything. However, it is challenging to ensure that these systems behave safely in all possible situations and to clearly define this "safe" behavior for each situation. In this work, we provide a methodology for testing the safe behavior of UAVs while considering their environment with the help of scenario-based testing and search-based techniques. Additionally, we explore two cases throughout the paper: (i) A safety distance is specified, and we can use it for testing. (ii) No safety distance is defined, but we still aim to test the safe behavior of UAVs. In our experiments, we show the effectiveness and applicability of the proposed methods by discovering several safety distance violations and questionable behaviors of the tested UAV for both cases and four scenarios that represent all alternatives to avoid an obstacle.

Keywords: Automated Vehicles, Image, Radar, Lidar Signal Processing, Vehicle Environment Perception
Abstract: Systems and functions that rely on machine learning (ML) are the basis of highly automated driving. An essential task of such ML models is to reliably detect and interpret unusual, new, and potentially dangerous situations. The detection of those situations, which we refer to as corner cases, is highly relevant for successfully developing, applying, and validating automotive perception functions in future vehicles where multiple sensor modalities will be used. A complication for the development of corner case detectors is the lack of consistent definitions, terms, and corner case descriptions, especially when taking into account various automotive sensors. In this work, we provide an application-driven view of corner cases in highly automated driving. To achieve this goal, we first consider existing definitions of the general outlier, novelty, anomaly, and out-of-distribution detection to show relations and differences to corner cases. Moreover, we extend an existing camerafocused systematization of corner cases by adding RADAR (radio detection and ranging) and LiDAR (light detection and ranging) sensors. For this, we describe an exemplary toolchain for data acquisition and processing, highlighting the interfaces of corner case detection. We also define a novel level of corner cases, the method layer corner cases, which appear due to uncertainty inherent in the methodology.

Keywords: Self-Driving Vehicles, Autonomous / Intelligent Robotic Vehicles, Automated Vehicles
Abstract: This paper proposes a discretionary lane selection algorithm. In particular, highway driving is considered as a targeted scenario, where each lane has a different level of traffic flow. When lane-changing is discretionary, it is advised not to change lanes unless highly beneficial, e.g., reducing travel time significantly or securing higher safety. Evaluating such "benefit" is a challenge, along with multiple surrounding vehicles in dynamic speed and heading with uncertainty. We propose a real-time lane-selection algorithm with careful cost considerations and with a modularity in design. The algorithm is search-based optimization method that evaluates uncertain dynamic positions of other vehicles under a continuous time and space domain. For demonstration, we incorporate a state-of-the-art motion planner framework (Neural Networks integrated Model Predictive Control) under a CARLA simulation environment.

Keywords: Unsupervised Learning, Self-Driving Vehicles, Deep Learning
Abstract: Autonomous Driving (AD) systems are heavily reliant on supervised models. In these approaches, a model is trained to detect only a predefined number of obstacles. However, for applications like railway obstacle detection, the training dataset is limited and not all possible obstacle classes are known beforehand. For such safety-critical applications, this situation is problematic and could limit the performance of obstacle detection in autonomous trains. In this paper, we propose an exploratory study using unsupervised models based on a large set of generated convolutional autoencoder models to detect obstacles on railway’s track level. The study was conducted based on three components: loss functions, activations and optimizers. Existing works rely on fixing thresholds to judge the performance of the model. We propose instead a methodology based on Multi-Criteria Decision Making (MCDM) to evaluate the performance of all models. Furthermore, we introduce the notion of gap-score to evaluate each model by calculating the average difference between the reconstruction score on images with and without obstacles. The aim is to find models maximizing the average of gap-scores and rank them according to their performances. Experimental results show that the evaluated models can provide up to 68% average gap-score.

Keywords: Vehicle Environment Perception, Vision Sensing and Perception, Deep Learning
Abstract: Current state-of-the-art in speed measurement technologies includes magnetic inductive loop detectors, Doppler radar, infrared sensors, and laser sensors. Many of these systems rely on intrusive methods that require intricate installation and maintenance processes that hinder traffic while leading to high acquisition and maintenance costs. Speed measurement from monocular videos appears as an alternative in this context. However, most of these systems present as a drawback the requirement of camera calibration -- a fundamental step to convert the vehicle speed from pixels per frame to some real-world unit of measurement (textit{e.g.} km/h). Considering that, we propose a speed measurement system based on monocular cameras with no need for calibration. Our proposed system was trained from a synthetic data set containing 12,290 instances of vehicle speeds. We extract the motion information of the vehicles that pass in a specific region of the image by using dense optical flow, using it as input to a regressor based on a customized VGG-16 network. The performance of our method was evaluated over the Luvizon's data set, which contains real-world scenarios with 7,766 vehicle speeds, ground-truthed by a high precision system based on properly calibrated and approved inductive loop detectors. Our proposed system was able to measure 85.4% of the speed instances within an error range of [-3, + 2] km/h, which is ideally defined by the regulatory authorities in several countries. Our proposed system does not rely on any distance measurements in the real world as input, eliminating the need for camera calibration.

Keywords: Deep Learning, Convolutional Neural Networks, Self-Driving Vehicles
Abstract: An understanding and classification of driving scenarios are important for testing and development of autonomous driving functionalities. Machine learning models are useful for scenario classification but most of them assume that data received during the testing are from one of the classes used in the training. This assumption is not true always because of the open environment where vehicles operate. This is addressed by a new machine learning paradigm called open-set recognition. Open-set recognition is the problem of assigning test samples to one of the classes used in training or to an unknown class. This work proposes a combination of Convolutional Neural Networks (CNN) and Random Forest (RF) for open set recognition of traffic scenarios. CNNs are used for the feature generation and the RF algorithm along with extreme value theory for the detection of known and unknown classes. The proposed solution is featured by exploring the vote patterns of trees in RF instead of just majority voting. By inheriting the ensemble nature of RF, the vote pattern of all trees combined with extreme value theory is shown to be well suited for detecting unknown classes. The proposed method has been tested on the highD and OpenTraffic datasets and has demonstrated superior performance in various aspects compared to existing solutions.

Keywords: Deep Learning, Unsupervised Learning, Self-Driving Vehicles
Abstract: Traffic scenario categorisation is an essential component of automated driving, for e.g., in motion planning algorithms and their validation. Finding new relevant scenarios without handcrafted steps reduce the required resources for the development of autonomous driving dramatically. In this work, a method is proposed to address this challenge by introducing a clustering technique based on a novel data-adaptive similarity measure, called Random Forest Activation Pattern (RFAP) similarity. The RFAP similarity is generated using a tree encoding scheme in a Random Forest algorithm. The clustering method proposed in this work takes into account that there are labelled scenarios available and the information from the labelled scenarios can help to guide the clustering of unlabelled scenarios. It consists of three steps. First, a self-supervised Convolutional Neural Network (CNN) is trained on all available traffic scenarios using a defined self-supervised objective. Second, the CNN is fine-tuned for classification of the labelled scenarios. Third, using the labelled and unlabelled scenarios an iterative optimisation procedure is performed for clustering. In the third step at each epoch of the iterative optimisation, the CNN is used as a feature generator for an unsupervised Random Forest. The trained forest, in turn, provides the RFAP similarity to adapt iteratively the feature generation process implemented by the CNN. Extensive experiments and ablation studies have been done on the highD dataset. The proposed method shows superior performance compared to baseline clustering techniques.

Keywords: Collision Avoidance, Convolutional Neural Networks, Vehicle Environment Perception
Abstract: This paper proposes a collision detection system to detect whether ego and target vehicles collide when both vehicles change from their lanes to the same lane. Although it is essential to predict this kind of collision for the active safety system, there is little literature on the case study. This paper presents the collision detection method using a Convolution Neural Network (CNN) consisting of four classes to predict collision risk on multi-lanes road conditions. The CNN is formed on stacked Occupancy Grid Maps (OGMs) based on point cloud data of the LiDAR and Radar sensors with in-vehicle sensor data for spatio-temporal information between vehicles. Further, we apply the open set recognition concept to the network to consider a conservative collision detection. The experimental results show the feasibility of the proposed collision detection system and the conservative decision about the confusing situation.

Keywords: Vehicle Environment Perception, Self-Driving Vehicles, Mapping and Localization
Abstract: One necessary condition for autonomous driving is an accurate and reliable representation of the environment around the vehicle. Current architectures rely on cameras, radars, and lidars to capture the visual environment and to localize and track other traffic participants. Human drivers can see but also hear and use a lot of auditory information for understanding the environment in addition to visual cues. In this paper, we present a pure sound localization and recognition system to extract an auditory representation of the environment. First, the environmental sound is classified into seven main categories of traffic objects followed by six specific kinds of sirens in an emergency case using a simple neural network layout. Second, each object is localized via a combined time-delay of arrival and amplitude-based localization algorithm. The system is evaluated on real-world data focusing on a robust detection and accurate localization of emergency vehicles.

Keywords: Vision Sensing and Perception, Deep Learning, Self-Driving Vehicles
Abstract: Understanding complex scenarios from in-vehicle cameras is essential for safely operating autonomous driving systems in densely populated areas. Among these, intersection areas are one of the most critical as they concentrate a considerable number of traffic accidents and fatalities. Detecting and understanding the scene configuration of these usually crowded areas is then of extreme importance for both autonomous vehicles and modern Advanced Driver Assistance Systems (ADAS), aimed at preventing road crashes and increasing the safety of Vulnerable Road Users (VRU). This work investigates how to classify intersection areas from RGB images using well-consolidated neural network approaches along with a method to enhance the results based on the teacher/student training paradigm. An extensive experimental activity aimed at identifying the best input configuration and evaluating different network parameters on both the well-known KITTI dataset and the new KITTI-360 sequences shows that our method outperforms current state-of-the-art intersection classification approaches on per-frame basis, proving the effectiveness of the proposed learning scheme.

Keywords: Collision Avoidance, Automated Vehicles
Abstract: Behaviour prediction in multi-agent and dynamic environments is crucial in the context of intelligent vehicles, due to the complex interactions and representations of road participants (such as vehicles, cyclists or pedestrians) and road context information (e.g. traffic lights, lanes and regulatory elements). This paper presents SmartMOT, a simple yet powerful pipeline that fuses the concepts of tracking-by-detection and semantic information of HD maps, in particular using the OpenDrive format specification to describe the road network's logic, to design a real-time and power-efficient Multi-Object Tracking (MOT) pipeline which is then used to predict the future trajectories of the obstacles assuming a CTRV (Constant Turn Rate and Velocity) model. The system pipeline is fed by the monitorized lanes around the ego-vehicle, which are calculated by the planning layer, the ego-vehicle status, that contains its odometry and velocity and the corresponding Bird's Eye View (BEV) detections. Based on some well-defined traffic rules, HD map geometric and semantic information are used in the initial stage of the tracking module, made up by a BEV Kalman Filter and Hungarian algorithm are used for state estimation and data association respectively, to track only the most relevant detections around the ego-vehicle, as well as in the subsequent steps to predict new relevant traffic participants or delete trackers that go outside the monitorized area, helping the perception layer to understand the scene in terms of behavioural use cases to feed the executive layer of the vehicle. First, our system pipeline is described, exploiting the concepts of lightweight Linux containers using Docker to provide the system with isolation, flexibility and portability, and standard communication in r

Keywords: Human-Machine Interface, Advanced Driver Assistance Systems, Novel Interfaces and Displays
Abstract: All state-of-the-art Head-up Displays (HUD) operate with s-polarized light, resulting in poor visibility when wearing polarized sunglasses. The p-polarized HUD system is a solution to this problem, as an additional optical layer within the windshield reflects p-polarized light, making the virtual image visible with polarized sunglasses. However, p-polarized HUDs may cause an unwanted double image, too. To save costs, a double image compensation by use of a wedge-angled film is not an option. As the double image causes discomfort to the driver, this major image defect has to be controlled. Therefore, the determination of a new quality score, the Double Image Contrast Ratio (DICR), is essential. Hence, we determine the DICR in a study (N=46), by evaluating the visual discomfort level for drivers, resulting in a minimum DICR of 90. Reaching this DICR is key to enhance user experience and safety during all driving conditions.

Keywords: Human-Machine Interface, Advanced Driver Assistance Systems, Automated Vehicles
Abstract: People park their vehicle depending on interior and exterior contexts. They do it naturally, even unconsciously. For instance, with a baby seat on the rear, the driver might leave more space on one side to be able to get the baby out easily; or when grocery shopping, s/he may position the vehicle to remain the trunk accessible. Autonomous vehicles are becoming technically effective at driving from A to B and parking in a proper spot, with a default way. However, in order to satisfy users’ expectations and to become trustworthy, they will also need to park or make a temporary stop, appropriate to the given situation. In addition, users want to understand better the capabilities of their driving assistance features, such as automated parking systems. A voice-based interface can help with this and even ease the adoption of these features. Therefore, we developed a voice-based in-car assistant (Park4U Mate), that is aware of interior and exterior contexts (thanks to a variety of sensors), and that is able to park autonomously in a smart way (with a constraints minimization strategy). The solution was demonstrated to thirty-five users in test-drives and their feedback was collected on the system’s decision-making capability as well as on the human-machine-interaction. The results show that: (1) the proposed optimization algorithm is efficient at deciding the best parking strategy; hence, autonomous vehicles can adopt it; (2) a voice-based digital assistant for autonomous parking is perceived as a clear and effective interaction method. However, the interaction speed remained the most important criterion for users. In addition, they clearly wish not to be limited on only voice-interaction, to use the automated parking function and rather appreciate a multi-modal interaction.

Keywords: Vehicle Control, Vehicle Environment Perception, Automated Vehicles
Abstract: Human safety is one of the most critical factors when a new technology is introduced to the everyday use. It was no different in the case of Autonomous Vehicles (AV), designed to replace generally available conventional vehicles (CV) in the future. AV rules, from the start, focus on guaranteeing safety for passengers and other road users, and these assumptions usually work during normal traffic conditions. However, there is still a problem with proper reaction time to sudden, dangerous and unexpected scenarios like a running animal on a rural road during the night. In this paper, we compare human and AV responses to sudden scenarios and accidents. As the AV topic can be analyzed as an ICT system, we review modern sensors, computer architectures and algorithms designed for this type of problems. Beside regular analysis, we also show which algorithms can run simultaneously and if vehicles have proper tools to guarantee safety during regular system delays. As a final result, we present a diagram which depicts Autonomous Vehicle logic and allows to identify bottlenecks. Additionally, the analysis shows how different refresh rates and algorithm execution times can affect the braking distance thus safety of other road users.

Keywords: Human-Machine Interface, Novel Interfaces and Displays, Assistive Mobility Systems
Abstract: Personal assistants like Alexa, Siri, and co. persuade users to treat technology like a friend. Studies indicate that humanized features like voice, the display of emotions and empathy help people to have more trust and pleasure and feel companionship when interacting with the technology. The professional truck driver has various challenges such as being isolated for several hours while performing a monotonous driving task, which is exacerbated by having to monitor increasing automation, growing pressure due to increasing demand, and just-in-time delivery, we believe that humanized human-machine interaction can improve the working situation of truck drivers. Therefore, we created a socially interactive device called ICo (the Intelligent Co-driver), which supports and accompanies the driver while driving. In a driving simulator study with 34 professional drivers, we investigated within a Wizard of Oz setup to what extent the addition of human characteristics through mimicry and gesture changes the driving performance, user experience, and technology acceptance. No differences could be shown in the subjective data, but the objective data captured in the study suggests that humanization affects driving performance and driver engagement.

Keywords: Automated Vehicles, Human-Machine Interface, Vulnerable Road-User Safety
Abstract: Compared to a manual driving vehicle (MV), an automated driving vehicle lacks a way to communicate with the pedestrian through the driver when it interacts with the pedestrian because the driver usually does not participate in driving tasks. Thus, an external human machine interface (eHMI) can be viewed as a novel explicit communication method for providing driving intentions of an automated driving vehicle (AV) to pedestrians when they need to negotiate in an interaction, e.g., an encountering scene. However, the eHMI may not guarantee that the pedestrians will fully recognize the intention of the AV. In this paper, we propose that the instruction of the eHMI's rationale can help pedestrians correctly understand the driving intentions and predict the behavior of the AV, and thus their subjective feelings (i.,e., dangerous feeling, trust in the AV, and feeling of relief) and decision-making are also improved. The results of an interaction experiment in a road-crossing scene indicate that the participants were more difficult to be aware of the situation when they encountered an AV w/o eHMI compared to when they encountered an MV; further, the participants' subjective feelings and hesitation in decision-making also deteriorated significantly. When the eHMI was used in the AV, the situational awareness, subjective feelings and decision-making of the participants regarding the AV w/ eHMI were improved. After the instruction, it was easier for the participants to understand the driving intention and predict driving behavior of the AV w/ eHMI. Further, the subjective feelings and the hesitation related to decision-making were improved and reached the same standards as that for the MV.

Keywords: Vehicle Environment Perception, Radar Sensing and Perception, Image, Radar, Lidar Signal Processing
Abstract: This paper presents vehicle occupant body imaging algorithms for detecting the presence of children that are left behind in vehicles. The proposed algorithms utilize occupancy grid mapping (OGM) techniques developed for autonomous driving, which are modified for in-cabin human detection purposes. Two different kinds of OGM techniques are presented to deal with two major difficulties in imaging the occupant bodies in an in-cabin environment. The processing flows of the proposed algorithms are detailed in this report. Two experimental results for 60 GHz millimeter-wave radars are also presented for performance verification.

Keywords: Image, Radar, Lidar Signal Processing, Radar Sensing and Perception, Deep Learning
Abstract: Autonomous vehicles are conceived to provide safe and secure services by validating the safety standards as indicated by SOTIF-ISO/PAS-21448 (Safety of the intended functionality). Keeping in this context, the perception of the environment plays an instrumental role in conjunction with localization, planning and control modules. As a pivotal algorithm in the perception stack, object detection provides extensive insights into the autonomous vehicle's surroundings. Camera and Lidar are extensively utilized for object detection among different sensor modalities, but these exteroceptive sensors have limitations in resolution and adverse weather conditions. In this work, radar-based object detection is explored provides a counterpart sensor modality to be deployed and used in adverse weather conditions. The radar gives complex data; for this purpose, a channel boosting feature ensemble method with transformer encoder-decoder network is proposed. The object detection task using radar is formulated as a set prediction problem and evaluated on the publicly available dataset in both good and good-bad weather conditions. The proposed method's efficacy is extensively evaluated using the COCO evaluation metric, and the best-proposed model surpasses its state-of-the-art counterpart method by 12.55% and 12.48% in both good and good-bad weather conditions.

Keywords: Radar Sensing and Perception, Vehicle Environment Perception, Autonomous / Intelligent Robotic Vehicles
Abstract: To solve the problem of obstacle detection in unstructured scene, we propose a novel obstacle detection method based on Radar. Compared with other sensors like 3D-Lidar and color camera, Radar has higher environmental robustness especially for sandy road with dust. According to the relative velocity information of detected obstacles, obstacles are divided into static ones and dynamic ones in the proposed method. For static obstacles, map reconstruction based on Bayesian probability model is used to overcome the problem of sparse obstacle detection result in single frame and obstacle retention in the blind area. The integrated navigation system of GNSS and IMU is used to provide position and posture information during map reconstruction. For dynamic obstacles, the historical trajectory of the same target is recorded in the global coordinate frame, then the Kalman predictor based trajectory tracking and prediction is used to predict the movement trend of moving obstacles. The proposed method has been verified in a variety of unstructured scene. The experimental results show that the proposed method can provide useful obstacle perception result and meet the autonomous navigation application of UGV in unstructured scene. We also compare the proposed method with that based on 3D-Lidar in terms of the same scene. And it has obvious advantages in environmental robustness.

Keywords: Vehicle Environment Perception, Lidar Sensing and Perception, Convolutional Neural Networks
Abstract: Light detection and ranging (LiDAR) technology plays an important role in achieving higher levels of autonomous driving. These sensors, although robust in clear weather conditions, including night scenes, tend to degrade in adverse weather conditions like fog, rain and snowfall. An autonomous vehicle relying on LiDAR should be able to assess in a real-time manner its limitations and raise an alarm in such scenarios. In this paper, we present a comprehensive statistical data analysis of the effects of real-world adverse weather conditions on the properties of LiDAR point clouds. Namely, we analyze the effect on range, reflectance and resolution of objects in point clouds recorded by LiDAR in challenging weather conditions. Furthermore, based on the results of the analysis, we propose RangeWeatherNet, a lightweight deep convolutional network architecture for classification of weather and road conditions. The classification accuracy of our network outperforms the existing methods by a large margin (+11.8%). The core network runs at 102 fps, which with the data pre-processing step, amounts to total 32 fps, which is higher than the usual LiDAR acquisition rate. To the best of our knowledge, this is the first approach that uses deep learning for classification of weather conditions on LiDAR point clouds.

Keywords: Radar Sensing and Perception, Deep Learning
Abstract: Radar has been employed as a key component of perception modules for more than two decades. However, radar image labeling requires expert knowledge. At the same time, it is much more time-consuming than for general RGB images, which impedes further developments of radar. In order to alleviate the high-cost annotation problem in radar datasets, we present a novel, semi-supervised deep learning method based on the spatio-temporal consistency. This way we explore the potential of unlabeled radar frames to enhance performance. We utilize the consecutive radar frames from different timeline directions to encourage the model to learn the target motion. Moreover, the proposed self-weighted mechanism avoids over-fitting on certain predominant targets, by exploiting the supervised classification loss dynamically. We evaluate the proposed method on semantic segmentation and Vulnerable Road Users (VRUs) detection problems. The quantitative results compare favourably to the state-of-the-art and demonstrate the effectiveness of the proposed concepts. The ablation studies also show the effectiveness of the proposed components.

Keywords: Radar Sensing and Perception, Image, Radar, Lidar Signal Processing, Vehicle Environment Perception
Abstract: Deep Neural Network (DNN) can provide highly accurate recognition for trained data, however, the estimation error for untrained data cannot be controlled. This is an essential challenge of generalizability in machine learning, and it is difficult to solve fundamentally. To solve this problem, we embed the inherent knowledge of the model to be estimated into the training model and impose parameter constraints to prevent deviation from the model so that generalizability is improved. Also, we trained the model with self-supervised learning using Variational Auto Encoder (VAE) on the training data, and evaluated the reconstruction error for the input data to judge the reliability of the estimation results. If the reliability is judged to be low, we switch to the deductive model, which does not cause large errors. This will suppress the maximum error. In this paper, we applied this method to the shape estimation of parking vehicles using a millimeter-wave radar. A parametric model with a minimum number of parameters was designed to represent the vehicle shape, which was trained by Convolutional Neural Network (CNN) to estimate the shape. The estimation reliability is predicted using the model trained with self-supervised learning by VAE. When the reliability is judged to be low, the position error is corrected based on the reflection pattern of a millimeter-wave radar. We confirmed that a large error for untrained data can be suppressed using actual parking data.

Keywords: Lidar Sensing and Perception, Vehicle Environment Perception, Deep Learning
Abstract: In autonomous driving, object detection and semantic segmentation are critical tasks for path planning and control of an autonomous vehicle. Recent approaches are based on supervised learning methods, with large datasets sampled in the target domain. However, annotating training data for supervised learning methods is a high resource and time-consuming task. In this work, we propose to exploit artificial LiDAR data for object detection and semantic segmentation. We use the CARLA simulator to generate artificial data of autonomous driving scenarios and propose ways to mitigate the differences between artificial and real-world data (domain generalization). We modeled both the noise and the missed reflections (denoted point dropout) that occur in real-world data collection, and show their effects in the detection and segmentation tasks. We assess the potential benefits of using pre-trained models on artificial data when fine-tuning with all, or a fraction, of the available real-world data (domain adaptation). We find clear improvements when using artificial data to pre-train a network, which allows to use a reduced amount of real-world data, and boost the performance of the trained models.

Keywords: Lidar Sensing and Perception, Mapping and Localization, Self-Driving Vehicles
Abstract: High-precision self-localization is one of the most important capabilities of automated vehicles. Not only accuracy but also localization robustness are crucial for self-driving vehicles in urban environments. The localization robustness decreases by misclassifications of landmarks and therefore false matches between dynamic objects and static landmarks listed in an a priori map. Here we show in the CARLA simulation environment, that the usage of semantic information prevents misclassifications of pedestrians as poles and so increases robustness in urban scenarios. In a simulated scenario of a road intersection pedestrians misclassified without semantic information could be filtered out by class label. In the presented experiments no mismatches of dynamic objects and map landmarks occurred and therefore the localization robustness was increased. Not only pole-like dynamic objects but also semistatic objects like parking cars or freight containers in terminal applications can be detected and excluded from map-based position estimation. The findings of this work show that the introduction of semantic class information leads to a higher self-localization robustness in urban scenarios and therefore should be included into current localization methods.

Keywords: Radar Sensing and Perception, Deep Learning, Advanced Driver Assistance Systems
Abstract: Localization and perception research for Autonomous Driving is mainly focused on camera and LiDAR data, rarely on radar data. We apply an automated labeling pipeline to semantically annotate real world radar measurements, manually correct point-wise labels to obtain ground-truth, and apply supervised learning models on this data. To assign an attribute, called class label, to every point of an input cloud is hereby referred to as semantic segmentation. Transferring approaches of LiDAR segmentation into the similar data structure, we research deep-learning semantic segmentation on radar point clouds. Compared to classical Cartesian coordinates, a polar coordinate input discretization benefits the dynamically changing number of radar detections per sensing cycle and simplifies to model the quasi-radial sensor resolution. Moreover, we evaluate different network architectures, examine radar feature channels and also temporal consistency by attention map concatenation. Our contribution is twofold. First, featuring a semantically labeled real world radar dataset for ground truth. Second, our supervised learning approach to solve semantic segmentation on radar point-cloud data. Our classification benchmark yields 56.1% weighted Intersection of a Union of relevant classes for radar, while reaching a real-time framerate of 12.4ms.

Keywords: Vehicle Environment Perception, Image, Radar, Lidar Signal Processing, Information Fusion
Abstract: This work gives a summary of our implemented algorithm for dynamic grid map with integrated cell-based velocity estimation. We propose a novel approach to generate prior information for the orientation of newly initialized particles. Our proposal does not rely on any additional input information as maps with the road topology, but does only rely on the vehicle motions and sensor data. Furthermore we show the improvement resulting from this prior information under certain conditions such as turning scenarios and that there is no significant degradation in other conditions.

Keywords: Image, Radar, Lidar Signal Processing, Lidar Sensing and Perception, Sensor and Data Fusion
Abstract: Accurate dense depth estimation is crucial for autonomous vehicles to analyze their environment. This paper presents a non-deep learning-based approach to densify a sparse LiDAR-based depth map using a guidance RGB image. To achieve this goal the RGB image is at first cleared from most of the camera-LiDAR misalignment artifacts. Afterward, it is over segmented and a plane for each superpixel is approximated. In the case a superpixel is not well represented by a plane, a plane is approximated for a convex hull of the most inlier. Finally, the pinhole camera model is used for the interpolation process and the remaining areas are interpolated. The evaluation of this work is executed using the KITTI depth completion benchmark, which validates the proposed work and shows that it outperforms the state-of-the-art non-deep learning-based methods, in addition to several deep learning-based methods.

Keywords: Image, Radar, Lidar Signal Processing, Convolutional Neural Networks
Abstract: The utilization of automatically generated image training data is a feasible way to enhance existing datasets, e.g., by strengthening underrepresented classes or by adding new lighting or weather conditions for more variety. Synthetic images can also be used to introduce entirely new classes to a given dataset. In order to maximize the positive effects of generated image data on classifier training and reduce the possible downsides of potentially problematic image samples, an automatic quality assessment of each enerated image seems sensible for overall quality enhancement of the training set and, thus, of the resulting classifier.

In this paper we extend our previous work on synthetic traffic sign images by assessing the quality of a fully generated dataset consisting of 215,000 traffic sign images using four different measures. According to each sample’s quality, we successively reduce the size of our training set and evaluate the performance with SVM and CNN classifiers to verify the approach. The comparability of real-world and synthetic training data is investigated by contrasting several classifiers trained on generated data to our baseline w.r.t. actual misclassifications during testing.

Keywords: Lidar Sensing and Perception, Vehicle Environment Perception, Automated Vehicles
Abstract: Semantic understanding of the surrounding environment is essential for automated vehicles. The recent publication of the SemanticKITTI dataset stimulates the research on semantic segmentation of LiDAR point clouds in urban scenarios. While most existing approaches predict sparse pointwise semantic classes for the sparse input LiDAR scan, we propose PillarSegNet to be able to output a dense semantic grid map. In contrast to a previously proposed grid map method, PillarSegNet uses PointNet to learn features directly from the 3D point cloud and then conducts 2D semantic segmentation in the top view. To train and evaluate our approach, we use both sparse and dense ground truth, where the dense ground truth is obtained from multiple superimposed scans. Experimental results on the SemanticKITTI dataset show that PillarSegNet achieves a performance gain of about 10% mIoU over the state-of-the-art grid map method.

Keywords: Lidar Sensing and Perception, Vehicle Environment Perception, Automated Vehicles
Abstract: Drivable area segmentation is an important feature for autonomous driving. Currently, state of the art techniques in this area focus on segmenting roads in urban areas with near perfect conditions. Roads with deteriorating conditions have received much less attention, even though they are common and present a unique set of challenges to the road segmentation tasks. These challenges include detecting obstacles (manholes and potholes) and determining whether or not it is safe to drive over them, and detecting road boundaries while lacking proper markings. This paper proposes a new method for drivable area segmentation in roads with deteriorating conditions based on a 3D LiDAR. Our framework represents the LiDAR point cloud data in an angular grid object which splits the data into smaller point cloud objects based on the laser scan number and the projection angle of each point. We apply multiple filtration steps in our framework in order to accurately detect the road boundaries and to detect and classify any road irregularities. The experiments on our collected datasets demonstrate the performance of our framework in detecting and classifying road drivable regions accurately and robustly. We reached a maximum precision of 92.78% in detection road boundaries and a max precision of 99.38% in detection and classifying road irregularities.

Keywords: Image, Radar, Lidar Signal Processing, Mapping and Localization, V2X Communication
Abstract: Visible light communication based intelligent transportation system (ITS-VLC) show great potential for future urban mobility. This study presents a performance evaluation of range estimation between vehicles and infrastructures in an ITS-VLC system. In the proposed ITS-VLC system, it is easy to simultaneously conduct communication and ranging using stereo cameras. However, the stereo camera calibration becomes a problem during simultaneous communication and ranging due to vehicle vibration. Using the data from LED transmitters and stereo cameras, it can obtain multiple measurements of distance. The monocular-stereo fusion algorithm is applied to visible light ranging in the proposed scheme using particle swarm optimization. We employed real data from the field trial experiment and achieved a ranging accuracy of 60 ± 1.0 m.

Keywords: Radar Sensing and Perception, Mapping and Localization, Vehicle Environment Perception
Abstract: This paper presents a robust detector of point-shaped landmarks by leveraging scattering information from polarimetric covariance radar gridmaps. Reliable landmarks are an essential part of any feature-based vehicle self-localization required for highly automated and autonomous driving. A polarimetric distance measure is used to extract cells with scattering behavior that is uniquely distinct from its neighborhood. Experiments on real-world data are evaluated and results from a single-, dual- and full-polarization utilization are compared. By association with a high definition map, a comparison of the suitability of different types of landmarks such as poles, guideposts and tree trunks for localization applications is made.

Keywords: Advanced Driver Assistance Systems, Image, Radar, Lidar Signal Processing, Vision Sensing and Perception
Abstract: We present a structured dataset for the research and development of automated driving systems (ADS) supported by hyperspectral imaging (HSI). The dataset contains per-pixel manually annotated images selected from videos recorded in real driving conditions that have been organized according to four environment parameters: season, daytime, road type, and weather conditions. The aim is to provide high data diversity and facilitate the automatic generation of data subsets for the evaluation of machine learning (ML) techniques applied to the research of ADS in different driving scenarios and environmental conditions. The video sequences have been captured with a small-size 25-band VNIR (Visible-NearInfraRed) snapshot hyperspectral camera mounted on a driving automobile. The current selection of classes for image annotation is aimed to provide reliable data for the spectral analysis of the items in the scenes; it is thus based on material surface reflectance patterns (spectral signatures). It is foreseen that future versions of the dataset will also incorporate alternative dense semantic labeling of the annotated images. The first version of the dataset, named HSI-Drive v1.0, is publicly available for download.

Keywords: Automated Vehicles, Lidar Sensing and Perception, Advanced Driver Assistance Systems
Abstract: Automated driving has been expected for decades. The first systems, which at least partially automate the vehicle, have been installed in higher priced vehicles for several years. In the near future, however, many more competencies are to be transferred to the systems and the vehicle will thus be fully automated. Such systems receive their data from various sensor systems such as Light Detection and Ranging (LiDAR). Therefore, it is essential that this information is transmitted correctly and reliably to the environmental perception system. In order to ensure this, redundancy of sensors is a key factor in addition to diversity. For example, multiple, independently controlled MEMS-based LiDAR systems can be operated synchronously. This requires the selection of a Master system which can be reliably followed by all Slave systems. In this publication, an architecture for MEMS-based Micro-Scanning LiDAR systems is proposed to determine the appropriate system as Master. The architecture has been implemented in an FPGA prototyping platform to demonstrate its feasibility and evaluate its performance.

Keywords: Autonomous / Intelligent Robotic Vehicles, Mapping and Localization, Information Fusion
Abstract: This paper presents a fully automatic large-scale point cloud mapping system for low-speed self-driving vehicles and robots operating in complicated unstructured environments. The proposed system robustly fuses multiple sensor inputs from IMU, RTK, wheel speed encoder, and LiDAR point clouds into a factor graph to obtain a globally consistent point cloud map. A robust two-stage optimization routine is proposed to tackle the practical issues that arise from real-world environments, such as handling unstable RTK signals, LiDAR degeneracy in structure-less areas, and cooperative mapping tasks. The system has been widely used for over 500 vehicles and 1,000 maps since 2019. We present a comparative evaluation with popular mapping algorithms in terms of accuracy and robustness to various challenging scenes.

Keywords: Mapping and Localization, Image, Radar, Lidar Signal Processing, Lidar Sensing and Perception
Abstract: Recently, various applications leveraging topological data analysis, in particular, persistent homology (PH), have been presented in many fields since PH provides a novel point cloud analysis method. In this work, we apply PH to LiDAR-based ego-vehicle localization applications. PH can extract translation and rotation invariant features from a point cloud. These features do not maintain local information of the point cloud, such as the edges and lines; however, they can abstract the global structure of the point cloud. A persistence image (PI) vectorizes the features and allows us to obtain fixed-size vectors despite the sizes of the source point clouds being different. Additionally, the size of the PI is not large even though the source point cloud is extremely big. We consider that these advantages are effective to loop closure detection, place categorization, and end-to-end global localization applications. Results reveal that it is difficult to improve the localization accuracy by simply applying PH owing to the basic concept of the topology that does not focus on exact shapes of the geometry. Therefore, we discuss how the advantages of PH can be utilized for the localization.

Keywords: Mapping and Localization, Sensor and Data Fusion, Autonomous / Intelligent Robotic Vehicles
Abstract: For fast and efficient vehicle localization, many researchers have used map matching algorithm to leverage inadequate traditional method of vehicle localization with HD map. The initialization process in map matching algorithm have always been problematic due to the nature of GNSS error, which might exceed the average lane width of approximately 3.5m. Thus, directly using the GNSS data to obtain the rough initial pose may cause the failure of the initialization process. The general solution is to randomly sample around the GNSS data and test the initialization with these hypotheses. However, this often leads to expensive computation and thus fails to run in real-time or online mode, as a dense sampling is required in order to achieve an acceptable level of initial estimation accuracy. As an viable alternative, we propose a multi-lane hypotheses approach to narrow down the search by limiting the sample pose to the number of lane within the error radius of the GNSS data. From these poses, we then perform an efficient map-matching to refine these pose candidates. Moreover, a novel belief function to evaluate the hypothesis is proposed to robustly select the best hypothesis for system initialization. Our evaluation result shows that we've outperformed the basic random sampling method in both accuracy and efficiency.

Keywords: Sensor and Data Fusion, Security, Intelligent Ground, Air and Space Vehicles
Abstract: Train localization based on the Global Navigation Satellite System (GNSS) is an important feature of the novel train control systems. Considering the complicated railway operation conditions, jamming signals from the environment may pose a severe threat to the GNSS-based train localization. Therefore, the accurate detection and perception of GNSS jamming will play a significant role in ensuring the safe operation of the trains. In this paper, a jamming identification method for GNSS-based train localization using singular value decomposition (SVD) is proposed. By extracting feature values from the singular value sequence, and modeling the mapping relationship between the feature values and the jamming characteristics, the discrimination of the jamming characteristics, including the type and the power of the jamming signal, is achieved. A satellite signal-level test platform with the jamming signal injection capability is built to verify the proposed solution. Results of the tests demonstrate the jamming recognition performance of the proposed solution under the Continuous Wave Interference (CWI), Linear Frequency Modulation (LFM) and the Band-limited White Noise (BLWN) jamming conditions.

Keywords: Mapping and Localization, Image, Radar, Lidar Signal Processing, Autonomous / Intelligent Robotic Vehicles
Abstract: This paper presents a vision-based metric localization method using pre-built point cloud maps. Matching the 3D structures reconstructed by visual SLAM to the point cloud map resolves the accumulative errors and scale ambiguity. In addition to the accuracy improvement, the proposed method achieves localization within given maps while ordinary visual SLAM constructs an on-line map and can only localize within this. Localization within a given map is crucial for autonomous driving, where various map types are employed. Point cloud maps are robust to appearance changes caused by illumination and seasonal changes. Once LiDAR sensors have built the point cloud maps, this paper demonstrates that localization is possible using solely low-cost and lightweight cameras. We verified the accuracy of the proposed method using real-world datasets. The results show that the accumulated error is suppressed even on extended vehicle trajectories. Also, we conducted experiments with various camera configurations and confirmed that the point cloud map improved the localization results for all configurations.

Keywords: Mapping and Localization, Automated Vehicles
Abstract: A high definition (HD) map, which provides prior knowledge to autonomous driving tasks, has been attracted in recent years. An HD map-based self-localization is a crucial technology for autonomous driving, but its accuracy is greatly affected by occlusions caused by dynamic obstacles in real environments. This paper focuses on clarifying the need for HD maps for stable self-localization in highly dynamic environments, especially in an urban canyon. By comparing occlusion effects with synthetically generated obstacles in a real environment, we show significant accuracy degradations in a general self-localization method due to obstacles in Shinjuku, Tokyo, Japan. In addition, we reveal that pole-like objects can be vital elements of an HD map to stabilize self-localization accuracy even with many obstacles by evaluating various patterns of high occlusion cases.

Keywords: Mapping and Localization, Autonomous / Intelligent Robotic Vehicles, Convolutional Neural Networks
Abstract: Visual Simultaneous Localization and Mapping is one of the main challenges for robotics and automated vehicles. In the state-of-the-art approaches, pixel-level correspondences are mostly used. In this paper, we address the problem of finding stable and repeatable pixel-level correspondences under challenging conditions. As a network basis, we use the D2-Net, which can select keypoints at positions that can be matched easily. Several techniques are implemented in this work to improve the keypoint detection and description performance. We first feed the network with rotated images to achieve a higher rotation invariance of point detections. Furthermore, we analyze the impact of a ranking of score values, adopting a cosine similarity and enforcing more dominant detections by defining a peakiness. The evaluation shows that combining a ranking with a peakiness can achieve the best result, especially to illumination changes. By using this combination, we achieve a mean matching accuracy increase of 12.5% on illumination scenes (9% overall) and a 9% higher repeatability rate at extremely low costs by only modifying the loss function.

Keywords: Vehicle Environment Perception, Convolutional Neural Networks, Lidar Sensing and Perception
Abstract: Evidential occupancy grid maps (OGMs) are a popular representation of the environment of automated vehicles. Inverse sensor models (ISMs) are used to compute OGMs from sensor data such as lidar point clouds. Geometric ISMs show a limited performance when estimating states in unobserved but inferable areas and have difficulties dealing with ambiguous input. Deep learning-based ISMs face the challenge of limited training data and they often cannot handle uncertainty quantification yet. We propose a deep learning-based framework for learning an OGM algorithm which is both capable of quantifying uncertainty and which does not rely on manually labeled data. Results on synthetic and on real-world data show superiority over other approaches. Source code and datasets are available at https://github.com/ika-rwth-aachen/EviLOG.

Keywords: Mapping and Localization, Vision Sensing and Perception, Autonomous / Intelligent Robotic Vehicles
Abstract: Speed bumps and expansion joints widely available on pavements cause pulse-like vibrations that can be measured by an in-vehicle accelerator. Such vibration patterns are called road-pulses in this paper, and we proposed a sliding Gaussian model (SGM) for detection. The detected road-pulses are associated with accurate positions and image features for road-pulse map construction. In the localization step, road-pulse patterns are detected and matched with the road- pulse map. And the corresponding image features and positions stored in the map allow accurate vehicle localization afterward. The proposed road-pulse-based localization is integrated into a downward pavement visual odometry with a Kalman filter based on the second-order Markov model (KF-MM2) for enhancing vehicle localization. The proposed method was validated on two different pavement conditions. Experimental results demonstrate that the proposed method can greatly improve vehicle localization with low cost and high feasibility.

Keywords: Mapping and Localization, Information Fusion
Abstract: Simultaneous localization and mapping (SLAM) is important to outdoor intelligent vehicle applications. Multi-vehicle cooperative SLAM which takes advantage of data sharing can outperform single vehicle SLAM. This paper proposes a multi-vehicle cooperative SLAM method using iterated split covariance intersection filter (Iterated Split CIF). In the proposed method, a vehicle can flexibly perform cooperative SLAM with other vehicles in decentralized way, without complicated monitoring and controlling of data flow. Moreover, the innovation and observation outliers caused by modeling errors or abnormal measurements can be solved reliably. A simulation-based comparative study demonstrates the potential and advantage of the proposed multi-vehicle cooperative SLAM using Iterated Split CIF in terms of accuracy and robustness.

Keywords: Mapping and Localization, Sensor and Data Fusion, Automated Vehicles
Abstract: A monocular VIO (Visual-Inertial Odometry) system provides a compact, low-cost, and easily-deployed configuration for relative localization. However, using thermal vision for VIO is much less studied than using a visible-spectrum camera. A thermal-vision camera works under all lighting conditions and has been used to detect pedestrians, cars and animals at nighttime to provide ADAS (Advanced Driver Assistance System) functions. Common problems in directly using thermal images in conventional VIO methods are: (1) lower signal-to-noise ratio and fewer reliable feature points in the texture-less thermal images, (2) periodic re-calibration hampers thermal image capturing and feature-point tracking, and (3) the ``jello" effect of the rolling shutter read-out architecture is sensitive to aggressive vehicle maneuvers. Extended VINS-Mono, proposed in our previous work, aims at providing relative and absolute localization of a vehicle in large-scale outdoor road environments by introducing (1) absolute localization methods to enable VINS-Mono to output local and global state estimates simultaneously, (2) vehicle speed readings for fast (re-)initialization and reliable scale estimates, and (3) DNN-based object detection methods to remove non-stationary objects from the visible scene. In this paper, we show that Extended VINS-Mono can use thermal images to provide relative and absolute localization even when light conditions are very poor. We conducted several experiments on a 25 Km-trip journey through highways, tunnels, urban areas and suburban areas in Pittsburgh, USA during daytime and nighttime to evaluate the performance of Extended Thermal VINS-Mono, including (re-)initialization, accuracy, rate, and latency. Our evaluation confirms t

Keywords: Mapping and Localization, Automated Vehicles
Abstract: High definition (HD) maps can fail by becoming outdated. To still use them safely for automated driving, they need to be verified or updated, both requiring methods for change detection. We propose two significant improvements for HD map change detection that do not require a highly accurate localization prior as localization quickly fails or cannot be trusted in an outdated map.

Given a very coarse localization prior, we group stored or measured map features in spatially and semantically separable feature groups. These feature groups are not only intuitive, like the sequence of leftmost dashed lane markings, but changes are also highly correlated within them.

The first contribution improves the way internal consistency of each feature group is assured by using boosted classification trees. Additionally, a mutual evaluation scheme is added for all seemingly unchanged feature groups. Always one feature group is used for localization by feature alignment while each other group’s alignment is checked for compatibility. Two voting schemes are presented that allow a more or less sensitive change detection on the level of proposed groups. In contrast to almost all other approaches, our approach allows to use still valid parts of the map for automated driving and to update the changed parts.

We evaluate our approach on a previously published map verification dataset, showing that the number of undetected map changes can be reduced by up to 31% compared to state of the art using boosted classification trees, at the same time reducing false positive rates by up to 50%. The additional mutual evaluation step is able to uncover a whole category of previously undetectable changes and reduces undetected changes by an extra 15%.

Keywords: Mapping and Localization, Automated Vehicles, Autonomous / Intelligent Robotic Vehicles
Abstract: Lane-level maps provide crucial detail to autonomous vehicle perception and decision making systems. Many common HD maps require human annotators to painstakingly label attributes such as lane geometry, connectivity, and speed limit. In this paper, we propose a method to derive these attributes automatically. Our method uses lane features and previous vehicle experience to generate potentials in a rectified raster map. We then infer the driving lanes within a Markov random field. Moreover, the lane topologies are extracted and represented by parsing the roadway into different segments based on the number of driving lanes and generating lane transitions between adjacent road segments. Lane attributes, such as speed limit and stop locations, are inferred from statistical analysis of previous vehicle experiences. Our approach is supported by experiments demonstrating effective automatic lane map generation on real-world roads.

Keywords: Autonomous / Intelligent Robotic Vehicles, Collision Avoidance, Active and Passive Vehicle Safety
Abstract: A solution for a tractor-trailer vehicle’s generic maneuver planning task can be introduced by an optimal control problem (OCP). However, the curse of dimensionality is excited along with the OCP solution due to the collision-avoidance constraints in a large scale. The collision-avoidance conditions are weakened by simply constructing a corridor along a homotopically guiding route such that the vehicle’s maneuvers are safely separated from obstacles. This approach is motivated by the safe flight corridor (SFC) applied for path planning of unmanned aerial vehicle (UAV). But SFC cannot be applied directly to the ground vehicle cases because a tractor- trailer vehicle cannot be modeled as a mass point in a narrow environment. An extension of the SFC is proposed, which requires different bodies of a multi-body vehicle to stay in different safe travel corridors. In this way, a reduced-scale OCP is formulated, and the problem scale becomes irrelevant to the environmental complexity. Simulation results illustrate that near-optimal maneuvers can be derived within less CPU time.

Keywords: Recurrent Networks, Situation Analysis and Planning, Hand-off/Take-Over
Abstract: In autonomous driving, complex traffic scenarios can cause situations that require human supervision to resolve safely. Instead of only reacting to such events, it is desirable to predict them early in advance. While predicting the future is challenging, there is a source of information about the future readily available in autonomous driving: the planned trajectory the car intends to drive. In this paper, we propose to analyze the trajectories planned by the vehicle to predict failures early on. We consider sequences of trajectories and use machine learning to detect patterns that indicate impending failures. Since no public data of disengagements of autonomous vehicles is available, we use data provided by development vehicles of the BMW Group. From over six months of test drives, we obtain more than 2600 disengagements of the automated system. We train a Long Short-Term Memory classifier with sequences of planned trajectories that either resulted in successful driving or disengagements. The proposed approach outperforms existing state-of-the-art failure prediction with low-dimensional data by more than 3% in a Receiver Operating Characteristic analysis. Since our approach makes no assumptions on the underlying system, it can be applied to predict failures in other safety-critical areas of robotics as well.

Keywords: Automated Vehicles, Vehicle Control, Collision Avoidance
Abstract: Motion planning and vehicle prediction play an important role for autonomous vehicle, which aims to guarantee the driving safety under cut-in scenarios. This paper presents a hybrid prediction model for computing the future trajectory of the surrounding vehicles and a dynamic speed planner based on model predictive control to avoid collisions. Firstly, the hybrid prediction model combines the physics-based model and behavior-based model through Mamdani fuzzy logic. The predicted physic trajectory is computed using the constant yaw rate and velocity vehicle model. In addition, the prediction of driving intention is accomplished by using information of the difference between current motion and driving lanes. Furthermore, the predicted behavior trajectory is selected from the candidate quintic polynomial trajectory cluster through the designed cost function. Then, Gaussian propagation is applied at the fusion trajectory to compute the uncertainty distribution. Secondly, the dynamic speed planner based on model predictive control provides the optimal control command for the collision avoidance maneuver, which considers the future trajectory distribution of surrounding vehicles. Finally, the effectiveness of the proposed method is verified through simulation in different cut-in scenarios.

Keywords: Reinforcement Learning, Situation Analysis and Planning, Autonomous / Intelligent Robotic Vehicles
Abstract: Data driven approaches for decision making applied to automated driving require appropriate generalization strategies, to ensure applicability to the world's variability. Current approaches either do not generalize well beyond the training data or are not capable to consider a variable number of traffic participants. Therefore we propose an invariant environment representation from the perspective of the ego vehicle. The representation encodes all necessary information for safe decision making. To assess the generalization capabilities of the novel environment representation, we train our agents on a small subset of scenarios and evaluate on the entire diverse set of scenarios. Here we show that the agents are capable to generalize successfully to unseen scenarios, due to the abstraction. In addition we present a simple occlusion model that enables our agents to navigate intersections with occlusions without a significant change in performance.

Keywords: Autonomous / Intelligent Robotic Vehicles, Automated Vehicles, Mapping and Localization
Abstract: It is evident that High-Definition Maps (HD Maps) are key enablers of autonomous driving systems, since they enable them to understand the nearby road structure in order to generate a localized corridor. However, the huge cost incurred in creating and maintaining HD Maps is being recognized as a big challenge. To deal with this problem, some researchers have been studying HD Map independent approaches such as online corridor estimation. They tend to require optimistic observations such as tracklets of surrounding vehicles. In this paper, we consider the use of self-driven trajectories, because they represent strong evidence that ego-vehicle could travel along. Based on this consideration, a concept of exemplary trajectory (or exemplar) generation by recombining parts of different trajectories is introduced. The recombination is achieved using genetic algorithm followed by a final selection. The crossover and selection operators are designed to preserve characteristics of parents; with the possibility to simultaneously maintain multiple exemplars covering different lanes. The final selection eliminates redundancy and provides the minimum set of exemplars. Our implementation demonstrated the effectiveness and usefulness of the proposed method.

Keywords: Autonomous / Intelligent Robotic Vehicles, Situation Analysis and Planning, Advanced Driver Assistance Systems
Abstract: Many everyday traffic situations require cooperation among traffic participants to establish the order in which they pass a shared part of the road. Behavior planners which do not take this cooperative aspect into account properly struggle to find efficient solutions if the situation is non-trivial. Improper modelling may lead to overly aggressive or conservative behavior. In this paper, we propose an extension to state-of-the-art systems that enables behavior planners to efficiently cope with narrow passage scenarios even without car-to-car communication. The extended system is based on an asymmetry measure which takes the shared traffic space and the cooperation partners into account. This measure is then used to continuously predict which potential outcome is more likely to occur, to infer the assumed strategy of the cooperation partner, and to match the own strategy accordingly. Experiments show that the proposed system significantly reduces the cumulative passing time of the shared traffic space as compared to baseline systems. The resulting solutions are robust against variations in the behavior of both cooperation partners, and explicitly account for oblivious traffic participants which behave uncooperatively.

Keywords: Situation Analysis and Planning, Collision Avoidance, Self-Driving Vehicles
Abstract: Ensuring the safety of autonomous vehicles (AVs) in uncertain traffic scenarios is a major challenge. In this paper, we address the problem of computing the risk that AVs violate a given safety specification in uncertain traffic scenarios, where state estimates are not perfect. We propose a risk measure that captures the probability of violating the specification and determines the average expected severity of violation. Using highway scenarios of the US101 dataset and Responsible Sensitive Safety (RSS) as an example specification, we demonstrate the effectiveness and benefits of our proposed risk measure. By incorporating the risk measure into a trajectory planner, we enable AVs to plan minimal-risk trajectories and to quantify trade-offs between risk and progress in traffic scenarios.

Keywords: Collision Avoidance, Self-Driving Vehicles, Vehicle Control
Abstract: The road's geometry strongly influences the path planner's performance, critical for autonomous navigation in high-speed dynamic scenarios (e.g., highways). Hence, this paper introduces the Curvature-aware Rapidly-exploring Random Trees (CA-CL-RRT), whose planning performance is invariant to the road's geometry. We propose a transformation strategy that allows us to plan on a virtual straightened road and then convert the planned motion to the curved road. It is shown that the proposed approach substantially improves path planning performance on curved roads as compared to prior RRT-based path planners. Moreover, the proposed CA-CL-RRT is combined with a Local Model Predictive Contour Controller (LMPCC) for path tracking while ensuring collision avoidance through constraint satisfaction. We present quantitative and qualitative performance results in two navigation scenarios: dynamic collision avoidance and structured highway driving. The results demonstrate that our proposed navigation framework improves the path quality on curved highway roads and collision avoidance with dynamic obstacles.

Keywords: Vehicle Control, Intelligent Ground, Air and Space Vehicles, Automated Vehicles
Abstract: The autonomous landing of a fixed-wing UAV on a moving UGV demands precise spatial synchronization of both vehicle systems. A promising strategy to achieve a successful landing in this heterogeneous rendezvous maneuver is the subdivision of the synchronization control task into a trajectory planner and underlying trajectory tracking controllers. For this purpose, a central trajectory planner, which is the subject of this paper, computes feasible trajectories for the UAV and UGV that converge towards each other. Current approaches mainly rely either on computationally expensive optimization problems or on potentially inaccurate linearization. In this paper, a new model predictive trajectory planning scheme based on the flatness property of kinematic models of the fixed-wing UAV and UGV is presented. By using vehicle models with this property, QPs are formulated which can be solved efficiently in each time instance without linearization of the nonlinear vehicle models, which is a novel approach in this application. Crucial requirements for the trajectory planner are that it reduces the horizontal distances between the vehicles to below 0.5m while maintaining safety constraints to allow safe landing, and a sampling rate of 40Hz to allow rapid replanning in case of disturbances. To overcome the issue of input constraint transformation between the original and the flat models, different approximation methods are investigated. Simulation results with simple kinematic vehicle models and high-fidelity dynamic models using underlying trajectory tracking controllers and navigation filters are presented that demonstrate that the proposed trajectory planning method allows safe landing maneuvers with the considered vehicle setup.

Keywords: Situation Analysis and Planning, Automated Vehicles, Self-Driving Vehicles
Abstract: To safely and effectively participate in road traffic, automated vehicles should explicitly consider compliance with traffic rules and high-level specifications. We propose a method that can incorporate traffic and handcrafted rules expressed in time-labeled propositional logic into our reachability analysis, which computes the over-approximative set of states reachable by vehicles. These reachable sets serve as low-level trajectory planning constraints to expedite the search for specification-compliant trajectories. Depending on the adopted specifications, related semantic labels are generated from predicates considering positions, velocities, accelerations, and general traffic situations. We exhibit the applicability of the proposed method with scenarios from the CommonRoad benchmark suite.

Keywords: Automated Vehicles, Situation Analysis and Planning, Vehicle Control
Abstract: To navigate complex driving scenarios, automated vehicles must be able to make decisions that reflect higher-level goals such as safety and efficiency, leveraging the vehicle’s full capabilities if necessary. We introduce an architecture that is capable of handling combinatorial decision making and control with a high fidelity vehicle model. This is accomplished by solving a nonlinear model predictive control optimization for each maneuver variant, or homotopy, identified in the drivable space. These locally optimal solutions are then evaluated on a criterion that reflects high-level objectives. Experimental results on a full-scale vehicle demonstrate this architecture’s effectiveness in an overtaking scenario with oncoming traffic that requires the ego vehicle to decide whether to pass before or after the oncoming traffic passes.

Keywords: Advanced Driver Assistance Systems, Deep Learning
Abstract: Understanding the behavior of road users is of vital importance for the development of trajectory prediction systems. In this context, the latest advances have focused on recurrent structures, establishing the social interaction between the agents involved in the scene. More recently, simpler structures have also been introduced for predicting pedestrian trajectories, based on Transformer Networks, and using positional information. They allow the individual modelling of each agent's trajectory separately without any complex interaction terms. Our model exploits these simple structures by adding augmented data (position and heading), and adapting their use to the problem of vehicle trajectory prediction in urban scenarios in prediction horizons up to 5 seconds. In addition, a cross-performance analysis is performed between different types of scenarios, including highways, intersections and roundabouts, using recent datasets (inD, rounD, highD and INTERACTION). Our model achieves state-of-the-art results and proves to be flexible and adaptable to different types of urban contexts.

Keywords: Reinforcement Learning, Autonomous / Intelligent Robotic Vehicles, Situation Analysis and Planning
Abstract: Overtaking the lead vehicle on two-way roads in the presence of several oncoming vehicles is a complex task for autonomous vehicles. In this paper, we formulate the overtaking behavior of an ego vehicle based on a deep reinforcement learning (DRL) method. First, a two-way urban road is created, wherein the ego vehicle aims to reach the destination safely and efficiently while considering multiple traffic participants. We use different intelligent driver model (IDM) parameters to account for different drivers’ habits. Furthermore, we introduce different responses of other vehicles when the ego vehicle takes overtaking maneuver. Then, a hierarchical control framework is proposed to manage vehicles on the road, which supervises vehicle behaviors at the high layer and controls the motion at the lower layer. The DRL method named Proximal Policy Optimization is applied to derive the high-level decision-making policies. A self-attention mechanism is further introduced to improve the performance of our algorithm. Finally, the overtaking maneuvers of the ego vehicle in different training timesteps are analyzed and how the responses of other vehicles affect the ego one’s overtaking behavior is investigated. Simulation results show that our approach can achieve good performance to deal with the two-way road autonomous overtaking task. Supplementary video is available at https://youtu.be/jPEGjM7cBuk.

Keywords: Collision Avoidance, Cooperative Systems (V2X), Automated Vehicles
Abstract: Connected and autonomous vehiles (CAVs) require an effective cooperative action planning strategy in an emergency situation. Monte Carlo Tree Search (MCTS) is a promising planning technique for such problems with large state spaces. However, traditional MCTS-based techniques do not scale well with the number of vehicles. In this paper, we present a novel MCTS-based cooperative action planning algorithm for CAVs driving in a coalition formation. Our proposed algorithm improves the reliability and the scalability of MCTS. Explicit communication is used to ensure that mitigation action plans chosen by the CAVs are conflict-free when possible. We perform the evaluation of the proposed algorithm in a large scale multi-agent based traffic simulation system. Our experiments show that our approach improves upon current state-of-the-art centralized and decentralized algorithms.

Keywords: Reinforcement Learning, Automated Vehicles
Abstract: In this survey, we systematically summarize the current literature on studies that apply reinforcement learning (RL) to the motion planning and control of autonomous vehicles. Many existing work can be attributed to the pipeline approach, which consists of many hand-crafted modules, each with a functionality selected for the ease of human interpretation. However, this approach does not automatically guarantee maximal performance due to the lack of a system-level optimization. Therefore, this paper also presents a growing trend of work that falls into the end-to-end approach, which typically offers better performance and smaller system scales. However, their performance also suffer from the lack of expert data and generalization issues. Finally, the remaining challenges applying deep RL algorithms on autonomous driving are summarized, and future research directions are also presented to tacked these challenges.

Keywords: Reinforcement Learning, Automated Vehicles, Self-Driving Vehicles
Abstract: Decision making under uncertainty can be framed as a partially observable Markov decision process (POMDP). Finding exact solutions of POMDPs is generally computationally intractable, but the solution can be approximated by sampling-based approaches. These sampling-based POMDP solvers rely on multi-armed bandit (MAB) heuristics, which assume the outcomes of different actions to be uncorrelated. In some applications, like motion planning in continuous spaces, similar actions yield similar outcomes. In this paper, we utilize variants of MAB heuristics that make Lipschitz continuity assumptions on the outcomes of actions to improve the efficiency of sampling-based planning approaches. We demonstrate the effectiveness of this approach in the context of motion planning for automated driving.

Keywords: Vision Sensing and Perception, Deep Learning, Self-Driving Vehicles
Abstract: Forecasting or anticipating the future scene is an important task in autonomous driving. The future scene, either in the form of visible images or semantic images, is predicted using current and past visible images. The forecast scene is used for planning, navigation and control tasks. However, visible camera images are susceptible to varying illumination, varying environmental and adverse weather conditions. Here, we address this limitation using a novel deep learning-based visible and thermal camera forecasting algorithm termed as the Seq2Seq. The Seq2Seq, a conditional GAN framework, forecasts the future visible-thermal camera scene using the current and past visible-thermal camera scene. The generator model is a deep sensor fusion based on the encoder-decoder architecture with a convolutional LSTM branch. The discriminator model is also a deep sensor fusion based on the patchGan architecture. The Seq2Seq is validated using the KAIST public dataset. The results show that the proposed framework can accurately forecast the future visible and thermal images. Moreover, we also demonstrate the application of the Seq2Seq, by performing semantic segmentation on the forecasted visible-thermal image using the MFNet.

Keywords: Lidar Sensing and Perception, Sensor and Data Fusion, Vehicle Environment Perception
Abstract: 3D object detection is a core component of automated driving systems. State-of-the-art methods fuse RGB imagery and LiDAR point cloud data frame-by-frame for 3D bounding box regression. However, frame-by-frame 3D object detection suffers from noise, field-of-view obstruction, and sparsity. We propose a novel Temporal Fusion Module (TFM) to use information from previous time-steps to mitigate these problems. First, a state-of-the-art frustum network extracts point cloud features from raw RGB and LiDAR point cloud data frame-by-frame. Then, our TFM module fuses these features with a recurrent neural network. As a result, 3D object detection becomes robust against single frame failures and transient occlusions. Experiments on the KITTI object tracking dataset show the efficiency of the proposed TFM, where we obtain ~6%, ~4%, and ~6% improvements on Car, Pedestrian, and Cyclist classes, respectively, compared to frame-by-frame baselines. Furthermore, ablation studies reinforce that the subject of improvement is temporal fusion and show the effects of different placements of TFM in the object detection pipeline. Our code is open-source and available at https://github.com/emecercelik/Temp-Frustum-Net.git.

Keywords: Sensor and Data Fusion, Mapping and Localization, Image, Radar, Lidar Signal Processing
Abstract: This paper presents a loosely-coupled multi-sensor fusion approach, which efficiently combines complementary visual and range sensor information to estimate the vehicle ego-motion. Descriptor-based and distance-based matching strategies are respectively applied to visual and range measurements for feature tracking. Nonlinear optimization optimally estimates the relative pose across consecutive frames and an uncertainty analysis using forward and backward covariance propagation is made to model the estimation accuracy. Covariance intersection filter paves the way for us to loosely couple stereo vision and LiDAR odometry considering respective uncertainties. We evaluate our approach with KITTI dataset which shows its robustness to fierce rotational motion and temporary lack of visual features, achieving the average relative translation error of 0.84% for the challenging 01 sequence.

Keywords: Deep Learning, Sensor and Data Fusion, Vision Sensing and Perception
Abstract: There is a growing interest in more intelligent natural user interaction with the car. Hand gestures and speech are already being applied for driver-car interaction. Moreover, multimodal approaches are also showing promise in the automotive industry. In this paper, we utilize deep learning for a multimodal fusion network for referencing objects outside the vehicle. We use features from gaze, head pose and finger pointing simultaneously to precisely predict the referenced objects in different car poses. We demonstrate the practical limitations of each modality when used for a natural form of referencing, specifically inside the car. As evident from our results, we overcome the modality specific limitations, to a large extent, by the addition of other modalities. This work highlights the importance of multimodal sensing, especially when moving towards natural user interaction. Furthermore, our user based analysis shows noteworthy differences in recognition of user behavior depending upon the vehicle pose.

Keywords: Vehicle Environment Perception, Vision Sensing and Perception, Radar Sensing and Perception
Abstract: Robust and reliable online 3D multi-object tracking is an essential component of autonomous driving. Recent research follows the tracking-by-detection paradigm and focuses mainly on lidar sensors, due to their superior range, resolution and depth accuracy compared to other automotive sensors. This simplifies the challenging data association in crowded urban road scenes, resulting in a predominant status of laser based methods. In contrast, we propose an online 3D multi-object tracker based solely on mono camera images and radar data to promote non-lidar based tracking research. By representing all detections of one frame as a Gaussian mixture model (GMM), we are able to avoid a fixed data association, which may include wrong assumptions. Instead, we assign the GMM to each tracked object and solve the data association implicitly and jointly by estimating the full 3D object tracks in our factor graph based optimization back end. By including all available information from the object detector, our algorithm achieves accurate, robust and reliable tracking results. We conduct real world experiments on the nuScenes tracking data set improving the state-of-the-art for non-lidar based methods from 17.7 % to 34.1 % AMOTA.

Keywords: Automated Vehicles, Advanced Driver Assistance Systems, Vehicle Environment Perception
Abstract: Development and testing of automated driving functions is complex and costly. Experts accord that it is necessary to cover most of the process by simulation. While tools and standards are evolving to satisfy this need, it is still challenging to generate appropriate driving scenarios for the simulation. In this paper we present a method for processing real driving data in order to generate maneuver-based scenarios for resimulation. We propose an automatic extraction of sequential, parametrized maneuvers - expressed in a high level format such as OpenSCENARIO. This enables to intuitively vary maneuver parameters and automatically generate whole sets of new discrete test scenarios, providing a link between simulation and real driving tests. The application of the method shows promising results in respect to the creation of meaningful scenarios with little loss in precision at reproducing the original driving tests.

Keywords: Automated Vehicles, Traffic Flow and Management, Vehicle Environment Perception
Abstract: Considering the recent advances in autonomous driving technology, conventional on-road based testing tools cannot meet the validation requirements with respect to time and cost-efficiency. Scenario-based virtual validation methods offer an efficient virtual testing approach that supports the development and contribute to decreased on-road-testing. X-in-the loop testing methods contribute to stepwise increase in test/validation quality by introducing hardware/software elements starting from the component up to full vehicle level and have been increasingly integrated into the process of design and validation of intelligent vehicle systems. In this paper, a novel framework that includes a highly dynamic vehicle-in-the-loop test bed, a traffic flow simulation method is introduced to precisely assess the impact of specific variables on the performance of an intelligent system with all vehicle components included.

Keywords: Automated Vehicles, Advanced Driver Assistance Systems, Vehicle Environment Perception
Abstract: For testing and validation of automated driving functions, simulations are absolutely essential to manage the required test effort. Therefore, the simulation models need to be modeled adequately in order to use the simulation results for virtual validation. As the required accuracy in virtual environment models is not clearly defined, this contribution investigates and quantifies accuracy requirements for the static domain of virtual environment models. By the use of an appropriate sensitivity analysis and a unique metric for the evaluation of simulation results suitable parameters are identified and statistically analyzed for validity and sensitivity assessment for a highway scenario. The results reveal that influences on the creation of virtual environment descriptions for automated driving could be derived and used for defining requirements in the generation and updating of virtual test fields.

Keywords: Self-Driving Vehicles, Automated Vehicles, Vulnerable Road-User Safety
Abstract: This paper introduces a method to validate autonomous navigation frameworks, in simulation using CARLA Simulator, fulfilling the requirements of the Euro-NCAP evaluation. We propose the protocol for evaluating an unexpected pedestrian scenario, where a walker suddenly invades the road and the vehicle has to react in a safe way. Standard validation metrics are created for this use case, which are generalizables for other use cases. To support the proposal, we describe our ROS (Robot Operating System) based Self-Driving architecture, open source and implemented in an electric vehicle. Then, we explain the procedures and requirements needed for the validation protocol that we propose. Finally, we show the metrics and results obtained in simulation for different ego-vehicle velocities and weather conditions. The scenarios implemented in Carla are publicly available.

Keywords: Cooperative ITS, Traffic Flow and Management, Automated Vehicles
Abstract: Several approaches have been presented during the last decades to reduce carbon pollution from transportation. One example is the use of platooning mode. This paper considers data obtained from daily trips to investigate the impact of platooning on travel time, emissions of CO2, CO, NO2 and HC and fuel consumption on a road network in Upper Austria. For this purpose, we studied fuel combustion-based engines relying on the extension of the 3DCoAutoSim simulation platform. The obtained results showed that the platooning mode not only increased driving efficiency but also decreased the total emissions by reducing fuel consumption.

Keywords: Societal Impacts, Autonomous / Intelligent Robotic Vehicles, Intelligent Ground, Air and Space Vehicles
Abstract: In 2016, the German Ministry for Traffic and Digital Infrastructure has granted 100 million euros on autonomous and connected vehicles. Soon after, in 2017, the German Ministry appointed Ethics Commission to regulate the use of such vehicles for social good. They identified transparency, trust, and safety as vital features for this purpose. Transparency requires that information about vehicles is available to customers, e.g. locations, fees, etc. Trust requires that vehicles are used in a fair and efficient manner. Safety requires that vehicles follow all road regulations and customers have top priority in cases of potential accidents. In this paper, we give scientific guarantees that respond to these requirements by providing preliminary fairness and efficiency analyses of the fleet of vehicles.

Keywords: Automated Vehicles, Self-Driving Vehicles, Advanced Driver Assistance Systems
Abstract: State-of-the-art route guidance systems aim at finding the optimal route based on the total estimated travel time. With the development of autonomous vehicles certain road links can already be driven autonomously. When obtaining route guidance travelers want to drive autonomously as long as possible. Thus, additional criteria with respect to autonomous driving (AD) need to be taken into account. This paper aims at finding the optimal route for AD. Besides the total travel time, we introduce the autonomous travel time and the number of take over requests by the vehicle to the driver as additional optimization criteria. Two methodologies are developed, which approach the problem of finding optimal AD routes. The first one is based on subjective perception and the second one is formulated as a multiobjective optimization problem. A case study is conducted and both methods are compared. It is shown that an optimal route for AD can be found uniquely.

Keywords: Human-Machine Interface, Cooperative ITS, Self-Driving Vehicles
Abstract: In this research, we proposed an active management method to tele-monitor and tele-operate more autonomous vehicles (AVs) with few observers by adjusting the movement of the AVs actively. A management system is created to get the status from the AVs and separate the monitoring requirement to the observers optimally. When the requirements might be intensive, the management system can also adjust the movements of the AVs actively to distribute the monitoring time, which can make the observers monitor more vehicles. We implement and verify the management system in an autonomous driving simulator - CARLA for the limited number of AVs and observers. Based on the data acquired from the driving simulator, we also create a numerical simulator and tested our method with the pairs of a large number of AVs and observers. The result of both simulators shows that our method can reduce the utilization degree of observers and make them monitor more AVs.

Keywords: Automated Vehicles, Cooperative Systems (V2X), Vehicle Environment Perception
Abstract: Safe and reliable automated driving is one key towards the future of mobility. We use smart road side infrastructure, which increases the field of view of road users, to step-up safety and reliability. We validate our claims regarding safety and reliability by extensive use of virtual testing. In this article we show our hardware-in-the-loop AV Testing Framework. We use a real vehicle computer, which is running open source planning and control algorithms, to demonstrate our findings. This computer is integrated in our testing framework, where we conduct complex urban traffic simulations with varying traffic demands and different traffic situations. The Field of View of the tested Automated Vehicle is increased using additional sensors mounted in the infrastructure. With road side infrastructure support, the 180 virtual test drives on an urban intersection, show a better performance in traffic efficiency and driving comfort. The simulated vehicle did not get stuck or was involved in collisions at any time.

Keywords: Advanced Driver Assistance Systems, Cooperative ITS, Electric and Hybrid Technologies
Abstract: With improvements to ranges and a growing charging infrastructure, electric vehicles are becoming increasingly popular. However, many prospective owners can not charge their vehicle at home. They thus would have to use public charging infrastructure, which reduces the appeal of electric vehicles to them due to the extra effort and time spent during charging. Many previous works tried to enhance this situation by using intelligent charging station scheduling and/or route planning to reduce the time overhead. But most solutions are focused on long distance travel or single trips without considering the often regular schedule of drivers in urban areas. We propose a charge stop planner that takes into account the activities of a driver's daily schedule to minimize the additional time the driver has to spend charging the vehicle. The charging stops can either be en-route charging at fast charging stations between two activities or destination charging at slow charging stations. To reduce waiting times, we coordinate the charging station visits between the vehicles with a centralized service. In an extensive set of simulation experiments, we demonstrate that our approach reduces the additional time for charging by about 40 %, compared to only destination charging and only en-route charging. The charging station coordination further reduces the waiting time by about 50 %.

Keywords: Impact on Traffic Flows, Traffic Flow and Management, Assistive Mobility Systems
Abstract: The introduction of connected and autonomous vehicles (CAVs) could bring practical solutions to the existing challenges with transportation infrastructures such as accidents and congestion. However, the transition to the era of CAVs would be gradual, and it could be expected that both CAVs and human-driven vehicles (HDVs) would exist in the network for some time, which could change the fundamental properties of urban networks. In this paper, the impact of CAVs on macroscopic fundamental diagram (MFD) is analyzed with microscopic traffic simulations, and the sensitivity analysis of market penetration rates of CAVs and network configurations is conducted. The analysis shows that one-way grid networks offer the most accessible and resilient environment during various phases of CAV introduction. Moreover, the introduction of CAVs not only improves the aggregated network performance but also improves the accessibility (trip completion rate) of regular HDVs.

Keywords: Traffic Flow and Management, Cooperative ITS, Vehicle Control
Abstract: This paper proposed a speed harmonization controller for partially connected and automated traffic. It regulated the flow rate of the entire traffic by adjusting only the target cruising speed of Connected and Automated Vehicles (CAVs). The major breakthrough of the proposed controller is that it is able to manage mesoscopic level traffic by controlling microscope level status (desired speed) of a small portion of vehicles. To evaluate the proposed controller, a VISSIM based microscopic simulation evaluation was conducted. Sensitivity analysis was performed for CAV Penetration Rate (PR) and demand level (v/c ratio). Results confirm that the control accuracy of the proposed controller is over 85% across all CAV PRs and demand levels.

Keywords: Smart Infrastructure, Image, Radar, Lidar Signal Processing, Information Fusion
Abstract: Urban Traffic Surveillance (UTS) is a surveillance system based on a monocular and calibrated video camera that detects vehicles in an urban traffic scenario with dense traffic on multiple lanes and vehicles performing sharp turning maneuvers. UTS then tracks the vehicles using a 3D bounding box representation and a physically reasonable 3D motion model relying on an unscented Kalman filter based approach. Since UTS recovers positions, shape and motion information in a three-dimensional world coordinate system, it can be employed to recognize diverse traffic violations or to supply intelligent vehicles with valuable traffic information. We build on YOLOv3 as a detector yielding 2D bounding boxes and class labels for each vehicle. A 2D detector renders our system much more independent to different camera perspectives as a variety of labeled training data is available. This allows for a good generalization while also being more hardware efficient. The task of 3D tracking based on 2D detections is supported by integrating class specific prior knowledge about the vehicle shape. We quantitatively evaluate UTS using self generated synthetic data and ground truth from the CARLA simulator, due to the non-existence of datasets with an urban vehicle surveillance setting and labeled 3D bounding boxes. Additionally, we give a qualitative impression of how UTS performs on real-world data. Our implementation is capable of operating in real time on a reasonably modern workstation. To the best of our knowledge, UTS is to date the only 3D vehicle tracking system in a surveillance scenario (static camera observing moving targets).

Keywords: Vehicle Control, Electric and Hybrid Technologies
Abstract: the present work proposes a novel state observer of the brake linings’ coefficient of friction (BLCF) based on vehicle longitudinal dynamics and wheel rotational dynamics without additional sensors. Kalman filter is adopted to polish the input signals of the observer such as the inertia measurement unit (IMU), vehicle speed, deceleration and brake pressure. The evolution of the BLCF is acquired and analyzed by vehicle test under different vehicle speed and brake pressure through the observer. A table-based model of the BLCF is proposed approximating the evolution of the BLCF as a linear function with time. Finally, this model is verified by vehicle test under regular driving conditions.

Keywords: Vehicle Control, Reinforcement Learning, Advanced Driver Assistance Systems
Abstract: Reinforcement Learning is a promising method for automated tuning of controllers, but is yet rarely applied to real systems like longitudinal vehicle control, since it struggles in the face of real-time tasks, noise, partially observed dynamics and delays. We propose a model-based reinforcement learning algorithm for the task of speed tracking control on constrained hardware. In order to cope with partially observed dynamics, delay and noise our algorithm relies on an autoregressive model with external inputs (ARX model) that is learned using a decaying step size. The output controller is updated by policy search on the learned model. Multiple experiments show that the proposed algorithm is capable of learning a controller in a real vehicle in different speed ranges and with a variety of exploration noise distribution and amplitudes. The results show that the proposed approach yields similar results to a recently published model-free reinforcement learning method in most conditions, e.g. when adapting the controller to very low speeds, but succeeds to learn with a wider variety of exploration noise types.

Keywords: Vehicle Control, Automated Vehicles, Reinforcement Learning
Abstract: The automated steering controller is crucial for smooth and accurate path tracking of autonomous ground vehicles (AGVs). However, time-varying uncertainties and disturbances may deteriorate the path tracking performance. Moreover, it is difficult for the steering system to strictly follow the desired steering angle in practice. Therefore, this paper proposes an automated steering control algorithm consisting of two parts: 1) an output feedback model predictive controller (MPC) to solve the path tracking problem, which is formulated as an optimization problem in this paper, with strong robustness against time-varying uncertainty and disturbance; 2) a feedforward compensator for the steering angle calculated by MPC using deep deterministic policy gradient (DDPG) algorithm so that the steering system can execute the desired steering angle more quickly and more accurately. Simulation results demonstrate that the proposed control scheme can significantly improve response speed and accuracy for path tracking of AGVs with strong robustness.

Keywords: Automated Vehicles, Vehicle Control
Abstract: In a rally competition, professional drivers usually adopt a very aggressive strategy. Agile maneuvering such as 'drifting' often occurs during the cornering process. In this paper, a controller framework is present for the vehicle's agile maneuver based on Model Predictive Control (MPC). We introduce the 3-state vehicle model and the brush tire model and analyze the trajectory characteristics of drift equilibrium. The proposed control system can track the variable drift state and the reference path simultaneously and is appliable for both regular driving and drift cornering. Different from the previous studies on drift stability or drift cornering, the system can not only realize lane keeping in complex scenarios but also achieve autonomous drift maneuvering during the cornering process. The effectiveness of the system is validated via simulations on the Matlab-Carsim platform.

Keywords: Self-Driving Vehicles, Vehicle Control, Collision Avoidance
Abstract: A computationally cheap method for computing collision-free trajectories with multiple moving obstacles is proposed here while meeting comfort and safety criteria. By avoiding search in the trajectory calculation and instead using a geometrical set to calculate the trajectory, the calculation time is significantly reduced. The geometrical set is calculated by solving a support vector machine problem and solving the SVM problem characterizes maximum separating surfaces between obstacles and the ego vehicle in the time-space domain. The trajectory on the separating surface might not be kinematically feasible. Therefore, a vehicle model and a Newton-Raphson based procedure is proposed to obtain a safe, kinematically feasible trajectory on the separating surface. A roundabout scenario and two take-over scenarios with different configurations are used to investigate the properties of the proposed algorithm. Robustness properties of the proposed algorithm is investigated by a large number of randomly initiated simulation scenarios.

Keywords: Vehicle Control, Self-Driving Vehicles, Automated Vehicles
Abstract: One central challenge for self-driving cars is a proper path-planning. Once a trajectory has been found, the next challenge is to accurately and safely follow the precalculated path. The model-predictive controller (MPC) is a common approach for the lateral control of autonomous vehicles. The MPC uses a vehicle dynamics model to predict the future states of the vehicle for a given prediction horizon. However, in order to achieve real-time path control, the computational load is usually large, which leads to short prediction horizons. To deal with the computational load, the control algorithm can be parallelized on the graphics processing unit (GPU). In contrast to the widely used stochastic methods, in this paper we propose a deterministic approach based on grid search. Our approach focuses on systematically discovering the search area with different levels of granularity. To achieve this, we split the optimization algorithm into multiple iterations. The best sequence of each iteration is then used as an initial solution to the next iteration. The granularity increases, resulting in smooth and predictable steering angle sequences. We present a novel GPU-based algorithm and show its accuracy and real-time abilities with a number of real-world experiments.

Keywords: Vehicle Control, Cooperative Systems (V2X), Situation Analysis and Planning
Abstract: With the increasing railway transportation demands, railway operators need to enhance the capacity in high traffic corridors to improve transportation efficiency. The establishment of the concept "virtual coupling (VC)" provides an effective method for achieving higher capacity. The core idea is to reduce the train separation and to enable a maximized number of trains on the track. Based on the emerging modern communication techniques, the synchronous operation between different trains to form the coupled state can be achieved without any physical connection. In this paper, a following-train speed profile optimization model is proposed by employing Mixed Integer Linear Programming (MILP) approach to achieving VC with the leading train. Therefore, the safety constraints (i.e. moving block signaling (MBS)) are considered as the most important physical border between each train under the VC operation. Three cases under different scenarios are conducted on a generic high-speed rail, which proves the effectiveness of the proposed method on the speed profile optimization for following trains under the VC operation. It also shows the factors influencing the time headway under minimized-separation operations.

Keywords: Vehicle Control, Automated Vehicles
Abstract: This research proposes a Model Predictive Control (MPC) based path tracker controller. It is designed for maneuvering an autonomous driving vehicle to follow its desired trajectory smoothly and accurately. The proposed path tracker has the following features: i) formulated in the time and space mixed-domain to improved control accuracy ii) with consideration of vehicle dynamics; iii) with consideration of vehicle control delay. Simulation and field test results demonstrate that the maximum longitudinal speed error is 2.3km/h and the maximum lateral position error is 11cm. It is 27% smaller than that of the conventional path-trackers. Moreover, the average computation time of the proposed path-tracker is 12 milliseconds on a laptop equipped with an Intel i7-4710MQ CPU. It indicates that the proposed path tracker is ready for real-time implementation.

Keywords: Vehicle Control, Self-Driving Vehicles, Automated Vehicles
Abstract: This study compares and evaluates the effect of the choice of the vehicle’s prediction model on the performance in designing a path-tracking controller for vehicles using Model Predictive Control (MPC). The Kinematic Ackermann Model (KAM), the Kinematic Bicycle Model (KBM), and the Dynamic Bicycle Model (DBM) are well known as nonlinear prediction models. The stability and tracking performance of these models are evaluated using simulations, and a newly proposed DBM improved in low-speed range (DBM-L) is also compared. As a result of the simulation, the proposed DBM-L was able to run in the widest 0 to 120km/h speed range among the models tested, and it was able to achieve the stop-and-go behavior that was not possible with the conventional DBM. In the future, if we can solve the problem that the tracking accuracy of the DBM-L is slightly decreased in the extremely low and high speed ranges, a vehicle prediction model that can be used in all speed ranges is expected to be realized.

Keywords: Self-Driving Vehicles, Reinforcement Learning, Vehicle Control
Abstract: This paper presents a Deep Reinforcement Learning (DRL) framework adapted and trained for Autonomous Vehicles (AVs) purposes. To do that, we propose a novel software architecture for training and validating DRL based control algorithms that exploits the concepts of standard communication in robotics using the Robot Operating System (ROS), the Docker approach to provide the system with portability, isolation and flexibility, and CARLA (CAR Learning to Act) as our hyper-realistic open-source simulation platform. First, the algorithm is introduced in the context of Self-Driving and DRL tasks. Second, we highlight the steps to merge the proposed algorithm with ROS, Docker and the CARLA simulator, as well as how the training stage is carried out to generate our own model, specifically designed for the AV paradigm. Finally, regarding our proposed validation architecture, the paper compares the trained model with other state-of-the-art traditional control approaches, demonstrating the full strength of our DL based control algorithm, as a preliminary stage before implementing it in our real-world autonomous electric car.

Keywords: Vehicle Control, Automated Vehicles
Abstract: Reversing general 2-trailers is often necessary, but challenging and time-consuming even for professional drivers. Existing approaches in literature attempt to automate such maneuvers based on ideas from control theory. In this work, we propose a fundamentally novel approach for finding steering functions to arrive at a given configuration of vehicle internal angles. Our method is based on a simple steering function model with only two degrees of freedom, and we assume that it models paths of optimally short length for every reachable configuration. To find such a solution, a novel search algorithm is presented as well. Inspired by bisection, it applies a similar idea in 2D parameter space using a mesh graph which is transformed into a topologically equivalent graph in configuration space. The solution is found by an iterative refinement process. The algorithm can be executed multiple times per second on a Raspberry Pi 4, and the resulting steering functions and paths are simple and short.

Keywords: Self-Driving Vehicles, Vehicle Control, Collision Avoidance
Abstract: This work proposes a novel framework for motion planning using trajectory optimization for autonomous driving. First, a two-phase behavioral policy maker (BPM) is proposed as a high-level decision maker to mimic human-like driving style by avoiding unnecessary tasks and early lane changes. Second, a comprehensive study on iterative adaptive weight tuning functions has been done to limit manual weight tuning in the Constrained Iterative Linear Quadratic Regulator (CILQR) motion planner. Third, a jerk-minimized CILQR is presented to ensure the comfort and safety of passengers by generating smooth trajectories. The simulation results show efficiency, safety, and comfort of generated trajectories.

Keywords: Vehicle Control, Autonomous / Intelligent Robotic Vehicles, Self-Driving Vehicles
Abstract: This paper proposes a novel method for reactive multiagent collision avoidance by characterizing the longitudinal and lateral intent uncertainty along a trajectory as a closed-form probability density function. Intent uncertainty is considered as the set of reachable velocities in a planning interval and distributed as a Gaussian distribution over the robot's instantaneous velocity. We utilize the Time Scaled Collision Cone(TSCC) approach, which characterizes the space of instantaneous collision avoidance velocities available to the ego-agent. We introduce intent uncertainty into the characteristic equation of the TSCC to derive the closed-form probability density function, which allows the collision avoidance problem to be rewritten as a deterministic optimization procedure. The formulation also allows the flexibility for the inclusion of confidence intervals for collision avoidance. We thus demonstrate the results and ablation studies of this derived collision avoidance formulation on various confidence intervals.

Keywords: Vision Sensing and Perception, Deep Learning
Abstract: Common domain shift problem formulations consider the integration of multiple source domains, or the target domain during training. Regarding the generalization of machine learning models between different car interiors, we formulate the criterion of training in a single vehicle: without access to the target distribution of the vehicle the model would be deployed to, neither with access to multiple vehicles during training. We performed an investigation on the SVIRO dataset for occupant classification on the rear bench and propose an autoencoder based approach to improve the transferability. The autoencoder is on par with commonly used classification models when trained from scratch and sometimes out-performs models pre-trained on a large amount of data. Moreover, the autoencoder can transform images from unknown vehicles into the vehicle it was trained on. These results are corroborated by an evaluation on real infrared images from two vehicle interiors.

Keywords: Deep Learning, Unsupervised Learning, Situation Analysis and Planning
Abstract: Detecting unknown and untested scenarios is crucial for scenario-based testing. Scenario-based testing is considered to be a possible approach to validate autonomous vehicles. A traffic scenario consists of multiple components, with infrastructure being one of it. In this work, a method to detect novel traffic scenarios based on their infrastructure images is presented. An autoencoder triplet network provides latent representations for infrastructure images which are used for outlier detection. The triplet training of the network is based on the connectivity graphs of the infrastructure. By using the proposed architecture, expert-knowledge is used to shape the latent space such that it incorporates a pre-defined similarity in the neighborhood relationships of an autoencoder. An ablation study on the architecture is highlighting the importance of the triplet autoencoder combination. The best performing architecture is based on vision transformers, a convolution-free attention-based network. The presented method outperforms other state-of-the-art outlier detection approaches.

Keywords: Active and Passive Vehicle Safety, Deep Learning, Vision Sensing and Perception
Abstract: Many mobile robotic platforms rely on an accurate knowledge of the extrinsic calibration parameters, especially systems performing visual stereo matching. Although a number of accurate stereo camera calibration methods have been developed, which provide good initial “factory” calibrations, the determined parameters can lose their validity over time as the sensors are exposed to environmental conditions and external effects. Thus, on autonomous platforms on-board diagnostic methods for an early detection of the need to repeat calibration procedures have the potential to prevent critical failures of crucial systems, such as state estimation or obstacle detection. In this work, we present a novel data-driven method to estimate the quality of extrinsic calibration and detect discrepancies between the original calibration and the current system state for stereo camera systems. The framework consists of a novel dataset generation pipeline to train CalQNet, a deep convolutional neural network. CalQNet can estimate the extrinsic calibration quality using a new metric that approximates the degree of miscalibration in stereo setups. We show the framework’s ability to predict the divergence of a state-of-the-art stereo-visual odometry system following a degraded calibration in two real-world experiments.

Keywords: Vehicle Environment Perception, Vision Sensing and Perception, Convolutional Neural Networks
Abstract: A self-supervised method to train a visual predictor of drivable surface roughness in front of a vehicle is proposed. A convolutional neural network taking a single camera image is trained on a dataset labeled automatically by a cross-modal supervision. The dataset is collected by driving a vehicle on various surfaces, while synchronously recording images and accelerometer data. The surface images are labeled by the local roughness measured using the accelerometer signal aligned in time. Our experiments show that the proposed training scheme results in accurate visual predictor. The correlation coefficient between the visually predicted roughness and the true roughness (measured by the accelerometer) is 0.9 on our independent test set of about 1000 images. The proposed method clearly outperforms a baseline method which has the correlation of 0.3 only. The baseline is based on surface texture strength without any training. Moreover, we show a coarse map of local surface roughness, that is implemented by scanning an input image with the trained convolutional network. The proposed method provides automatic and objective road condition assessment, enabling a cheap and reliable alternative to manual data annotation, which is infeasible in a large scale.

Keywords: Vision Sensing and Perception, Vehicle Environment Perception, Convolutional Neural Networks
Abstract: We propose a modular framework for 3D tracking not only of paved roads but also of dirt roads. It is based on recursive state estimation of lane boundary points connected by clothoid pieces. While our tracking is flexible to integrate every kind of measurement, we specifically propose two image-based measurements. They combine traditional with modern computer vision: On the one hand, we show how to use directed edge detection to robustly measure road and lane boundaries. On the other hand, we introduce a innovative CNN-based measurement utilizing the self-similarity of (dirt) road areas.

We demonstrate the performance of our approach in challenging scenarios. On a marked road, we achieve a median error of 0.13m for the ego lane's boundaries in 25m look-ahead. A difficult dirt road can also be tracked reliably with a look-ahead length of 25m, resulting in a median error of 0.3m. The tracking, as well as both measurements, are real-time capable.

Keywords: Self-Driving Vehicles, Convolutional Neural Networks, Vision Sensing and Perception
Abstract: Lane detection is one of the most important task in autonomous driving. While the semantic segmentation based method is widely explored and recognized in recent decade, some post-processing are required to estimate the exact location of the predicted lane markings and can be easily failed in complex scenarios. To tackle these limitations, this paper proposes a novel lane detection network named PCRLaneNet. Firstly, we use a fully convolutional network to predict the coordinates of lane marking points directly, which can better meet with the requirements of autonomous driving. Secondly, to take the fully advantage of the correlation of these lane marking points, a point feature fusion strategy is designed to fuse feature maps of the points on the same lane marking, which makes our method capable of handling challenging scenarios. Lastly, the robustness, accuracy and latency of the proposed method are extensively verified in two datasets (CULane and TuSimple).

Keywords: Vehicle Environment Perception, Autonomous / Intelligent Robotic Vehicles, Convolutional Neural Networks
Abstract: As a recently proposed task, video instance segmentation (VIS) can classify, detect, segment and track each instance in a given video, which is very useful for driving environment perception of autonomous vehicles. In this paper, we propose a novel method called bi-directional attention feature enhancement (BAFE) to make up for the lack of feature processing in existing works of VIS. BAFE contains a top-down attention branch and a bottom-up attention branch. It uses attention to perform two-way information transmission between high-level, semantic features and low-level, local features to combine them efficiently and can be utilized in any network that uses both high-level and low-level features. We add BAFE before the mask-specialized regression branch of the latest work, spatial information preservation for VIS (SipMask-VIS), to enhance features for mask generating. Besides, we introduce modified path aggregation network (Mod-PAN) to further enhance features. Different from existing works which first train their models on image datasets and then use the pre-trained models to finish the VIS task, we obtain VIS results in an end-to-end way without any pre-training. Our method outperforms SipMask-VIS by an absolute gain of 2.5%, which strongly proves its effectiveness.

Keywords: Vision Sensing and Perception, Convolutional Neural Networks, Deep Learning
Abstract: The field of continual deep learning is an emerging filed and a lot of progress has been made. However, concurrently most of the approaches are only tested on the task of image classification, which is not of much use in the field of intelligent vehicles. Only recently approaches for class-incremental semantic segmentation were proposed. However, all of those approaches are based on some form of knowledge distillation. There are no investigations on other classes of approaches that are commonly used for object recognition in a continual setting. Therefore, the goal of our works is to evaluate and adapt established solutions for continual object recognition to the task of semantic segmentation and to provide baseline methods and evaluation protocols for the task of continual semantic segmentation. We firstly introduce evaluation protocols for the class- and domain-incremental segmentation and analyze selected approaches. We show that the nature of the task of semantic segmentation changes which methods are most effective in mitigating forgetting compared to image classification. Especially, in class-incremental learning knowledge distillation proves to be a vital tool, whereas replay methods struggle to distinguish between old and new classes.

Keywords: Autonomous / Intelligent Robotic Vehicles, Vision Sensing and Perception, Convolutional Neural Networks
Abstract: Moving object detection (MOD) from a moving vehicle has been a challenging problem and critical to autonomous driving especially in urban scenarios. Recent literature has laid focus on this task, as many approaches were devoted to moving object detection. These approaches consist of multistage pipelines including semantic segmentation and optical flow estimation and require various sources of information from active and passive sensors. However, they fail to accurately segment moving objects due to the large ego-camera motion, in addition to the high processing time. This paper proposes a novel approach for moving object detection by processing information only from a camera. Our approach is based on the integration of an encoder-decoder network (EDNet) with a semantic segmentation model (Mask R-CNN), where Mask R-CNN detects the objects of interest and the EDNet classifies their motion (moving /static) over two consecutive frames. We compare our proposed model results against existing MOD models on three SOTA benchmarks. We achieved SOTA performance in terms of visual quality, accuracy and speed. We outline our work with qualitative results in a short video at https://youtu.be/H5mUfWXBRvY

Keywords: Vision Sensing and Perception, Vehicle Environment Perception, Convolutional Neural Networks
Abstract: The precise 3D localization of non-ego vehicles is a crucial task for the long-term goal of autonomous driving. In urban scenarios, where pedestrians frequently interact with vehicles, this task also requires a precise modeling of dynamic vehicle parts, e.g., doors. Current state-of-the-art computer vision algorithms are in fact able to estimate a vehicle pose but do not model doors by any means. To provide a solution solely based on a monocular camera, our proposed pipeline first performs a six degree-of-freedom pose estimation and then predicts the respective states of the vehicle doors. For both problems we utilize a perspectiven-point fitting method based on key points. To this end, we jointly detect the two required sets of correspondences for the vehicle body and the doors with a neural network. Since little insight is published for the application of key point based vehicle detection in the literature, we compare different implementations of the key point prediction module and investigate algorithm details, i.e., the role of a key point visibility analysis and two differing key point layouts. Results for the body estimation and the door detection with respect to these implementation details are presented on a proprietary dataset, in which we utilize an exact vehicle model to receive precise ground truth.

Keywords: Image, Radar, Lidar Signal Processing, Deep Learning, Mapping and Localization
Abstract: Local image features matching is a useful technique for 3D reconstruction and localization in ITS field. In recent years, machine learning-based local feature matching methods have been proposed. However, it is impractical to manually generate the ground truth of corresponding points used as training data on these methods. Therefore, for in-vehicle camera images we propose an automatic training data collection method using geometric constraints and continuity with the movement of the vehicle. The experimental results show that the proposed method can provide effective corresponding points for training a machine learning-based matcher.

Keywords: Driver Recognition, Vision Sensing and Perception, Self-Driving Vehicles
Abstract: Recent innovation in highly automated driving in industrial and scientific domains has created a growing demand for logical description of statistically meaningful real-world motion data. On one hand this data supports learning-based probabilistic methods in software development while on the other it allows validation and testing. The AUTOMATUM DATA dataset is a new dataset which is now available at automatum-data.com, and was generated initially using 12 characteristic highway-like scenes from 30 hours of drone videos. The processing pipeline for determining the object trajectories was validated with reference vehicles, where the relative speed error was less than 0.2 percent. To generate the dataset described in this study, the objects from the drone videos were first identified and classified. The detected objects were then linked to their coordinate system results to produce valid object trajectories. The presented dataset is freely available for future research and development-based endeavors (Creative Commons license model CC BY-ND).

Keywords: Advanced Driver Assistance Systems, Image, Radar, Lidar Signal Processing, Deep Learning
Abstract: Image captioning is a task to generate a sentence explaining an input image. In autonomous driving, image captioning is expected for providing linguistic explanations of autonomous driving control decision-making because it can reduce the psychological burden on passengers and prevent accidents. Current image-captioning methods are limited to generating a caption for an input image and not generating captions for events in the near future. It is important to generate captions for any event that will happen in the near future to prevent accidents and alert passengers. Therefore, we created a task to generate an explanatory sentence of near-future events using images observed from past to present. For this task, we propose a near-future image-captioning method suitable for in-vehicle camera images. Our experiments using the Berkeley Deep Drive eXplanation Dataset showed that the proposed method can appropriately generate captions for near-future events.

Keywords: Traffic Flow and Management, Vision Sensing and Perception, Unsupervised Learning
Abstract: Reconstructing 4D vehicular activity (3D space and time) from cameras is useful for autonomous vehicles, commuters and local authorities to plan for smarter and safer cities. Traffic is inherently repetitious over long periods, yet current deep learning-based 3D reconstruction methods have not considered such repetitions and have difficulty generalizing to new intersection-installed cameras. We present a novel approach exploiting longitudinal (long-term) repetitious motion as self-supervision to reconstruct 3D vehicular activity from a video captured by a single fixed camera. Starting from off-the-shelf 2D keypoint detections, our algorithm optimizes 3D vehicle shapes and poses, and then clusters their trajectories in 3D space. The 2D keypoints and trajectory clusters accumulated over long-term are later used to improve the 2D and 3D keypoints via self-supervision without any human annotation. Our method improves reconstruction accuracy over state of the art on scenes with a significant visual difference from the keypoint detector's training data, and has many applications including velocity estimation, anomaly detection and vehicle counting. We demonstrate results on traffic videos captured at multiple city intersections, collected using our smartphones, YouTube, and other public datasets.