Keywords:
Abstract: Professor A. Alahi's keynote presentation explores the burgeoning field of representation learning in AI, particularly focusing on its implications for Autonomous Mobility. Despite the transformative impacts of representation learning in areas like Computer Vision and Natural Language Processing, challenges abound when applying these methods to the autonomous mobility domain, notably due to data scarcity in diverse and adverse conditions. Prof. Alahi emphasizes the limitations of traditional data-heavy approaches, such as their inadequacy in handling novel or rare events crucial for robust perception in traffic scenes. His work critically evaluates the prevalent assumption that massive datasets are sufficient for accurate forecasting, revealing key deficiencies in current deep learning methods when faced with diverse real-world interactions. Additionally, he discusses the shortcomings of existing planning algorithms in dynamic and uncertain environments. To address these gaps, Prof. Alahi introduces seven foundational principles aimed at enhancing the robustness of representation learning for the unique demands of Autonomous Mobility, advocating for a paradigm shift towards more adaptable and resilient AI systems.

Keywords: Automotive Cyber Physical Systems, Simulation and Real-World Testing Methodologies, Future Mobility and Smart City
Abstract: As urban road traffic systems evolve into intelligent and intricate networks, real-time collaborative decision-making becomes increasingly vital. This paper addresses the challenge of cooperative path planning in complex urban road conditions, considering different vehicle types and priorities. Traditional path-planning algorithms prove inefficient in multi-vehicle collaboration, while training reinforcement learning algorithms in novel environments is complex. This study introduces an innovative parallel multi-agent reinforcement learning path planning approach, formalizing the problem into a multi-agent Markov Decision Process. The key contribution lies in a parallelized training methodology that significantly reduces training times and enhances path optimization. Comparative analysis demonstrates the superiority of the proposed approach, with a 0.84% training time compared to MA-QL and only a 6.06% probability of path overlap with traditional methods. These findings highlight the efficacy of the parallel multi-agent reinforcement learning approach for addressing the complexities of cooperative path planning in urban road networks.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques
Abstract: Scenario-based testing of automated driving functions has become a promising method to reduce time and cost compared to real-world testing. In scenario-based testing automated functions are evaluated in a set of pre-defined scenarios. These scenarios provide information about vehicle behaviors, environmental conditions, or road characteristics using parameters. To create realistic scenarios, parameters and parameter dependencies have to be fitted utilizing real-world data. However, due to the large variety of intersections and movement constellations found in reality, data may not be available for certain scenarios. This paper proposes a methodology to systematically analyze relations between parameters of scenarios. Bayesian networks are utilized to analyze causal dependencies in order to decrease the amount of required data and to transfer causal patterns creating unseen scenarios. Thereby, infrastructural influences on movement patterns are investigated to generate realistic scenarios on unobserved intersections. For evaluation, scenarios and underlying parameters are extracted from the inD dataset. Movement patterns are estimated, transferred and checked against recorded data from those initially unseen intersections.

Keywords: Automated Vehicles
Abstract: Due to the considerable data amount produced by the vehicle’s perception sensors, there is the need to investigate techniques to reduce the datarate, e.g. for camera, well established lossy compression techniques can be explored and evaluated. These techniques must be analysed in combination with the consumer of the data, which will most likely be a perception algorithm based on deep neural networks (DNNs). This work shows that compression tuned DNNs have enhanced performance with respect to traditionally trained DNNs, and the performance is higher when evaluating not only compressed data, but also uncompressed data. Overall, the DNN performance is steady when transmitting data with increasing lossy compression rate (up to~130:1), but above this value there is a performance decrease. The results presented in this work demonstrate that compression can be used in automotive sensors, particularly leveraging the hereby proposed and optimised compression-tuned DNNs. The full paper is available at DOI: 10.1109/TITS.2023.3308344.

Keywords: Automated Vehicles
Abstract: short

Keywords: Automated Vehicles
Abstract: short

Keywords: Automated Vehicles
Abstract: Off-road autonomous ground vehicles (AGVs) have gained increased attention in recent years due to their promising potential to be deployed in many areas to replace humans in harsh and/or boring working environments. Different from on-road conditions, off-road AGVs suffer from more complicated working conditions, in which vehicle-terrain interaction is much more complex, leading to unpredictable vehicle dynamic responses. Besides, uncertainty prevails in off-road terrain which is attributed to lacking precise environmental information. To solve the problems, this research will be carried out from three aspects. First, high-resolution terrain reconstruction employs low spatial resolution satellite images and soil maps. The spatial-dependent uncertainty of the reconstructed terrain (including both elevation and soil properties) induced by lacking precise environmental information is characterized. Second, the vehicle-terrain interaction model is constructed based on terramechanics, based on which a Bayesian machine learning model is used to learn the relationship between vehicle mobility in terms of speed-made-good and terrain properties (elevation and soil properties) and quantify the uncertainty of vehicle mobility. Third, a motion planning method integrating vehicle-terrain interaction with taking mobility uncertainty into account is developed, which will be verified by numerical simulations. This study could contribute part of the theoretical basis for motion planning of off-road AGVs under uncertain terrain conditions.

Keywords: Automated Vehicles
Abstract: Optimization problems for trajectory planning in autonomous vehicle racing are characterized by their nonlinearity and nonconvexity. Instead of solving these optimization problems, usually a convex approximation is solved instead to achieve a high update rate. We present a real-time-capable model predictive control (MPC) trajectory planner based on a nonlinear single-track vehicle model and Pacejka’s magic tire formula for autonomous vehicle racing. After formulating the general nonconvex trajectory optimization problem, we form a convex approximation using sequential convex programming (SCP). The state of the art convexifies track constraints using sequential linearization (SL), which is a method of relaxing the constraints. Solutions to the relaxed optimization problem are not guaranteed to be feasible in the nonconvex optimization problem. We propose sequential convex restriction (SCR) as a method to convexify track constraints. SCR guarantees that resulting solutions are feasible in the nonconvex optimization problem. We show recursive feasibility of solutions to the restricted optimization problem. The MPC is evaluated on a scaled version of the Hockenheimring racing track in simulation. The results show that MPC using SCR yields faster lap times than MPC using SL, while still being real-time capable.

Keywords: Automated Vehicles
Abstract: Heterogeneous autonomous vehicle (HAV) coordinated motion planning must guide each vehicle out of the conflict zone based on the differences in vehicle platform characteristics.Decomposing complex driving tasks into primitives is an effective way to improve algorithm efficiency. Hence, the purpose of this paper is to complete the coordinated motion planning tasks through offline driving behavior primitive (DBP) library generation, online extension and selection of DBPs. The proposed algorithm applies dynamic movement primitives and singular value decomposition to learn driving behavior patterns from driving data, integrates them into a model-based optimization generation method as constraints, and builds a DBP library by fusing driving data and vehicle model. Based on the generated DBP library and primitive association probabilities learned from labeled driving segments via stochastic context-free grammar, the planning method completes the independent DBP extension of each vehicle in the conflict zone, generates an interaction DBP tree, and uses the mixed-integer linear programming algorithm to optimally select the primitives to be executed. This study demonstrates that the generated DBP library not only expands the types of primitives, but also distinguishes the characteristics of HAVs. We also present how to utilize the DBP libraries to obtain coordinated motion planning results with spatiotemporal information in the form of DBP extension and selection. The results obtained by real vehicle platforms and simulation show that the proposed method can accomplish coordinated motion planning tasks without relying on specific scene elements and highlight the unique motion characteristics of HAVs. The online version of paper is https://ieeexplore.ieee.org/document/10159576/

Keywords: Automated Vehicles
Abstract: Occlusion is a major challenge for LiDAR-based object detection methods as it renders regions of interest unobservable to the ego vehicle. A proposed solution to this problem comes from collaborative perception via Vehicle-to-Everything (V2X) communication, which leverages a diverse perspective thanks to the presence of connected agents (vehicles and intelligent roadside units) at multiple locations to form a complete scene representation. The major challenge of V2X collaboration is the performance- bandwidth tradeoff which presents two questions (i) which information should be exchanged over the V2X network, and (ii) how the exchanged information is fused. The current state-of-the-art resolves to the mid-collaboration approach where Birds-Eye View (BEV) images of point clouds are communicated to enable a deep interaction among connected agents while reducing bandwidth consumption. While achieving strong performance, the real-world deployment of most mid-collaboration approaches are hindered by their overly complicated architectures and unrealistic assumptions about inter-agent synchronization. In this work, we devise a simple yet effective collaboration method based on exchanging the outputs from each agent that achieves a better bandwidth-performance trade-off while minimising the required changes to the single-vehicle detection models. Moreover, we relax the assumptions used in existing state-of-the-art approaches about inter-agent synchronization to only require a common time reference among connected agents, which can be achieved in practice using GPS time. Experiments on the V2X-Sim dataset show that our collaboration method reaches 76.72 mean average precision which is 99% the performance of the early collaboration method.

Keywords: Automated Vehicles
Abstract: A novel vehicle traction algorithm solving the traction force allocation problem based on vehicle center point motion feedback controller is proposed in this paper. The center point motion feedback control system proposed utilizes individual wheel torque actuation assuming all wheels are individually driven. The approach presented is an alternative to the various direct optimization-based traction force/torque allocation schemes. The proposed system has many benefits, such as significant reduction of the algorithm complexity by merging most traction system functionalities into one. Such a system enables significant simplification, unification, and standardization of powertrain control design. Moreover, many signals needed by conventional traction force allocation methods are not required to be measured or estimated with the proposed approach, which are among others vehicle mass, wheel loading (normal force), and vehicle center of gravity location. Vehicle center point trajectory setpoints and measurements are transformed to each wheel, where the tracking is ensured using the wheel torque actuation. The proposed control architecture performance and analysis are shown using the nonlinear twin-track vehicle model implemented in Matlab & Simulink environment. The performance is then validated using high fidelity FEE CTU in Prague EFORCE formula model implemented in IPG CarMaker environment with selected test scenarios. Finally, the results of the proposed control allocation are compared to the state-of-the-art approach.

https://ieeexplore.ieee.org/document/10194411

Keywords: Automated Vehicles
Abstract: Autonomous vehicles currently suffer from a time-inefficient driving style caused by uncertainty about human behavior in traffic interactions. Accurate and reliable prediction models enabling more efficient trajectory planning could make autonomous vehicles more assertive in such interactions. However, the evaluation of such models is commonly oversimplistic, ignoring the asymmetric importance of prediction errors and the heterogeneity of the datasets used for testing. We examine the potential of recasting interactions between vehicles as gap acceptance scenarios and evaluating models in this structured environment. To that end, we develop a framework aiming to facilitate the evaluation of any model, by any metric, and in any scenario. We then apply this framework to state-of-the-art prediction models, which all show themselves to be unreliable in the most safety-critical situations.

Link: https://ieeexplore.ieee.org/abstract/document/10043012

Keywords: Automated Vehicles
Abstract: Despite significant advances in vehicle safety, road- way accidents remain a substantial danger. To bring about safer vehicles, researchers and manufacturers continue to develop new systems to assist drivers in dangerous situations. Current approaches often implement several independent systems, each of which can assist in only specific situations, leaving open critical paths for accidents to occur. We propose a general approach to driver assistance based on nonlinear model predictive control (NMPC). This system can intervene in both lateral and longitudi- nal commands to keep the vehicle safely within the boundaries of the road, but allows the driver freedom to maneuver the vehicle as they wish when they act safely. This work builds on previous MPC approaches by incorporating a notion of safe operating speed. Properly modulating speed ensures the vehicle’s limits are never exceeded, and not reached unless necessary. Experimental results on a full-size steer-, brake-, and throttle-by-wire vehicle, validate the performance of this system. These experiments show that the controller effectively matches the driver’s commands when possible but will deviate from those commands when necessary for safe operation.

Keywords: Automated Vehicles
Abstract: Detecting dangerous traffic agents in videos captured by a dashboard camera (dashcam) mounted on vehicles is essential to ensure safe navigation in complex driving environments. Crash-related videos are corner cases in driving-related big data, and pre-crash process is transient and complex. Besides, risky and non-risky traffic agents can be similar in their appearance. These make the localization of risky traffic agents in driving videos particularly challenging. In addressing the challenges, this paper proposes an attention-guided multistream feature fusion network (AM-Net) to localize dangerous traffic agents from dashcam videos ahead of potential accidents. Two Gated Recurrent Unit (GRU) networks use object bounding box and optical flow features extracted from consecutive video frames to capture spatio-temporal cues for distinguishing risky traffic agents. An attention module, coupled with the GRUs, learns to identify traffic agents that are relevant to a crash. Fusing the two streams of global and object-level features, AM-Net predicts the riskiness scores of traffic agents in the video. This paper also introduces a new benchmark dataset called Risky Object Localization (ROL), which contains spatial, temporal, and categorical annotations of the crash, object, and scene-level attributes. The proposed AM-Net achieves a promising performance of 85.59% AUC on the ROL dataset. Additionally, the AM-Net outperforms the current state-of-the-art for video anomaly detection by 3.5% AUC on the public DoTA dataset. A thorough ablation study further reveals AM-Net’s merits by assessing the contributions of its functional constituents. The full paper is published in IEEE Transactions on Intelligent Vehicles, vol. 9, no. 1, pp. 1792-1803, Jan. 2024, doi: 10.1109/TIV.2023.3275543.

Keywords: Automated Vehicles
Abstract: Drifting, or cornering with rear tires that exceed slip limits, represents a trade-off of stability for controllability while operating at the limits of friction. Recent work has demonstrated exceptional performance by autonomous systems of stabilization and path tracking a vehicle around an unstable drifting equilibrium. However, safely navigating unexpected or challenging road conditions that require an autonomous vehicle to operate at the limits of friction is likely to require dynamic, non-equilibrium maneuvers. These trajectories activate underlying dynamics, such as weight transfer and wheelspeed, which significantly affect the forces acting on the vehicle. In this paper, we present a modeling and control framework for dynamic drifting trajectories. First, a novel vehicle model is proposed that strikes an appropriate balance of fidelity and complexity. Then, this vehicle model is embedded into a Nonlinear Model Predictive Control policy that can maintain stability and path tracking while performing dynamic drifting maneuvers. This work is validated experimentally using ``Takumi", an autonomous Toyota Supra, that demonstrates root mean squared path tracking error of 13 centimeters and a peak error of just 47 cm. Finally, a simulation study suggests parameter uncertainty, rather than additional model fidelity, is the primary limitation of further increasing controller performance.

Keywords: Automated Vehicles
Abstract: shortThere is a need for a large-scale, real world, diverse, and context rich vehicle acceleration catalog that can be used to design, analyze, and compare various intelligent transportation systems. This paper fulfills three primary objectives. First, it provides such a catalog through the Surface Accelerations Reference, which is openly available as an interactive analytics tool as well as an open and downloadable dataset. The Surface Accelerations Reference statistically describes the driving profiles of about 3,500 individuals contributing 34 million miles of continuous driving data collected in the Second Strategic Highway Research Program Naturalistic Driving Study (SHRP 2 NDS). These profiles were created by summarizing billions of longitudinal and lateral acceleration epochs experienced by the participants. Second, this paper introduces a standardized methodology for creating such a catalog so that similar acceleration profiles can be produced for other human cohorts or automated driving systems. Finally, the data are used to analyze the effect of roadway speed category on the rates of lateral and longitudinal acceleration epochs at various thresholds. It is observed that, for the median driver, the rates of epochs are upto 3 orders of magnitude higher on low-speed roads as compared to high-speed roads. This catalog will facilitate intelligent vehicle system designers to compare and tune their systems for safer driving experiences. It will also allow agencies with similar data to create comparable catalogs facilitating safety and behavioral comparisons between populations.

Keywords: Automated Vehicles
Abstract: Rendering accurate multisensory feedback is critical to ensure natural user behavior in driving simulators. In this work, we present a virtual reality (VR)-based Vehicle-in-the-Loop (ViL) simulator that provides visual, vestibular, and haptic feedback to drivers in high speed driving conditions. Designing our simulator around a four-wheel steer-by-wire vehicle enables us to emulate the dynamics of a vehicle traveling significantly faster than the test vehicle and to transmit corresponding haptic steering feedback to the driver. By scaling the speed of the test vehicle through a combination of VR visuals, vehicle dynamics emulation, and steering wheel force feedback, we can safely and immersively run experiments up to highway speeds within a limited driving space. In double lane change and highway weaving experiments, our high speed emulation method tracks yaw motion within human perception limits and provides sensory feedback comparable to the same maneuvers driven manually. The online version of this journal paper can be found at: https://ieeexplore.ieee.org/abstract/document/9743209.

Keywords: Automated Vehicles
Abstract: We introduce a goal-aware extension of responsibility-sensitive safety (RSS), a recent methodology for rule-based safety guarantee for automated driving systems (ADS). Making RSS rules guarantee goal achievement---in addition to collision avoidance as in the original RSS---requires complex planning over long sequences of manoeuvres. To deal with the complexity, we introduce a compositional reasoning framework based on program logic, in which one can systematically develop RSS rules for smaller subscenarios and combine them to obtain RSS rules for bigger scenarios. As the basis of the framework, we introduce a program logic dFHL that accommodates continuous dynamics and safety conditions. Our framework presents a dFHL-based workflow for deriving goal-aware RSS rules; we discuss its software support, too. We conducted experimental evaluation using RSS rules in a safety architecture. Its results show that goal-aware RSS is indeed effective in realising both collision avoidance and goal achievement.

The paper appeared in IEEE Transactions on Intelligent Vehicles, Vol. 8, Issue 4, April 2023. https://doi.org/10.1109/TIV.2022.3169762

Its preprint version is available at https://arxiv.org/abs/2207.02387

Keywords: Automated Vehicles
Abstract: The target of reducing travel time only is insufficient to support the development of future smart transportation systems. To align with the United Nations Sustainable Development Goals (UN-SDG), a further reduction of fuel and emissions, improvements of traffic safety, and the ease of infrastructure deployment and maintenance should also be considered. Different from existing work focusing on optimizing the control in either traffic light signal (to improve the intersection throughput), or vehicle speed (to stabilize the traffic), this paper presents a multi-agent Deep Reinforcement Learning (DRL) system called CoTV, which Cooperatively controls both Traffic light signals and Connected Autonomous Vehicles (CAV). Therefore, our CoTV can well balance the reduction of travel time, fuel, and emissions. CoTV is also scalable to complex urban scenarios by cooperating with only one CAV that is nearest to the traffic light controller on each incoming road. This avoids costly coordination between traffic light controllers and all possible CAVs, thus leading to the stable convergence of training CoTV under the large-scale multi-agent scenario. We describe the system design of CoTV and demonstrate its effectiveness in a simulation study using SUMO under various grid maps and realistic urban scenarios with mixed-autonomy traffic.

Keywords: Automated Vehicles
Abstract: Test the safety of Autonomous Driving Systems (ADS) with realistic traffic conditions is important to the insurance industry, legislators, and third-party technical services. Approaches for ADS testing can be divided into two main categories: physical test and virtual test, as shown in Fig.1. The physical test mainly includes closed-field tests and open-road tests, providing a comprehensive but time-consuming measure for autonomous vehicles operating in the real world. The virtual test mimics the real world and vehicles with digital models and preset simulation scenarios, which have the advantages of high efficiency and low cost. However, the scarcity of risky driving events distributed in real-world driving often makes sampling inefficient and biased.

To address these issues, a unified and hierarchical framework for ADS safety test is proposed in this paper, as shown in Fig.2. The four steps are integrated to explore the ADS probability of failure under challenging and realistic traffic conditions in an efficient and unbiased manner: (1) Risk subspace is set up using domain knowledge, driving events cleaning and clustering. (2) The joint probability density function (PDF) of the decision variables determined by the risk subspace is modeled. (3) A Kriging model-based optimization problem is formulated for finding the appropriate IS distribution parameters, where the Kriging model is trained iteratively within limited computational resources. (4) Risk test cases (or concrete scenarios in PEGASUS) that may lead to autonomous vehicle crashes are further generated.

Keywords: Automated Vehicles
Abstract: With the rapid development of autonomous driving, Autonomous Vehicles (AVs) have started to appear on public roads, which has inevitably affected current traffic conditions and the operations of Manual Vehicles (MVs). Current research on AVs’ influence has mainly been conducted at individual level of driving behaviors, while few studies have focused on the overall network level to consider the traffic flow pattern due to mixed traffic. In this work, considering varying signal control schemes and demand loading patterns, we conducted simulation experiments based on a grid network and a real-world network in Beijing using SUMO. Traffic flow with the mixture of MVs, low-level AVs (LAVs), and high-level AVs (HAVs) were emulated so to investigate how the network performs at various levels of mixed traffic. Driving behaviors between the three types of vehicles were calibrated using driving data drawn from OpenACC dataset, and Waymo Open Dataset. The capacity and critical accumulation of the Macroscopic Fundamental Diagram (MFD) were chosen as the key indicators of network performance. We found that AVs positively boost network capacity (up to 19.0% increase) but a negative influence on critical accumulation was also observed (up to 9.0% decrease). However, the positive impact of OpenACC and Waymo's AVs on macroscopic traffic is still far from ideal since they may be too conservative. AVs can boost flow when traffic is in unsaturated or saturated states. However, when traffic flow is oversaturated, AVs can instead cause flow and average speed to drop faster than that in the MV-only scenario. The web link for this paper is: https://ieeexplore.ieee.org/document/10102700

Keywords: Automated Vehicles
Abstract: Rapid prototyping of Connected and Automated Vehicles (CAV) is challenging because of the physical distribution of vehicles. Furthermore, experiments with CAV may be subject to external influences which prevent reproducibility. This article presents an architecture for the experimental testing of CAVs, focusing on decision-making. Our architecture for experiments of CAV is strictly modular and hierarchical, and therefore it supports an easy and rapid exchange of every single controller as well as of optimization libraries. Additionally, the architecture synchronizes the whole network of sensors, computation devices, and actuators. Thus, it achieves deterministic and reproducible results, even for time- variant network topologies. Using this architecture, we can include active and passive vehicles and vehicles with heterogeneous dynamics in the experiments. The architecture also allows for handling communication uncertainties, e.g., data packet drop and time delay. The resulting architecture supports performing different in-the-loop tests and experiments. We demonstrate the architecture in the Cyber-Physical Mobility Lab (CPM Lab) using 20 vehicles on a 1:18 scale. The architecture can be applied to other domains.

Keywords: Automated Vehicles
Abstract: Formal models for the safety validation of autonomous vehicles have become increasingly important. To this end, we present a safety framework for longitudinal automated driving. This framework allows calculating minimum safe inter-vehicular distances for arbitrary ego vehicle control policies. We use this framework to enhance the Responsibility-Sensitive Safety (RSS) model and models based on it, which fail to cover situations where the ego vehicle has a higher decelerating capacity than its preceding vehicle. For arbitrary ego vehicle control policies, we show how our framework can be applied by substituting real (possibly computationally intractable) controllers with upper bounding functions. This comprises a general approach for longitudinal safety, where safety guarantees for the upper-bounded system are equivalent to those for the original system but come at the expense of larger inter-vehicular distances. https://ieeexplore.ieee.org/document/9904311

Keywords: Automated Vehicles
Abstract: In the field of lidar odometry for autonomous navigation, the Iterative Closest Point (ICP) algorithm is a prevalent choice for estimating robot motion by comparing point clouds. However, ICP accuracy is strictly dependent on the nature of the features involved, but also on the directional choice of the extraction and matching, either from the current to the reference point cloud or viceversa. Previous approaches focused on involving different kind of features, extracted in the incoming frame and matched with a reference. This paper introduces Balanced ICP, a novel formulation that addresses these challenges by exploiting the feature extraction step for lines or planes, and consequent matching, in both the directions. Then, the cost function is designed to perform a simultaneous optimization of all available data. The experiments, conducted both on simulated and real data from the KITTI dataset, reveal that our method outperform the classical monodirectional formulations, in terms of robustness, accuracy and stability.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR), Automated Vehicles
Abstract: In recent years, much progress has been made in LiDAR-based 3D object detection mainly due to advances in detector architecture designs and availability of large-scale LiDAR datasets. Existing 3D object detectors tend to perform well on the point cloud regions closer to the LiDAR sensor as opposed to on regions that are farther away. In this paper, we investigate this problem from the data perspective instead of detector architecture design. We observe that there is a learning bias in detection models towards the dense objects near the sensor and show that the detection performance can be improved by simply manipulating the input point cloud density at different distance ranges without modifying the detector architecture and without data augmentation. We propose a model-free point cloud density adjustment pre-processing mechanism that uses iterative MCMC optimization to estimate near-optimal parameters for altering the point density at different distance ranges. We conduct experiments using four state-of-the-art LiDAR 3D object detectors on two public LiDAR datasets, namely Waymo and ONCE. Our results demonstrate that our range-based point cloud density manipulation technique can improve the performance of the existing detectors, which in turn could potentially inspire future detector designs.

Keywords: Automated Vehicles, Sensor Signal Processing
Abstract: Autonomous vehicles require an accurate understanding of the surrounding road scene for navigation. One crucial task in this understanding is the bird's-eye view (BEV) road network estimation. However, accurately extracting the BEV road network around the vehicle in complex scenes, considering variations in lane curvature and shape, remains a challenge. This paper aims to accurately represent and learn the BEV road network around the vehicle for structured road scene understanding. Specifically, we propose a road network representation, emph{i.e.}, representing the lane centerline as an ordered point set and the road network as a directed graph, which accurately describes lane centerline instances and lane topological relationships in complex scenes. Then, we introduce an online road network estimation framework that takes on-board surround-view video as input and utilizes hierarchical query embedding to extract the BEV road network around the vehicle. Furthermore, we present a temporal aggregation module to alleviate occlusion issues in road scenes and enhance the accuracy of road network estimation by incorporating historical frame information flexibly. Finally, we conducted extensive experiments on the nuScenes dataset to validate the effectiveness of the proposed method in structured BEV road scene understanding.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR)
Abstract: Radar and camera fusion yields robustness in perception tasks by leveraging the strength of both sensors. The typical extracted radar point cloud is 2D without height information due to insufficient antennas along the elevation axis, which challenges the network performance. This work introduces a learning-based approach to infer the height of radar points associated with 3D objects. A novel robust regression loss is introduced to address the sparse target challenge. In addition, a multi-task training strategy is employed, emphasizing important features. The average radar absolute height error decreases from 1.69 to 0.25 meters compared to the state-of-the-art height extension method. The estimated target height values are used to preprocess and enrich radar data for downstream perception tasks. Integrating this refined radar information further enhances the performance of existing radar camera fusion models for object detection and depth estimation tasks.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR)
Abstract: This paper addresses the challenges in multi-target domain adaptive (MTDA) for semantic segmentation, aiming to learn a single model capable of adapting to multi-target domains. Existing methods solely focus on visual appearance (style) discrepancies, overlooking contextual variations across multi-target domains, resulting in limited performance. We propose a novel approach termed Masking-augmented Collaborative Domain Congregation (MacDC) to handle both style gap and contextual gap among multi-target domains. MacDC achieves this goal by generating image-level and region-level intermediate domains among multi-target domains. To further strengthen contextual alignment, MacDC applies multi-context masking that enforces the model's understanding of diverse contexts. Notably, MacDC directly learns a single model for multi-target domain adaptation, significantly reducing training times and model parameters. Despite its simplicity, MacDC demonstrates superior performance compared to state-of-the-art MTDA segmentation methods on the syn-to-real and real-to-real benchmarks.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR)
Abstract: Knowledge distillation has recently been proven to be effective for model compression and acceleration of point cloud-based 3D object detection. However, the complementary network pruning is often overlooked during knowledge distillation. In this paper, we propose a pre-pruned distillation framework that combines network pruning and knowledge distillation to better transfer knowledge from the teacher to the student. To maintain the feature consistency between the student and the teacher, we train a teacher model and then generate a compact student model by structural channel pruning. Then, we employ multi-source knowledge distillation to transfer both mid-level and high-level information to the student model. Additionally, to improve the object detection performance of the student model, we propose a soft pivotal position selection mask to emphasize the features of the foreground regions during distillation. We conduct experiments on both pillar- and voxel-based 3D object detectors on the Waymo datasets, demonstrating the effectiveness of our approach in compressing point cloud-based 3D detectors.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR)
Abstract: With the advances of deep neural networks, there is progress on the detection and recognition of traffic lights for advanced driver assistance systems (ADAS). However, existing approaches most rely on the identification of traffic light boxes, followed by the recognition of signal lights. It is considered as a major drawback since light bulbs can be arranged in different directions or irregular patterns in different geographic regions. In this paper, we present a traffic light detection method based on direct recognition of individual signal lights. Our two-stage technique utilizes data augmentation and ensemble learning to detect the light bulbs with least miss rate. By learning the color characteristics from validation sets for data augmentation, it is able to achieve a signal light candidate detection rate at 97.26%. Followed by the classification stage, the recognition accuracy is given by 98.6%, which outperforms state-of-the-art traffic light detection algorithms. The source code and dataset are available at https://github.com/981124/yolov7 traffic light detect.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR)
Abstract: Road surface condition (RSC) estimation is useful to warn a driver or for automatic speed control in slippery road conditions. By using front-facing camera images, the RSC few meters ahead of the vehicle can be estimated, which could provide the valuable milliseconds to act in critical conditions before reaching the slippery road region. Convolutional neural networks (CNNs) have been used in previous work to classify or segment the RSC from front-facing camera images. In this work, we look for ways in improving the performance for RSC classification by fusion additional information. In-contrast to the widely used approach of classifying the most prominent RSC of the whole image, we propose to separate the road region into a collection of cells using a 2D grids and classifying the RSC of each grid-cell into dry-moist, wet-water, slush, snow-ice classes. Three additional information sources are investigated to fuse with road grid-cell region of the image, which are from the temperature sensor and other areas of the image, in an alternative manner. Our results indicates clear improvements on RSC estimation performance when fusing with each of the three sources of additional information, compared to only using image grid-cell region. The best performance with a 5.8% increase in average F1-score is obtained when using the full image in addition to the grid-cell image region.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR)
Abstract: Bird's-eye-view (BEV) representation is crucial for the perception function in autonomous driving tasks. It is difficult to balance the accuracy, efficiency and range of BEV representation. The existing works are restricted to a limited perception range within 50 meters. Extending the BEV representation range can greatly benefit downstream tasks such as topology reasoning, scene understanding, and planning by offering more comprehensive information and reaction time. The Standard-Definition (SD) navigation maps can provide a lightweight representation of road structure topology, characterized by ease of acquisition and low maintenance costs. An intuitive idea is to combine the close-range visual information from onboard cameras with the beyond line-of-sight (BLOS) environmental priors from SD maps to realize expanded perceptual capabilities. In this paper, we propose BLOS-BEV, a novel BEV segmentation model that incorporates SD maps for accurate beyond line-of-sight perception, up to 200m. Our approach is applicable to common BEV architectures and can achieve excellent results by incorporating information derived from SD maps. We explore various feature fusion schemes to effectively integrate the visual BEV representations and semantic features from the SD map, aiming to leverage the complementary information from both sources optimally. Extensive experiments demonstrate that our approach achieves state-of-the-art performance in BEV segmentation on nuScenes and Argoverse benchmark. Through multi-modal inputs, BEV segmentation is significantly enhanced at close ranges below 50m, while also demonstrating superior performance in long-range scenarios, surpassing other methods by over 20% mIoU at distances ranging from 50-200m.

Keywords: Sensor Signal Processing, Sensor Fusion for Localization
Abstract: Although LiDAR sensors are crucial for autonomous systems due to providing precise depth information, they struggle with capturing fine object details, especially at a distance, due to sparse and non-uniform data. Recent advances introduced pseudo-LiDAR, i.e., synthetic dense point clouds, using additional modalities such as cameras to enhance 3D object detection. We present a novel LiDAR-only framework that augments raw scans with denser pseudo point clouds by solely relying on LiDAR sensors and scene semantics, omitting the need for cameras. Our framework first utilizes a segmentation model to extract scene semantics from raw point clouds, and then employs a multi-modal domain translator to generate synthetic image segments and depth cues without real cameras. This yields a dense pseudo point cloud enriched with semantic information. We also introduce a new semantically guided projection method, which enhances detection performance by retaining only relevant pseudo points. We applied our framework to different advanced 3D object detection methods and reported up to 2.9% performance upgrade. We also obtained comparable results on the KITTI 3D object detection test set, in contrast to other state-of-the-art LiDAR-only detectors.

Keywords: Sensor Signal Processing, Sensor Fusion for Localization
Abstract: Modern autonomous systems typically use several sensors for perception. For best performance, accurate and reliable extrinsic calibration is necessary. In this research, we propose a reliable technique for the extrinsic calibration of several lidars on a vehicle without the need for odometry estimation or fiducial markers. First, our method generates an initial guess of the extrinsics by matching the raw signals of IMUs co-located with each lidar. This initial guess is then used in ICP and point cloud feature matching which refines and verifies this estimate. Furthermore, we can use observability criteria to choose a subset of the IMU measurements that have the highest mutual information - rather than comparing all the readings. We have successfully validated our methodology using data gathered from Scania test vehicles.

Keywords: Sensor Signal Processing, Sensor Fusion for Localization
Abstract: Autonomous vehicles should be capable of operating in all types of weather conditions. Drivable road region detection is a core component of the perception stack of self-driving vehicles. Current approaches for detecting road regions perform well in good weather but lack in inclement weather conditions. In this paper, we examine the effect of inclement weather on the camera-based state-of-the-art deep learning approaches and introduce a new camera and automotive radar-based multimodal deep learning model to efficiently detect drivable road regions in all weather conditions. We also propose a novel approach to overcome the sparse resolution problem of automotive radars and a way to effectively use it in higher precision tasks such as image segmentation. To validate our work, we have augmented the nuScenes data with rain and fog to add challenging weather conditions. Experimental results show that the performance of the state-of-the-art techniques drops 18% in bad weather conditions while our proposed method improves the performance by 12% compared to the state-of-the-art.

Keywords: Sensor Signal Processing, Software-Defined Vehicle for Intelligent Vehicles, Verification and Validation Techniques
Abstract: Time-series modelling (TSM) is a critical task for intelligent vehicles (IVs), covering areas like fault detection, health monitoring, and inference of road user intentions. In this study, we present a novel TSM approach for enhancing the accuracy of multi-variate signal forecasting in intelligent vehicles. Our method leverages advanced Transformer networks within a relativistic generative adversarial network (RGAN) training framework. The RGAN training framework efficiently improves the accuracy of vehicle states forecasting for IV, demonstrating effective learning of long-time dependencies for more accurate predictions over extended sequences. Additionally, we introduce a high-dimensional extension (HDE) built-in block for the time-series Transformer to explore the impact of higher-dimensional features on representing long-term sequences. The experimental data is collected from a real-world electric vehicle testing bed. We evaluate the proposed RGANFormer framework and the HDE block on two popular time-series models, namely, Autoformer and FiLM. The results demonstrate that the RGANFormer, along with the built-in HDE block, significantly enhances long-term sequential forecasting accuracy for both multivariate and univariate tasks.

Keywords: Sensor Signal Processing
Abstract: The spoofing attack would be a serious threat to location-based applications based on Global Navigation Satellite System (GNSS). To mitigate the negative effect of the spoofing attack that makes the GNSS receiver obtain fake and misleading positioning information, the identification of the spoofing attack is a significant step before the countermeasure is adopted. In this paper, considering constraints of existing methods, SpoofSpike network, which is a novel post-correlation solution is proposed using the Spiking Neural Network (SNN). This GNSS spoofing identification scheme is based on the differences between the practically measured Cross Ambiguity Functions (CAFs) and the predicted one in the GNSS receiver information processing. Under the overall solution architecture, details about the spiking neuron model and the SpoofSpike network are given. The decision-making mechanism to identify the spoofing attack is analyzed. Results from the test and comparisons using the TEXBAT datasets illustrate that the SpoofSpike network-based solution is capable of realizing effective identification according to the comparison of the spoofing score with the threshold, and it outperforms other SNN-based models and the Artificial Neural Network (ANN) counterpart.

Keywords: Sensor Signal Processing
Abstract: Multi-modal 3D object detection models for automated driving have demonstrated exceptional performance on computer vision benchmarks like nuScenes. However, their reliance on densely sampled LiDAR point clouds and meticulously calibrated sensor arrays poses challenges for real-world applications. Issues such as sensor misalignment, miscalibration, and disparate sampling frequencies lead to spatial and temporal misalignment in data from LiDAR and cameras. Additionally, the integrity of LiDAR and camera data is often compromised by adverse environmental conditions such as inclement weather, leading to occlusions and noise interference. To address this challenge, we introduce MultiCorrupt, a comprehensive benchmark designed to evaluate the robustness of multi-modal 3D object detectors against ten distinct types of corruptions. We evaluate five state-of-the-art multi-modal detectors on MultiCorrupt and analyze their performance in terms of their resistance ability. Our results show that existing methods exhibit varying degrees of robustness depending on the type of corruption and their fusion strategy. We provide insights into which multi-modal design choices make such models robust against certain perturbations. The dataset generation code and benchmark are open-sourced at https://github.com/ika-rwth-aachen/MultiCorrupt.

Keywords: Automated Vehicles
Abstract: Understanding the interdependence between autonomous and human-operated vehicles remains an ongoing challenge, with significant implications for the safety and the feasibility of autonomous driving.

This interdependence arises from inherent interactions among road users. Thus, it is crucial for Autonomous Vehicles (AVs) to understand and analyze the intentions of human-driven vehicles, and to display a behavior comprehensible to other traffic participants. To this end, this paper presents GTP-UDRIVE a unified game-theoretic trajectory planner and decision-maker considering a mixed-traffic environment.

Our model considers the intentions of other vehicles in the decision-making process and provides the AV with a human-like trajectory, based on the clothoid interpolation technique. Among highly interactive traffic scenarios, the intersection crossing is particularly challenging. Hence, we choose to illustrate the feasibility and effectiveness of our method in real traffic conditions, using an experimental autonomous vehicle at an unsignalized intersection.

Testing results reveal that our approach is suitable for 1) Making decisions and generating trajectories simultaneously. 2) Describing the vehicle’s trajectory as a piecewise clothoid and enforcing geometric constraints. 3) Reducing search space dimensionality for the trajectory optimization problem.

Keywords: Drone and Urban Air Mobility, Human Factors for Intelligent Vehicles, Future Mobility and Smart City
Abstract: As the industry advances towards the technical realization of passenger drones, there is a noticeable trend where the wishes and acceptance factors of future users are not given primary consideration. Although several companies are progressing with prototype development and market-entry preparations, ongoing research on the requirements for electric Vertical Take-Off and Landing (eVTOL) vehicles remains crucial. For such a novel form of aerial transportation, understanding passenger needs is key, as their acceptance and willingness to use these services will ultimately drive revenue for manufacturers and airlines. To this end, our study, involving 51 participants in South Korea, delved into three key areas of user acceptance: safety considerations of eVTOLs, preferred vertiport locations, and concerns about weight. The findings indicate that piloted passenger drones are acceptable to the public, particularly when they assure safety and offer sufficient luggage storage, among other factors. These insights are vital for the engineering design and development of eVTOLs, highlighting opportunities for weight and energy savings by aligning with passenger preferences and requirements.

Keywords: Policy, Ethics, and Regulations, Human Factors for Intelligent Vehicles, Future Mobility and Smart City
Abstract: This study proposes to find the most appropriate transport modes with an awareness of user preferences (e.g., costs, times) and trip characteristics (e.g., purpose, distance). The work was based on real-life trips obtained from a map application. Several methods including gradient boosting tree, learning to rank, multinomial logit model, automated machine learning, random forest, and shallow neural network have been tried. For some methods, feature selection and over-sampling techniques were also tried. The results show that the best-performing method is a gradient-boosting tree model with the synthetic minority over-sampling technique (SMOTE). Also, results of the multinomial logit model show that (1) an increase in travel cost would decrease the utility of all the transportation modes; (2) people are less sensitive to the travel distance for the metro mode or a multi-modal option that contains metro, i.e., compared to other modes, people would be more willing to tolerate long-distance metro trips. This indicates that metro lines might be a good candidate for large cities.

Keywords: Automated Vehicles, Human Factors for Intelligent Vehicles, Vehicular Active and Passive Safety
Abstract: As a safety critical task, autonomous driving requires accurate predictions of road users' future trajectories for safe motion planning, particularly under challenging conditions. Yet, many recent deep learning methods suffer from a degraded performance on the challenging scenarios, mainly because these scenarios appear less frequently in the training data. To address such a long-tail issue, existing methods force challenging scenarios closer together in the feature space during training to trigger information sharing among them for more robust learning. These methods, however, primarily rely on the motion patterns to characterize scenarios, omitting more informative contextual information, such as interactions and scene layout. We argue that exploiting such information not only improves prediction accuracy but also scene compliance of the generated trajectories. In this paper, we propose to incorporate richer training dynamics information into a prototypical contrastive learning framework. More specifically, we propose a two-stage process. First, we generate rich contextual features using a baseline encoder-decoder framework. These features are split into clusters based on the model's output errors, using the training dynamics information, and a prototype is computed within each cluster. Second, we retrain the model using the prototypes in a contrastive learning framework. We conduct empirical evaluations of our approach using two large-scale naturalistic datasets and show that our method achieves state-of-the-art performance by improving accuracy and scene compliance on the long-tail samples. Furthermore, we perform experiments on a subset of the clusters to highlight the additional benefit of our approach in reducing training bias.

Keywords: Automated Vehicles, Human Factors for Intelligent Vehicles
Abstract: This paper investigates the development of self-aware mechanisms for automated vehicles, introducing the notion of an automation state estimation system. This system is capable to understand its capabilities in a given context, and can leverage that knowledge to estimate the current and near-future automation performance based on internal metrics, as well as external, static (e.g. lane geometry) and dynamic environmental elements (e.g. traffic and weather information). From an application perspective, we consider automation state estimation in the scope of automation mediation, as part of a broader and holistic mediation system, with the goal to tackle challenging aspects related to transitions of control, mode confusion, and driver engagement. We used real-world data for system design, and implemented the proposed automation estimation system in a prototype vehicle. Based on 70 hours of real-world driving, we also validated the performance of the automation state estimation for automation mediation purposes.

Keywords: Infotainment Systems and Human-Machine Interface Design, Human Factors for Intelligent Vehicles
Abstract: Dynamic hand gesture recognition is crucial for human-machine interfaces in the automotive domain. However, creating a diverse and comprehensive dataset of hand gestures can be challenging and time-consuming, especially in dynamic dual-task situations like driving. To address these challenges, we propose using synthetic gesture datasets generated by virtual 3D models as an alternative. Our framework synthesizes realistic hand gestures using a combination of 3D models and animation software, particularly utilizing Unreal Engine. This approach enables the creation of diverse and customizable gesture datasets, reducing the risk of overfitting and improving the model's generalizability. Specifically, our framework generates natural-looking dynamic hand gestures with multiple variants, including gesture speed, performance, and hand shape. Moreover, we simulate various camera locations, such as above the driver and behind the wheel, and different camera types, such as RGB, infrared, and depth cameras, without incurring additional time and cost to obtain these cameras. Our experiments demonstrate that our proposed framework, SynthoGestures (available at https://github.com/amrgomaaelhady/SynthoGestures), can augment or replace existing real-hand datasets with additional enhancement in gesture recognition accuracy. Our tool for generating synthetic static and dynamic hand gestures saves time and effort in creating large datasets, facilitating the faster development of gesture recognition systems for automotive applications.

Keywords: Teleoperation of Intelligent Vehicles, Human Factors for Intelligent Vehicles
Abstract: Since Automated Driving Systems are not expected to operate flawlessly, Automated Vehicles will require human assistance in certain situations. For this reason, teleoperation offers the opportunity for a human to be remotely connected to the vehicle and assist it. The Remote Operator can provide extensive support by directly controlling the vehicle, eliminating the need for Automated Driving functions. However, due to the physical disconnection to the vehicle, monitoring and controlling is challenging compared to driving in the vehicle. Therefore, this work follows the approach of simplifying the task for the Remote Operator by separating the path and velocity input. In a study using a miniature vehicle, different operator-vehicle interactions and input devices were compared based on collisions, task completion time, usability and workload. The evaluation revealed significant differences between the three implemented prototypes using a steering wheel, mouse and keyboard or a touchscreen. The separate input of path and velocity via mouse and keyboard or touchscreen is preferred but is slower compared to parallel input via steering wheel.

Keywords: Human Factors for Intelligent Vehicles, Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: Driving automation is being pushed towards widespread adoption, with significant progress being made continuously. Once the automated vehicle takes over the driving task, the question arises as to how people want to be driven by automation. In order to gain insights into this, a driving simulator study was conducted, in which textit{N} = 49 participants experienced an automated urban drive where pedestrians crossed or attempted to cross the road in front of the automated vehicle at various points. The driving style of the automated vehicle was manipulated (aggressive/defensive), while participants rated their desire for control, trust in automation, and acceptance. The results show that there is no general preference for one driving style over the other. Rather, the preferred behavior of the automation depended on the respective traffic scenario, with drivers preferring defensive driving in some crossing situations and aggressive driving in other situations. The present study indicates that, generally, defensive driving behavior is not necessarily the solution preferred by the user. Instead, a more nuanced approach based on the traffic scenario is recommended.

Keywords: Human Factors for Intelligent Vehicles, Drone and Urban Air Mobility, Vehicle Control and Motion Planning
Abstract: Active control inceptors have been used successfully in conventional aviation. However, with the imminent arrival of electric Vertical Take-Off and Landing (eVTOL) aircraft, the use of force feedback as an assistance system should be revisited. In this paper, a concept for a haptic assistance system using an active force feedback sidestick was developed to assist a pilot in precise maneuver execution with an eVTOL. It uses soft stops, detents, tick cues, and stick vibrations to support the take-off, landing, and standard rate turn maneuvers in a generic eVTOL, operating under the Simplified Vehicle Operations (SVO) concept. The concept was validated in a thinking-aloud experiment with N = 10 subject matter experts in a simulator setup. Supporting the use of haptic detents for nominal stick positions and soft stops to indicate system limits, the study also provides insights into further findings on soft stops, tick cues, and vibrations.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Human Factors for Intelligent Vehicles
Abstract: Revealing differences in vehicle characteristics is critical to enhancing the accessibility of heterogeneous vehicles in off-road environments. Decomposing complex motions into primitives facilitates the maintenance of the algorithm's solution efficiency while considering various constraints. Therefore, this paper proposes a heterogeneous vehicle motion planning method for off-road scenarios based on the generation, extension, and selection of driving behavior primitives. Based on the library of heterogeneous vehicle driving behavior primitives (HDBPs) extracted from driving data in our previous study, this paper proposes a primitive offline optimization generation method that integrates driving behavior constraints, vehicle kinematics constraints, reserved power constraints, and ground adhesion constraints. The generation of spatiotemporal coupled planning results is accomplished by HDBP extension and selection using the optimized HDBP library as the source. In particular, the extension and selection cost considers the interaction with the ground under the constraints of the suspension system, as well as the capacity of the drive system. This paper demonstrates that the proposed HDBP-based planning method can generate highly adaptable and differentiated primitive sequences based on diverse terrain conditions and heterogeneous vehicle characteristic constraints. Moreover, benefiting from the suspension-based pose estimation and drive system characteristic limitations, the method proposed in this paper has a significant advantage over the comparison methods in terms of terrain traversability in real-scene motion planning experiments.

Keywords: Automated Vehicles
Abstract: As the advent of autonomous vehicle (AV) technology revolutionizes transportation, it simultaneously introduces new vulnerabilities to cyber-attacks, posing significant challenges to vehicle safety and security. The complexity of these systems, coupled with their increasing reliance on advanced computer vision and machine learning algorithms, makes them susceptible to sophisticated AV attacks. This paper explores the potential of Large Multimodal Models (LMMs) in identifying Natural Denoising Diffusion (NDD) attacks on traffic signs. Our comparative analysis show the superior performance of LMMs in detecting NDD samples with an average accuracy of 82.52% across the selected models compared to 37.75% for state-of-the-art deep learning models. We further discuss the integration of LMMs within the resource-constrained computational environments to mimic typical autonomous vehicles and assess their practicality through latency benchmarks. Results show substantial superiority of GPT models in achieving lower latency, down to 4.5 seconds per image for both computation time and network latency (RTT), suggesting a viable path towards real-world deployability. Lastly, we extend our analysis to LMMs’ applicability against a wider spectrum of AV attacks, particularly focusing on the Automated Lane Centering systems, emphasizing the potential of LMMs to enhance vehicular cybersecurity.

Keywords: Automated Vehicles, Sensor Signal Processing
Abstract: Precise object detection is crucial in automated driving. In contrast to lidar and radar sensors, cameras provide high-resolutional measurements at comparatively low cost. A state-of-the-art method for object detection using camera images is panoptic segmentation, which combines semantic and object instance information. Current public datasets use multiple cameras to cover a larger area of the environment. But, the limited field of view occludes objects. As a results, on the one hand, the correct dimensions of objects cannot be captured and, on the other hand, false detections can occur. Objects can also be detected multiple times in the overlapping image area. To track dynamic objects, duplicate detections must be filtered. Rather than directly segmenting all camera images individually, we first stitch them into a horizontal panorama. Using a stitched surround view avoids detection difficulties at the boundaries of the individual images. For this purpose, we leverage the EfficientPS pre-trained network architecture and adapt it for use with panoramic images. In our evaluation, we demonstrate the improvement in panoptic quality of a stitched surround view. In addition, we separately compare the panoptic quality in the overlapping image areas between the panorama and the individual images. Finally, we show further advantages of panoramic images in terms of inference time in runtime analysis.

Keywords: Automated Vehicles, End-To-End (E2E) Autonomous Driving, Human Factors for Intelligent Vehicles
Abstract: In the context of autonomous driving, pedestrian behavior prediction is a key component for improving road safety. Presently, many existing prediction models prioritize achieving reliable results, however, they often lack insights into the explainability of each prediction. In this work, we propose a novel approach to pedestrian behavior prediction using knowledge graphs (KG), knowledge graph embeddings (KGE), and a Bayesian Inference process, enabling fully inductive reasoning on KGEs. Our approach aims to consolidate knowledge from annotated datasets through explainable pedestrian features and fuzzy rules, evaluating the importance of these two components within the KG. The entire pipeline has been trained and tested using two datasets: Joint Attention for Autonomous Driving (JAAD) and Pedestrian Situated Intent (PSI). Preliminary results demonstrate the effectiveness of this system in providing explainable clues for pedestrian behavior predictions, even improving results by up to 15% compared to other models. Our approach achieves an F1 score of 0.84 for PSI and 0.82 for JAAD.

Keywords: Software-Defined Vehicle for Intelligent Vehicles
Abstract: With the development of intelligent connected vehicles and Ethernet-based In-Vehicle Network (IVN), the data-oriented data distribution service (DDS) has gradually been entering the vision of in-vehicle network researchers. Thanks to the perfect dynamic discovery mechanism of DDS, applications just need to know the topic of interest to complete the connection with publishing and subscribing nodes, which eliminates cumbersome configurations. However, the current implementations of DDS consume a relatively large quantity of resources, which makes it difficult to be deployed on resource-constrained embedded microcontroller unit (MCU). Considering the advantages of Time Sensitive Networking (TSN) in improving the reliability and latency of network transmission, this paper analyzes and carries out a lightweight design of DDS stack from the perspective of convergence of TSN and DDS, and deploys this LightWeight (LW) DDS on the AUTOSAR Classic Platform. Experiments show that the LW DDS has less resource consumption reduced by 75% and lower end-to-end latency reduced by up to 18% compared to standard implementation.

Keywords: Advanced Driver Assistance Systems (ADAS), Automated Vehicles, Sensor Signal Processing
Abstract: Although lidar sensors have emerged as a cornerstone sensing modality in autonomous driving, they face significant challenges in adverse weather conditions. A particular detrimental effect is spray — a phenomenon where water particles are whirled up by vehicles driving with high velocities on wet roads. Spray often causes clutter points in lidar data to be falsely classified as vehicles by downstream object detectors. In this work, a phenomenological spray simulation model, suitable as an augmentation method for object detection algorithms, is presented. Two distinct datasets featuring real-world spray scenarios are recorded and analyzed, with the first serving for calibrating the simulation model through extensive experiments that vary vehicle speeds, types, and pavement wetness levels. The second dataset functions as a spray test set to evaluate the effectiveness of the simulation model in the context of object detection. Employing the simulation model as an augmentation tool reveals an improvement of up to 17% in Average Precision for state-of-the-art object detection methods in real spray conditions.

Keywords: Human Factors for Intelligent Vehicles, Policy, Ethics, and Regulations
Abstract: The advent of autonomous vehicles (AVs) represents a transformative shift in transportation, promising to redefine mobility and alter our interaction with vehicles. Understanding public perceptions of AVs is crucial, as it influences the adoption and integration of this technology into society. This research conducts a comprehensive investigation into the factors that influence people’s attitudes toward AVs, examining the associated benefits and concerns, as well as the extent of trust placed in this emerging technology. The primary objective is to gain a deeper understanding of the elements that contribute to human-machine trust in the context of AVs. The findings reveal a consistent pattern in the propensity to trust AVs and concerns regarding performance failures, both at individual and societal levels. From a societal perspective, enhanced locomotion independence is the primary benefit of AV deployment, contributing to increased accessibility and reduced reliance on conventional transportation systems. At the individual level, increased free time emerges as the foremost advantage. These findings provide AV developers and policymakers the critical insight when deploying autonomous vehicle systems.

Keywords: Integration of HD map and Onboard Sensors, Sensor Signal Processing
Abstract: To reduce the expensive labor costs of manually labeling autonomous driving datasets, an alternative is to automatically label the datasets using an offline perception system. However, objects might be temporarily occluded. Such occlusion scenarios in the datasets are common yet underexplored in offline auto labeling. In this work, we propose an offline tracking model that focuses on occluded object tracks. It leverages the concept of object permanence, which means objects continue to exist even if they are not observed anymore. The model contains three parts: a standard online tracker, a re-identification (ReID) module that associates tracklets before and after occlusion, and a track completion module that completes the fragmented tracks. The Re-ID module and the track completion module use the vectorized lane map as a prior to refine the tracking results with occlusion. The model can effectively recover the occluded object trajectories. It significantly improves the original online tracking result, demonstrating its potential to be applied in offline auto labeling as a useful plugin to improve tracking by recovering occlusions.

Keywords: Pedestrian Protection, Human Factors for Intelligent Vehicles, Automated Vehicles
Abstract: Understanding and predicting pedestrian crossing behavior is essential for enhancing automated driving and improving driving safety. Predicting gap selection behavior and the use of zebra crossing enables driving systems to proactively respond and prevent potential conflicts. This task is particularly challenging at unsignalized crossings due to the ambiguous right of way, requiring pedestrians to constantly interact with vehicles and other pedestrians. This study addresses these challenges by utilizing simulator data to investigate scenarios involving multiple vehicles and pedestrians. We propose and evaluate machine learning models to predict gap selection in non-zebra scenarios and zebra crossing usage in zebra scenarios. We investigate and discuss how pedestrians' behaviors are influenced by various factors, including pedestrian waiting time, walking speed, the number of unused gaps, the largest missed gap, and the influence of other pedestrians. This research contributes to the evolution of intelligent vehicles by providing predictive models and valuable insights into pedestrian crossing behavior.

Keywords: Perception Including Object Event Detection and Response (OEDR), End-To-End (E2E) Autonomous Driving
Abstract: In autonomous vehicles, understanding the surrounding 3D environment of the ego vehicle in real-time is essential. A compact way to represent scenes while encoding geometric distances and semantic object information is via 3D semantic occupancy maps. State of the art 3D mapping methods leverage transformers with cross-attention mechanisms to elevate 2D vision-centric camera features into the 3D domain. However, these methods encounter significant challenges in real-time applications due to their high computational demands during inference. This limitation is particularly problematic in autonomous vehicles, where GPU resources must be shared with other tasks such as localization and planning. In this paper, we introduce an approach that extracts features from front-view 2D camera images and LiDAR scans, then employs a sparse convolution network (Minkowski Engine), for 3D semantic occupancy prediction. Given that outdoor scenes in autonomous driving scenarios are inherently sparse, the utilization of sparse convolution is particularly apt. By jointly solving the problems of 3D scene completion of sparse scenes and 3D semantic segmentation, we provide a more efficient learning framework suitable for real-time applications in autonomous vehicles. We also demonstrate competitive accuracy on the nuScenes dataset.

Keywords: Perception Including Object Event Detection and Response (OEDR), Sensor Fusion for Localization, Sensor Signal Processing
Abstract: Leveraging inputs over multiple consecutive frames has been shown to benefit 3D object detection. However, existing approaches often demonstrate unsatisfactory scaling with increasing temporal histories. In this work, we propose SF3D, a late fusion module which addresses this issue by better modeling temporal relationships via a two-stream factorization. Concretely, SF3D operates on an input sequence of consecutive bird's-eye view (BEV) features, which is partitioned into ``short-term'' and ``long-term'' frames. A more heavily parameterized short-term branch using adapters and deformable attention aggregates features closer to the current timestep. In parallel, a long-term branch composed of efficiently implemented global convolution layers aggregates a larger window of temporally distant historical features. This two-stream paradigm allows SF3D to effectively consume near-term information, while scaling to efficiently leverage longer historical windows. We show that SF3D works with arbitrary upstream BEV encoders and downstream detectors, achieving improvements over recent state-of-the-art on the Waymo Open and nuScenes benchmarks.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR), Sensor Fusion for Localization
Abstract: Online 3D multi-object tracking (MOT) has recently received significant research interests due to the expanding demand of 3D perception in advanced driver assistance systems (ADAS) and autonomous driving (AD). Among the existing 3D MOT frameworks for ADAS and AD, conventional point object tracking (POT) framework using the tracking-by-detection (TBD) strategy has been well studied and accepted for LiDAR and 4D imaging radar point clouds. In contrast, extended object tracking (EOT), another important framework which accepts the joint-detection-and-tracking (JDT) strategy, has rarely been explored for online 3D MOT applications. This paper provides the first systematical investigation of the EOT framework for online 3D MOT in real-world ADAS and AD scenarios. Specifically, the widely accepted TBD-POT framework, the recently investigated JDT-EOT framework, and our proposed TBD-EOT framework are compared via extensive evaluations on two open source 4D imaging radar datasets: View-of-Delft and TJ4DRadSet. Experiment results demonstrate that the conventional TBD-POT framework remains preferable for online 3D MOT with high tracking performance and low computational complexity, while the proposed TBD-EOT framework has the potential to outperform it in certain situations. However, the results also show that the JDT-EOT framework encounters multiple problems and performs inadequately in evaluation scenarios. After analyzing the causes of these phenomena based on various evaluation metrics and visualizations, we provide possible guidelines to improve the performance of these MOT frameworks on real-world data. These provide the first benchmark and important insights for the future development of 4D imaging radar-based online 3D MOT.

Keywords: Simulation and Real-World Testing Methodologies, Automated Vehicles, Human Factors for Intelligent Vehicles
Abstract: Enhancing simulation environments to replicate real-world driver behavior is essential for developing Autonomous Vehicle technology. While some previous works have studied the yielding reaction of lag vehicles in response to a merging car at highway on-ramps, the possible lane-change reaction of the lag car has not been widely studied. In this work we aim to improve the simulation of the highway merge scenario by including the lane-change reaction in addition to yielding behavior of main-lane lag vehicles, and we evaluate two different models for their ability to capture this reactive lane-change behavior. To tune the payoff functions of these models, a novel naturalistic dataset was collected on U.S. highways that provided several hours of merge-specific data to learn the lane change behavior of U.S. drivers. To make sure that we are collecting a representative set of different U.S. highway geometries in our data, we surveyed 50,000 U.S. highway on-ramps and then selected eight representative sites. The data were collected using roadside-mounted lidar sensors to capture various merge driver interactions. The models were demonstrated to be configurable for both keep-straight and lane-change behavior. The models were finally integrated into a high-fidelity simulation environment and confirmed to have adequate computation time efficiency for use in large-scale simulations to support autonomous vehicle development.

Keywords: Verification and Validation Techniques, Simulation and Real-World Testing Methodologies, Automated Vehicles
Abstract: The selection of relevant test scenarios for the scenario-based testing and safety validation of automated driving systems (ADSs) remains challenging. An important aspect of the relevance of a scenario is the challenge it poses for an ADS. Existing methods for calculating the challenge of a scenario aim to express the challenge in terms of a metric value. Metric values are useful to select the least or most challenging scenario. However, they fail to provide human-interpretable information on the cause of the challenge which is critical information for the efficient selection of relevant test scenarios. Therefore, this paper presents the Challenge Description Method that mitigates this issue by analyzing scenarios and providing a description of their challenge in terms of the minimum required lane changes and their difficulty. Applying the method to different highway scenarios showed that it is capable of analyzing complex scenarios and providing easy-to-understand descriptions that can be used to select relevant test scenarios.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Sensor Signal Processing
Abstract: As urban road traffic systems evolve into intelligent and intricate networks, real-time collaborative decision-making becomes increasingly vital. This paper addresses the challenge of cooperative path planning in complex urban road conditions, considering different vehicle types and priorities. Traditional path-planning algorithms prove inefficient in multi-vehicle collaboration, while training reinforcement learning algorithms in novel environments is complex. This study introduces an innovative parallel multi-agent reinforcement learning path planning approach, formalizing the problem into a multi-agent Markov Decision Process. The key contribution lies in a parallelized training methodology that significantly reduces training times and enhances path optimization. Comparative analysis demonstrates the superiority of the proposed approach, with a 0.84% training time compared to MA-QL and only a 6.06% probability of path overlap with traditional methods. These findings highlight the efficacy of the parallel multi-agent reinforcement learning approach for addressing the complexities of cooperative path planning in urban road networks.

Keywords: Automated Vehicles, Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS)
Abstract: Prediction of vehicle lane change maneuvers has gained a lot of momentum in the last few years. Some recent works focus on predicting a vehicle's intention by predicting its trajectory first. This is not enough, as it ignores the context of the scene and the state of the surrounding vehicles (as they might be risky to the target vehicle). Other works assessed the risk made by the surrounding vehicles only by considering their existence around the target vehicle, or by considering the distance and relative velocities between them and the target vehicle as two separate numerical features. In this work, we propose a solution that leverages Knowledge Graphs (KGs) to anticipate lane changes based on linguistic contextual information in a way that goes well beyond the capabilities of current perception systems. Our solution takes the Time To Collision (TTC) with surrounding vehicles as input to assess the risk on the target vehicle. Moreover, our KG is trained on the HighD dataset using the TransE model to obtain the Knowledge Graph Embeddings (KGE). Then, we apply Bayesian inference on top of the KG using the embeddings learned during training. Finally, the model can predict lane changes two seconds ahead with 97.95% f1-score, which surpassed the state of the art, and three seconds before changing lanes with 93.60% f1-score.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Autonomous vehicle controllers capable of drifting can improve safety in dynamic emergency situations. However, drifting involves operating at high sideslip angles, which is a fundamentally unstable operating regime that typically requires an accurate vehicle model for reliable operation; such models may not be available after environmental or vehicle parameter changes. Towards that goal, this work presents a Nonlinear Model Predictive Control approach which is capable of initiating and controlling a drift in a production vehicle even when changes in vehicle parameters degrade the original model. A neural network model of the vehicle dynamics is used inside the optimization routine and updated with online learning techniques, giving a higher fidelity and more adaptable model. Experimental validation on a full size, nearly unmodified Lexus LC500 demonstrates the increased modeling fidelity, adaptability, and utility of the presented controller framework. As the LC500 is a difficult car to drift, previous approaches which rely on physics based vehicle models could not complete the autonomous drift tests on this vehicle. Furthermore, the tires on the experimental vehicle are then switched, changing the vehicle parameters, and the capability of the controller to adapt online is demonstrated.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, End-To-End (E2E) Autonomous Driving
Abstract: This study explores efficient and safe Automated Valet Parking (AVP) strategies in unstructured and dynamic environments. Existing approaches utilizing reinforcement learning neglected the impact of dynamic agents on ego vehicle and disregarded human driving patterns, leading to their ineffectiveness in unstructured dynamic contexts. We propose a novel hybrid attention mechanism that comprehends the mixed interactions between static and dynamic elements, aiding autonomous vehicles in advanced planning. We implemented a guidance system based on human preferences, eliminating the need for expert data at the outset and expediting the training process via intermediate planning stages, thereby facilitating parking maneuvers akin to human drivers. The model was trained and validated in a range of parking situations. The experimental outcomes indicate that our method possesses robust adaptability and navigation skills in static and dynamic environments.

Keywords: Automotive Cyber Physical Systems, Simulation and Real-World Testing Methodologies, Future Mobility and Smart City
Abstract: As urban road traffic systems evolve into intelligent and intricate networks, real-time collaborative decision-making becomes increasingly vital. This paper addresses the challenge of cooperative path planning in complex urban road conditions, considering different vehicle types and priorities. Traditional path-planning algorithms prove inefficient in multi-vehicle collaboration, while training reinforcement learning algorithms in novel environments is complex. This study introduces an innovative parallel multi-agent reinforcement learning path planning approach, formalizing the problem into a multi-agent Markov Decision Process. The key contribution lies in a parallelized training methodology that significantly reduces training times and enhances path optimization. Comparative analysis demonstrates the superiority of the proposed approach, with a 0.84% training time compared to MA-QL and only a 6.06% probability of path overlap with traditional methods. These findings highlight the efficacy of the parallel multi-agent reinforcement learning approach for addressing the complexities of cooperative path planning in urban road networks.

Keywords: Human Factors for Intelligent Vehicles, Functional Safety in Intelligent Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: Despite the continual advances in Advanced Driver Assistance Systems (ADAS) and the development of high-level autonomous vehicles (AV), there is a consensus that for the short to medium term, there is a requirement for a human supervisor to handle the edge cases that inevitably arise. Given this requirement, the state of the autonomous vehicle operator (referred to as the safety driver) must be monitored to ensure their contribution to the vehicle's safe operation. This paper introduces a dual-source approach integrating data from an infrared camera facing the safety driver and vehicle perception systems to produce a metric for safety driver alertness to promote and ensure safe operator behaviour. The infrared camera detects the safety driver's head, enabling the calculation of head orientation, which is relevant as the head typically moves according to the individual's focus of attention. By incorporating environmental data from the perception system, it becomes possible to determine whether the safety driver observes objects in the surroundings. Experiments were conducted using data collected in Sydney, Australia, simulating AV operations in an urban environment. Our results demonstrate that the proposed system effectively determines a metric for the attention levels of the safety driver, enabling interventions such as warnings or reducing autonomous functionality as appropriate. The results indicate reduced awareness on subsequent laps during the study, demonstrating the ``automation complacency'' phenomenon. This comprehensive solution shows promise in contributing to ADAS and AVs' overall safety and efficiency in a real-world setting.

Keywords: Sensor Signal Processing, Advanced Driver Assistance Systems (ADAS), Software-Defined Vehicle for Intelligent Vehicles
Abstract: Integrating multi-modal sensor capabilities is imperative in the current landscape of technological advancements aimed at achieving fully autonomous driving systems. LiDAR sensors are pivotal in demonstrating exceptional reliability in adverse weather conditions, day-night scenarios, and various complex situations because of their laser pulse emission properties. LiDAR predicts object distance with remarkable precision by leveraging time-of-flight measurements from laser pulse refraction. However, challenges arise when laser pulses encounter highly reflective surfaces, leading to a phenomenon known as Blooming. Especially on high reflectors, blooming poses a significant issue as it can obscure the accurate determination of an object's dimension. This can impact the performance of object detection and classification algorithms in autonomous driving systems. More comprehensive LiDAR-Blooming datasets and straightforward algorithms must be developed in state-of-the-art research to effectively perceive and understand the blooming effect in real-time. In response to this challenge, our paper proposes a novel algorithm designed to generate and validate synthetic blooming datasets, offering a comprehensive understanding of the LiDAR-based phenomenon. Furthermore, we introduce an advanced deep-learning model named BloomNet, which addresses LiDAR-blooming issues. Experiments are conducted with state-of-the-art models, and our proposed model, BloomNet, outperforms existing approaches by huge margins. The results in our artificially created synthetic dataset and real-time blooming scenarios are also promising.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS)
Abstract: Low-latency instance segmentation of LiDAR point clouds is crucial in real-world applications because it serves as an initial and frequently-used building block in a robot's perception pipeline, where every task adds further delay. Particularly in dynamic environments, this total delay can result in significant positional offsets of dynamic objects, as seen in highway scenarios. To address this issue, we employ a new technique, which we call continuous clustering. Unlike most existing clustering approaches, which use a full revolution of the LiDAR sensor, we process the data stream in a continuous and seamless fashion. Our approach does not rely on the concept of complete or partial sensor rotations with multiple discrete range images; instead, it views the range image as a single and infinitely horizontally growing entity. Each new column of this continuous range image is processed as soon it is available. Obstacle points are clustered to existing instances in real-time and it is checked at a high-frequency which instances are completed in order to publish them without waiting for the completion of the revolution or some other integration period. In the case of rotating sensors, no problematic discontinuities between the points of the end and the start of a scan are observed. In this work we describe the two-layered data structure and the corresponding algorithm for continuous clustering. It is able to achieve an average latency of just 5 ms with respect to the latest timestamp of all points in the cluster. We are publishing the source code at https://github.com/UniBwTAS/continuous_clustering.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques
Abstract: Scenario-based testing of automated driving functions has become a promising method to reduce time and cost compared to real-world testing. In scenario-based testing automated functions are evaluated in a set of pre-defined scenarios. These scenarios provide information about vehicle behaviors, environmental conditions, or road characteristics using parameters. To create realistic scenarios, parameters and parameter dependencies have to be fitted utilizing real-world data. However, due to the large variety of intersections and movement constellations found in reality, data may not be available for certain scenarios. This paper proposes a methodology to systematically analyze relations between parameters of scenarios. Bayesian networks are utilized to analyze causal dependencies in order to decrease the amount of required data and to transfer causal patterns creating unseen scenarios. Thereby, infrastructural influences on movement patterns are investigated to generate realistic scenarios on unobserved intersections. For evaluation, scenarios and underlying parameters are extracted from the inD dataset. Movement patterns are estimated, transferred and checked against recorded data from those initially unseen intersections.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: Driving behavior modeling is crucial in autonomous driving systems for preventing traffic accidents. Inverse reinforcement learning (IRL) allows autonomous agents to learn complicated behaviors from expert demonstrations. Similar to how humans learn by trial and error, failed demonstrations can help an agent avoid failures. However, expert and failed demonstrations generally have some common behaviors, which could cause instability in an IRL model. To improve the stability, this work proposes a novel method that introduces time-series labeling for the optimization of IRL to help distinguish the behaviors in demonstrations. Experimental results in a simulated driving environment show that the proposed method converged faster than and outperformed other baseline methods. The results also show consistency for various data balances of the number of expert and failed demonstrations.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, End-To-End (E2E) Autonomous Driving
Abstract: Optimization of vehicle maneuvers using dynamic models in constrained spaces is challenging. Homotopic optimization, which has shown success for vehicle maneuvers with kinematic models, is studied in the case where the vehicle model is governed by dynamic equations considering road-tire interactions. This method involves a sequence of optimization problems that start with a large free space. By iteration, this space is progressively made smaller until the target problem is reached. The method uses a homotopy index to iterate the sequence of optimizations, which is verified by solving challenging maneuvering problems with different road surfaces and entry velocities using a double-track vehicle dynamics model. The main takeaway is that homotopic optimization is also efficient for dynamic vehicle models at the limit of road-tire friction, and it demonstrates capabilities in solving demanding maneuvering problems compared with alternative methods like stepwise initialization and driver model-based initialization.