Keywords:
Abstract: Model predictive control (MPC) solves repeatedly a constrained optimization problem updated by the actual state. This basis fits how a human driver drives a car and provides a good framework for intelligent driving. This talk discusses the old and the new of the development.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS)
Abstract: Low-latency instance segmentation of LiDAR point clouds is crucial in real-world applications because it serves as an initial and frequently-used building block in a robot's perception pipeline, where every task adds further delay. Particularly in dynamic environments, this total delay can result in significant positional offsets of dynamic objects, as seen in highway scenarios. To address this issue, we employ a new technique, which we call continuous clustering. Unlike most existing clustering approaches, which use a full revolution of the LiDAR sensor, we process the data stream in a continuous and seamless fashion. Our approach does not rely on the concept of complete or partial sensor rotations with multiple discrete range images; instead, it views the range image as a single and infinitely horizontally growing entity. Each new column of this continuous range image is processed as soon it is available. Obstacle points are clustered to existing instances in real-time and it is checked at a high-frequency which instances are completed in order to publish them without waiting for the completion of the revolution or some other integration period. In the case of rotating sensors, no problematic discontinuities between the points of the end and the start of a scan are observed. In this work we describe the two-layered data structure and the corresponding algorithm for continuous clustering. It is able to achieve an average latency of just 5 ms with respect to the latest timestamp of all points in the cluster. We are publishing the source code at https://github.com/UniBwTAS/continuous_clustering.

Keywords: Human Factors for Intelligent Vehicles, Functional Safety in Intelligent Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: Despite the continual advances in Advanced Driver Assistance Systems (ADAS) and the development of high-level autonomous vehicles (AV), there is a consensus that for the short to medium term, there is a requirement for a human supervisor to handle the edge cases that inevitably arise. Given this requirement, the state of the autonomous vehicle operator (referred to as the safety driver) must be monitored to ensure their contribution to the vehicle's safe operation. This paper introduces a dual-source approach integrating data from an infrared camera facing the safety driver and vehicle perception systems to produce a metric for safety driver alertness to promote and ensure safe operator behaviour. The infrared camera detects the safety driver's head, enabling the calculation of head orientation, which is relevant as the head typically moves according to the individual's focus of attention. By incorporating environmental data from the perception system, it becomes possible to determine whether the safety driver observes objects in the surroundings. Experiments were conducted using data collected in Sydney, Australia, simulating AV operations in an urban environment. Our results demonstrate that the proposed system effectively determines a metric for the attention levels of the safety driver, enabling interventions such as warnings or reducing autonomous functionality as appropriate. The results indicate reduced awareness on subsequent laps during the study, demonstrating the ``automation complacency'' phenomenon. This comprehensive solution shows promise in contributing to ADAS and AVs' overall safety and efficiency in a real-world setting.

Keywords: Sensor Signal Processing, Advanced Driver Assistance Systems (ADAS), Software-Defined Vehicle for Intelligent Vehicles
Abstract: Integrating multi-modal sensor capabilities is imperative in the current landscape of technological advancements aimed at achieving fully autonomous driving systems. LiDAR sensors are pivotal in demonstrating exceptional reliability in adverse weather conditions, day-night scenarios, and various complex situations because of their laser pulse emission properties. LiDAR predicts object distance with remarkable precision by leveraging time-of-flight measurements from laser pulse refraction. However, challenges arise when laser pulses encounter highly reflective surfaces, leading to a phenomenon known as Blooming. Especially on high reflectors, blooming poses a significant issue as it can obscure the accurate determination of an object's dimension. This can impact the performance of object detection and classification algorithms in autonomous driving systems. More comprehensive LiDAR-Blooming datasets and straightforward algorithms must be developed in state-of-the-art research to effectively perceive and understand the blooming effect in real-time. In response to this challenge, our paper proposes a novel algorithm designed to generate and validate synthetic blooming datasets, offering a comprehensive understanding of the LiDAR-based phenomenon. Furthermore, we introduce an advanced deep-learning model named BloomNet, which addresses LiDAR-blooming issues. Experiments are conducted with state-of-the-art models, and our proposed model, BloomNet, outperforms existing approaches by huge margins. The results in our artificially created synthetic dataset and real-time blooming scenarios are also promising.

Keywords: Automated Vehicles, Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS)
Abstract: Prediction of vehicle lane change maneuvers has gained a lot of momentum in the last few years. Some recent works focus on predicting a vehicle's intention by predicting its trajectory first. This is not enough, as it ignores the context of the scene and the state of the surrounding vehicles (as they might be risky to the target vehicle). Other works assessed the risk made by the surrounding vehicles only by considering their existence around the target vehicle, or by considering the distance and relative velocities between them and the target vehicle as two separate numerical features. In this work, we propose a solution that leverages Knowledge Graphs (KGs) to anticipate lane changes based on linguistic contextual information in a way that goes well beyond the capabilities of current perception systems. Our solution takes the Time To Collision (TTC) with surrounding vehicles as input to assess the risk on the target vehicle. Moreover, our KG is trained on the HighD dataset using the TransE model to obtain the Knowledge Graph Embeddings (KGE). Then, we apply Bayesian inference on top of the KG using the embeddings learned during training. Finally, the model can predict lane changes two seconds ahead with 97.95% f1-score, which surpassed the state of the art, and three seconds before changing lanes with 93.60% f1-score.

Keywords: Automated Vehicles
Abstract: Banquet Speech, General Chair

Seoul Robotics [Silver] sponsor promotion

Entertainment

Dr. Ahmed Hussein to promote ITSS activities and membership

Introduction of next year’s IV Chair.

Keywords: Advanced Driver Assistance Systems (ADAS), Human Factors for Intelligent Vehicles
Abstract: Recent advancements in video transformers have significantly impacted the field of human action recognition. Leveraging these models for distracted driver action recognition could potentially revolutionize road safety measures and enhance Human-Machine Interaction (HMI) technologies. A factor that limits their potential use is the need for extensive data for model training. In this paper, we propose DRVMon-VM, a novel approach for the recognition of distracted driver actions. This is based on a large pre-trained video transformer called VideoMaeV2 (backbone) and a classification head as decoder, which are fine-tuned using a dual learning rate strategy and a medium-sized driver actions database complemented by various data augmentation techniques. Our proposed model exhibits a substantial improvement, exceeding previous results by 7.34% on the challenging Drive&Act dataset, thereby setting a new benchmark in this field.

Keywords: Advanced Driver Assistance Systems (ADAS), Human Factors for Intelligent Vehicles
Abstract: During the use of Advanced Driver Assistance Systems (ADAS), drivers can intervene in the active function and take back control due to various reasons. However, the specific reasons for driver-initiated takeovers in naturalistic driving are still not well understood. In order to get more information on the reasons behind these takeovers, a test group study was conducted. There, 17 participants used a predictive longitudinal driving function for their daily commutes and annotated the reasons for their takeovers during active function use. In this paper, the recorded takeovers are analyzed and the different reasons for them are highlighted. The results show that the reasons can be divided into three main categories. The most common category consists of takeovers which aim to adjust the behavior of the ADAS within its Operational Design Domain (ODD) in order to better match the drivers’ personal preferences. Other reasons include takeovers due to leaving the ADAS’s ODD and corrections of incorrect sensing state information. Using the questionnaire results of the test group study, it was found that the number and frequency of takeovers especially within the ADAS’s ODD have a significant negative impact on driver satisfaction. Therefore, the driver satisfaction with the ADAS could be increased by adapting its behavior to the drivers’ wishes and thereby lowering the number of takeovers within the ODD. The information contained in the takeover behavior of the drivers could be used as feedback for the ADAS. Finally, it is shown that there are considerable differences in the takeover behavior of different drivers, which shows a need for ADAS individualization.

Keywords: Advanced Driver Assistance Systems (ADAS), Infotainment Systems and Human-Machine Interface Design, Human Factors for Intelligent Vehicles
Abstract: The driver-assisted navigation system is an invaluable tool. However, in intricate scenarios, drivers frequently commit navigation errors. To mitigate this issue, this study focuses on the F-type intersection with the highest incidence of deviations as the research subject. Road scenarios are replicated, and driver behavior data is collected through driving simulator technology. Speed and speed standard deviation are indicators for investigating the influence of driveway distance (DD), navigation prompt timing (NPT), and driver attributes on driving efficiency and safety stability using a generalized linear mixed model (GLMM). Findings reveal that excessively large or small driveway distances and navigation messages that are either premature or delayed negatively affect driving efficiency and safety stability. Consequently, it is recommended to adhere to a driveway distance range of 15-30m, accompanied by the prompt mode of {-300m, -150m, Confirmation}. Furthermore, although no random effects of driver attributes were identified, it is essential to recognize that driver attributes heavily influence their driving behavior in complex road scenarios. This study lays the foundation for optimizing the design of road facilities and navigation systems.

Keywords: Advanced Driver Assistance Systems (ADAS), Integration of Onboard Systems and Cloud-Based Services, Smart Infrastructure
Abstract: Leveraging the power of large language models, asTech Insights marks a revolutionary stride in automotive repair, skillfully interpreting complex vehicle diagnostics to produce custom repair instructions. This innovation, born from creative problem-solving, boasts an impressive accuracy rate exceeding 90% and operates over 60 times faster than conventional methods, dramatically cutting operational expenses. asTech Insights elevates service quality in the industry through using certified recommendations, ensuring swift, evidence-based, and precise repair tactics. The paper thoroughly explores the sophisticated technical design of asTech Insights, examines its real-world applications, and underscores its capacity to redefine the automotive repair and maintenance sector's future.

Keywords: Advanced Driver Assistance Systems (ADAS), Perception Including Object Event Detection and Response (OEDR), Human Factors for Intelligent Vehicles
Abstract: Accurate prediction of drivers' gaze is an important component of vision-based driver monitoring and assistive systems. Of particular interest are safety-critical episodes, for example, performing maneuvers or crossing intersections. In such scenarios, drivers' gaze distribution changes significantly and becomes difficult to predict, especially if the task and context information is represented implicitly, as is common in many state-of-the-art (SOTA) models. However, explicit modeling of top-down factors affecting drivers' attention often requires additional information and annotations that may not be readily available.

In this paper, we address the challenge of effective modeling of task and context with common sources of data for use in practical systems. To this end, we introduce SCOUT+, a task- and context-aware model for drivers' gaze prediction, which leverages route and map information inferred from commonly available GPS data. We evaluate our model on two datasets, DR(eye)VE and BDD-A, and demonstrate that using maps improves results compared to SOTA bottom-up models and reaches performance comparable to the top-down model SCOUT which relies on privileged ground truth information. Code is available at https://github.com/ykotseruba/SCOUT.

Keywords: Advanced Driver Assistance Systems (ADAS), Software-Defined Vehicle for Intelligent Vehicles, Simulation and Real-World Testing Methodologies
Abstract: This paper presents a cost-effective approach to utilizing large language models (LLMs) as part of advanced driver-assistance systems (ADAS) through a knowledge-distilled model for driving assessment. LLMs have recently been employed across various domains. However, due to their size, they require sufficient computing infrastructure for deployment and ample time for generation. These characteristics make LLMs challenging to integrate into applications requiring real-time feedback, including ADAS. An existing study employed a vector database containing responses generated from an LLM to act as a surrogate model. However, this approach is limited when handling out-of-distribution (OOD) scenarios, which LLMs excel at. We propose a novel approach that utilizes a distilled model obtained from an established knowledge distillation technique to perform as a surrogate model for a target LLM, offering high resilience in handling OOD situations with substantially faster inference time. To assess the performance of the proposed approach, we also introduce a new dataset for driving scenarios and situations (DriveSSD), containing 124,248 records. Additionally, we augment randomly selected 12,425 records, 10% of our DriveSSD, with text embeddings generated from an embedding model. We distill the model using 10,000 augmented records and test all approaches on the remaining 2,425 records. We find that the distilled model introduced in this study has better performance across metrics, with half of the inference time used by the previous approach. We make our source code and data publicly available.

Keywords: Advanced Driver Assistance Systems (ADAS), Vehicle Control and Motion Planning, Automated Vehicles
Abstract: This paper examines various non-linear state estimators for accuracy and robustness, estimating vehicle sideslip angle which is crucial for improving vehicle handling, stability, and safety in modern vehicle dynamic control systems. The study compares the performance of different state estimators under various driving conditions and driving scenarios, with a particular focus on a novel two-stage observer (of order mathbit{n}=mathbf{2}) that combines a super-twisting sliding mode observer and a conventional sliding mode filter. The results demonstrate the effectiveness of the proposed observer, which outperforms other state estimators in terms of accuracy and robustness.

Keywords: Advanced Driver Assistance Systems (ADAS), Vehicle Control and Motion Planning, Policy, Ethics, and Regulations
Abstract: This paper presents a comprehensive review of trajectory datasets from vehicles equipped with Advanced Driver Assistance Systems, with the aim of precisely modeling the behavior of Autonomous Vehicles (AVs). This study emphasizes the importance of trajectory data in the development of AV models, especially in car-following scenarios. We introduce and evaluate several datasets: the OpenACC Dataset, the Connected & Autonomous Transportation Systems Laboratory Open Dataset, the Vanderbilt ACC Dataset, the Central Ohio Dataset, and the Waymo Open Dataset. Each dataset offers unique insights into AV behaviors, yet they share common challenges in terms of data availability, processing, and standardization. After a series of data cleaning, outlier removal, and statistical analysis, this paper transforms datasets of varied formats into a uniform standard, thereby improving their applicability for modeling AV car-following behavior. Key contributions of this study include: 1. the transformation of all datasets into a unified standard format, enhancing their utility for broad research applications; 2. a comparative analysis of these datasets, highlighting their distinct characteristics and implications for car-following model development; 3. the provision of guidelines for future data collection projects.

Keywords: Advanced Driver Assistance Systems (ADAS), Vehicle Control and Motion Planning, Sensor Signal Processing
Abstract: Automated and electrified ground vehicles depend on optimal chassis control to synergistically orchestrate different actuators. This integration aims to improve performance, comfort, handling, and stability. However, this optimal control problem necessitates several key parameters that are expensive or impractical to measure. Virtual sensing emerges as an efficient and cost-effective alternative, utilizing existing vehicle sensors to overcome these challenges. The present study proposes a Virtual Sensing Module (VSM) designed to cope with a chassis controller. It is capable of estimating tire forces, the vehicle's SideSlip Angle (SSA), and the Tire-Road Friction Coefficient (TRFC). Furthermore, the VSM has the benefit of inferring the vehicle mass and adjusting the tire parameters. Importantly, the VSM operates effectively across coupled dynamics and under both standard and aggressive driving conditions. The proposed framework adopts a Data to Features to Decision structure: The Feature segment tracks the tire forces and SSA via Model-Based (MB) dynamic estimators. Fed by the outputs of the previous sub-module, the Decision block tackles the TRFC estimation challenge using a novel MB strategy. This strategy was benchmarked against various Data-Driven techniques for validation. The effectiveness of the VSM was verified through both simulation and real-world experimental data under varying road conditions. The findings demonstrate the module's high accuracy, with minimal estimation errors, and its efficiency in both low and high excitation scenarios.

Keywords: Advanced Driver Assistance Systems (ADAS), Vehicle Control and Motion Planning, Vehicular Active and Passive Safety
Abstract: Safe driving envelope can help drivers maneuver vehicles without risking instability. Traditional open-loop safe envelop is conservative, while closed-loop can enhance the vehicle performance through the control inputs feedback. This article firstly introduces the classic open-loop safe envelope determination method, based on which the closed-loop safe envelope is calculated through a numerical optimal control method. The open-loop and closed-loop safe envelope can work together to assess the potential risk of current vehicles. The optimal control law is also analyzed and verified through simulation in Simulink. The driver assistance potential applications are finally discussed.

Keywords: Advanced Driver Assistance Systems (ADAS), Vehicle Control and Motion Planning
Abstract: The cut-in behavior of adjacent vehicles presents a challenge for the Adaptive Cruise Control (ACC) system. Inability to proactively discern adjacent vehicles' cut-in actions could impact driving safety. In addition, abrupt changes in ego vehicle's following target might provoke excessive reactions, undermining passenger comfort. To address this challenge, this paper integrates trajectory prediction model into a deep reinforcement learning(DRL)-based car-following policy. Utilizing Finite State Machine(FSM), we proactively identify cut-in vehicles based on the predicted trajectories to enhance safety. In designing the DRL-based car-following policy, we propose a novel reward function by analyzing human driving data distribution and considering lateral information of cut-in vehicles. This method enhances driving comfort by significantly reducing abrupt maneuvers in both car-following and cut-in scenarios. Additionally, we investigate the impact of state observation configurations on the performance of the DRL policy. Our experimental findings reveal that incorporating the ego vehicle's acceleration into the observation state contributes to optimizing comfort and enhancing robustness in scenarios where the observation of other vehicles' motion state is not precise.

Keywords: Advanced Driver Assistance Systems (ADAS), Vehicular Active and Passive Safety, Sensor Signal Processing
Abstract: For in-vehicle systems, head pose estimation (HPE) is a primitive task for many safety indicators, including driver attention modeling, visual awareness estimation, behavior detection, and gaze detection. The driver's head pose information is also used to augment human-vehicle interfaces for infotainment and navigation. HPE is challenging, especially in the context of driving, due to the sudden variations in illumination, extreme poses, and occlusions. Due to these challenges, driver HPE based only on 2D color data is unreliable. These challenges can be addressed by 3D-depth data to an extent. We observe that features from 2D and 3D data complement each other. The 2D data provides detailed localized features, but is sensitive to illumination variations, whereas 3D data provides topological geometrical features and is robust to lighting conditions. Motivated by these observations, we propose a robust HPE model which fuses data obtained from color and depth cameras (i.e., 2D and 3D). The depth feature representation is obtained with a model based on PointNet++. The color images are processed with the ResNet-50 model. In addition, we add temporal modeling to our framework to exploit the time-continuous nature of head pose trajectories. We implement our proposed model using the multimodal driving monitoring (MDM) corpus, which is a naturalistic driving database. We present our model results with a detailed ablation study with unimodal and multimodal implementations, showing improvement in head pose estimation. We compare our results with baseline HPE models using regular cameras, including OpenFace 2.0 and HopeNet. Our fusion model achieves the best performance, obtaining an average root mean square error (RMSE) equal to 4.38 degrees.

Keywords: Advanced Driver Assistance Systems (ADAS), Vehicular Active and Passive Safety
Abstract: Safety is the cornerstone of L2+ autonomous driving and one of the fundamental tasks is forward collision warning that detects potential rear-end collisions. Potential collisions are also known as conflicts, which have long been indicated using Time-to-Collision with a critical threshold to distinguish safe and unsafe situations. Such indication, however, focuses on a single scenario and cannot cope with dynamic traffic environments. For example, TTC-based crash warning frequently misses potential collisions in congested traffic, and issues false alarms during lane-changing or parking. Aiming to minimise missed and false alarms in conflict detection, this study proposes a more reliable approach based on vehicle spacing patterns. To test this approach, we use both synthetic and real-world conflict data. Our experiments show that the proposed approach outperforms single-threshold TTC unless conflicts happened in the exact way that TTC is defined, which is rarely true. When conflicts are heterogeneous and when the information of conflict situation is incompletely known, as is the case with real-world conflicts, our approach can achieve less missed and false detection. This study offers a new perspective for conflict detection, and also a general framework allowing for further elaboration to minimise missed and false alarms. Less missed alarms will contribute to fewer accidents, meanwhile, fewer false alarms will promote people's trust in collision avoidance systems. We thus expect this study to contribute to safer and more trustworthy autonomous driving.

Keywords: Advanced Driver Assistance Systems (ADAS)
Abstract: Vehicle navigation systems use both GPS and map data, primarily information derived from map data. Conventional navigation systems assume that the user will look directly at the display to check information. Simultaneously provided text and voice often play only a supplementary role, which can lead to driver distraction and misinterpretation. In contrast, human navigation utilizes visual information, potentially reducing the cognitive load on drivers. Human-like Guidance is aimed at realizing a driving assistance system that supports navigation akin to human guidance. Implementing Human-like Guidance, requires the handling of video footage from in-vehicle cameras during vehicle operation, suggesting the need for an approach combining image recognition and language model. However, images captured during operation often contain superfluous information, making the selection of relevant objects for navigation challenging. Moreover, relying solely on image information makes it difficult to consider the relationship with surrounding objects. Therefore, this study proposes a Spatial-Temporal Scene Graph that can represent spatial and temporal information of objects from driving scene videos. Furthermore, we achieve Human-like Guidance through navigation generation using features extracted from the Spatial-Temporal Scene Graph. Our results show that our proposed method improves the accuracy of navigation generation accuracy compared to traditional image-based navigation methods. In addition, the use of a Spatial-Temporal Scene Graph enables the generation of human-like navigation that focuses on the movements of surrounding vehicle objects.

Keywords: Software-Defined Vehicle for Intelligent Vehicles, Advanced Driver Assistance Systems (ADAS), Automated Vehicles
Abstract: Design automation and synthesis are becoming increasingly crucial in the creation of vehicle electrical and/or electronic (E/E) architectures. This significance arises from the necessity to manage complexity, integrate hardware and software, adapt to technological advancements, and optimize performance. We explore novel aspects aimed at facilitating the synthesis of E/E architectures by extending a developed framework. Our newly incorporated features encompass reliability-based optimization and the integration of hypervisor technologies. To assess the scalability and applicability of our framework to real-world use cases, we conduct multiple experiments, evaluating computation times for architecture generation.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Human Factors for Intelligent Vehicles
Abstract: Optimizing the composition of the desired trajectory and creating an appropriate control method are essential to improve the tracking control effect. Decomposition and optimal combination of primitives is a general and practical way of composing desired trajectories. Therefore, the purpose of this paper is to generate control-system-adapted driving behavior primitives (CDBPs) and design the corresponding control strategy. Based on the pre-constructed driving behavior primitive library extracted from driving data, a nonlinear optimization method is applied to optimize the trajectories that do not conform to the vehicle kinematic constraints during the primitive offline generalization process. In addition to providing time-series trajectory points for tracking control, the optimized primitives contain reference control quantities for linearizing the online control system, as well as optimal controller parameters generated based on fuzzy logic with respect to the category of primitives. Moreover, the control optimization problem at the primitive transition segment is solved by introducing a game-coordinated control strategy. Simulation results demonstrate that the CDBP-based control method proposed in this paper can enhance the control accuracy within the primitives and also effectively solve the smooth transition issue between the primitives.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Integration of HD map and Onboard Sensors
Abstract: For an autonomous vehicle to plan a path in its environment, it must be able to accurately forecast the trajectory of all dynamic objects in its proximity. While many traditional methods encode observations in the scene to solve this problem, there are few approaches that consider the effect of the ego vehicle's behavior on the future state of the world. In this paper, we introduce VRD, a vectorized world model-inspired approach to the multi-agent motion forecasting problem. Our method combines a traditional open-loop training regime with a novel dreamed closed-loop training pipeline that leverages a kinematic reconstruction task to imagine the trajectory of all agents, conditioned on the action of the ego vehicle. Quantitative and qualitative experiments are conducted on the Argoverse 2 multi-world forecasting evaluation dataset and the intersection drone (inD) dataset to demonstrate the performance of our proposed model. Our model achieves state-of-the-art performance on the single prediction miss rate metric on the Argoverse 2 dataset and performs on par with the leading models for the single prediction displacement metrics.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, Sensor Signal Processing
Abstract: Slip parameter prediction is crucial for motion planning and control of unmanned skid-steering vehicles in off-road environments. Slip parameter prediction methods based on nonlinear observers and those based on machine learning models both have limitations under various conditions. Therefore, this paper presents a multi-layer Adaptive Unscented Kalman Filter (AUKF) slip parameter prediction method based on a fusion framework of nonlinear observers and machine learning models. The method first constructs the 0-layer AUKF using the vehicle kinematic model and sensor data to initialize the slip parameters. Then, with the input of the desired sequences of wheel speeds generated by the autonomous driving system, the 1-layer AUKF is constructed by combining the machine learning predictive model and running N times to obtain the future slip parameter sequence. Experimental data was collected by driving on paved and dirt roads with a skid-steering vehicle. The experimental results show that the method in this paper outperforms methods based on nonlinear observers in terms of slip parameter prediction accuracy when the prediction time domain is long. Furthermore, when faced with unknown conditions, this method shows superior robustness compared to methods based on machine learning models.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, SLAM (Simultaneous Localization and Mapping)
Abstract: Long wavelength light detection and ranging (LiDAR) sensors have emerged as an essential component for increasing the accuracy and range of perception of autonomous vehicles because they employ directed lasers with wavelengths longer than 1μm. However, adverse weather conditions like fog, rain, and snow pose a major challenge. Long-wavelength lasers generally exhibit increased absorption and scattering by water-based ambient particles compared to those with short wavelengths, which reduces sensor accuracy. Filtering out ambient particles is crucial for accurately representing the surrounding environment to ensure safe navigation. Despite extensive research on filtering snow particles from LiDAR point clouds, there is little documented research on long-wavelength LiDAR. Furthermore, existing filters that can be used with long-wavelength LiDAR sensors are limited in speed and accuracy, impeding their implementation in autonomous vehicles. In this paper, we propose a Network-Adjusted Reflectance Filter (NARF), a novel two-phase, physics-informed filtering method for long-wavelength LiDAR that outperforms the state-of-the-art geometric filters in terms of both speed and accuracy. The NARF first uses a physics-based range-corrected directional reflectance (RCDR) filter for initial snow particle classification, followed by a CNN-based RestoreNet to refine the RCDR predictions. Due to the lack of open-source datasets collected from long-wavelength LiDAR systems, we use a custom experimental dataset obtained during a snow event to train and validate the proposed filter.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Optimal task and motion planning (TAMP) has seen an increase in interest in recent years. In this paper we propose methods for using numerical optimal control to improve upon a feasible solution to a TAMP rearrangement problem. The methods are extensions of existing improvement methods for pure motion planning. The first method poses an optimal control problem (OCP) to simultaneously improve all motions in the plan. The second method, which we denote multiple finite horizons (MFH), takes inspiration from finite horizon control and poses a sequence of finite horizon OCPs involving variables for the positions of temporary placements of movable objects as well as motions in the plan, such that after solving each problem a feasible plan is maintained and the plan cost is non-increasing after each step. The methods are evaluated on example plans in numerical experiments, and the results show that both methods improve the plan for the evaluated problems. Results also show that MFH can reduce the computation time compared to the first method, and that depending on the horizon length and the problem it is sometimes possible to achieve plans of similar quality as when all motions are optimized at the same time.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Automated vehicles are expected to enter the market in the coming years. To achieve this goal, it is essential to develop vehicles that prioritize both safety and reliability, but they should also ensure a comfortable user experience. As a result, autonomous driving functions should adapt to the individual preferences and requirements of drivers. This paper proposes an algorithm to dynamically update the parameters of a motion planner while driving to fit a driving style based on human-driver data. The motion planner generates multiple trajectory candidates and it can adjust the selection criteria online to prioritize the trajectory that matches better to a given behavior. The system was implemented in a real vehicle and compared to different human drivers, showing that it can reproduce their driving styles.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Motion planning at intersections is a challenging problem in autonomous driving due to the complicated interactions. The existing pipeline of "planning after predicting" is too conservative, can reduce traffic efficiency. Using game theory to model the non-cooperative coupling relationships between multiple vehicles can resolve the above problems, but such methods cannot guarantee safety without collision. This paper presents motion planning for autonomous driving with safe differential games based on Control Barrier Function (CBF), and also provides a safety-critical generalized Nash equilibrium seeking algorithm. We handle the hard CBF constraints through augmented Lagrangian multiplier method. Motivated by iterative Linear-Quadratic Game (iLQG) algorithm, we use the Taylor expansion method to approximate the model into an LinearQuadratic (LQ) structure, and then incrementally solve this problem with an iterative feedback LQ game algorithm. Through Carla simulation and hardware testing, our results indicate that the algorithm can find a balance between safety and efficiency while maintaining real-time implementation performance.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Navigating in an unknown, unstructured environment is a crucial task in autonomous driving navigation. However, the majority of existing path planners struggle to efficiently navigate in environments with partial sensor observations and irregularly placed obstacles. In this paper, we introduce a novel path planner referred to as the Incremental Distance Map-based Path Planner (IDM Planner). The IDM Planner initially employs an Incremental Distance Map to facilitate precise representations of the environment. To enhance the efficiency of map updating, a real-time update method using a hierarchical data structure is proposed. Subsequently, the planning stage involves constructing a search tree from the initial and target states, integrating efficient node expansion and rapid collision checking mechanisms. Finally, a planning framework with an event-triggered replanning mechanism is designed to achieve high safety performance in real-time applications. Extensive experiments conducted using the TPCAP benchmark and CARLA simulation validate the capability of our planner to achieve enhanced efficiency and safety performance.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: We present MBAPPE, a novel approach to motion planning for autonomous driving combining tree search with a partially-learned model of the environment. Leveraging the inherent explainable exploration and optimization capabilities of the Monte-Carlo Search Tree (MCTS), our method addresses complex decision-making in a dynamic environment. We propose a framework that combines MCTS with supervised learning, enabling the autonomous vehicle to effectively navigate through diverse scenarios. Experimental results demonstrate the effectiveness and adaptability of our approach, showcasing improved real-time decision-making and collision avoidance. This paper contributes to the field by providing a robust solution for motion planning in autonomous driving systems, enhancing their explainability and reliability.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: This paper introduces a novel numerical approach to achieving smooth lane-change trajectories in autonomous driving scenarios. Our trajectory generation approach leverages particle swarm optimization (PSO) techniques, incorporating Neural Network (NN) predictions for trajectory refinement. The generation of smooth and dynamically feasible trajectories for the lane change maneuver is facilitated by combining polynomial curve fitting with particle propagation, which can account for vehicle dynamics. The proposed planning algorithm is capable of determining feasible trajectories with real-time computation capability. We conduct comparative analyses with two baseline methods for lane changing, involving analytic solutions and heuristic techniques in numerical simulations. The simulation results validate the efficacy and effectiveness of our proposed approach.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Navigating off-road terrains is crucial for military, agricultural, and rescue operations. Existing algorithms for off-road path planning offer limited adaptability to complex terrains and often lack the computational efficiency required for real-time applications. This is largely due to the nonconvex and nonsmooth characteristics of terrain geometry. Our research introduces an innovative terrain representation technique that streamlines the complexity of the terrain into a manageable path optimization problem, focusing on optimizing vehicle attitude concerning the path. By employing discrete curves to represent lateral terrain elevation changes, our method facilitates the direct integration of vehicle attitude into the optimization framework, thereby diminishing the need for computationally intensive traversability maps typical of traditional approaches. We tackle the resulting nonlinear optimization problem with a constrained iterative linear quadratic regulator (iLQR), achieving real-time path planning capabilities. The proposed method demonstrates improved computational efficiency and enhanced path quality, demonstrating significant time savings in planning while ensuring high-quality outcomes.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Trajectory augmentation serves as a means to mitigate distributional shift in imitation learning. However, imitating trajectories that inadequately represent the original expert data can result in undesirable behaviors, particularly in safety-critical scenarios. We propose a trajectory augmentation method designed to maintain similarity with expert trajectory data. To accomplish this, we first cluster trajectories to identify minority yet safety-critical groups. Then, we combine the trajectories within the same cluster through geometrical transformation to create new trajectories. These trajectories are then added to the training dataset, provided that they meet our specified safety-related criteria. Our experiments exhibit that training an imitation learning model using these augmented trajectories can significantly improve closed-loop performance.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Alongside optimization-based planners, sampling-based approaches are often used in trajectory planning for autonomous driving due to their simplicity. Model predictive path integral control is a framework that builds upon optimization principles while incorporating stochastic sampling of input trajectories. This paper investigates several sampling approaches for trajectory generation. In this context, normalizing flows originating from the field of variational inference are considered for the generation of sampling distributions, as they model transformations of simple to more complex distributions. Accordingly, learning-based normalizing flow models are trained for a more efficient exploration of the input domain for the task at hand. The developed algorithm and the proposed sampling distributions are evaluated in two simulation scenarios.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Planning the trajectory of the controlled ego vehicle is a key challenge in automated driving. As for human drivers, predicting the motions of surrounding vehicles is important to plan the own actions. Recent motion prediction methods utilize equivariant neural networks to exploit geometric symmetries in the scene. However, no existing method combines motion prediction and trajectory planning in a joint step while guaranteeing equivariance under roto-translations of the input space. We address this gap by proposing a lightweight equivariant planning model that generates multi-modal joint predictions for all vehicles and selects one mode as the ego plan. The equivariant network design improves sample efficiency, guarantees output stability, and reduces model parameters. We further propose equivariant route attraction to guide the ego vehicle along a high-level route provided by an off-theshelf GPS navigation system. This module creates a momentum from embedded vehicle positions toward the route in latent space while keeping the equivariance property. Route attraction enables goal-oriented behavior without forcing the vehicle to stick to the exact route. We conduct experiments on the challenging nuScenes dataset to investigate the capability of our planner. The results show that the planned trajectory is stable under roto-translations of the input scene which demonstrates the equivariance of our model. Despite using only a small split of the dataset for training, our method improves L2 distance at 3 s by 20.6% and surpasses the state of the art.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Automated vehicles need to estimate tire-road friction information, as it plays a key role in safe trajectory planning and vehicle dynamics control. Notably, friction is not solely dependent on road surface conditions, but also varies significantly depending on the tire temperature. However, tire parameters such as the friction coefficient have been conventionally treated as constant values in automated vehicle motion planning. This paper develops a simple thermodynamic model that captures tire friction temperature variation. To verify the model, it is implemented into trajectory planning for automated drifting - a challenging application that requires leveraging an unstable, drifting equilibrium at the friction limits. The proposed method which captures the hidden tire dynamics provides a dynamically feasible trajectory, leading to more precise tracking during experiments with an LQR (Linear Quadratic Regulator) controller.

Keywords: Perception Including Object Event Detection and Response (OEDR), Sensor Fusion for Localization, Sensor Signal Processing
Abstract: In autonomous driving, addressing occlusion scenarios is crucial yet challenging. Robust surrounding perception is essential for handling occlusions and aiding navigation. State-of-the-art models fuse LiDAR and Camera data to produce impressive perception results, but detecting occluded objects remains challenging. In this paper, we emphasize the crucial role of temporal cues in reinforcing resilience against occlusions in the bird’s eye view (BEV) semantic grid segmentation task. We proposed a novel architecture that enables the processing of temporal multi-step inputs, where the input at each time step comprises the spatial information encoded from fusing LiDAR and camera sensor readings. We experimented on the real-world nuScenes dataset and our results outperformed other baselines, with particularly large differences when evaluating on occluded and partially-occluded vehicles. Additionally, we applied the proposed model to downstream tasks, such as multi-step BEV prediction and trajectory forecasting of the ego-vehicle. The qualitative results obtained from these tasks underscore the adaptability and effectiveness of our proposed approach.

Keywords: Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS), Automated Vehicles
Abstract: Detecting oncoming vehicles at night as early as possible is important for highly automated driving and Advanced-Driver-Assistance-Systems (ADAS). The sooner objects are detected, the earlier autonomous systems can take them into consideration to plan more anticipatory and safe actions. Previous work showed that on rural land roads at night, oncoming vehicles can already be detected before they are actually directly visible. This is done based on their emitted light. However, no work exists on covering the problem for more complex scenarios such as urban areas in cities. In this paper, we present a new approach to annotate light reflections in urban scenarios caused by oncoming vehicles at night before they are directly visible. We revisit design decisions in previous work for rural land road scenarios and find several improvements. We propose a pipeline which takes relatively cheap-to-get, yet highly subjective human Bounding-Box (BB) annotations, and automatically turns them into normally expensive-to-get, more objective binary masks. In our annotation experiment, we show that labeling light reflections is far more challenging and complex than conventional objects. Also, we show that our method can improve inter-annotator agreement and filter out annotator subjectivity. We train several State-Of-The-Art (SOTA) neural networks for semantic segmentation to demonstrate that our annotations can be used to detect light reflections from oncoming vehicles in urban scenarios before they are directly visible.

Keywords: Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS), Automated Vehicles
Abstract: In the domain of autonomous driving, the integration of multi-modal perception techniques based on data from diverse sensors has demonstrated substantial progress. Effectively surpassing the capabilities of state-of-the-art single-modality detectors through sensor fusion remains an active challenge. This work leverages the respective advantages of cameras in perspective view and radars in Bird's Eye View (BEV) to greatly enhance overall detection and tracking performance. Our approach, Camera-Radar Associated Fusion Tracking Booster (CRAFTBooster) represents a pioneering effort to enhance radar-camera fusion in the tracking stage, contributing to improved 3D MOT accuracy. The superior experimental results on K-Radar dataset, which exhibit 5-6% on IDF1 tracking performance gain, validate the potential of effective sensor fusion in advancing autonomous driving.

Keywords: Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS), Automated Vehicles
Abstract: In this paper, we present an innovative trajectory prediction algorithm that is specifically crafted for high-speed autonomous racing, with a focus on the mutual influence between vehicles. This algorithm was developed in the context of the Hyundai Autonomous Challenge 2023, a pioneering event that featured the world's first competitive racing scenario involving three autonomous vehicles simultaneously navigating a road course race track. Stable overtaking in 1:N races requires accurate prediction of the trajectories of surrounding vehicles, taking into account their inter-vehicle dynamics. To meet this challenge, our approach leverages the Model Predictive Path Integral technique, which not only considers information from neighboring vehicles but also incorporates prior knowledge of the race track. Furthermore, we have augmented our algorithm with maneuver intention estimation-based trajectory prediction, an approach that leverages a vehicle's historical trajectory data to forecast its future path. By integrating these two methodologies, our algorithm adeptly anticipates the motion of other vehicles under a variety of conditions on the race track. The efficacy of our proposed solution has been substantiated through extensive simulation and real-world testing, demonstrating its capability to deliver real-time performance in high-speed environments, with a processing time as low as 20 milliseconds.

Keywords: Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS), Automated Vehicles
Abstract: For the development of advanced driver assistance systems (ADAS) and autonomous driving, a perception of the vehicle's environment is necessary. This includes, among others, road gradients, cross-slopes, and the road surface condition, with the maximum friction coefficient of the tire-road contact as a safety-relevant parameter. However, these three road parameters cannot be measured directly while driving by sensors installed in modern vehicles. Current estimation methods provide either the maximum friction coefficient or the road gradient and cross-slope but never combined. Since the road angles influence the maximum friction coefficient estimation and vice versa, separate estimation of these parameter, in general, leads to incorrect estimation results. In this paper, a new Unscented Kalman Filter (UKF)-based approach is proposed for simultaneous estimation of all three mentioned road parameters. For this purpose, a dynamic vehicle model considering road gradients and cross-slopes is introduced and integrated into the UKF. It is demonstrated that, in contrast to a state-of-the-art UKF, the proposed algorithm yields improved accuracy and correct maximum friction coefficient estimates even on roads with gradients or cross-slopes.

Keywords: Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS), Pedestrian Protection
Abstract: Autonomous vehicles are equipped with a wide range of sensors and corresponding object detectors tuned to reliably detect vulnerable road users in a variety of conditions. However, the activation scores of these object detectors are easily influenced by contextual factors. To address this challenge, we propose a probabilistic, sensor-agnostic and context-adaptive calibration layer that translates the activation scores of the candidate detections into likelihood ratios that are tuned to that specific context. Our method, seamlessly integrated with the underlying object detector, effectively enhances detection precision by mitigating contextual biases. As a proof of concept, we demonstrate that calibrating the activation scores for four pre-trained state-of-the-art detectors achieves an average precision improvement of up to 4% on the Waymo open dataset for the specific task of pedestrian detection using regular RGB cameras. In challenging scenarios, the average precision can improve up to 9%. Additionally, we showcase that context-calibration emerges as a viable alternative to conventional transfer learning when dealing with limited datasets.

Keywords: Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS)
Abstract: In this work, we present a novel approach for general object segmentation from a monocular image, eliminating the need for manually labeled training data and enabling rapid, straightforward training and adaptation with minimal data. Our model initially learns from LiDAR during the training process, which is subsequently removed from the system, allowing it to function solely on monocular imagery. This study leverages the concept of the Stixel-World to recognize a medium level representation of its surroundings. Our network directly predicts a 2D multi-layer Stixel-World and is capable of recognizing and locating multiple, superimposed objects within an image. Due to the scarcity of comparable works, we have divided the capabilities into modules and present a free space detection in our experiments section. Furthermore, we introduce an improved method for generating Stixels from LiDAR data, which we use as ground truth for our network.

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: This work addresses the problem of object segmentation and pixel-wise distance (disparity) estimation in a Multi- Task Learning (MTL) framework, where semantic segmentation enjoys robustness against unexpected objects via Out-of- Distribution (OoD) segmentation. The proposed MTL network, referred to as ’OSDNet’, leverages the positive transfer between in-distribution, OoD and disparity learning, while kept sufficiently less complex for real-time applications. This positive transfer resulted in 0.52 % improvement in mean Intersection over Union (mIoU), 3 % increase in Area Under the Precision Recall Curve (AUPRC) with 7 % improvement in error (FPR95) of MTL in semantic segmentation and OoD detection, respectively, compared to Single-Task Learning (STL) approach for semantic segmentation task (’OSNet’ is used in STL approach which has only semantic segmentation head while rest of the architecture is same as in ’OSDNet’). Due to the missing semantic labels of publicly-available dataset for training, we propose a semi-automatic relabeling technique. In this work, we orient ourselves in both data-centric and model-centric approaches, i.e., a new set of data is derived for the learning process, and a novel model is proposed for the target MTL. The code and the resultant dataset are made publicly available at https://github.com/ravivaghasiya1998/MTL_OSDNet.

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: The construction of local HD (High Definition) Map and Lane Network with onboard sensors is critical for autonomous vechicles and facilitates downstream tasks. In contrast to previous studies that treated building HD Map and Lane Network as two individual tasks, in this paper a unified BEV (Bird's-Eye-View) perception framework is proposed with seperate decoders to realize two tasks simultaneously. In this paper, the gap between object detection and curve regression when using a DETR-like decoder is discussed and a curve region aware method is proposed to make up for the above gap. Specifically, a mechanism called Curve Region Aware Deformable Attention is designed with a bezier grid sampling module to guide the attention learning in bev features and structual loss regarding shapes of lanelines is also included. Moreover, a BEV spatial-temporal fusion method is introduced to better utilize historical features with minimal loss of spatial information. The results on NuScenes dataset show that our work has been close to or exceeded SOTAs (state-of-the-art) on both two tasks simultaneously.

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: The foundational role of datasets in defining the capabilities of deep learning models has led to their rapid proliferation. At the same time, published research focusing on the process of dataset development for environment perception in automated driving has been scarce, thereby reducing the applicability of openly available datasets and impeding the development of effective environment perception systems. Sensor-based, mapless automated driving is one of the contexts where this limitation is evident. While leveraging real-time sensor data, instead of pre-defined HD maps promises enhanced adaptability and safety by effectively navigating unexpected environmental changes, it also increases the demands on the scope and complexity of the information provided by the perception system.

To address these challenges, we propose a scenario- and capability-based approach to dataset development. Grounded in the principles of ISO 21448 (safety of the intended functionality, SOTIF), extended by ISO/TR 4804, our approach facilitates the structured derivation of dataset requirements. This not only aids in the development of meaningful new datasets but also enables the effective comparison of existing ones. Applying this methodology to a broad range of existing lane detection datasets, we identify significant limitations in current datasets, particularly in terms of real-world applicability, a lack of labeling of critical features, and an absence of comprehensive information for complex driving maneuvers.

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: This paper introduces a novel hybrid architecture that enhances radar-based Dynamic Occupancy Grid Mapping (DOGM) for autonomous vehicles, integrating deep learning for state-classification. Traditional radar-based DOGM often faces challenges in accurately distinguishing between static and dynamic objects. Our approach addresses this limitation by introducing a neural network-based DOGM state correction mechanism, designed as a semantic segmentation task, to refine the accuracy of the occupancy grid. Additionally a heuristic fusion approach is proposed which allows to enhance performance without compromising on safety. We extensively evaluate this hybrid architecture on the NuScenes Dataset, focusing on its ability to improve dynamic object detection as well grid quality. The results show clear improvements in the detection capabilities of dynamic objects, highlighting the effectiveness of the deep learning-enhanced state correction in radar-based DOGM.

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles, Integration of Infrastructure and Intelligent Vehicles
Abstract: Semantic segmentation of 3D LiDAR point clouds, essential for autonomous driving and infrastructure management, is best achieved by supervised learning, which demands extensive annotated datasets and faces the problem of domain shifts. We introduce a new 3D semantic segmentation pipeline that leverages aligned scenes and state-of-the-art 2D segmentation methods, avoiding the need for direct 3D annotation or reliance on additional modalities such as camera images at inference time. Our approach generates 2D views from LiDAR scans colored by sensor intensity and applies 2D semantic segmentation to these views using a camera-domain pretrained model. The segmented 2D outputs are then back-projected onto the 3D points, with a simple voting-based estimator that merges the labels associated to each 3D point. Our main contribution is a global pipeline for 3D semantic segmentation requiring no prior 3D annotation and not other modality for inference, which can be used for pseudo-label generation. We conduct a thorough ablation study and demonstrate the potential of the generated pseudo-labels for the Unsupervised Domain Adaptation task.

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles, Sensor Signal Processing
Abstract: In the field of autonomous vehicles, the detection of road line markings is a crucial yet versatile component. It provides real-time guidance for navigation and low-level vehicle control, while it also enables the generation of lane-level HD maps. These maps require high precision to provide low-level details to all future map users. At the same time, control-oriented detection pipelines require increased inference frequency and high robustness to be deployed on a safety-critical system. With this work, we present OptimusLine, a versatile line detection pipeline tackling with ease both scenarios. Built around a frame-by-frame transformer-based neural model operating in image segmentation, we show that OptimusLine achieves state-of-the-art performance and analyze its computational impact. To provide robustness to perturbations when deployed on an actual vehicle, OptimusLine introduces a scheme exploiting temporal links between consecutive frames. Enforcing temporal consistency on each new line prediction, OptimusLine can generate more robust line descriptions and produce an estimate of its prediction uncertainty.

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles, Sensor Signal Processing
Abstract: 3D object detection at long-range is crucial for ensuring the safety and efficiency of self-driving vehicles, allowing them to accurately perceive and react to objects, obstacles, and potential hazards from a distance. But most current state-of-the-art LiDAR based methods are range limited due to sparsity at long-range, which generates a form of domain gap between points closer to and farther away from the ego vehicle. Another related problem is the label imbalance for faraway objects, which inhibits the performance of Deep Neural Networks at long-range. To address the above limitations, we investigate two ways to improve long-range performance of current LiDAR-based 3D detectors. First, we combine two 3D detection networks, referred to as range experts, one specializing at near to mid-range objects, and one at long-range 3D detection. To train a detector at long-range under a scarce label regime, we further weigh the loss according to the labelled point’s distance from ego vehicle. Second, we augment LiDAR scans with virtual points generated using Multimodal Virtual Points (MVP), a readily available image-based depth completion algorithm. Our experiments on the long-range Argoverse2 (AV2) dataset indicate that MVP is more effective in improving long range performance, while maintaining a straightforward implementation. On the other hand, the range experts offer a computationally efficient and simpler alternative, avoiding dependency on image-based segmentation networks and perfect camera-LiDAR calibration.

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles
Abstract: We address the challenge of domain adaptation in LiDAR-based 3D object detection by introducing a simple yet effective training strategy known as Gradual Batch Alternation. This method enables adaptation from a well-labeled source domain to an insufficiently labeled target domain. Initially, training commences with alternating batches of samples from both the source and target domains. As the training progresses, we gradually reduce the number of samples from the source domain. Consequently, the model undergoes a gradual transition towards the target domain, resulting in improved adaptation. Domain adaptation experiments for 3D object detection on four benchmark autonomous driving datasets, namely ONCE, PandaSet, Waymo, and nuScenes, demonstrate significant performance gains over prior works and strong baselines.

Keywords: Cooperative Vehicles, Automated Vehicles
Abstract: Accurate and comprehensive Bird's Eye View (BEV) semantic segmentation is essential for ensuring safe and proactive navigation in autonomous driving. Although cooperative perception has exceeded the detection capabilities of single-agent systems, prevalent camera-based algorithms in cooperative perception neglect valuable information derived from historical observations. This limitation becomes critical during sensor failures or communication issues as cooperative perception reverts to single-agent perception, leading to degraded performance and incomplete BEV segmentation maps. This paper introduces TempCoBEV, a temporal module designed to incorporate historical cues into current observations, thereby improving the quality and reliability of BEV map segmentations. We propose an importance-guided attention architecture to effectively integrate temporal information that prioritizes relevant properties for BEV map segmentation. TempCoBEV is an independent temporal module that seamlessly integrates into state-of-the-art camera-based cooperative perception models. We demonstrate through extensive experiments on the OPV2V dataset that TempCoBEV performs better than non-temporal models in predicting current and future BEV map segmentations, particularly in scenarios involving communication failures. We show the efficacy of TempCoBEV and its capability to integrate historical cues into the current BEV map, improving predictions under optimal communication conditions by up to 2% and under communication failures by up to 19%. The code is available at https://github.com/cvims/TempCoBEV.

Keywords: Cooperative Vehicles, Automated Vehicles
Abstract: This survey analyzes intermediate fusion methods in collaborative perception for autonomous driving, categorized by real-world challenges. We examine various methods, detailing their features and the evaluation metrics they employ. The focus is on addressing challenges like transmission efficiency, localization errors, communication disruptions, and heterogeneity. Moreover, we explore strategies to counter adversarial attacks and defenses, as well as approaches to adapt to domain shifts. The objective is to present an overview of how intermediate fusion methods effectively meet these diverse challenges, highlighting their role in advancing the field of collaborative perception in autonomous driving.

Keywords: Cooperative Vehicles, Automotive Cyber Physical Systems
Abstract: Connected and Automated Vehicles (CAVs) have the potential to enhance traffic safety and efficiency. In contrast, aligning both vehicles' utility with system-level interests in scenarios with conflicting road rights is challenging, hindering cooperative driving. This paper advocates a game theory model, which strategically incorporates deceptive information within incomplete information vehicle games, operating under the premise of imprecise perceptions. The equilibria derived reveal that CAVs can exploit deceptive strategies, not only gaining advantages that undermine the utility of the other vehicle in the game but also posing hazards to the overall benefits of the transportation system. Vast experiments were conducted, simulating diverse inbound traffic conditions at an intersection, validating the detrimental impact on efficiency and safety resulting from CAVs with perception uncertainties, and employing deceptive maneuvers within connected and automated transportation systems. Finally, the paper proposes feasible solutions and potential countermeasures to address the adverse consequences of deception in connected and automated transportation systems. It concludes by calling for integrating these insights into future research endeavors and pursuing to fully realize the potential and expectations of CAVs in enhancing the whole traffic performance.

Keywords: Cooperative Vehicles, Eco-Driving and Energy-Efficient Vehicles, Vehicle Control and Motion Planning
Abstract: This paper treats the uncertainties in practical train operations as disturbances in train dynamics and proposes a robust optimization method for multi-train operations. First, the indeterministic problem is transformed into a deterministic problem, by linearizing the train dynamics, analyzing the propagation of disturbances, and introducing new state variables. Then, a nonlinear program (NLP) is developed to solve the deterministic problem. The speed profiles of each train are simultaneously optimized to minimize the total traction energy while ensuring safe-separation among adjacent trains. Our results show that operation constraints and absolute safe time headway can always be guaranteed, even in the worst-case scenarios caused by disturbances.

Keywords: Cooperative Vehicles, Functional Safety in Intelligent Vehicles, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications
Abstract: Ensuring safety is the most important factor in connected and autonomous vehicles, especially in emergency braking situations. As such, assessing the safety benefits of one information topology over other is a necessary step towards evaluating and ensuring safety. In this paper, we compare the safety benefits of a cooperative adaptive cruise control which utilizes information from one predecessor vehicle (CACC) with the one that utilizes information from multiple predecessors (CACC+) for the maintenance of spacing under an emergency braking scenario. A constant time headway policy is employed for maintenance of spacing (that includes a desired standstill spacing distance and a velocity dependent spacing distance) between the vehicles in the platoon. The considered emergency braking scenario consists of braking of the leader vehicle of the platoon at its maximum deceleration and that of the following vehicles to maintain the spacing as per CACC or CACC+. By focusing on the standstill spacing distance and utilizing Monte Carlo simulations, we assess the safety benefits of CACC+ over CACC by utilizing the following safety metrics: (1) probability of collision, (2) expected number of collisions, and (3) severity of collision (defined as the relative velocity of the two vehicles at impact). We present and provide discussion of these results.

Keywords: Cooperative Vehicles, Future Mobility and Smart City
Abstract: While autonomous intersection management (AIM) emerges to facilitate signal-free scheduling for connected and autonomous vehicles (CAVs), several challenges arise for planning secure and swift trajectories. Existing works mainly focus on addressing the challenge of multi-CAV interaction complexity. In this context, multi-agent reinforcement learning-based methods exhibit higher scalability and efficiency compared with other traditional methods. However, current AIM methods omit discussions on the practical challenge of CAV heterogeneity. As CAVs exhibit different dynamics features and perception capabilities, it is inappropriate to adapt identical control schemes. Besides, existing MARL methods that lack heterogeneity adaptability may experience a performance decline. In response, this paper exploits MARL to model the decision-making process among CAVs and proposes a novel heterogeneous-agent attention gated trust region policy optimization (HAG-TRPO) method. The proposed method can accomplish more effective and efficient AIM with CAV discrepancies by applying a sequential update schema that boosts the algorithm adaptability for MARL tasks with agentlevel heterogeneity. In addition, the proposed method utilizes the attention mechanism to intensify vehicular cognition on disordered ambience messages, as well as a gated recurrent unit for temporal comprehension on global status. Numerical experiments verify that our method results in CAVs passing at the intersection with fewer collisions and faster traffic flow, showing the superiority of our method over existing benchmarks in terms of both traffic safety and efficiency.

Keywords: Cooperative Vehicles, Human Factors for Intelligent Vehicles, Future Mobility and Smart City
Abstract: Intersections are a major source of traffic accidents and congestion. The development of Connected and Autonomous Vehicles (CAVs) is expected to improve safety and traffic efficiency at intersections, but there will be a lengthy period with the existence of Human-driven Vehicles (HVs). For mixed-traffic intersection management, an iconic reservation-based scheduling approach, H-AIM [1], introduces a concept of "active green trajectory" and has a noteworthy performance. However, the traffic model of H-AIM schedules traffic lights for different roads in a rotating manner, which causes lower scheduling flexibility. Therefore, we propose a trigger-based scheduling approach which allows an Intersection Manager (IM) to detect HVs on lanes and trigger the traffic lights correspondingly. Besides, to encourage CAV adoption for more efficient traffic, we propose principles of designing the combinations of turning policies favoring CAVs. The experimental results show that our proposed trigger-based scheduling approach outperforms H-AIM [1], and the turning policy assignment provides insights to mixed-traffic intersection management.

Keywords: Cooperative Vehicles, Smart Infrastructure, Automated Vehicles
Abstract: This survey offers a comprehensive examination of collaborative perception datasets in the context of Vehicle-to-Infrastructure (V2I), Vehicle-to-Vehicle (V2V), and Vehicle-to-Everything (V2X). It highlights the latest developments in large-scale benchmarks that accelerate advancements in perception tasks for autonomous vehicles. The paper systematically analyzes a variety of datasets, comparing them based on aspects such as diversity, sensor setup, quality, public availability, and their applicability to downstream tasks. It also highlights the key challenges such as domain shift, sensor setup limitations, and gaps in dataset diversity and availability. The importance of addressing privacy and security concerns in the development of datasets is emphasized, regarding data sharing and dataset creation. The conclusion underscores the necessity for comprehensive, globally accessible datasets and collaborative efforts from both technological and research communities to overcome these challenges and fully harness the potential of autonomous driving.

Keywords: Cooperative Vehicles, Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Truck platooning is a promising technology, especially reducing fuel consumption. However, conventional truck platooning methods are short of field tests and lack the capability of platoon lane-change. It impedes the large-scale implementation, since a truck platoon may usually be blocked by a slow-moving vehicle. To address this challenge, we propose a truck platooning controller with lane change capability. This controller is highlighted for the following features: i) enhancing mobility by cutting through traffic one-by-one like a snake; ii) enhancing string stability by formulating in the spatial domain; iii) enhancing planning accuracy by utilizing a tractor-semitrailer-based truck dynamics model; iv) ready for large-scale implementation since it passes our field tests. Results have demonstrated that the proposed controller enhances the mobility of truck platooning by 7.44%.

Keywords: Cooperative Vehicles, Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Multi-lane formation control can significantly enhance the efficiency of traffic systems in multi-lane scenarios. However, existing multi-lane formation control methods are mostly developed for fully Connected and Automated Vehicle (CAV) environments and lack a mechanism for multi-lane formation control in mixed traffic. This paper proposes a CAV formation grouping method and an interaction mechanism of CAVs and Human-driven Vehicles (HDVs), which is suitable for different traffic volumes and penetration rates. Depending on the positional relationship between the CAV formation and HDVs, it flexibly chooses between conventional formation control methods or improved mixed traffic formation control methods. By conducting simulations at various input flow volumes and penetration rates, the applicability of this method to different conditions is verified. The paper also explores the extent to which multi-lane formation control methods can improve traffic efficiency and their relationship with traffic volume and penetration rate.

Keywords: Cooperative Vehicles, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Integration of Infrastructure and Intelligent Vehicles
Abstract: While autonomous driving roadmaps envision large data to be exchanged in V2X scenarios, current V2X communication standards only focus on reliable exchange of small objects. The Wireless Reliable Real-Time Protocol (W2RP) addresses this challenge, however, cannot properly handle scenarios where multiple applications share the channel especially as there are predetermined periodic and dynamic retransmissions phases for each data object. If transient error spikes occur, applications will compete for dynamic channel resources, potentially creating critical overload situations that can lead to safety violations. Furthermore, the channel is shared by multiple applications with different criticality, properties and constraints. We propose a two-level hierarchical approach that assigns a fixed budget to each node, with periodic and dynamic transmission parts of a node's applications being scheduled by the node itself. Prioritization is used to differentiate applications based on their criticality, thereby enabling reliable data exchange for safety-critical applications. The concept was evaluated using an OMNeT++-based simulation of an Automated Valet Parking use case and proved highly effective in enabling safe sample exchange even under varying channel conditions, while offering significantly more efficient resource reservation compared to a static configuration.

Keywords: Cooperative Vehicles, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Simulation and Real-World Testing Methodologies
Abstract: Negotiation is a class of cooperation enabled by vehicle-to-everything (V2X) communication, which involves the exchange of maneuver requests and responses between road users. In this paper, we develop criteria for request initiation and response generation under a unified conflict analysis framework. This leads to guaranteed maneuver feasibility in request and response that satisfy user-based behavior preferences. We implement negotiation via commercially available V2X devices, and experimentally evaluate the benefits of negotiation in conflict resolution. We demonstrate that negotiation can significantly benefit time efficiency of maneuvers while ensuring safety, compared to lower levels of cooperation such as status-sharing and intent-sharing. These benefits and their degradation under communication delays are quantified.

Keywords: Cooperative Vehicles, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications
Abstract: Shared situation awareness (SSA) has been garnering explosive interest in various applications for intelligent transportation system (ITS). In addition, the delay-constrained nature of supporting vehicular networks makes it critical to precisely analyze the performance of a SSA procedure. Extending the relevant literature, this paper provides an analysis framework that evaluates the performance of SSA in spatial and temporal aspects simultaneously. Specifically, this paper provides a closed-form probability distribution for the length of time taken for constitution of a SSA among a group of connected vehicles. This paper extends the calculation to investigation of feasibility of SSA in supporting various types of safety messages defined by the SAE J2735.

Keywords: Cooperative Vehicles, Vehicular Active and Passive Safety, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications
Abstract: Consider a fleet of autonomous vehicles traversing an adversarial terrain that includes obstacles and mines. The goal of the fleet is to ensure that they can complete their mission safely (with minimal casualty) and efficiently (as quickly as possible). In Distributed Coordinated Fleet Management (DCFM), fleet members coordinate with one another while traversing the terrain, e.g., a vehicle encountering an obstacle at a location l can inform other agents so that they can recompute their route to avoid l. In this paper, we consider the problem of cyber-resilient DCFM, i.e., DCFM, in an environment where the adversary can additionally tamper with the cyber-communication performed by the fleet members. Our framework, DRIFT, enables fleet members to coordinate in the presence of such adversaries. Our extensive evaluations demonstrate that DRIFT can achieve a high degree of safety and efficiency against a large spectrum of communication adversaries.

Keywords: Cooperative Vehicles
Abstract: In the field of autonomous driving, collaborative perception has emerged as a promising solution to augment the capabilities of individual sensors by enabling vehicles to share their sensor information with each other, thereby enhancing their situational awareness. This paper addresses the limitations of classical perception in autonomous vehicles by proposing a novel intermediate collaborative perception methodology employing a graph attention network (GAT) to incorporate multiple feature maps and to selectively emphasize important regions within the feature maps. We construct the graph structure as a set of nodes embedding the ego and the neighboring connected vehicles feature maps, as well as establish edge weights between those nodes based on their relationship to each other which is defined by the attention coefficients. The proposed approach leverages both channel and spatial attention-based aggregation and enables the model to determine inter-feature map relationships at a specific channel and spatial regions, while adaptively highlighting the informative regions. This adaptive highlighting mechanism directs the aggregation algorithm towards the most informative areas within the ego and the received feature maps, thereby enhancing the representation power of the ego vehicle’s feature map leading to improved precision in object detection. We quantitatively and qualitatively evaluate the performance of our proposed approach against existing state-of-the-art in collaborative perception. We validate our methodology using V2XSim, a large-scale multi-agent perception dataset. The results demonstrate that our methodology achieves superior performance in enhancing object detection average precision.

Keywords: End-To-End (E2E) Autonomous Driving, Automated Vehicles, Vehicle Control and Motion Planning
Abstract: The network of end-to-end automatic driving algorithms can be divided into perception network part and planning network part. Most of the research on the end-to-end automatic driving algorithm focuses on the part of the perception network, while the improvement of the planning network is less. However, the existing planning network can not effectively use the perceptual features, which may lead to traffic accidents. In this paper, we propose a Transformer-based trajectory prediction network for end-to-end autonomous driving without rules called Target-point Attention Transformer network (TAT). Leveraging the attention mechanism, our proposed model facilitates interaction between the predicted trajectory and perception features, along with target-points. Comparative evaluations with existing conditional imitation learning and GRU-based methods show the superior performance of our approach, particularly in reducing accident occurrences and improving route completion. Extensive assessments conducted in complex closed-loop driving scenarios within urban settings, utilizing the CARLA simulator, affirm the state-of-the-art proficiency of our proposed method.

Keywords: End-To-End (E2E) Autonomous Driving, Automated Vehicles, Human Factors for Intelligent Vehicles
Abstract: Imitation Learning (IL) algorithms such as behavior cloning are a promising direction for learning human-level driving behavior. However, these approaches do not explicitly infer the underlying causal structure of the learned task. This often leads to misattribution about the relative importance of scene elements towards the occurrence of a corresponding action, a phenomenon termed causal confusion or causal misattribution. Causal confusion is made worse in highly complex scenarios such as urban driving, where the agent has access to a large amount of information per time step (visual data, sensor data, odometry, etc.). Our key idea is that while driving, human drivers naturally exhibit an easily obtained, continuous signal that is highly correlated with causal elements of the state space: eye gaze. We collect human driver demonstrations in a CARLA-based VR driving simulator, DReyeVR, allowing us to capture eye gaze in the same simulation environment commonly used in prior work. Further, we propose a contrastive learning method to use gaze-based supervision to mitigate causal confusion in driving IL agents -- exploiting the relative importance of gazed-at and not-gazed-at scene elements for driving decision-making. We present quantitative results demonstrating the promise of gaze-based supervision improving the driving performance of IL agents.

Keywords: End-To-End (E2E) Autonomous Driving, Automated Vehicles
Abstract: In this paper, our focus is on enhancing steering angle prediction for autonomous driving tasks. We initiate our exploration by investigating two veins of widely adopted deep convolutional neural architectures, namely ResNets and InceptionNets. Within both families, we systematically evaluate various model sizes to understand their impact on performance. Notably, our key contribution lies in the incorporation of an attention mechanism to augment steering angle prediction accuracy and robustness. By introducing attention, our models gain the ability to selectively focus on crucial regions within the input data, leading to improved predictive outcomes. Our findings showcase that our attention-enhanced models not only achieve state-of-the-art results in terms of steering angle Mean Squared Error (MSE) but also exhibit enhanced adversarial robustness, addressing critical concerns in real-world deployment. For example, in our experiments on the Kaggle SAP and our created publicly available datasets, attention can lead to over 6% error reduction in steering angle prediction and boost model robustness by up to 56.09%.

Keywords: End-To-End (E2E) Autonomous Driving, Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Reinforcement Learning (RL) over motion skill space has been verified to generate more diverse behaviors than that over low-level control space, and has exhibited superior autonomous driving performance in complex traffic scenarios. However, the incomplete observations pose challenges in achieving efficient skill exploration under unsupervised conditions, hampering the driving performance and applicability. In this paper, we propose a dynamic-aware RL with hybrid network (Da-HnRL) to develop a pseudo-hierarchical planning framework for better motion skill learning in challenging dense traffics. Based on the semi-POMDP modeling, we construct a hybrid network with skip connections as the RL backbone, facilitating a better understanding of the underlying system dynamics. Then we design an efficiency-oriented reward shaping mechanism to incentivize active skill exploration, promoting enhanced trade-off between exploration and exploitation. Furthermore, we provide a comprehensive scoring mechanism for policy identification, ensuring the near-optimality. We validate the proposed methods on challenging dense-traffic tasks. The results demonstrate the superiority of our approach over previous methods, with improved learning efficiency, driving stability and generalization.

Keywords: End-To-End (E2E) Autonomous Driving, Automated Vehicles
Abstract: Vision-based end-to-end driving models trained by imitation learning can lead to affordable solutions for autonomous driving. However, training these well-performing models usually requires a huge amount of data, while still lacking explicit and intuitive activation maps to reveal the inner workings of these models while driving. In this paper, we study how to guide the attention of these models to improve their driving quality and obtain more intuitive activation maps by adding a loss term during training using salient semantic maps. In contrast to previous work, our method does not require these salient semantic maps to be available during testing time, as well as removing the need to modify the model's architecture to which it is applied. We perform tests using perfect and noisy salient semantic maps with encouraging results in both, the latter of which is inspired by possible errors encountered with real data. Using CIL++ as a representative state-of-the-art model and the CARLA simulator with its standard benchmarks, we conduct experiments that show the effectiveness of our method in training better autonomous driving models, especially when data and computational resources are scarce.

Keywords: End-To-End (E2E) Autonomous Driving, Automated Vehicles, Vehicle Control and Motion Planning
Abstract: End-to-end autonomous driving models are widely used currently to avoid error transmission. The training of these models is affected by the imbalance of the training labels, such as imbalanced distributions of trajectory waypoint, steering, throttle, and brake labels. Therefore, the models cannot be trained enough on low-frequency labels. To mitigate this problem, a balanced training method is proposed in this paper. First, a general kernel density estimation method is used to estimate the probability function of labels. Then, to conduct balanced training, we design a cost-sensitive loss function that assigns different weights to each training data according to the probability of the labels. Comparative experiments on the Longest6 Benchmark show that the balanced training method proposed in this article improves the baseline significantly. We also conduct ablation studies to discuss the balanced training among different tasks in multi-task learning, as well as between two output forms: waypoint and action. Due to the generalization of kernel density estimation, the balanced training method proposed in this paper can be extended to any label distribution of two output forms in the end-to-end autonomous driving models.

Keywords: End-To-End (E2E) Autonomous Driving, Sensor Signal Processing, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications
Abstract: With cutting-edge sensors and learning algorithms developed for vehicular perception, breakthrough advancements have been made in the perception-based end-to-end autonomous driving in recent years. However, the reliability of autonomous driving systems could be compromised by the vulnerability of perception module to occlusion. To address this issue, the integration of vehicle-to-vehicle communication enabled perception data sharing in the dynamic driving task has been proposed and has yielded notable results, as demonstrated by COOPERNAUT, a cooperative system based on distributed lidar perception. In this paper, we introduce ICOP, an end-to-end driving system based on multi-agent camera cooperative perception, to select sensor sharing nodes and to fuse intermediate image data features for learning a driving policy. In the ICOP system, each agent encodes image information into Bird’s Eye View (BEV) representations individually, and these representations are then transmitted as payloads of V2X (vehicle-to-everything) messages via wireless connection, thus enables capturing global spatial interactions among agents to form comprehensive BEV perception information used for final control decision-making. Supported by our designed mechanism of vehicle-to-vehicle communication and transformer block to achieve acceptable image sensory data size for transmission, the experiments suggest that the proposed cooperative perception driving system achieves better results than lidar-based systems in challenging driving situations compared to prior works.

Keywords: End-To-End (E2E) Autonomous Driving, Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Autonomous parking is a crucial application for intelligent vehicles, especially in crowded parking lots. The confined space requires highly precise perception, planning, and control. Currently, the traditional Automated Parking Assist (APA) system, which utilizes geometric-based perception and rule-based planning, can assist with parking tasks in simple scenarios. With noisy measurement, the handcrafted rule often lacks flexibility and robustness in various environments, which performs poorly in super crowded and narrow spaces. On the contrary, there are many experienced human drivers, who are good at parking in narrow slots without explicit modeling and planning. Inspired by this, we expect a neural network to learn how to park directly from experts without handcrafted rules. Therefore, in this paper, we present an end-to-end neural network to handle parking tasks. The inputs are the images captured by surrounding cameras and basic vehicle motion state, while the outputs are control signals, including steer angle, acceleration, and gear. The network learns how to control the vehicle by imitating experienced drivers. We conducted closed-loop experiments on the CARLA Simulator to validate the feasibility of controlling the vehicle by the proposed neural network in the parking task. The experiment demonstrated the effectiveness of our end-to-end system in achieving the average position and orientation errors of 0.3 meters and 0.9 degrees with an overall success rate of 91%. The source code will be released after the paper's acceptance.

Keywords: End-To-End (E2E) Autonomous Driving, Vehicle Control and Motion Planning, Automated Vehicles
Abstract: The end-to-end neural network has become a hot topic in recent years. Compared with traditional module-based solutions, the end-to-end paradigm is able to reduce the accumulated error and avoid information loss, so that it earns great attention in autonomous driving tasks. revise{However, the current end-to-end network designs easily lose useful information during training due to the complexity of mapping high-dimensional visual observation to navigation waypoints.} revise{Since the future navigation point is reasoned from the former one, the planning task is like a sequence generation task.} Inspired by the great power of the neural language model, we propose an end-to-end framework, which transfers the planning task as a language sequence generation task conditioned on pixel inputs. The proposed method firstly extracts and transforms the image feature from camera-view to bird-eye-view (BEV). Then the target navigation point is revise{constructed into a text sequence}, as the prompt of the visual-language transformer. Finally, the auto-regressive transformer decoder receives the BEV feature and revise{the text sequences} to generate sequential waypoints. Overall, our proposed method can make full use of the environmental information and express the planning trajectory as a language sequence to revise{learn the correspondence between trajectory sequences and images.} We have conducted extensive experiments on CARLA benchmarks and our model achieves state-of-the-art performance compared with other visual methods. The source code will be open-sourced after the paper's acceptance.

Keywords: Automated Vehicles, End-To-End (E2E) Autonomous Driving, Advanced Driver Assistance Systems (ADAS)
Abstract: Reinforcement learning has been shown to have great potential applications in autonomous driving. For collaborative driving scenarios, multi-agent reinforcement learning can be used to explore efficient and collaborative driving strategies. However, it still faces the challenge of non-stationary. Traditional methods focus on evaluating the similarities between the real state of the teammate and the modeled state. There is also the issue of partial observability. It can be addressed by establishing communication to share information with other surrounding agents. However, prior approaches overlook the efficient communication problem caused by unprocessed and redundant information. To tackle these two challenges, we propose an approach named Multi-Agent Collaboration via Cross-Attention and Communication (MACAC). MACAC leverages the agent's local observations to analyze and capture environment and interaction information, while also incorporating teammate modeling through the exchange of concise state information via communication. In addition, to improve the learning process, we integrate the noisy advantage technique into MACAC to enhance the agent's exploration capabilities. As a result, vehicles can effectively adapt to dynamic environments and exhibit efficient collaborative driving skills. In all, experiments conducted on an autonomous driving simulator demonstrate that our approach surpasses the performance of the baseline algorithms.

Keywords: Automated Vehicles, End-To-End (E2E) Autonomous Driving
Abstract: In autonomous driving tasks, trajectory prediction in complex traffic environments requires adherence to real-world context conditions and behavior multimodalities. Existing methods predominantly rely on prior assumptions or generative models trained on curated data to learn road agents' stochastic behavior bounded by scene constraints. However, they often face mode averaging issues due to data imbalance and simplistic priors, and could even suffer from mode collapse due to unstable training and single ground truth supervision. These issues lead the existing methods to a loss of predictive diversity and adherence to the scene constraints. To address these challenges, we introduce a novel trajectory generator named Controllable Diffusion Trajectory (CDT), which integrates map information and social interactions into a Transformer-based conditional denoising diffusion model to guide the prediction of future trajectories. To ensure multimodality, we incorporate behavioral tokens to direct the trajectory's modes, such as going straight, turning right or left. Moreover, we incorporate the predicted endpoints as an alternative behavioral token into the CDT model to facilitate the prediction of accurate trajectories. Extensive experiments on the Argoverse 2 benchmark demonstrate that CDT excels in generating diverse and scene-compliant trajectories in complex urban settings.

Keywords: Automated Vehicles, End-To-End (E2E) Autonomous Driving
Abstract: Addressing hard cases in autonomous driving, such as anomalous road users, extreme weather conditions, and complex traffic interactions, presents significant challenges. To ensure safety, it is crucial to detect and manage these scenarios effectively for autonomous driving systems. However, the rarity and high-risk nature of these cases demand extensive, diverse datasets for training robust models. Vision-Language Foundation Models (VLMs) have shown remarkable zero-shot capabilities as being trained on extensive datasets. This work explores the potential of VLMs in detecting hard cases in autonomous driving. We demonstrate the capability of VLMs such as GPT-4v in detecting hard cases in traffic participant motion prediction on both agent and scenario levels. We introduce a feasible pipeline where VLMs, fed with sequential image frames with designed prompts, effectively identify challenging agents or scenarios, which are verified by existing prediction models. Moreover, by taking advantage of this detection of hard cases by VLMs, we further improve the training efficiency of the existing motion prediction pipeline by performing data selection for the training samples suggested by GPT. We show the effectiveness and feasibility of our pipeline incorporating VLMs with state-of-the-art methods on NuScenes datasets. The code is accessible at https://github.com/KTH-RPL/Detect_VLM.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, End-To-End (E2E) Autonomous Driving
Abstract: This study introduces a novel approach to autonomous motion planning, informing an analytical algorithm with a reinforcement learning (RL) agent within a Frenet coordinate system. The combination directly addresses the challenges of adaptability and safety in autonomous driving. Motion planning algorithms are essential for navigating dynamic and complex scenarios. Traditional methods, however, lack the flexibility required for unpredictable environments, whereas machine learning techniques, particularly reinforcement learning (RL), offer adaptability but suffer from instability and a lack of explainability. Our unique solution synergizes the predictability and stability of traditional motion planning algorithms with the dynamic adaptability of RL, resulting in a system that efficiently manages complex situations and adapts to changing environmental conditions. Evaluation of our integrated approach shows a significant reduction in collisions, improved risk management, and improved goal success rates across multiple scenarios. The code used in this research is publicly available as open-source software and can be accessed at the following link: url{https://github.com/TUM-AVS/Frenetix-RL}.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, End-To-End (E2E) Autonomous Driving
Abstract: A primary challenge in autonomous driving is achieving safe and efficient trajectory planning in complex dynamic environments. This task requires adherence to traffic laws and vehicle dynamics models as well as an understanding of the spatial distributions and behavior of various traffic participants in densely populated areas. Model Predictive Control (MPC) and its variants typically employ a fixed prediction horizon, which results in limited adaptability in dynamic environments. A long prediction horizon escalates computational costs, while a short prediction horizon may impact real-time performance adversely. To tackle this challenge, our study introduces an optimal horizon MPC planning approach. This method incorporates a sliding horizon window founded on reinforcement learning and interactive MPC planning, making it versatile for a variety of driving scenarios. Additionally, our approach implicitly models the spatio-temporal interactions among traffic participants, thereby enriching the information pool for effective planning. Rigorous tests and validations conducted using the real-world dataset nuPlan affirm that our proposed method delivers robust planning performance, facilitating safe and efficient trajectory planning for autonomous vehicles.

Keywords: Simulation and Real-World Testing Methodologies, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Cooperative Vehicles
Abstract: Due to the increasing automation and connectivity of future, software-defined vehicles, there is a growing interest in safety in this area. Vehicle-to-Everything (V2X) communication is a crucial aspect of this. However, testing V2X applications in the real environment poses a number of challenges, such as the cost of renting test tracks and integrating V2X technology. To develop and test V2X applications more cheaply and at early development stages, simulation is a key enabler. There are several open-source tools that can simulate V2X communication, but a comprehensive and easy-to-use solution that enables to simulate V2X communication combined with sensor data required for automated driving is missing. Therefore, this paper presents an open-source approach that extends the CARLA simulation platform with a new module to create a unified environment for developing V2X applications in simulation. Different parameters can be set individually to account for real-world sensor’s differences. Mechanisms for generating, transmitting and receiving Cooperative Awareness Messages (CAM) are implemented. For the transmission, different propagation models are included, taking into account the outlines of vehicles and buildings: Line of sight (LOS), non-line of sight due to vehicles (NLOSv) and non-line of sight due to static objects (NLOSb). We perform a benchmark measurement with a varying number of simulated vehicles, indicating that simulating the V2X communication introduces only a small overhead to the simulated sensors, such as cameras. The code for the V2X additions to CARLA is available online.

Keywords: Simulation and Real-World Testing Methodologies, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Security and Privacy
Abstract: Misbehavior detection which verifies the semantics of the V2X shared messages is a crucial research topic in Cooperative Intelligent Transport Systems (C-ITS). Misbehavior detection solutions aim to detect and identify the potential attackers which generate V2X messages with erroneous data. Providing efficient misbehavior detection solutions is even more challenging in the context of Cooperative Perception Services (CPS) in which communicating entities share their perception of the environment. This is because of the complexity of the attacks and the lack of the available experimental platforms that allow to evaluate and validate the misbehavior detection solutions. For these reasons, we propose a unified simulation framework to the research community that enables exploration and development of misbehavior detection and mitigation solutions as integrated parts of the CPS in various scenarios. We demonstrate the effectiveness of our framework in generating performance results and provide the corresponding datasets.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques, Advanced Driver Assistance Systems (ADAS)
Abstract: High-definition road maps play a crucial role for the functionality and verification of highly automated driving functions. These contain precise information about the road network, geometry, condition, as well as traffic signs. Despite their importance for the development and evaluation of driving functions, the generation of high-definition maps is still an ongoing research topic. While previous work in this area has primarily focused on the accuracy of road geometry, we present a novel approach for automated large-scale map generation for use in industrial applications. Our proposed method leverages a minimal number of external information about the road to process LiDAR data in segments. These segments are subsequently combined, enabling a flexible and scalable process that achieves high-definition accuracy. Additionally, we showcase the use of the resulting OpenDRIVE in driving function simulation.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques, Automated Vehicles
Abstract: Before an automated vehicle can be commercialised, it requires extensive tests, to prove its safety in the real world. One way to perform safer tests is in simulated environment, due to its comparative lower costs than real world testing. For these tests to be meaningful, the simulation-to-reality gap must be minimal. To minimise this gap, many simulation factors must be considered, one of them is how to accurately include weather in the simulated environment. This work tackles this issue by the creation of a dataset with multiple scenes, across two different test tracks, one in outdoor environment and the other in indoor environment. The selection of sensors to collect data includes RGB camera, thermal camera, 4D RADAR, MEMS LiDAR and spinning LiDAR. The data are collected in 3 weather conditions, namely rain, fog, and clear weather, and at daytime and nighttime. The chosen targets in the scenes are EuroNCAP validated targets representing Pedestrian, Cyclist, and Car; EuroNCAP targets have RADAR and LiDAR reflections similar to the real road stakeholders. Auxiliary targets such as LiDAR and RADAR reflectors, and camera board, with known properties, are also used. In both facilities, the data are collected with targets at set distances from the sensors. A total of 412 test runs, with 30s data collection each, has been carried out. The dataset is publicly available in the Open Simulation Interface (OSI) format, and provides a dataset creator, converting rosbag into OSI format. The REHEARSE dataset contributes to the research community as the first weather dataset focused for automated driving system, which includes the ground truth of target position as well as the label of the weather conditions on the level of rain and fog distributions, visibility, and droplet size.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques, Functional Safety in Intelligent Vehicles
Abstract: Safety of the Intended Functionality (SOTIF) testing is critical in developing autonomous vehicles, as it identifies functional limitations and performance boundaries of autonomous driving systems (ADS). Current scenario-based approaches are predominant in SOTIF testing, but lack diversity and overlook non-collision risk factors. Therefore, we propose an adaptive SOTIF testing and evaluation framework based on a multi-population genetic algorithm, capable of effectively combining risk forms to expand a variety of testing scenarios and simultaneously optimizing parameters to enhance the criticality of test cases. Moreover, apart from common collision factors, our assessment also considers the impacts of rule compliance, aiming to unearth additional potential risk scenarios. We conduct experiments by taking urban intersection passage scenarios as an example, exposing issues of Autoware, a known open-source ADS. This contribution provides insights into the extraction, testing, and evaluation of SOTIF risk scenarios in autonomous driving.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques, Human Factors for Intelligent Vehicles
Abstract: We propose a new method to create plausible pedestrian crossing trajectories that cover a given arbitrarily shaped crosswalk area for simulation-based testing of autonomous vehicles. This method addresses the crossing area coverage problem where the trajectories produced by the generative methods do not cover the entire area that pedestrians may possibly walk on. The actual area covered by pedestrians often differs from marked crosswalks on the road. Furthermore, in the case of jaywalking, the area can take a variety of shapes based on the road structure and surrounding places of interest. Our method is a constructive process that generates trajectories conditioned on an area defined with polygons. We demonstrate that the method can generate trajectories that cover a wide range of crossing areas, including ones from the InD dataset.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques, Perception Including Object Event Detection and Response (OEDR)
Abstract: Recent advances in automated vehicle technology rely heavily on simulated environments for training and testing. However, a significant challenge lies in bridging the gap between simulated and real-world scenarios, as discrepancies between these environments can affect the performance and reliability after that transition, especially in perception. Particularly, LiDAR sensors are highly affected in this matter due to disparities in pointcloud distribution and intensity. Therefore, this paper presents an innovative approach to bridge the gap between simulation and reality. For it, we test and validate a realistic LiDAR library, PCSim, within the CARLA simulator, providing an enhanced simulation environment. Our method involves integrating perception models, pre-trained on real-world datasets, in this environment. Then, we develop a Real2Sim domain adaptation method to transfer these models into the library, leveraging their performance. Finally, we evaluate the 3D object detection models in PCSim LiDARs to prove our methodology.

We have assessed this proposal in PCSim, obtaining promising results in mitigating the simulation-reality gap. Our evaluations provide a guidance for future effective transition from virtual environments to real-world applications.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques, Policy, Ethics, and Regulations
Abstract: Getting real data from safety critical systems for specific, rare situations (e.g., edge cases) is challenging. Moreover, data gathered during operations (e.g., crash reports) are not often publicly accessible and reports might be incomplete. However, covering all scenarios is very important for Verification and Validation (V&V) of safety-critical systems such as AVs. Therefore, synthetic data could be used for V&V to fill that gap. Synthetic data generation, labelling, and validation are open challenges, though. To the best of the authors' knowledge, no standard methods for integrating synthetic data into V&V are shared across automotive-domain companies. Therefore, this study (i) gathers expert knowledge on current practices for Digital Twins for V&V development, (ii) proposes a general 6-stage pipeline for synthetic data usage within an early V&V process, and (iii) discusses open source tools and formats standardisation of synthetic data use within V&V. The open-source tools and format standardisation may facilitate the integration of synthetic data into the V&V process. The proposed pipeline and mapping study, provide a foundation for future research on synthetic data use within V&V.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques, Vehicular Active and Passive Safety
Abstract: Verifying highly automated driving functions can be challenging, requiring identifying relevant test scenarios. Scenario-based testing will likely play a significant role in verifying these systems, predominantly occurring within simulation. In our approach, we use traffic scenes as a starting point (seed-scene) to address the individuality of various highly automated driving functions and to avoid the problems associated with a predefined test traffic scenario. Different highly autonomous driving functions, or their distinct iterations, may display different behaviors under the same operating conditions. To make a generalizable statement about a seed-scene, we simulate possible outcomes based on various behavior profiles. We utilize our lightweight simulation environment and populate it with rule-based and machine learning behavior models for individual actors in the scenario. We analyze resulting scenarios using a variety of criticality metrics. The density distributions of the resulting criticality values enable us to make a profound statement about the significance of a particular scene, considering various eventualities.

Keywords: Automated Vehicles
Abstract: In recent years, remarkable strides have been made in the field of autonomous driving, with a particular focus on enhancing perception and prediction capabilities through the integration of big data and advanced deep learning algorithms. Despite these advancements, the persistent challenge of effectively validating the performance of autonomous vehicles (AVs) remains a critical issue. In the realm of microscopic traffic simulation, a noteworthy challenge persists – that of bridging the behavior gap between simulated scenarios and real-world driving situations. Efforts to define agent behavior in simulations manually or replay observed behaviors have proven to be inefficient and prone to inaccuracies, mainly because simulated agents often fail to authentically react to actions initiated by AVs. While rule-based traffic simulation models offer plausible behaviors, they exhibit limitations in adapting to diverse and data-driven behavior patterns within complex driving interactions. Ad-dressing these challenges, we propose a learning-based method for traffic agent simulation, emphasizing realism and reactivity. This involves learning data-driven agent models from re-al-world driving data with detailed interaction information and high-definition (HD) maps. Utilizing convolutional neural net-work (CNN), we extract features and predict future trajectories, achieving realism and reactivity through closed-loop simulation at inference. The proposed model is evaluated using real-world data, demonstrating its effectiveness in simulating diverse and realistic traffic behaviors, like stopping at red traffic lights, yielding to other vehicles during right-turn-on-red, and car following.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques
Abstract: To select optimal routes for vehicles endurance tests, it is necessary to have a road map characterized by events of interest. In this context, we define events as effects on the vehicle triggered by some proprieties of the routes. Such a characterization strongly relies on data from previous test drives. If new road maps are to be considered, e.g., in a different region, the route selection rather depends on the experience of engineers, which can lead to suboptimal decisions. To overcome this problem, we propose using the existing data from prior test drives to train a machine learning (ML) model, which then transfers this knowledge to unseen road maps. To this end, we formulate a sequential problem that can be solved with state-of-the-art ML architectures. Our experimental results based on real-world data show the potential of the proposed approach as we illustrate for the case of testing energy recuperation in electric vehicles.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques
Abstract: Long Short-term Memory (LSTM) is a Recurrent Neural Network (RNN) widely used in time series traffic forecasting. LSTM captures both short-term and long-term trends and dependency in sequential data like time series data, as it contains specialized memory cells to store information in memory for longer periods. Existing traffic forecasting approaches lack features to forecast the traffic speed of interconnected road links and provide multivariate (multi-input and multi-output) and multi-step traffic forecasting both in the short and long term. We propose an Encoder-Decoder LSTM-based sequence-to-sequence architecture to capture the traffic speed of interconnected road links and provide multivariate multistep traffic forecasting both in the short term (15 minutes) and long term (two days). We apply a sliding-window approach to feed the short-term traffic forecasting as an input to the model to project long-term traffic forecasting. Our model can incorporate multiple interconnected road links and provide traffic speed forecasting for multiple future steps. We conduct our experiment at an intersection of Oshawa, ON, Canada, and evaluate the performance using the error distribution and Mean Absolute Error. Performance evaluation shows that the model can forecast the traffic speed of interconnected road links in multiple steps with negligible error, both in the short term and in the long term.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques
Abstract: The increasing spread of automated driving functions, ranging from relatively simple parking assistants to fully automated driving systems, has led to a surge in the demand for traffic data used for development, validation, and verification in order to minimize the safety risks. Synthetic data plays an ever-growing role at this point since real-world data alone cannot satisfy all demands due to costs and complexity of data acquisition. To generate such synthetic data, realistic, agent-based traffic flow simulations can be used where numerous individual agents interact according to parametric behavior models. In order to be able to realistically simulate traffic using such models, sufficiently accurate data-based optimization is crucial. Until now, the optimization of such models has required the use of gradient-free algorithms or machine learning methods, which can become computationally disproportionate for complex models. The concept for sample-based gradient approximation presented in this work has the potential to make the optimization of such complex parametric simulation models feasible and efficient by making gradient-based optimization algorithms usable. The application of this concept is presented at the example of optimizing model parameters in the traffic simulation framework PTV Vissim.

Keywords: Simulation and Real-World Testing Methodologies, Verification and Validation Techniques
Abstract: This paper presents a novel solution to the demand of a realistic virtual test environment (VTE) for the development and safety assurance of Automated Driving Systems (ADSs). The current VTE offerings suffer limitations when it comes to creating a rich environment (from the Operational Design Domain perspective) based on the ASAM OpenDRIVE files (a formatted description of the scenery), and hence there is no automatic OpenDRIVE scenery generation available that resembles the richness required to thoroughly test an ADS in a virtual environment. Therefore, through the use of Unreal Engine, leveraging the power of esmini's roadmanager and developing on a primitive OpenDRIVE plugin, a comprehensive scenery with complex lighting, dynamic environmental conditions and photoscanned meshes can be achieved. There is then a demonstration of how this is incorporated into an end-to-end testing framework, using just one user interface to take the user from the creation and retrieval of scenarios through to the execution.

Keywords: Simulation and Real-World Testing Methodologies
Abstract: In traffic safety research, simulation tools are considered more straightforward and cost-effective than direct observations of real-world conditions, especially when dealing with scenarios that may not exist in reality. The tools include traffic micro-simulation tools (e.g., SUMO) and driver simulators developed in game engines (e.g., Unity). However, the tools also have limitations. For example, the equations used to simulate human behavior may not always reflect real-world behavior accurately, and driver simulators’ lack of realistic traffic systems affect the interaction between the simulator vehicle and other vehicles. Co-simulation allows two different simulation tools to exchange data to enhance the capabilities of each tool, but many traffic safety researchers currently spend significant amounts of time, effort, and budget working on their own version of a co-simulation tool to integrate, for example, a traffic micro-simulation tool such as SUMO with a driver simulator such as the Unity game engine. This situation takes time away from focusing on the goal of improving traffic safety. In this paper, we developed an open-source traffic co-simulation tool. Development involved three tasks: 1. integration of SUMO and Unity; 2. development of a 2D and 3D environment (a 3D road environment in Unity was generated from a 2D road environment in SUMO); and 3. development of a 3D model of a simulator vehicle and development of a VR-based driver simulator. We named our tool SUMO2Unity and believe that it can significantly help traffic safety researchers to conduct future research aimed at improving traffic safety.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Sensor Signal Processing
Abstract: Many state-of-the-art methods for safety assessment and motion planning for automated driving require estimation of the probability of collision (POC). To estimate the POC, a shape approximation of the colliding actors and probability density functions of the associated uncertain kinematic variables are required. Even with such information available, the derivation of the POC is in general, i.e., for any shape and density, only possible with Monte Carlo sampling (MCS). Random sampling of the POC, however, is challenging since computational resources are limited in real-world applications. We present expressions for the POC in the presence of Gaussian uncertainties, based on multi-circular shape approximations. In addition, we show that the proposed approach is computationally more efficient than MCS. Lastly, we provide a method for upper and lower bounding the estimation error for the POC induced by the used shape approximations.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: Driving behavior modeling is crucial in autonomous driving systems for preventing traffic accidents. Inverse reinforcement learning (IRL) allows autonomous agents to learn complicated behaviors from expert demonstrations. Similar to how humans learn by trial and error, failed demonstrations can help an agent avoid failures. However, expert and failed demonstrations generally have some common behaviors, which could cause instability in an IRL model. To improve the stability, this work proposes a novel method that introduces time-series labeling for the optimization of IRL to help distinguish the behaviors in demonstrations. Experimental results in a simulated driving environment show that the proposed method converged faster than and outperformed other baseline methods. The results also show consistency for various data balances of the number of expert and failed demonstrations.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Autonomous vehicle controllers capable of drifting can improve safety in dynamic emergency situations. However, drifting involves operating at high sideslip angles, which is a fundamentally unstable operating regime that typically requires an accurate vehicle model for reliable operation; such models may not be available after environmental or vehicle parameter changes. Towards that goal, this work presents a Nonlinear Model Predictive Control approach which is capable of initiating and controlling a drift in a production vehicle even when changes in vehicle parameters degrade the original model. A neural network model of the vehicle dynamics is used inside the optimization routine and updated with online learning techniques, giving a higher fidelity and more adaptable model. Experimental validation on a full size, nearly unmodified Lexus LC500 demonstrates the increased modeling fidelity, adaptability, and utility of the presented controller framework. As the LC500 is a difficult car to drift, previous approaches which rely on physics based vehicle models could not complete the autonomous drift tests on this vehicle. Furthermore, the tires on the experimental vehicle are then switched, changing the vehicle parameters, and the capability of the controller to adapt online is demonstrated.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, End-To-End (E2E) Autonomous Driving
Abstract: This study explores efficient and safe Automated Valet Parking (AVP) strategies in unstructured and dynamic environments. Existing approaches utilizing reinforcement learning neglected the impact of dynamic agents on ego vehicle and disregarded human driving patterns, leading to their ineffectiveness in unstructured dynamic contexts. We propose a novel hybrid attention mechanism that comprehends the mixed interactions between static and dynamic elements, aiding autonomous vehicles in advanced planning. We implemented a guidance system based on human preferences, eliminating the need for expert data at the outset and expediting the training process via intermediate planning stages, thereby facilitating parking maneuvers akin to human drivers. The model was trained and validated in a range of parking situations. The experimental outcomes indicate that our method possesses robust adaptability and navigation skills in static and dynamic environments.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, End-To-End (E2E) Autonomous Driving
Abstract: Optimization of vehicle maneuvers using dynamic models in constrained spaces is challenging. Homotopic optimization, which has shown success for vehicle maneuvers with kinematic models, is studied in the case where the vehicle model is governed by dynamic equations considering road-tire interactions. This method involves a sequence of optimization problems that start with a large free space. By iteration, this space is progressively made smaller until the target problem is reached. The method uses a homotopy index to iterate the sequence of optimizations, and the method is verified by solving challenging maneuvering problems with different road surfaces and entry velocities using a double-track vehicle dynamics model. The main takeaway is that homotopic optimization is also efficient for dynamic vehicle models at the limit of road-tire friction, and it demonstrates capabilities in solving demanding maneuvering problems compared with alternative methods like stepwise initialization and driver model-based initialization.

Keywords: Sensor Signal Processing, Pedestrian Protection
Abstract: In recent years, wearable sensors and mobile devices have become popular tools in the field of positioning. In pedestrian navigation, Pedestrian Dead Reckoning (PDR) is the primary algorithm, with numerous previous research cases available. However, traditional PDR algorithms' stride length calculations rely on empirical formulas that are influenced by factors such as user height, walking frequency, and walking habits. Without appropriate parameters, stride length estimation can result in significant errors, leading to poor positioning outcomes. Apart from the stride length calculation issue, IMUs contain bias and noise themselves, leading to drift errors over time, especially in consumer-grade IMUs. To address these issues, this study introduces a CNN velocity estimation model to calculate users' 1D velocity. The trained velocity estimation model can overcome the user dependency issue caused by traditional PDR because the training data covers different users. For the heading calculation, this study employs a novel 9D IMU AHRS algorithm (Laidig et al., 2022) to address attitude estimation problems that traditional PDR cannot handle effectively under high motion conditions. Finally, incorporating GNSS through the principles of Extended Kalman Filter(EKF) to compensate for IMU’s drift over time. In the experiment, we use NovAtel Pwrpak as ground truth. It contains high-quality GNSS and IMU, which can provide reliable reference trajectory. A comparison of trajectories is conducted using a certain-brand smartphone in different modes.

Keywords: Automated Vehicles, Simulation and Real-World Testing Methodologies, Pedestrian Protection
Abstract: Predicting the movement of other road users is beneficial for improving automated vehicle (AV) performance. However, the relationship between horizon of these predictions and AV performance remains unclear. Despite the existence of numerous trajectory prediction algorithms, no studies have been conducted on how varying prediction horizons affect AV safety and other vehicle performance metrics, resulting in undefined horizon requirements for prediction methods. Our study addresses this gap by examining the effects of different prediction horizons on AV performance, focusing on safety, comfort, and efficiency. Through multiple experiments using a state-of-the-art, risk-based predictive trajectory planner, we simulated predictions with horizons up to 20 seconds. Based on our simulations, we propose a framework for specifying the minimum required and optimal prediction horizons depending on specific AV performance criteria and application needs. Our results indicate that a horizon of 1.6 seconds is required to prevent collisions with crossing pedestrians, horizons of 7-8 seconds yield the best efficiency, and horizons up to 15 seconds improve passenger comfort. We conclude that prediction horizon requirements are application-dependent, and recommend aiming for a prediction horizon of 11.8 seconds as a general guideline for applications involving crossing pedestrians.

Keywords: Pedestrian Protection, Smart Infrastructure, Vehicular Active and Passive Safety
Abstract: This work present a novel approach to identify possible early accident-prone zones in a large city-scale road network using geo-tagged collision alert data from a vehicle fleet. The alert data has been collected for a year from 200 city buses installed with the Advanced Driver Assistance System (ADAS). To best of our knowledge, no research paper has used ADAS alert to identify the early accident-prone zones. A nonparametric technique called Kernel Density Estimation (KDE) is employed to model the distribution of alert data across stratified time intervals. A novel recall-based measure is introduced to assess the degree of support provided by our density-based approach for existing, manually determined accident-prone zones (‘blackspots’) provided by civic authorities. This show that our KDE approach significantly outperforms existing approaches in terms of the recall-based measure. Introducing a novel linear assignment Earth Mover Distance based measure to predict previously unidentified accident-prone zones. The results and findings support the feasibility of utilizing alert data from vehicle fleets to aid civic planners in assessing accident-zone trends and deploying traffic calming measures, thereby improving overall road safety and saving lives.

Keywords: Pedestrian Protection, Simulation and Real-World Testing Methodologies, Perception Including Object Event Detection and Response (OEDR)
Abstract: Pedestrians are among the most endangered traffic participants in road traffic. While pedestrian detection in nominal conditions is well established, the sensor and, therefore, the pedestrian detection performance degrades under adverse weather conditions. Understanding the influences of rain and fog on a specific radar and lidar sensor requires extensive testing, and if the sensors' specifications are altered, a retesting effort is required. These challenges are addressed in this paper, firstly by conducting comprehensive measurements collecting empirical data of pedestrian detection performance under varying rain and fog intensities in a controlled environment, and secondly, by introducing a dedicated Weather Filter (WF) model that predicts the effects of rain and fog on a user-specified radar and lidar on pedestrian detection performance. We use a state-of-the-art baseline model representing the physical relation of sensor specifications, which, however, lacks the representation of secondary weather effects, e.g., changes in pedestrian reflectivity or droplets on a sensor, and adjust it with empirical data to account for such. We find that our measurement results are in agreement with existent literature and our WF outperforms the baseline model in predicting weather effects on pedestrian detection while only requiring a minimal testing effort.

Keywords: Software-Defined Vehicle for Intelligent Vehicles, Advanced Driver Assistance Systems (ADAS), Vehicular Active and Passive Safety
Abstract: Many established vision perception systems for autonomous driving scenarios ignore the influence of light conditions, one of the key elements for driving safety. To address this problem, we present HawkDrive, a novel perception system with hardware and software solutions. Hardware that utilizes stereo vision perception, which has been demonstrated to be a more reliable way of estimating depth information than monocular vision, is partnered with the edge computing device Nvidia Jetson Xavier AGX. Our software for low light enhancement, depth estimation, and semantic segmentation tasks, is a transformer-based neural network. Our software stack, which enables fast inference and noise reduction, is packaged into system modules in Robot Operating System 2 (ROS2). Our experimental results have shown that the proposed end-to-end system is effective in improving the depth estimation and semantic segmentation performance. Our dataset and codes will be released at https://github.com/ZionGo6/HawkDrive.

Keywords: Sensor Signal Processing, Advanced Driver Assistance Systems (ADAS), Perception Including Object Event Detection and Response (OEDR)
Abstract: Target detection in road clutter environment is a challenge for automotive radar. The performance of model-based methods degrades when the prior model is mismatched or the target energy is overwhelmed by the clutter. In contrast, deep learning methods can nonlinearly fit clutter distributions and extract deep features to identify targets from clutter backgrounds. Considering that the spatial-temporal feature in multi-frame data helps distinguish targets from clutter, we use the multi-frame data for detection. This paper proposes a multi-frame detection network for radar moving targets in clutter environment. First, we use transformer as the backbone to fit the large-scale clutter background by extracting the global spatio-temporal feature. Second, we proposed a multi-frame detection head to predict multi-frame bounding boxes in parallel by utilizing the spatio-temporal feature. Third, we proposed a segmentation-assisted refinement module to refine the objectness of bounding boxes, thus further suppressing the false alarms caused by clutter. Through experiments on simulation and measured datasets, the proposed method effectively reduces false alarms while maintaining a high detection probability. In addition, compared with the segmentation-based method, our method distinguishes adjacent targets more robustly.

Keywords: Sensor Fusion for Localization, Advanced Driver Assistance Systems (ADAS), Sensor Signal Processing
Abstract: Vehicle localization plays an important role in effectively perceiving location information for intelligent vehicles. In this paper, a novel array calibration method for intelligent vehicle localization is proposed. Utilizing the stability of triangles, three identical GNSS localization receivers are placed at the three vertices of an equilateral triangle, and the centroid of the equilateral triangle is calculated as the position of the vehicle positioning receiver. A receiver array localization enhancement model for vehicle motion process, including linear and curved motion, is established. In the localization calculation process, by comparing the localization error of each receiver, the point with the smallest error is selected to reconstruct an equilateral triangle based on geometric relationships, and the position of the centroid point is recalculated to obtain the corrected position information. The proposed method is simulated and validated in both static and dynamic driving environments, and the experimental results showed that the method can effectively improve positioning accuracy.

Keywords: Sensor Signal Processing, Advanced Driver Assistance Systems (ADAS), Automated Vehicles
Abstract: Focusing on the strength of 4D (4-Dimensional) radar, research about robust 3D object detection networks in adverse weather conditions has gained attention. To train such networks, datasets that contain large amounts of 4D radar data and ground truth labels are essential. However, the existing 4D radar datasets (e.g., K-Radar) lack sufficient sensor data and labels, which hinders the advancement in this research domain. Furthermore, enlarging the 4D radar datasets requires a time-consuming and expensive manual labeling process. To address these issues, we propose the auto-labeling method of 4D radar tensor (4DRT) in the K-Radar dataset. The proposed method initially trains a LiDAR-based object detection network (LODN) using calibrated LiDAR point cloud (LPC). The trained LODN then automatically generates ground truth labels (i.e., auto-labels, ALs) of the K-Radar train dataset without human intervention. The generated ALs are used to train the 4D radar-based object detection network (4DRODN), Radar Tensor Network with Height (RTNH). The experimental results demonstrate that RTNH trained with ALs has achieved a similar detection performance to the original RTNH which is trained with manually annotated ground truth labels, thereby verifying the effectiveness of the proposed auto-labeling method. All relevant codes will be soon available at the following GitHub project: https://github.com/kaist-avelab/K-Radar

Keywords: Eco-Driving and Energy-Efficient Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: One of the constraints that limits regenerative braking in electric vehicles is its inability to regenerate energy under low-speed conditions, due to insufficient back-electromotive force developed in the traction motor. Below a certain speed threshold known as the low-speed cutoff point, energy is extracted from the battery instead of being returned, while overcoming electrical losses in the motor drive system. Although methods implemented to determine dynamic low-speed threshold are reported to result in higher braking efficiencies than fixed point methods, former implementations typically involve dynamic detection of battery current direction, which poses sensing challenges due to factors such as current sensor accuracy, offset, filtering and delays. As an alternative method, this paper proposes a model-based approach to analytically determine the dynamic low-speed cutoff point. Additionally, a loss minimization framework is developed to enhance the amount of energy recovered by achieving a lower value of the cutoff point, thereby extending the braking duration. Simulation studies have been done on a vector-controlled induction motor drive used for a passenger electric car to ascertain braking efficiency improvement. Results indicate that the proposed loss-reducing approach results in the highest efficiency improvement during low torque, high speed operation, when compared to other regions. The magnitude of efficiency improvement depends on system parameters and actual operating conditions of the vehicle. The practical effectiveness of the proposed approach is demonstrated on the US06 drive cycle which verifies the observations from the analysis.

Keywords: Integration of HD map and Onboard Sensors, Advanced Driver Assistance Systems (ADAS), Sensor Fusion for Localization
Abstract: Vision-based ego-lane inference using High- Definition (HD) maps is essential in autonomous driving and advanced driver assistance systems. The traditional approach necessitates well-calibrated cameras, which confines variation of camera configuration, as the algorithm relies on intrinsic and extrinsic calibration. In this paper, we propose a learning-based ego-lane inference by directly estimating the ego-lane index from a single image. To enhance robust performance, our model incorporates the two-head structure inferring ego-lane in two perspectives simultaneously. Furthermore, we utilize an attention mechanism guided by vanishing point-and-line to adapt to changes in viewpoint without requiring accurate calibration. The high adaptability of our model was validated in diverse environments, devices, and camera mounting points and orientations.

Keywords: Eco-Driving and Energy-Efficient Vehicles, Advanced Driver Assistance Systems (ADAS), Simulation and Real-World Testing Methodologies
Abstract: This study proposes a novel climate control strategy for electric vehicles (EVs) by addressing door-opening interruptions, an overlooked aspect in EV thermal management. We create and validate an EV simulation model that incorporates door-opening scenarios. Three controllers are compared using the simulation model: (i) a hierarchical nonlinear model predictive control (NMPC) with a unique coolant dividing layer and a component for cabin air inflow regulation based on door-opening signals; (ii) a single MPC controller; and (iii) a rule-based controller. The hierarchical controller outperforms, reducing door-opening temperature drops by 46.96% and 51.33% compared to single layer MPC and rule-based methods in the relevant section. Additionally, our strategy minimizes the maximum temperature gaps between the sections during recovery by 86.4% and 78.7%, surpassing single layer MPC and rule-based approaches, respectively. We believe that this result opens up future possibilities for incorporating the thermal comfort of passengers across all sections within the vehicle.

Keywords: Pedestrian Protection, Future Mobility and Smart City, Smart Infrastructure
Abstract: Ensuring the safety of Vulnerable Road Users (VRUs) at intersections is crucial to enhancing urban traffic systems. This paper introduces a novel intelligent warning system specifically designed to increase the safety of VRUs crossing intersections. The proposed system leverages the COSMOS testbed to obtain real time vehicle information and employs Message Queuing Telemetry Transport (MQTT) as a standards-based messaging protocol for device communication and data transmission and utilizes a transformer model and Time To Collision (TTC) method to predict the collision. To validate the effectiveness and reliability of our intelligent alert system, we conducted comprehensive tests using the CARLA simulator, incorporating hardware in the loop simulation Approach. The results demonstrate enhanced situational awareness and reduced risk factors associated with VRUs at intersections. Our work supports the integration of this intelligent alert system as a viable solution for reducing accidents and enhancing the overall safety of urban intersections in real time.

Keywords: Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: Multi-object tracking (MOT) requires the crucial step of associating measurements with tracks. To manage the exponentially growing number of track hypotheses, truncation becomes necessary. In the δ-Generalized Labeled Multi-Bernoulli (δ-GLMB) filter application, this truncation typically involves the ranked assignment problem, solved by Murty's algorithm or the Gibbs sampling approach, both with limitations in terms of complexity or accuracy, respectively. With the motivation to improve these limitations, this paper addresses the ranked assignment problem arising from data association tasks with an approach that employs Graph Neural Networks (GNNs). The proposed Ranked Assignment Prediction Graph Neural Network (RAPNet) uses bipartite graphs to model the problem, harnessing the computational capabilities of deep learning. The conclusive evaluation compares the RAPNet with Murty's algorithm and the Gibbs sampler, showing accuracy improvements compared to the Gibbs sampler.

Keywords: Sensor Signal Processing, Advanced Driver Assistance Systems (ADAS), Integration of HD map and Onboard Sensors
Abstract: This work proposes NavPathNet, a graph neural network based trajectory prediction system capable of accurately predicting the trajectory of a two-wheeler with onboard sensors for up to 6 seconds using the vehicle state and only navigation map information. Possible paths are generated from the local road network in a regular navigation map using Bézier curves and a multimodal prediction captures the different possibilities of the road network. A kinematic model integrated into the network allows to give guarantees on physical realism and on motion constraints, essential for safety-relevant systems. The proposed system was trained and evaluated on an in-house e-bike data set and was able to reach a top 1 final displacement error of 2 m for four seconds and 3.8 m for six seconds of prediction time, significantly outperforming other baselines. The improvement in prediction quality compared to purely physical models opens up new possibilities for driver assistance systems in connected vehicles.

Keywords: Automated Vehicles, Advanced Driver Assistance Systems (ADAS), Vehicle Control and Motion Planning
Abstract: Level-4 autonomous trucks are essential for operating in challenging real-world environments. Accurate estimation of the hitch angle in the tractor-trailer system is crucial for safe maneuvering. Traditional sensor-based measurement techniques for estimating the hitch angle can be complex and expensive. To address this challenge, we propose a lidar-assisted hitch angle estimation (HAE) approach, leveraging existing two-sided lidars installed on the tractor. Overcoming challenges related to accurate lidar extrinsic calibration, time synchronization, online calibration of the trailer's hitch point, and robustness to environmental changes, our system achieves highly accurate and robust HAE, even in adverse conditions. Our contributions include introducing a novel lidar-assisted HAE system, successfully implementing it in real-time level-4 self-driving trucks, and curating a challenging real-world dataset for comprehensive evaluation and benchmarking of HAE in autonomous driving systems.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: With the flourishing development of intelligent warehousing systems, the technology of automated guided vehicle (AGV) has experienced rapid growth. Within intelligent warehousing environments, AGV is required to safely and rapidly plan an optimal path in complex and dynamic environments. Most research has studied deep reinforcement learning to address this challenge. However, in the environments with sparse extrinsic rewards, these algorithms often converge slowly, learn inefficiently or fail to reach the target. Random network distillation (RND), as an exploration enhancement, can effectively improve the performance of proximal policy optimization, especially enhancing the additional intrinsic rewards of the AGV agent which is in sparse reward environments. Moreover, most of the current research continues to use 2D grid mazes as experimental environments. These environments have insufficient complexity and limited action sets. To solve this limitation, we present simulation environments of AGV path planning with continuous actions and positions for AGVs, so that it can be close to realistic physical scenarios. Based on our experiments and comprehensive analysis of the proposed method, the results demonstrate that our proposed method enables AGV to more rapidly complete path planning tasks with continuous actions in our environments. A video of part of our experiments can be found at https://youtu.be/lwrY9YesGmw.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: The key to achieving path-following and emergency obstacle avoidance in autonomous vehicles is the Steer-By-Wire (SBW) system. While enhancing control accuracy and response speed, the SBW system introduces an increased risk of electronic and electrical failures. To address this problem, the commonly adopted approach is dual steering motor redundancy, ensuring reliability and safety while considering structural and cost constraints. This paper focuses on the Dual-Motor Steer-By-Wire (DMSBW) system, specifically the master-master redundancy mode. Existing studies have paid less attention to the control after the failure of one steering motor. Therefore, this paper proposed an adaptive nonsingular fast terminal sliding mode redundant control algorithm to achieve accurate and swift steering angle tracking. This algorithm accounts for system uncertainty and parameter perturbation, enhancing system robustness while ensuring precise control accuracy. Additionally, considering cost implications in engineering applications, we propose an unknown input observer for accurate estimation of the front wheel steering angle. Finally, the effectiveness of the proposed strategy is validated through Carsim and Simulink, which demonstrate that the strategy exhibits outstanding control performance and robustness, enabling accurate and swift steering tracking at diverse working conditions.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning
Abstract: In addressing the trade-off between prediction model accuracy and computational cost in the context of path tracking control, this paper proposes a learning-based model predictive control (LB-MPC) strategy for autonomous buses. A three-degree-of-freedom (DOF) single-track vehicle dynamic model is established, and an in-depth analysis is conducted on its step response error with respect to variations in vehicle speed, pedal position, and front wheel steering angle compared to the IPG TruckMaker model. Methods for constructing error datasets and receding horizon updates are designed, and a Gaussian process regression (GPR) is employed to establish an error fitting model for real-time error compensation and correction of the nominal single-track model. The error correction model is utilized as the prediction model, and a path tracking cost function is designed to formulate a quadratic programming (QP) optimization problem, proposing an LB-MPC path tracking control architecture. Through joint simulations using the IPG TruckMaker & Simulink platform and real bus experiment, the real-time performance and effectiveness of the proposed GPR error correction model and LB-MPC path tracking control strategy are verified. Results demonstrate that compared to traditional MPC path tracking control strategy, the proposed LB-MPC strategy reduces the average path tracking error by 79.00%.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: This paper addresses trajectory planning for overtaking maneuvers in a dynamic, two-way, two-lane environment. The proposed novel five-stage planning algorithm comprises initial lane change, acceleration, final lane change, abort, and final lane change abort. This algorithm allows overtaking to be aborted in oncoming lane danger, reverting to the original lane. Employing a point mass model-based model predictive control (MPC) framework, separate longitudinal and lateral planning is conducted in each phase, combined to generate the overall trajectory. A practical nonlinear bicycle model is employed for trajectory tracking to address kinematic infeasibility. Furthermore, the tracking controller utilizes adaptive MPC. Numerical simulations validate the performance of the proposed algorithm.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: Curvilinear coordinate frames are a widespread representation for motion planners of automated vehicles. In structured environments, the required reference path is often extracted from map data, e.g., by linearly interpolating the center points of lanes. Often, these reference paths are not directly suited for curvilinear frames, as the representation of points is not guaranteed to be unique for relevant parts of the road. Artifacts arising from faulty coordinate conversions can impede the robustness of downstream planning tasks and may result in safety-critical situations. We present an iterative procedure to adapt a reference path, ensuring a unique representation of all points within a provided subset of a map. Our numerical experiments demonstrate the efficacy of our method when combined with two motion planning tasks: Computing the reachable set of the ego vehicle and planning trajectories using a sampling-based approach.

Keywords: Simulation and Real-World Testing Methodologies, Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Motion planning algorithms should be tested on a large, diverse, and realistic set of scenarios before deploying them in real vehicles. However, existing 3D simulators usually focus on perception and end-to-end learning, lacking specific interfaces for motion planning. We present an interface for the CARLA simulator focusing on motion planning, e.g., to create configurable test scenarios and execute motion planners in interactive environments. Additionally, we introduce a converter from lanelet-based maps to OpenDRIVE, making it possible to use CommonRoad and Lanelet2 maps in CARLA. Our evaluation shows that our interface is easy to use, creates new scenarios efficiently, and can successfully integrate motion planners to solve CommonRoad scenarios. Our tool is published as an open-source toolbox at commonroad.in.tum.de.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: Deploying a certain trajectory tracking control algorithm to a different autonomous vehicle is time-consuming as most control parameters need to be fine-tuned, which necessitates a series of experiments and computations to ascertain the appropriate parameters considering factors such as tire stiffness, inertia, wheelbase, and speed levels. This paper introduces a data-driven method for automatic optimization of the parameters for fast migration. We leverage preview control as the fundamental algorithm whose effectiveness has been proven on an automated Lincoln MKZ in our previous studies and aim to deploy it to other autonomous cars more efficiently. To learn the control parameters, naturalistic driving data is collected, but one challenge is that the data lacks tracking error information. The proposed strategy decouples the algorithm into feedforward control and feedback control, then learns the two control parts separately with well-designed experiments. Multiple schemes of neural networks are devised to identify feedforward and feedback gains, then the optimal scheme is selected to form the preview control for the given target vehicle. The deployment and validation take place on both simulation and a 4×4 off-road platform. Tests show that the proposed pipeline can learn the control gains for a certain speed range with only around 0.5 hours of driving data, which accelerates the control deployment compared to empirical human-based fine tuning. Compared to the pure pursuit control, the control identified by the proposed method demonstrates outstanding performance in terms of accuracy and stability at the specific speed range.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: Path tracking is a key problem in autonomous vehicles, but uncertainties in vehicle models and environmental parameters as well as a couple of lateral and longitudinal behaviors pose a major challenge to the path following accuracy and efficiency. To solve this problem, a layered vehicle lateral and longitudinal controller based on model predictive control is proposed in this paper. Firstly, a lateral controller based on a synthesis of robust model predictive control is used to reduce the influence of model uncertainty. Then a longitudinal nonlinear model predictive controller is proposed to achieve the coordination between the lateral following error and the longitudinal forward speed of the vehicle. Finally, the bottom PID controller is used to control the throttle and brake of the vehicle. The effectiveness of the proposed method is verified by experiments, which show that it is robust to the uncertain parameters and the computation is relatively small.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Automotive Cyber Physical Systems
Abstract: This work presents an adaptive open-source implementation of ArduPilot: the adaptive mechanisms in the autopilot are inspired by model reference stabilization (MRS) and are seamlessly embedded into the open-source ArduPilot suite. We illustrate MRS ArduPilot for the ArduPlane and ArduCopter modules (fixed-wing and rotary-wing vehicles): yet, the approach is general enough to be applicable to all aerial/surface/marine vehicles of ArduPilot, and even to PX4. Our tests show that the embedded adaptation makes the vehicle capable of handling uncertain scenarios like wind and varying payloads. The source code of MRS ArduPilot is released at https://github.com/Sunsun24/MRS.git

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Cooperative Vehicles
Abstract: This paper addresses the distributed adaptive coordinated control for high-speed train (HST) fleet with uncertain parameters. The motion of the train in the fleet is constrained by its adjacent trains, necessitating dynamic adjustment mechanism facilitated through inter-train communication. For the uncertainty, radial basic function neural network (RBFNN) is introduced into the distributed adaptive coordinated control algorithm, which ensures behavioral consistency and short inter-train intervals for each train in the fleet. This paper compares the proposed method with distributed adaptive sliding mode control (DASMC). The simulation demonstrates better performance and benefits of this new algorithm. We show that the algorithm substantially reduces inter-train distance and ensures heightened level of behavioral consistency among all individual trains within the train fleet.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Cooperative Vehicles
Abstract: Real-world driving involves intricate interactions among vehicles navigating through dense traffic scenarios. Recent research focuses on enhancing the interaction awareness of autonomous vehicles to leverage these interactions in decision-making. These interaction-aware planners rely on neural-network-based prediction models to capture inter-vehicle interactions, aiming to integrate these predictions with traditional control techniques such as Model Predictive Control. However, this integration of deep learning-based models with traditional control paradigms often results in computationally demanding optimization problems, relying on heuristic methods. This study introduces a principled and efficient method for combining deep learning with constrained optimization, employing knowledge distillation to train smaller and more efficient networks, thereby mitigating complexity. We demonstrate that these refined networks maintain the problem-solving efficacy of larger models while significantly accelerating optimization. Specifically, in the domain of interaction-aware trajectory planning for autonomous vehicles, we illustrate that training a smaller prediction network using knowledge distillation speeds up optimization without sacrificing accuracy.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Drone and Urban Air Mobility
Abstract: Existing local dynamic route planning algorithms, when directly applied to terrain following/terrain avoidance, or dynamic obstacle avoidance for large and medium-sized fixed-wing aircraft, fail to simultaneously meet the requirements of real-time performance, long-distance planning, and the dynamic constraints of large and medium-sized aircraft. To deal with this issue, this paper proposes the Motion Dynamic RRT based Fluid Field - PPO for dynamic TF/TA routing planning. Firstly, the action and state spaces of the proximal policy gradient algorithm are redesigned using disturbance flow fields and artificial potential field algorithms, establishing an aircraft dynamics model, and designing a state transition process based on this model. Additionally, a reward function is designed to encourage strategies for obstacle avoidance, terrain following, terrain avoidance, and safe flight. Experimental results on real DEM data demonstrate that our algorithm can complete long-distance flight tasks through collision-free trajectory planning that complies with dynamic constraints, without the need for prior global planning.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, End-To-End (E2E) Autonomous Driving
Abstract: Vehicle slipping may cause an accident while driving. However, professional drivers usually perform high side-slip angle maneuvers, such as drifting to minimize lap time or avoid obstacles. Tracking the desired trajectory while maintaining drift is a challenging task due to the complexity of the vehicle model. In this paper, we first solve a series of minimum-time cornering problems under different initial conditions. The results show that an aggressive cornering can be divided into three segments, including the entry corner stage, the drifting stage, and the exiting stage. We then propose a complete trajectory planning and motion control framework to conduct the drift cornering maneuver. The trajectory planner calculates the speed profile and then updates the initial path by optimizing curvature. A switch-mode control system is proposed for the above three stages to track the reference trajectory, which is based on pure pursuit control and Model Predictive Control (MPC). Finally, we validate the cornering framework by simulation on the Simulink-Carsim software and experiments on a 1/10 scale RC vehicle.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, Sensor Signal Processing
Abstract: State-estimation is an integral method for automated driving as the need for more measurement data for vehicle control increases, despite them not always being directly measurable. In the field of state estimation, AI-based algorithms are increasingly attracting interest. However, an uncertainty measure is pivotal to use AI-based state estimation for safety-critical applications. This paper presents the implementation of a vehicle state estimator based on a recurrent neural network and a novel method for uncertainty quantification. The uncertainty quantification method comprises the sequential evaluation of four parts: feature importance algorithms to remove input features lacking informative value, novelty detection filtering data beyond the range of the training data, and prediction of an uncertainty measure and confidence interval with Monte Carlo dropout. The performance of the proposed approach is demonstrated using AI-based state estimation of the vehicle sideslip angle based on the simulation data from a nonlinear two-track model. The results achieved imply that the novel method can provide a reliable confidence interval and successfully identify cases where the estimation and uncertainty quantification are not trustworthy.

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles
Abstract: The BERT-style (Bidirectional Encoder Representations from Transformers) pre-training paradigm has achieved remarkable success in both NLP (Natural Language Processing) and CV (Computer Vision). However, due to the sparsity and lack of semantic information in point cloud data, directly applying this paradigm faces significant obstacles. To address these issues, we propose LSV-MAE, a masked autoencoding pre-training scheme designed for voxel representations. We pre-train the backbone to reconstruct the masked voxels features extracted by PointNN. To enhance the feature extraction capability of the encoder, the point cloud is voxelized with different voxel sizes at different pre-training stages. Meanwhile, to minimize the effect of masking key points during the masking stage, the masked voxel features is re-integrated during the decoder processing. To test the proposed approach, experiments are conducted on well-known datasets. It is shown that the proposed method can improve detection accuracy by 1% to 18%, compared to the model without pre-training, across datasets of different sizes.

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles
Abstract: Sensor fusion is crucial for an accurate and robust perception system on autonomous vehicles. Most existing datasets and perception solutions focus on fusing cameras and LiDAR. However, the collaboration between camera and radar is significantly under-exploited. Incorporating rich semantic information from the camera and reliable 3D information from the radar can achieve an efficient, cheap, and portable solution for 3D perception tasks. It can also be robust to different lighting or all-weather driving scenarios due to the capability of mmWave radars. In this paper, we introduce the CRUW3D dataset, including 66K synchronized and well-calibrated camera, radar, and LiDAR frames in various driving scenarios. Unlike other large-scale autonomous driving datasets, our radar data is in the format of radio frequency (RF) tensors that contain not only 3D location information but also spatio-temporal semantic information. This kind of radar format can enable machine learning models to generate more reliable object perception results after interacting and fusing the information or features between the camera and radar. We run several camera- and radar-based baseline methods for 3D object detection and multi-object tracking on our dataset. We hope the CRUW3D dataset will foster radar and multi-modal 3D perception research. CRUW3D is available at https://huggingface.co/datasets/uwipl/CRUW3D

Keywords: Perception Including Object Event Detection and Response (OEDR), Automated Vehicles
Abstract: Multi-sensor object detection is an active research topic in automated driving, but the robustness of such detection models against missing sensor input (modality missing), e.g., due to a sudden sensor failure, is a critical problem which remains under-studied. In this work, we propose UniBEV, an end-to-end multi-modal 3D object detection framework designed for robustness against missing modalities: UniBEV can operate on LiDAR plus camera input, but also on LiDAR-only or camera-only input without retraining. To facilitate its detector head to handle different input combinations, UniBEV aims to create well-aligned Bird’s Eye View (BEV) feature maps from each available modality. Unlike prior BEV-based multi-modal detection methods, all sensor modalities follow a uniform approach to resample features from the original sensor coordinate systems to the BEV features. We furthermore investigate the robustness of various fusion strategies w.r.t. missing modalities: the commonly used feature concatenation, but also channel-wise averaging, and a generalization to weighted averaging termed Channel Normalized Weights. To validate its effectiveness, we compare UniBEV to state-of-the-art BEVFusion and MetaBEV on nuScenes over all sensor input combinations. In this setting, UniBEV achieves better performance than these baselines for all input combinations. An ablation study shows the robustness benefits of fusing by weighted averaging over regular concatenation, and of sharing queries between the BEV encoders of each modality. Our code is available at https://github.com/tudelft-iv/UniBEV.

Keywords: Perception Including Object Event Detection and Response (OEDR), Cooperative Vehicles, Automated Vehicles
Abstract: Perception serves as the vital cornerstone of autonomous driving system, influencing the decision-making and control performance of vehicles. The rich semantic color information of images, the low cost of cameras and the support of deep learning make visual perception play a pivotal role. However, there are occlusions and blind areas when capturing data using only the on-board camera. With the development of vehicle-to-everything (V2X), information interaction can be achieved based on vehicle-to-vehicle (V2V), cooperative perception of connected autonomous vehicles (CAVs) based on information interaction has become a new trend. This study delves into visual perception based on Transformer attention, and enhances the encoder-decoder through multi-scale feature extraction and queries initialization. Furthermore, a visual cooperative perception method driven by V2V interaction is proposed. Based on spatial registration, data association and multi-source cooperation, perception enhancement of far-sight and see-through is achieved. Experiments were conducted on the real-world KITTI dataset and the PreScan simulator, evaluating the proposed method under various traffic state and density scenarios. Experimental results demonstrate that visual cooperative perception can improve the perception effect of CAVs and adapt to more complex traffic environments.

Keywords: Automated Vehicles, Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR)
Abstract: Object detection in complex scenes and long-distance small object detection are still challenging issues that need to be addressed in the field of autonomous driving. This article proposes a novel method that utilizes 2D image detection results to enhance point cloud semantic and positional features (PFE). Firstly, the method utilizes 2D image object detection box projection to generate a 3D frustum, and excludes non-object point cloud data such as ground and background through filtering. Then, the 2D image classification results are used to provide semantic features of point clouds, enhancing the identification ability of small objects with sparse point cloud data. On the other hand, by projecting a 3D point cloud onto a 2D image and defining positional features based on the distance from the projection point to the center of the 2D detection box, the discrimination ability of point cloud data in complex scenes is further enhanced. The experimental results show that the method proposed in this paper significantly improves the performance of existing LiDAR detection models in complex scenes of the KITTI and NuScenes datasets, and achieves state-of-the-art detection accuracy on long-distance small objects.

Keywords: Perception Including Object Event Detection and Response (OEDR), Functional Safety in Intelligent Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: This work presents a method for monitoring the quality of the data captured by the cameras of an Advanced Driver Assistance System or an Autonomous Driving vehicle. Adverse weather conditions such as rain, snow, fog, or extreme light conditions can diminish the quality of images. Identifying the areas of an image affected by loss of quality due to different phenomena offers valuable information for subsequent critical tasks like object or lane tracking. Using simple features derived from the power spectrum of small image patches and a search strategy based on an evolutionary algorithm, we are able to efficiently locate the affected regions in the image.

Keywords: Perception Including Object Event Detection and Response (OEDR), Functional Safety in Intelligent Vehicles
Abstract: LiDAR-based 3D object detection has become an essential part of automated driving due to its ability to localize and classify objects precisely in 3D. However, object detectors face a critical challenge when dealing with unknown objects, particularly those that were not present in their original training data. These out-of-distribution (OOD) objects can lead to misclassifications, posing a significant risk to the safety and reliability of automated vehicles. Currently, LiDAR-based OOD object detection has not been well studied. We address this problem by generating synthetic training data for OOD objects by perturbing known object categories. Our idea is that these synthetic OOD objects produce different responses in the feature map of an object detector compared to in-distribution (ID) objects. We then extract features using a pre-trained and fixed object detector and train a simple multilayer perceptron (MLP) to classify each detection as either ID or OOD. In addition, we propose a new evaluation protocol that allows the use of existing datasets without modifying the point cloud, ensuring a more authentic evaluation of real-world scenarios. The effectiveness of our method is validated through experiments on the newly proposed nuScenes OOD benchmark. The source code is available at https://github.com/uulm-mrm/mmood3d.

Keywords: Perception Including Object Event Detection and Response (OEDR), Future Mobility and Smart City
Abstract: Applying existing vehicle re-identification methods directly to the re-identification task of E-bikes comes with high costs for capturing and annotating a specific dataset, and it is prone to missing small E-bikes in dense street scenes. In this paper, an innovative E-bikes re-identification system (E-reID) is proposed to address the challenge of E-bikes re-identification for dense small packed object in complex street scenes with only need for a small detection dataset of E-bikes. This system decomposes the task of re-identification for small E-bikes in complex backgrounds into two sub-tasks: instance segmentation and instance retrieval. The instance segmentation is composed of a specific object detection branch that trained with the custom detection dataset to avoid missing the small E-bikes and a MASK branch trained with publicly available datasets containing similar objects such as motorcars and bicycles. For the instance retrieval task, this paper tested methods such as SIFT matching and HSV histogram for matching the same E-bike in different scenarios. The E-reID system built in this paper demonstrates good performance in the custom re-identification dataset of E-bikes. This paper provides an effective and cost-efficient solution to the re-identification of small-target E-bikes in complex scenes.

Keywords: Perception Including Object Event Detection and Response (OEDR), Integration of HD map and Onboard Sensors, Automated Vehicles
Abstract: Vector maps are essential in autonomous driving for tasks like localization and planning, yet their creation and maintenance are notably costly. While recent advances in online vector map generation for autonomous vehicles are promising, current models lack adaptability to different sensor configurations. They tend to overfit to specific sensor poses, leading to decreased performance and higher retraining costs. This limitation hampers their practical use in real-world applications. In response to this challenge, we propose a modular pipeline for vector map generation with improved generalization to sensor configurations. The pipeline leverages probabilistic semantic mapping to generate a bird's-eye-view (BEV) semantic map as an intermediate representation. This intermediate representation is then converted to a vector map using the MapTRv2 decoder. By adopting a BEV semantic map robust to different sensor configurations, our proposed approach significantly improves the generalization performance. We evaluate the model on datasets with sensor configurations not used during training. Our evaluation sets includes larger public datasets, and smaller scale private data collected on our platform. Our model generalizes significantly better than the state-of-the-art methods. The code will be available at https://github.com/AutonomousVehicleLaboratory/SemVecNet .

Keywords: Perception Including Object Event Detection and Response (OEDR), Pedestrian Protection, Human Factors for Intelligent Vehicles
Abstract: Data-driven approaches have been recently proven effective in pedestrian action prediction by extensive works. However, frame-level %labels annotations of pedestrian actions require a significant amount of manpower and time. Furthermore, existing models usually take high-dimensional spatiotemporal data as inputs, which increases the risk of overfitting. In this paper, we propose a simple yet effective contrastive learning framework that enables pedestrian action prediction models to be trained on data without action labels. First of all, we regard disentangled visual observations, such as appearance, motion and trajectories, as multiple modalities. Then we construct a joint latent space where multimodal features from the same sample are encouraged to be close, whereas features from different samples are encouraged to be far from each other. Since most existing models use a similar architecture composed of separate feature extractors and fusion modules, our proposed framework can be applied directly to existing methods to boost the feature extractors. We pretrained state-of-the-art models on datasets without action labels, nuScenes and BDD100k, and evaluated these models on PIE, JAAD and TITAN. Quantitative results show that the pretrained with only the fusion parameters fine-tuned can compete with or even outperform models that are completely trained the one dataset.

Keywords: Automated Vehicles, Perception Including Object Event Detection and Response (OEDR), Sensor Signal Processing
Abstract: For safe driving of autonomous vehicles, it is crucial to detect 3D objects within point clouds with both real-time performance and accuracy. Currently, many autonomous driving research groups adopt simple and efficient pillar-based 3D detectors for real-time performance. However, these detectors often apply grids of large size, which can potentially lead to the loss of significant point information. Moreover, these detectors commonly rely on post-processing steps such as NMS and pseudo-label generation. This reliance can adversely impact detection accuracy because it does not fully reflect the object's location information. To address these issues, we propose 3DSimDet, an efficient and compact 3D object detection framework suitable for autonomous driving platforms. This framework comprises two key components. The proposed SimBackbone module is designed to enhance the feature encoding capabilities of traditional methods. BIoU Head module includes a classification branch and an IoU prediction branch, which considers the object's location information during inference. Moreover, we introduce a high-quality pseudo-label generator for semi-supervised learning, leveraging the prediction outcomes from the BIoU Head module during the semi-supervised learning phase. Through extensive experimentation on diverse public autonomous driving datasets such as WOD, H3D, and A3D, we demonstrate the effectiveness of our proposed method. The experimental findings demonstrate that our 3DSimDet exhibits superior overall performance in terms of accuracy and runtime compared to conventional pillar-based supervised learning models. Furthermore, the pseudo-label generator proves to enhance detection performance within a semi-supervised learning framework.

Keywords: Automated Vehicles, Infotainment Systems and Human-Machine Interface Design, Perception Including Object Event Detection and Response (OEDR)
Abstract: The rapid integration of augmented reality (AR) and virtual reality (VR) technologies into contemporary automotive development has led to unprecedented opportunities and challenges. This work addresses the integration of edge computing and AR/VR applications within connected autonomous vehicles, focusing on the pivotal role of object detection. The edge-assisted object detection problem is formulated as a constrained optimization problem, aiming to minimize the adverse effects on the object detection process. To solve the problem, we introduce an innovative edge-assisted algorithm, transmitting live camera frames to an edge server for detailed processing. Only essential detection data is then relayed to AR/VR devices, marking a significant advancement over existing strategies. Notable outcomes include a reduction in latency (averaging between 37.06% and 44.76%), enhanced data throughput (ranging from 27.66% to 41.18%), improved freshness loss (between 36.36% and 69.57%), and a frame loss reduction to 7.5%, surpassing baseline methods by 6.5% to 36%. These findings underscore the potential of this methodology for optimizing AR/VR applications in vehicular environments.

Keywords: Sensor Signal Processing, Automated Vehicles, Perception Including Object Event Detection and Response (OEDR)
Abstract: Multi-modal fused images can provide reliable perceptual information for intelligent vehicles in various weather and lighting conditions. However, most existing fusion algorithms neglect the information interactions among different modalities, leading to a loss of essential information in transportation systems characterized by strong information correlations. To enhance the quality of multi-modal semantic information fusion perception in intelligent vehicles, we propose the Cross Rubik Cube Attention Fusion Network (RCAFusion). Inspired by the shape and recovery process of a Rubik's Cube, RCAFusion establishes an information interaction pathway among different modalities, and it achieves a more comprehensive information crossover through the simultaneous spatial attention, channel attention, and self-attention mechanisms, which enhance the feature extraction effect in the fusion architecture. Experimental results demonstrate that RCAFusion surpasses existing fusion algorithms in several metrics and improves 5.04% in the objective fused image metric Qabf. Moreover, the fused images output by RCAFusion have good results in the image object detection task and can achieve 95.4% mAP using the yolov8m model in the MSRS open source datasets. Our code and pre-trained model will be available at https://github.com/vehicle-AngLi/RCAFusion upon acceptance.

Keywords: Automated Vehicles, Perception Including Object Event Detection and Response (OEDR)
Abstract: For autonomous vehicles to proactively plan safe trajectories and make informed decisions, they must be able to predict the future occupancy states of the local environment. However, common issues with occupancy prediction include predictions where moving objects vanish or become blurred, particularly at longer time horizons. We propose an environment prediction framework that incorporates environment semantics for future occupancy prediction. Our method first semantically segments the environment and uses this information along with the occupancy information to predict the spatiotemporal evolution of the environment. We validate our approach on the real-world Waymo Open Dataset. Compared to baseline methods, our model has higher prediction accuracy and is capable of maintaining moving object appearances in the predictions for longer prediction time horizons.

Keywords: Sensor Signal Processing, Automated Vehicles, Perception Including Object Event Detection and Response (OEDR)
Abstract: Adverse weather conditions such as snow, rain, and fog are natural phenomena that can impair the performance of the perception algorithms in autonomous vehicles. Although LiDARs provide accurate and reliable scans of the surroundings, its output can be substantially degraded by precipitation (e.g., snow particles) leading to an undesired effect on the downstream perception tasks. Several studies have been performed to battle this undesired effect by filtering out precipitation outliers, however, these works have large memory consumption and long execution times which are not desired for onboard applications. To that end, we introduce a novel outlier detector for 3D LiDAR point clouds captured under adverse weather conditions. Our proposed detector 3D-OutDet is based on a novel convolution operation that processes nearest neighbors only, allowing the model to capture the most relevant points. This reduces the number of layers, resulting in a model with a low memory footprint and fast execution time, while producing a competitive performance compared to state-of-the-art models. We conduct extensive experiments on three different datasets (WADS, SnowyKITTI, and SemanticSpray) and show that with a sacrifice of 0.16% mIOU performance, our model reduces the memory consumption by 99.92%, number of operations by 96.87%, and execution time by 82.84% per point cloud on the real-scanned WADS dataset. Our experimental evaluations also showed that the mIOU performance of the downstream semantic segmentation task on WADS can be improved up to 5.08% after applying our proposed outlier detector. We release our source code, supplementary material and videos in https://sporsho.github.io/3DOutDet. Upon clicking the link you will have to option to go to source code

Keywords: Eco-Driving and Energy-Efficient Vehicles, Automated Vehicles, Future Mobility and Smart City
Abstract: The ongoing shift towards electric vehicles and the simultaneous integration of sensor, computing and communication systems for automated driving pose challenges for the interplay between batteries and onboard autonomous mobility functions and services. Research on autonomous test vehicle (ATV) power consumption does not match the current realities of industry solutions brought to early markets worldwide. Current research predicts high power demands of commercial AVs, reflected in extrapolating data points of underlying hardware data sheets, simulations, or the piloting of ATVs. Although we can relay to significant power demands in ATVs, we identified different realities and higher efficiency in emerging industry vehicles. We show a counteracting reality in how AVs power demand is perceived due to a mismatch between ATVs and early market-ready AVs. To bridge the gap between research and practice we pose a power-demand model and core indicators to be considered in estimating power demands.

Keywords: Eco-Driving and Energy-Efficient Vehicles, Automotive Cyber Physical Systems, Battery Management Systems and State-of-Charge (SoC) Estimation
Abstract: We present a study focusing on the joint optimization of the sizing of hardware components as well as strategic decisions for a race car in a Formula 1 setting. Our research leverages a monotone theory of co-design, which allows for hardware and software considerations to achieve optimal, synergistic performance improvements. We aim to identify the Pareto optimal curves that illustrate the optimal balance between conflicting objectives, such as lap time, energy allocation, and component choice, within the tight constraints imposed by the regulations. The results of the study demonstrate the versatility of our framework by showing optimal component sizing on two structurally different track layouts on a single lap. Moreover, by increasing the amount of laps under consideration, we show the ability of our tool to consider strategic energy allocation decisions.

Keywords: Eco-Driving and Energy-Efficient Vehicles, Future Mobility and Smart City, Integration of Infrastructure and Intelligent Vehicles
Abstract: Informing drivers about the predicted state of upcoming traffic lights is considered a key solution to reduce unneeded energy expenditure and dilemma zones at intersections. However, newer traffic lights can react to traffic demand, resulting in spontaneous switching behavior and poor predictability. To assess whether future traffic light assistance services are viable, it is crucial to understand how strongly predictability is affected by such spontaneous switching behavior. Previous studies have so far only reported percentages of adaptivity-capable traffic lights, but the actual switching behavior has not been measured. Addressing this research gap, we conduct a large-scale predictability evaluation based on 424 million recorded switching cycles over four weeks for 18,009 individual traffic lights in Hamburg. Two characteristics of predictability are studied: cycle discrepancy and wait time diversity. Results indicate that fewer traffic lights exhibit hard-to-predict switching behavior than suggested by previous work, considering a reported number of 90.7% adaptive traffic lights in Hamburg. Contrasting previous work, we find that not all traffic lights capable of adaptiveness may necessarily exhibit low predictability. We critically review these results and derive avenues for future research.

Keywords: Eco-Driving and Energy-Efficient Vehicles, Policy, Ethics, and Regulations, Integration of Infrastructure and Intelligent Vehicles
Abstract: The introduction of automated trucks into the ground freight sector will yield significant transformations, such as the alleviation of labor shortages, the reduction of energy consumption, and the improvement of road safety. Given the current mixed-traffic environment where autonomous vehicles share roads with their human-driven counterparts, support from intelligent transportation infrastructure becomes imperative for ensuring the safe and efficient operations of autonomous trucks. This study proposes an integrated framework for designing dedicated lanes for automated trucks (DLAT) operating under Hours of Service (HOS) Regulations for total cost minimization. This framework considers the truck fleets' routing and scheduling behavioral changes brought by the implementation of DLAT. We formulate a mixed-integer program (MIP) model and propose an origin-destination(OD)-clustering-based iterative algorithm to tackle it. The algorithm contains two parts: the spectral clustering method for OD pairs partitions and DLAT design within each cluster, and the iterative algorithm addressing the dedicated lanes design problem on overlapping links across clusters. Three numerical experiments on the U.S. freight highway network and algorithm demonstrate the effectiveness of time-varying planning of DLAT, corroborating its benefits to stakeholders and providing managerial insights for future implementations.

Keywords: Eco-Driving and Energy-Efficient Vehicles, Simulation and Real-World Testing Methodologies, Sensor Signal Processing
Abstract: Luxury car manufacturers are facing the challenge of transitioning their customers to electric mobility without compromising on comfort levels. While efforts to enhance efficiency have traditionally focused on powertrain and aerodynamics, the rise in efficiency of electric drive trains has highlighted the significance of low-voltage power consumers. To address this, our study proposes a data-driven modelling strategy that leverages real-time in-car communication data to gain insights into the power consumption behavior of low-voltage power consumers across various real-world driving scenarios. Our methodology involves a selection process for simulation-worthy consumers, an appropriate regression algorithm, feature selection techniques, and Key Performance Indicators to assess the models' quality and efficiency. By utilizing our data-driven modelling strategy, we can provide an individual and transparent evaluation of each power consumer, enabling automotive development engineers to optimize their vehicles' power distribution network and deliver an unparalleled driving experience.

Keywords: Eco-Driving and Energy-Efficient Vehicles, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Vehicle Control and Motion Planning
Abstract: This paper addresses the eco-driving problem for connected vehicles on urban roads, considering localization uncertainty. Eco-driving is defined as longitudinal speed planning and control on roads with the presence of a sequence of traffic lights. We solve the problem by using a data-driven model predictive control (MPC) strategy. This approach involves learning a cost-to-go function and constraints from state-input data. The cost-to-go function represents the remaining energy-to-spend from the given state, and the constraints ensure that the controlled vehicle passes the upcoming traffic light timely while obeying traffic laws. The resulting convex optimization problem has a short horizon and is amenable for real-time implementations. We demonstrate the effectiveness of our approach through real-world vehicle experiments. Our method demonstrates 12% improvement in energy efficiency compared to the traditional approaches, which plan longitudinal speed by solving a long-horizon optimal control problem and track the planned speed using another controller, as evidenced by vehicle experiments.

Keywords: Eco-Driving and Energy-Efficient Vehicles
Abstract: The equivalent consumption minimization strategy (ECMS) is a well-established energy management strategy for hybrid vehicles, which is easily real-time implementable and can provide optimal energy management. However, optimality requires knowledge of the optimal equivalence factor, which highly depends on the driving cycle and is therefore unknown in advance. This work proposes a predictive ECMS for fuel cell hybrid vehicles, which derives a map describing the optimal equivalence factor for any vehicle position and battery state of charge from the optimal cost-to-go provided from an offline optimization. The offline optimization is conducted with dynamic programming before departure and considers a long-term driving cycle prediction derived from static route information such as speed limits and altitude. Based on the optimal equivalence factor map, the ECMS implicitly considers the long-term prediction in each instant allowing for continuous adaption to the current situation while driving. The performance of the predictive ECMS is demonstrated in a numerical study based on real-world driving cycles highlighting its robustness against unpredicted changes in traffic conditions.

Keywords: Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Eco-Driving and Energy-Efficient Vehicles, Automated Vehicles
Abstract: Many studies have demonstrated the eco-driving capabilities of connected and automated vehicles (CAVs) to significantly enhance mobility systems. The majority of these studies have been conducted using simulations, which fail to capture the effects of practical uncertainties encountered in vehicle-to-anything (V2X) communications. In this paper, we investigated the performance of current cellular V2X (C-V2X) communications through systematic testing and provided a quantitative analysis of key performance indices (e.g., inter-packet gap and packet error rate) across various test scenarios. As one use case to demonstrate the benefits of C-V2X communication on the road, we tested the feasibility of eco-driving for a SAE level 3 (L3) automated vehicle (AV) communicating with a connected urban corridor capable of transmitting traffic light information (i.e., signal phase and timing). To achieve this, we implemented the eco-speed planning algorithm at a high-level in the AV control software system and ensured its interactions with other existing low-level control algorithms, as well as the C-V2X onboard unit. Finally, we experimentally demonstrated eco-driving of the L3 CAV on a scaled-down corridor with two signal-controlled intersections, revealing the AV's ability to maintain smoother trajectories and avoid unnecessary stops compared to human-driven vehicles.

Keywords: Automated Vehicles, Future Mobility and Smart City, Eco-Driving and Energy-Efficient Vehicles
Abstract: With consistently increasing amounts of transported goods, autonomous cargo transport has gained increasing interest as a potential solution. In addition to reliable autonomous driving functions, Autonomous cargo transport requires a wide range of additional software and hardware components to ensure a safe and efficient transport of cargo as well as a pleasant user experience for the customers. This work presents a general concept for an autonomous and flexible cargo transport system, targeting point-to-point transports in the range of the typical last mile. The proposed concept provides flexibility for demand-responsive passenger transport as a mixed cargo-passenger transport solution. Furthermore, the proposed concept is realized through designing a removable cargo hold with electronic locks, and software modules such as a Booking App, a Scanner App, and a central backend. The implementation was developed, deployed in an autonomous shuttle, and extensively tested in a peri-urban quarter of the Test Area Autonomous Driving Baden-Württemberg

Keywords: Future Mobility and Smart City, Simulation and Real-World Testing Methodologies, Eco-Driving and Energy-Efficient Vehicles
Abstract: Rising demand for last-mile deliveries in the logistics sector has prompted the adoption of Autonomous Delivery Robots (ADRs) and electric trucks (eTrucks) for their efficiency and cost-effectiveness. This paper proposes an optimization model for an integrated eTruck-and-ADR system. The model employs a range of information sources to optimize vehicle routing and robot allocation, emphasizing energy efficiency and operating cost. This includes incorporating Geographic Information System (GIS) to estimate customer demand based on demographics and utilizing a battery aging/degradation model to account for hardware depreciation. A metaheuristic Genetic Algorithm is employed to solve optimal vehicle routing and customer node clustering. In a simulated case study conducted with real GIS and geographic data, the proposed model demonstrates efficacy in determining the optimal number of ADRs for specific census tracts, with a cost breakdown highlighting the dominance of human labor costs.

Keywords: Battery Management Systems and State-of-Charge (SoC) Estimation
Abstract: The energy consumed to overcome gravity during elevation gain is a significant factor in the energy consumption of electric vehicles (EVs). Assessing elevation influence can help improve the accuracy of estimated energy consumption, which will alleviate drivers' range anxiety. This study explores how to improve the accuracy of energy consumption prediction for EVs using elevation features. The trip dataset is supplemented with elevation features, and then a voting ensemble model of machine learning is proposed to predict energy consumption. Also, a total of 10,847 trip records from 16 hilliness cities and 13 flatness cities in the United States are studied. The experimental results show that the prediction accuracy of EVs energy consumption improves with the inclusion of elevation features, where the Mean Absolute Error (MAE) of the prediction result decreases from 796 Wh to 695 Wh, and the R-squared (R²) score of the prediction result increases by 1.6% to finally reach 94.4%.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, Sensor Fusion for Localization
Abstract: When academic researchers develop and validate autonomous driving algorithms, there is a challenge in balancing high-performance capabilities with the cost and complexity of the vehicle platform. Much of today's research on autonomous vehicles (AV) is limited to experimentation on expensive commercial vehicles that require large skilled teams to retrofit the vehicles and test them in dedicated facilities. On the other hand, 1/10th-1/16th scaled-down vehicle platforms are more affordable but have limited similitude in performance and drivability. To address this issue, we present the design of a one-third-scale autonomous electric go-kart platform with open-source mechatronics design along with fully functional autonomous driving software. The platform's multi-modal driving system is capable of manual, autonomous, and teleoperation driving modes. It also features a flexible sensing suite for the algorithm deployment across perception, localization, planning, and control. This development serves as a bridge between full-scale vehicles and reduced-scale cars while accelerating cost-effective algorithmic advancements. Our experimental results demonstrate the AV4EV platform's capabilities and ease of use for developing new AV algorithms. All materials are available at AV4EV.org to stimulate collaborative efforts within the AV and electric vehicle (EV) communities.

Keywords: Human Factors for Intelligent Vehicles, End-To-End (E2E) Autonomous Driving, Verification and Validation Techniques
Abstract: Autonomous vehicles often make complex decisions via machine learning-based predictive models applied to collected sensor data. While this combination of methods provides a foundation for real-time actions, self-driving behavior primarily remains opaque to end users. In this sense, explainability of real-time decisions is a crucial and natural requirement for building trust in autonomous vehicles. Moreover, as autonomous vehicles still cause serious traffic accidents for various reasons, timely conveyance of upcoming hazards to road users can help improve scene understanding and prevent potential risks. Hence, there is also a need to supply autonomous vehicles with user-friendly interfaces for effective human-machine teaming. Motivated by this problem, we study the role of explainable AI and human-machine interface jointly in building trust in vehicle autonomy. We first present a broad context of the explanatory human-machine systems with the “3W1H” (what, whom, when, how) approach. Based on these findings, we present a situation awareness framework for calibrating users’ trust in self-driving behavior. Finally, we perform an experiment on our framework, conduct a user study on it, and validate the empirical findings with hypothesis testing.

Keywords: Human Factors for Intelligent Vehicles, Automated Vehicles
Abstract: Advancements in autonomous vehicle (AV) technologies necessitate precise estimation of perceived risk to enhance user comfort, acceptance and trust. This paper introduces a novel AV-Occupant Risk (AVOR) model designed for perceived risk estimation during AV cut-in scenarios. An empirical study is conducted with 18 participants with realistic cut-in scenarios. Two factors were investigated: scenario risk and scene population. 76 % of subjective risk responses indicate an increase in perceived risk at cut-in initiation. The existing perceived risk model did not capture this critical phenomenon. Our AVOR model demonstrated a significant improvement in estimating perceived risk during the early stages of cut-ins, especially for the high-risk scenario, enhancing modelling accuracy by up to 54%. The concept of the AVOR model can quantify perceived risk in other diverse driving contexts characterized by dynamic uncertainties, enhancing the reliability and human-centred focus of AV systems.

Keywords: Human Factors for Intelligent Vehicles, Infotainment Systems and Human-Machine Interface Design, Automated Vehicles
Abstract: Interactions between humans and automated vehicles (AVs) will become increasingly common in future traffic. Cooperative behavior enables comfortable and efficient resolutions of such interactions through joint actions. Thus, AVs need the ability to communicate their intention and planned behavior to outside road users to facilitate cooperation among the interaction partners. Many recent studies suggest external human-machine-interfaces (eHMIs) as a feasible way to realize the communication between AVs and other road users. These studies often focus on the design of eHMIs and evaluate their usability based on subjective measures from the perspective of the outside road user. There is only little research on the objective benefits of explicit external communication on the resolution of interactions with AVs. The decision making behind the activation of the communication and the consideration of the actual interaction in the communication are still open research topics. In this work, we present a communication framework that facilitates cooperation between an AV and pedestrians by communicating the vehicle’s yielding intention to give right of way. We demonstrate the proposed system’s ability to improve the joint utility as well as the driving comfort by empirically evaluating simulated interactions between AVs and pedestrians. The results indicate that timely communication leads to more efficient and more predictable pedestrian-vehicle interactions.

Keywords: Human Factors for Intelligent Vehicles, Infotainment Systems and Human-Machine Interface Design, Automated Vehicles
Abstract: This study takes a fresh perspective by focusing on the drivers of surrounding cars near to level 3 automated vehicles (AVs). We advocates for level 3 AVs using an external human-machine interface (eHMI) to provide high-risk warning information to drivers of surrounding cars during AVs issuing a request-to-intervene (RtI). Through a driving simulator-based subjects experiments, we have established that the proposed eHMI can assist the surrounding MV's driver in better comprehending the AV's driving intentions and predicting its driving behavior. This leads to increased the surrounding MV's driver confidence in handling potential risks from the AV during the take-over period. While we did not observe a significant impact of the proposed eHMI on the driving behavior of the MV drivers, i.e. participants, they reported a greater willingness to have AVs equipped with the proposed eHMI drive around them in their daily life.

Keywords: Human Factors for Intelligent Vehicles, Infotainment Systems and Human-Machine Interface Design, Future Mobility and Smart City
Abstract: This study aims to investigate the causal relationships from pedestrians' psychological states to their walking behavior during interactions with an autonomous personal mobility vehicle (APMV) featuring automation capabilities ranging from SAE levels 3 to 5. A subjective experiment was conducted, where various external human-machine interfaces (eHMIs) were designed to induce participants to experience different levels of subjective feelings and generate corresponding walking behaviors. By employing a structural equation model named DirectLiNGAM to analyze the collected data for causal discovery, the results of causal discovery align with the hypothesized model of the pedestrian's cognition-decision-behavior process. Furthermore, the experimental results have enriched the detailed causal relationships within the hypothesized model, i.e., the outcomes of situation awareness lead to a sense of danger, trust in APMV and a sense of relief; the outcomes of situation awareness, the sense of danger and trust in APMV lead to hesitation in decision-making; and the outcomes of situation awareness, the sense of danger and hesitation lead to walking behaviors.

Keywords: Human Factors for Intelligent Vehicles, Infotainment Systems and Human-Machine Interface Design
Abstract: Drowsy driving has a high potential for causing serious accidents because it leads to collisions without hazard avoidance. Against this backdrop, in Europe, the installation of driver drowsiness and attention warning (DDAW) systems became mandatory under the General Safety Regulations (GSR) in 2022. The DDAW systems use the Karolinska Sleepiness Scale (KSS) to assess the extent of driver drowsiness. However, although various representative indices to quantify drowsiness, such as KSS, are available, they present issues in terms of reproducibility and practical use, and there is currently no uniform index. This present research therefore sought to propose practical new indices of drowsiness. At the same time, this research aims to provide indices based on features extracted from images by focusing on the relationship with the comparatively severe range of drowsiness represented by KSS scores of 7 or higher, which must be detected for compliance with DDAW. The results demonstrated that the occurrence of “Eye closure“ and “Yawning“ could serve as substitutes for states involving KSS scores of 7 or higher. These drowsiness behaviors can be objectively and reproducibly annotated, leading to the conclusion that they could be candidates for new indicators.

Keywords: Human Factors for Intelligent Vehicles, Infotainment Systems and Human-Machine Interface Design
Abstract: Conditionally automated driving enables drivers to engage in non-driving-related activities, with the responsibility to take over vehicle control upon request. This takeover process increases the risk of collisions, especially when drivers fail to safely complete takeovers within limited time budgets (the time offered by automation for takeovers). This phenomenon underlines the significance of providing time budgets that sufficiently accommodate drivers' takeover time (the time required by drivers to resume conscious vehicle control). Considering that drivers' takeover time varies significantly across scenarios, this study centres on understanding the role of driver perception in takeover time using the Task-Capability Interface (TCI) theory. TCI theory suggests that drivers adjust their behaviours based on perceived task demands and driver capabilities. Accordingly, in a driving simulator experiment featuring diverse traffic densities and distractions, we investigated drivers' takeover time while capturing their perceived task demands and capabilities through a takeover-oriented questionnaire based on established instruments. The results show that drivers generally have longer takeover time as their perceived task demand rises, perceived capability diminishes, and perceived saturation (perceived driver capability minus perceived task demand) decreases. These patterns fluctuate under conditions of low perceived task demand or high perceived driver capability. When both conditions coincide, drivers necessitate considerably longer takeover time. Our findings contribute to the development of strategies aimed at predicting drivers' takeover time, optimizing time budgets, fostering human-centred vehicle design, and enhancing the safety of conditionally automated driving.

Keywords: Human Factors for Intelligent Vehicles, Policy, Ethics, and Regulations, Teleoperation of Intelligent Vehicles
Abstract: We design a survey-and-scenario-based method for investigating the demand of real people for ride-pooling services in Berlin. We deploy it among public transport users to study where, why, how, and when they would use ride-pooling. In particular, we learn about their socio-demographic features, trip purposes, and deciding factors, as well as their preferences for microtransit service waiting times and fares. Our results inform service operators about dispatching decisions, allow for encoding user behavior in utility functions for reasoning, and enable the design of fair pricing schemes to improve service accessibility.

Keywords: Human Factors for Intelligent Vehicles, Simulation and Real-World Testing Methodologies
Abstract: Situational awareness of the driving environment is crucial for making safe and informed driving decisions. It can be deteriorated by distractions or environmental properties such as low light. In order to study and model the effects of environmental, perceptual, and behavioral features within the binocular field of view on the drivers' awareness of pedestrians, an experimental setup consisting of a Virtual Reality (VR) based driving simulator, data generation, and analysis solution has been developed. An experimental study was conducted to evaluate the developed setup and analyze drivers' awareness of pedestrians in the context of varying external conditions. Results show that driver awareness can be measured from the properties of eye tracking and a secondary detection task to detect pedestrians. It is demonstrated how this research aims towards developing more human-aware driver monitoring systems with assistive functionalities such as attention guidance, taking perceptual and cognitive factors into account.

Keywords: Human Factors for Intelligent Vehicles, Simulation and Real-World Testing Methodologies
Abstract: Scenario-based testing for automated driving systems (ADS) is an industry norm for safety assurance. A scenario describes situations that an automated driving systems may encounter during its operation. To ensure accurate representation of real-world situations, including human behavior and system interactions, a formal language is essential. It ensures consistent testing across diverse scenarios and facilitates compatibility with simulation tools. However, while existing scenario languages excel in describing environmental and road structure aspects, they lack the same detail for road users and drivers. We have developed a methodology to identify and incorporate relevant human factors elements into scenario languages. Our methodology focuses on understanding diverse individuals and their interactions with ADS on the road, enabling their representation in scenarios. We offer practical examples to improve language representation of human elements and actions, in WMG-SDL Level-2 for logical scenarios and BSI Flex 1889 for abstract scenario descriptions. This methodology serves as a starting point for language designers to accurately represent all road users and their interactions with ADS.

Keywords: Human Factors for Intelligent Vehicles, Software-Defined Vehicle for Intelligent Vehicles, Integration of Onboard Systems and Cloud-Based Services
Abstract: As the demand for electric vehicles (EVs) has increased in recent years, automotive manufacturers for EV productions face technical challenges addressed from a different perspective than conventional internal combustion engine vehicles. In particular, the battery capability in EVs determines most of the vehicle's performance and service features. Among them, the development of a range estimation algorithm for distance to empty in EVs is very important because drivers directly perceive the driving distance available under the current battery condition. Therefore, an accurate estimation of the distance to empty is able to improve customer satisfaction from a service point of view. In this paper, the model-based distance to empty algorithm is proposed to improve the prediction accuracy and scalability of additional service development. The concept of a driving behavior model is also proposed to represent personalization for driving energy patterns. The experimental results show the effectiveness of the proposed method only using in-vehicle measurement signals.

Keywords: Automated Vehicles, Vehicular Active and Passive Safety, Human Factors for Intelligent Vehicles
Abstract: Road user trajectory prediction in dynamic environments is a challenging but crucial task for various applications, such as autonomous driving. One of the main challenges in this domain is the multimodal nature of future trajectories stemming from the unknown yet diverse intentions of the agents. Diffusion models have shown to be very effective in capturing such stochasticity in prediction tasks. However, these models involve many computationally expensive denoising steps and sampling operations that make them a less desirable option for real-time safety-critical applications. To this end, we present a novel framework that leverages diffusion models for predicting future trajectories in a computationally efficient manner. To minimize the computational bottlenecks in iterative sampling, we employ an efficient sampling mechanism that allows us to maximize the number of sampled trajectories for improved accuracy while maintaining inference time in real time. Moreover, we propose a scoring mechanism to select the most plausible trajectories by assigning relative ranks. We show the effectiveness of our approach by conducting empirical evaluations on common pedestrian (UCY/ETH) and autonomous driving (nuScenes) benchmark datasets on which our model achieves state-of-the-art performance on several subsets and metrics.

Keywords: Infotainment Systems and Human-Machine Interface Design, Simulation and Real-World Testing Methodologies, Verification and Validation Techniques
Abstract: In-mold electronics can reduce deformation by integrating multiple components and encapsulating them with plastic injection molding. However, complex components and electronic functional films can cause heat concentration problems. We used an in-mold radiating structure to heat-dissipating from electronic devices, and thermal simulation analysis was used to model the structure and process, and the results showed that local contact with heat-dissipating ink can effectively reduce heat energy by 25%. SMD LED components with plastic injection molding can increase thermal energy by 36%. Proper heat dissipation structures and sealing with plastic injection molding can effectively discharge heat energy, but if the heat-dissipating structure in contact with the heat source does not match, the temperature can reach 154.2°C, causing the LED to malfunction.

Keywords: Simulation and Real-World Testing Methodologies, Automated Vehicles, Verification and Validation Techniques
Abstract: In this paper, we introduce a method for estimating blind spots for sensor setups of autonomous or automated vehicles and/or robotics applications. In comparison to previous methods that rely on geometric approximations, our presented approach provides more realistic coverage estimates by utilizing accurate and detailed 3D simulation environments. Our method leverages point clouds from LiDAR sensors or camera depth images from high-fidelity simulations of target scenarios to provide accurate and actionable visibility estimates. A Monte Carlo-based reference sensor simulation enables us to accurately estimate blind spot size as a metric of coverage, as well as detection probabilities of objects at arbitrary positions.

Keywords: Simulation and Real-World Testing Methodologies, Cooperative Vehicles, Software-Defined Vehicle for Intelligent Vehicles
Abstract: Mine fleet management algorithms can significantly reduce operational costs and enhance productivity in mining systems. Most current fleet management algorithms are evaluated based on self-implemented or proprietary simulation environments, posing challenges for replication and comparison. This paper models the simulation environment for mine fleet management from a complex systems perspective. Building upon previous work, we introduce probabilistic, user-defined events for random event simulation and implement various evaluation metrics and baselines, effectively reflecting the robustness of fleet management algorithms against unforeseen incidents. We present ''OpenMines'', an open-source framework encompassing the entire process of mine system modeling, algorithm development, and evaluation, facilitating future algorithm comparison and replication in the field. Code is available at https://github.com/370025263/openmines.

Keywords: Simulation and Real-World Testing Methodologies, Drone and Urban Air Mobility, Future Mobility and Smart City
Abstract: This study introduces a novel model known as the Parallel Scheduling Vehicle Routing Problem (PSVRP) in an endeavor to revolutionize package delivery by enhancing its efficiency, accessibility, and cost-effectiveness. The PSVRP represents a state-of-the-art approach to vehicle routing problems, incorporating a diversified fleet of innovative delivery modes. The multi-modal fleet of electric vans, ADVs, drones, and truck-drone systems works in unison to minimize operational costs in various settings. The model can also be adapted to different scenarios, such as variations in customer numbers and package weights, by strategically deploying the most suitable mode of transport.

Keywords: Simulation and Real-World Testing Methodologies, Human Factors for Intelligent Vehicles, Pedestrian Protection
Abstract: In the field of autonomous driving, a key challenge is the “reality gap”: transferring knowledge gained in simulation to real-world settings. Despite various approaches to mitigate this gap, there’s a notable absence of solutions targeting agent behavior generation which are crucial for mimicking spontaneous, erratic, and realistic actions of traffic participants. Recent advancements in Generative AI have enabled the representation of human activities in semantic space and generate real human motion from textual descriptions. Despite current limitations such as modality constraints, motion sequence length, resource demands, and data specificity, there’s an opportunity to innovate and use these techniques in the intelligent vehicles domain. We propose Walk-the-Talk, a motion generator utilizing Large Language Models (LLMs) to produce reliable pedestrian motions for high-fidelity simulators like CARLA. Thus, we contribute to autonomous driving simulations by aiming to scale realistic, diverse long-tail agent motion data - currently a gap in training datasets. We employ Motion Capture (MoCap) techniques to develop the Walk-the-Talk dataset, which illustrates a broad spectrum of pedestrian behaviors in street-crossing scenarios, ranging from standard walking patterns to extreme behaviors such as drunk walking and near-crash incidents. By utilizing this new dataset within a LLM, we facilitate the creation of realistic pedestrian motion sequences, a capability previously unattainable. Additionally, our findings demonstrate that leveraging the Walk-the-Talk dataset enhances cross-domain generalization and significantly improves the Fréchet Inception Distance (FID) score by approximately 15% on the HumanML3D dataset.

Keywords: Simulation and Real-World Testing Methodologies, Integration of Infrastructure and Intelligent Vehicles, Smart Infrastructure
Abstract: In autonomous driving systems, infrastructure LiDAR technology provides advanced point cloud information of the road, allowing for preemptive analysis, which increases decision-making time. 3D object detection affords autonomous vehicles the ability to recognize and understand surrounding environmental objects accurately. To further investigate the optimal deployment locations and impacts of infrastructure LiDAR in autonomous driving systems, we have developed an automated annotation framework integrated into an autonomous driving simulator. This framework enables the automated labeling of point cloud data and the rapid construction of datasets, significantly reducing the time required for users to create such datasets. Additionally, we enhanced the usability of the autonomous driving simulator, allowing for real-time adjustments of LiDAR settings during operation, and the generation of vehicle NPCs in accordance with the OpenSCENARIO 2.0 standard. Finally, utilizing this automatic annotation framework, we conducted an evaluation of the impact of various types of LiDAR (dense point clouds and sparse point clouds) and their quantities on the accuracy of 3D object detection models. The experimental evaluation shows that the number of points in infrastructure point clouds and the detection range have a significant impact on 3D detection models. Upon replacing VLP-16 with MID70, the performance of various models improved significantly, with a maximum increase of 50% mAP.

Keywords: Simulation and Real-World Testing Methodologies, Perception Including Object Event Detection and Response (OEDR), Automated Vehicles
Abstract: Datasets are essential for training and testing vehicle perception algorithms. However, the collection and annotation of real-world images is time-consuming and expensive. Driving simulators offer a solution by automatically generating various driving scenarios with corresponding annotations, but the simulation-to-reality (Sim2Real) domain gap remains a challenge. While most of the Generative Artificial Intelligence (AI) follows the de facto Generative Adversarial Nets (GANs)-based methods, the recent emerging diffusion probabilistic models have not been fully explored in mitigating Sim2Real challenges for driving data synthesis. To explore the performance, this paper applied three different generative AI methods to leverage semantic label maps from a driving simulator as a bridge for the creation of realistic datasets. A comparative analysis of these methods is presented from the perspective of image quality and perception. New synthetic datasets, which include driving images and auto-generated high-quality annotations, are produced with low costs and high scene variability. The experimental results show that although GAN-based methods are adept at generating high-quality images when provided with manually annotated labels, ControlNet produces synthetic datasets with fewer artefacts and more structural fidelity when using simulator-generated labels. This suggests that the diffusion-based approach may provide improved stability and an alternative method for addressing Sim2Real challenges. These insights contribute to the intelligent vehicle community's understanding of the potential for diffusion models to mitigate the Sim2Real gap.

Keywords: Simulation and Real-World Testing Methodologies, Security and Privacy, Functional Safety in Intelligent Vehicles
Abstract: This work presents a simulation framework for CAN communication including attacks, with a particular focus on bus-off and WeepingCAN attacks. The simulation is validated against existing experiments, demonstrating its accuracy in replicating attack scenarios proposed by Cho et al. [11] and Bloom [2]. An Intrusion Detection System (IDS) is evaluated using the Hacking and Countermeasures Research Lab (HCRL) data set, and limitations in detecting WeepingCAN attacks are identified. To address these limitations, an extension to the IDS is proposed that incorporates an error frequency criterion to improve detection accuracy. The extended IDS is evaluated in different environments, demonstrating its effectiveness in reliably distinguishing between normal communications and WeepingCAN attacks. Overall, this work contributes to the understanding and detection of sophisticated attacks on CAN and offers support to develop and evaluate further detection mechanisms.

Keywords: Simulation and Real-World Testing Methodologies, Sensor Signal Processing, Advanced Driver Assistance Systems (ADAS)
Abstract: Autonomous vehicles rely on camera, LiDAR, and radar sensors to navigate the environment. Adverse weather conditions like snow, rain, and fog are known to be problematic for both camera and LiDAR-based perception systems. Currently, it is difficult to evaluate the performance of these methods due to the lack of publicly available datasets containing multimodal labeled data. To address this limitation, we propose the SemanticSpray++ dataset, which provides labels for camera, LiDAR, and radar data of highway-like scenarios in wet surface conditions. In particular, we provide 2D bounding boxes for the camera image, 3D bounding boxes for the LiDAR point cloud, and semantic labels for the radar targets. By labeling all three sensor modalities, the SemanticSpray++ dataset offers a comprehensive test bed for analyzing the performance of perception methods when vehicles travel on wet surface conditions. Together with comprehensive label statistics, we also evaluate multiple baseline methods across different tasks and analyze their performances. The dataset will be available upon acceptance.

Keywords: Simulation and Real-World Testing Methodologies, Smart Infrastructure, Vehicular Active and Passive Safety
Abstract: Intersections are conflict zones where the paths of vehicles and pedestrians intersect. They are particularly prone to accidents, with a significant portion of both fatal and nonfatal crashes occurring at these locations. Understanding the diverse behaviors of traffic participants at intersections is crucial for mitigating risks and improving safety. However, conducting experiments in real-world settings to study such behaviors is impractical and hazardous. In this work, we propose a data-driven approach for predicting and simulating vehicular and pedestrian traffic at signalized intersections. Our method focuses on accurately modeling their behavior and interactions at a busy intersection. We introduce a multi-agent trajectory prediction model, utilizing the Conditional Variational Autoencoder (CVAE) framework, to generate plausible future trajectories. By incorporating intersection geometry, agent history, and signal state, our model produces a realistic prediction of traffic dynamics. Our model outperforms a strong baseline model—-Trajectron++-—by 17% in terms of final displacement errors. We then apply a rule-based trajectory sampling approach to simulate long-horizon behaviors, enabling the proactive identification of potential risks and the exploration of counterfactual scenarios. This research contributes to the development of predictive traffic analytics systems, facilitating safer and more efficient intersection management strategies.

Keywords: Simulation and Real-World Testing Methodologies, Software-Defined Vehicle for Intelligent Vehicles
Abstract: Future mobility systems and their components are increasingly defined by their software. The complexity of these cooperative intelligent transport systems (C-ITS) and the ever-changing requirements posed at the software require continual software updates. The dynamic nature of the system and the practically innumerable scenarios in which different software components work together necessitate efficient and automated development and testing procedures that use simulations as one core methodology. The availability of such simulation architectures is a common interest among many stakeholders, especially in the field of automated driving. That is why we propose CARLOS - an open, modular, and scalable simulation framework for the development and testing of software in C-ITS that leverages the rich CARLA and ROS ecosystems. We provide core building blocks for this framework and explain how it can be used and extended by the community. Its architecture builds upon modern microservice and DevOps principles such as containerization and continuous integration. In our paper, we motivate the architecture by describing important design principles and showcasing three major use cases - software prototyping, data-driven development, and automated testing. We make CARLOS and example implementations of the three use cases publicly available at github.com/ika-rwth-aachen/carlos.

Keywords: Simulation and Real-World Testing Methodologies, Vehicle Control and Motion Planning, Software-Defined Vehicle for Intelligent Vehicles
Abstract: This paper introduces a novel methodology for implementing a practical Decision Making module within an Autonomous Driving Stack, focusing on merge scenarios in urban environments. Our approach leverages Deep Reinforcement Learning and Curriculum Learning, structured into three stages: initial training in a lightweight simulator (SUMO), refinement in a high-fidelity simulation (CARLA) using a Digital Twin approach, and final validation in real-world scenarios with Parallel Execution. We propose a Partially Observable Markov Decision Process framework and employ the Trust Region Policy Optimization algorithm to train our agent. Our method significantly narrows the gap between simulated training and real-world application, offering a cost-effective and flexible solution for Autonomous Driving development. The paper details the experimental setup and outcomes in each phase, demonstrating the effectiveness of the proposed methodology.