Keywords:
Abstract: For over a century, automobile manufacturers have used the challenge of racing to better understand and improve upon vehicle design. Can the development of autonomous race cars advance the development of driver assistance systems and automated vehicles in a similar way? This talk explores our work with automated race cars at Stanford’s Dynamic Design Lab, identifying the basic challenges of racing and how control systems can handle these challenges. While automated vehicles hold significant advantages in computation and response time, head-to-head comparison with expert drivers shows humans can still teach the machine a few tricks. How, then, should we close this gap? Should we rely on our knowledge of physics to harness increasingly detailed models of the vehicle dynamics? Should we instead turn to AI to learn models directly from data and potentially eliminate the need to estimate physical parameters like friction? Or is there a path forward that can leverage the benefits of these two very different approaches? The talk at concludes with a look at some of our latest experiments, the current state of the art and open questions on the road to the future.

Keywords: Simulation and Real-World Testing Methodologies, Automated Vehicles, Human Factors for Intelligent Vehicles
Abstract: Enhancing simulation environments to replicate real-world driver behavior is essential for developing Autonomous Vehicle technology. While some previous works have studied the yielding reaction of lag vehicles in response to a merging car at highway on-ramps, the possible lane-change reaction of the lag car has not been widely studied. In this work we aim to improve the simulation of the highway merge scenario by including the lane-change reaction in addition to yielding behavior of main-lane lag vehicles, and we evaluate two different models for their ability to capture this reactive lane-change behavior. To tune the payoff functions of these models, a novel naturalistic dataset was collected on U.S. highways that provided several hours of merge-specific data to learn the lane change behavior of U.S. drivers. To make sure that we are collecting a representative set of different U.S. highway geometries in our data, we surveyed 50,000 U.S. highway on-ramps and then selected eight representative sites. The data were collected using roadside-mounted lidar sensors to capture various merge driver interactions. The models were demonstrated to be configurable for both keep-straight and lane-change behavior. The models were finally integrated into a high-fidelity simulation environment and confirmed to have adequate computation time efficiency for use in large-scale simulations to support autonomous vehicle development.

Keywords: Advanced Driver Assistance Systems (ADAS), Automated Vehicles, Sensor Signal Processing
Abstract: Although lidar sensors have emerged as a cornerstone sensing modality in autonomous driving, they face significant challenges in adverse weather conditions. A particular detrimental effect is spray — a phenomenon where water particles are whirled up by vehicles driving with high velocities on wet roads. Spray often causes clutter points in lidar data to be falsely classified as vehicles by downstream object detectors. In this work, a phenomenological spray simulation model, suitable as an augmentation method for object detection algorithms, is presented. Two distinct datasets featuring real-world spray scenarios are recorded and analyzed, with the first serving for calibrating the simulation model through extensive experiments that vary vehicle speeds, types, and pavement wetness levels. The second dataset functions as a spray test set to evaluate the effectiveness of the simulation model in the context of object detection. Employing the simulation model as an augmentation tool reveals an improvement of up to 17% in Average Precision for state-of-the-art object detection methods in real spray conditions.

Keywords: Human Factors for Intelligent Vehicles, Policy, Ethics, and Regulations
Abstract: The advent of autonomous vehicles (AVs) represents a transformative shift in transportation, promising to redefine mobility and alter our interaction with vehicles. Understanding public perceptions of AVs is crucial, as it influences the adoption and integration of this technology into society. This research conducts a comprehensive investigation into the factors that influence people’s attitudes toward AVs, examining the associated benefits and concerns, as well as the extent of trust placed in this emerging technology. The primary objective is to gain a deeper understanding of the elements that contribute to human-machine trust in the context of AVs. The findings reveal a consistent pattern in the propensity to trust AVs and concerns regarding performance failures, both at individual and societal levels. From a societal perspective, enhanced locomotion independence is the primary benefit of AV deployment, contributing to increased accessibility and reduced reliance on conventional transportation systems. At the individual level, increased free time emerges as the foremost advantage. These findings provide AV developers and policymakers the critical insight when deploying autonomous vehicle systems.

Keywords: Pedestrian Protection, Human Factors for Intelligent Vehicles, Automated Vehicles
Abstract: Understanding and predicting pedestrian crossing behavior is essential for enhancing automated driving and improving driving safety. Predicting gap selection behavior and the use of zebra crossing enables driving systems to proactively respond and prevent potential conflicts. This task is particularly challenging at unsignalized crossings due to the ambiguous right of way, requiring pedestrians to constantly interact with vehicles and other pedestrians. This study addresses these challenges by utilizing simulator data to investigate scenarios involving multiple vehicles and pedestrians. We propose and evaluate machine learning models to predict gap selection in non-zebra scenarios and zebra crossing usage in zebra scenarios. We investigate and discuss how pedestrians' behaviors are influenced by various factors, including pedestrian waiting time, walking speed, the number of unused gaps, the largest missed gap, and the influence of other pedestrians. This research contributes to the evolution of intelligent vehicles by providing predictive models and valuable insights into pedestrian crossing behavior.

Keywords: Advanced Driver Assistance Systems (ADAS), Automated Vehicles, Vehicle Control and Motion Planning
Abstract: This paper proposes collision-free optimal trajectory planning for autonomous vehicles in highway traffic, where vehicles need to deal with the interaction among each other. To address this issue, a novel optimal control framework is suggested, which couples the trajectory of surrounding vehicles with collision avoidance constraints. Additionally, we describe a trajectory optimization technique under state constraints, utilizing a planner based on Pontryagin's Minimum Principle, capable of numerically solving collision avoidance scenarios with surrounding vehicles. Simulation results demonstrate the effectiveness of the proposed approach regarding interaction-based motion planning for different scenarios.

Keywords: Automated Vehicles, Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS)
Abstract: Object detection and subsequent perception of the environment surrounding a vehicle play a very important role in autonomous driving applications. Existing perception algorithms do not generalize well since most algorithms are trained in well-structured driving environment datasets. To deploy self-driving cars on the road, they should have a reliable and robust perception system to handle all corner cases. This paper introduces a multimodal dataset, TIAND (TiHAN-IITH Autonomous Navigation Dataset), collected from structured and unstructured environments seen in and around the city of Hyderabad, India, as an aid to further research in the generalization of object detection algorithms. The sensor suite contains four cameras, six radars, one Lidar, and GPS and IMU. TIAND comprises 150 scenes, each spanning a duration ranging from 2 minutes to 4 minutes. Subsequently, we present the object detection model’s performance using camera, radar, and Lidar data. Additionally, we offer insights into projecting data from Lidar to camera and from radar to camera.

Keywords: Automated Vehicles, Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS)
Abstract: The main objective of few-shot semantic segmentation (FSSS) is to segment novel objects within query images by leveraging a limited set of support images. Being capable of segmenting the novel classes plays an essential role in the development of perception functions for automated vehicles. However, existing few-shot semantic segmentation work strives to improve the performance of the models on object-centric datasets. In our work, we evaluate the few-shot semantic segmentation on the more challenging driving scene understanding tasks. As a use case specific study, we give a systematic analysis of the disparity between commonly used FSSS datasets and driving datasets. Based on that, we proposed methodologies to integrate knowledge from the class hierarchy of the datasets, utilize more effective feature extraction, and choose more representative support images during inference. These approaches are evaluated extensively on the Cityscapes and Mapillary datasets to indicate their effectiveness. We point out the remaining challenges of training, evaluating, and employing FSSS models for complex road scenes in real practice.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS)
Abstract: This paper intends to address the problem of multi-vehicle tracking (MVT) by considering road constraints and interactions between vehicles. Most existing tracking algorithms assume that vehicles move independently in an open-field environment. However, due to traffic volume, road networks, and traffic rules, the movements of vehicles have to be affected by the surrounding environment (e.g., neighboring vehicles, physical road, speed limit) for the purposes of ensuring a safe distance and avoiding collisions. To address these problems, this paper proposes a novel MVT algorithm that incorporates the prior information from the surrounding environment into the estimation process by modeling it as the noisy control input of a dynamic model. In this paper, the kinematic state of vehicles is decomposed into longitudinal and lateral components, and the corresponding interacting multiple model (IMM) estimator is utilized to individually estimate longitudinal and lateral motions. Furthermore, by employing a two-stage IMM-based estimation scheme with different transition probability matrices (TPM) for lateral motions of the vehicle, the proposed algorithm significantly enhances tracking quality and continuity, especially in lane changing scenes. The effectiveness of the proposed algorithm is validated through numerical simulations and real-measured data.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS)
Abstract: In high-resolution radar, an observed target is segmented into isolated scattering centers (SCs) along the range domain, known as a range spread target (RST). Due to RST return energy is spilled into adjacent range cells, it is a formidable task for robust detection of RST. Moreover, both of range migration (RM) and Doppler frequency migration (DFM) induced by its high speed and maneuverability will also disturb the detection performance. To address these problems, this paper proposes a joint coherent integration (CI) and detection method called KT-SR for high speed and maneuvering RST detection. It can simultaneously perform the integration of inter-pulse and intra-pulse components of the target return. Firstly, the Keystone transform (KT) method is applied for range walk correction. Then, the DFM is mitigated by searching parameters of RST in the generalized likelihood ratio test (GLRT) detection model. Furthermore, the sparse representation (SR) is applied to estimate the target SCs to circumvent the drawback of target range spread, aiming to achieve incoherent integration among target SCs. Finally, the effectiveness of the proposed algorithm is validated through numerical simulations.

Keywords: Simulation and Real-World Testing Methodologies, Vehicular Active and Passive Safety, Advanced Driver Assistance Systems (ADAS)
Abstract: Abstract— Autonomous driving holds great potential for reducing traffic accidents. Despite many advancements in autonomous vehicle functions, challenges persist in assessing their safety. Scenario-Based Testing (SBT) has gained prominence for evaluating these vehicles. This review succinctly analyzes established and innovative strategies used to generate scenarios for SBT, outlining crucial challenges and current research focal points. Valuable insights are provided for researchers and engineers addressing concerns in scenario-based testing for autonomous driving systems.

Keywords: Vehicular Active and Passive Safety, Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Safe and efficient interactions with surrounding vehicles in multilane driving are essential for autonomous vehicles. However, achieving smooth and flexible responses to surrounding vehicles' lane changes remains a challenge due to the uncertainties in the behavior prediction progress. Deep learning-based methods were manifested powerful in modeling agents' motion uncertainties for making stochastic intention classification and trajectory prediction. Nevertheless, performance degradation are likely to occur when the black-box model makes multi-modal predictions in unseen situations. This paper proposes a novel AV planning framework that combines deep learning-based behavior prediction and optimization-based uncertainty-aware motion planning to resolve these challenges. We hierarchically address uncertainties inherent in both behavior patterns and model performance through an adaptive motion planning approach, using an improved constrained iterative linear quadratic regulator that handles non-convex constraints and non-Gaussian uncertainties while minimizing travel costs. Evaluations using INTERACTION and HighD datasets demonstrate the effectiveness of uncertainty-aware planning in enhancing AV safety performance.

Keywords: Vehicular Active and Passive Safety, Functional Safety in Intelligent Vehicles, Pedestrian Protection
Abstract: Vision-based object detection and tracking play a critical role in the field of autonomous driving as they enable the ego-car to identify potential dangers and detect objects that emerge suddenly. However, the current frequency of out-of-state vision-based object detection and tracking perception is limited to approximately 10-20Hz, which is insufficient for the autonomous driving system to promptly respond to events such as the sudden appearance of objects in close proximity or high-speed scenarios. In contrast, animals possess the ability to promptly avoid dangers despite lacking precise knowledge of an object’s exact distance, owing to their eyes’ high flicker fusion rates. Consequently, this paper introduces a novel vision-based hybrid object tracking technique. The proposed approach incorporates two distinct perception systems: a multi-camera perception system (MCP) and a high-frequency perception system (HFP). These systems operate independently to detect objects, with the MCP offering higher accuracy albeit lower update rates, and the HFP providing lower accuracy but higher update rates. Experimental results demonstrate that our method surpasses traditional perception approaches by achieving faster detection of real-time pop-out motorbikes across various ego-car speeds. Moreover, when both MCP and HFP information are available, our proposed method effectively combines the strengths of both systems to estimate the object's status.

Keywords: Vehicular Active and Passive Safety, Functional Safety in Intelligent Vehicles, Pedestrian Protection
Abstract: This paper introduces an innovative near-field monitoring system designed to enhance the active safety features of autonomous vehicles during low-speed driving. The proposed system has dedicated Light Detecting and Ranging (LiDAR) components. It can independently detect and respond to potential hazards, even without the high-level system's intervention, and apply the emergency brake promptly. Unlike traditional systems that depend on a comprehensive mapping of the entire surrounding environment, our approach focuses on immediate decision-making within the keep-out area (KOA) upon detecting obstacles. The monitoring system utilizes information from the planned driving route to delineate specific monitoring boundaries. This paper explains different monitoring strategies that are suitable for various driving conditions. These strategies incorporate the use of 2-dimensional layouts, including rectangles and parallelograms, to cover the driving path area and fulfill the monitoring requirements of the autonomous vehicle in diverse scenarios.

Keywords: Vehicular Active and Passive Safety, Sensor Signal Processing, Advanced Driver Assistance Systems (ADAS)
Abstract: In the context of spatial-temporal autonomous driving, the accurate and real-time trajectory prediction of the surrounding vehicle (SV) is crucial. This paper aims to design an efficient, accurate, and interpretable trajectory prediction approach capable of incrementally learning from feedback information in dynamic driving environments. To achieve this objective, we employ Sparse Gaussian Process Regression (SGPR), which enables large data set learning and efficient inference of future trajectories. This approach ensures accurate predictions while maintaining high computational efficiency. To further enhance the robustness of the prediction module, we propose the translation and rotation transformation strategy, which effectively simplifies the prediction problem. Additionally, we utilize an instant evaluation algorithm to assess the prediction performance and maintain a streaming data set for incremental learning and adaptation. In our experimental evaluation, we compare our proposed trajectory prediction approach with a series of existing methods. The results demonstrate that our work achieves superior prediction accuracy while requiring less inference time. It is noteworthy that, the proposed SGPR-based trajectory prediction approach with rotation equivalence is able to swiftly infer and incrementally learn from dynamic environments, which makes it a promising tool for enhancing safety and efficiency in autonomous driving systems.

Keywords: Vehicular Active and Passive Safety, Vehicle Control and Motion Planning, Advanced Driver Assistance Systems (ADAS)
Abstract: Distributed drive electric vehicles (DDEV) utilize differential torque to generate direct yaw moment (DYM) to improve vehicle safety and controllability, making the DYM control an active safety research hotspot. However, the generation of DYM relies on additional longitudinal tire forces, which may exceed the feasible tire force region, leading to vehicle drift. In addition, the DYM and traction force are highly coupled and can come into conflict under extreme handling operations. Thus, a novel concept of maneuverable stability region is proposed to describe the feasible safety boundaries of DYM and traction force. According to the different maneuverable stability region, four modes of vehicle operation are defined. Subsequently, the multi-modes judgment criterion is formulated using linear matrix inequality (LMI) to determine the boundaries of each mode and identify the current vehicle mode. Finally, a multi-mode torque distribution strategy (MTDS) is developed to meet the control requirements of the different modes, taking into account both energy saving and mechanical fatigue of the motors. Simulation and experimental results demonstrate that the multi-mode torque distribution strategy outperforms both the distributed torque distribution strategy and the single-mode torque distribution strategy. This strategy effectively mitigates the trade-off between mobility and stability, while maintaining vehicle safety, controllability, and energy saving at extreme handling limits.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, Vehicular Active and Passive Safety
Abstract: Previous studies on car-following controllers for autonomous vehicles (AVs) in mixed traffic have a narrow focus on maximizing the AV's utility, neglecting the utility of the entire traffic flow. This leads to self-centered AVs that may not be beneficial to surrounding vehicles. Thus, this study aims to develop a leading cruise controller for AVs that considers not only the AV's behaviors, but also the behaviors of both the lead human-driven vehicle (LHDV) and the following human-driven vehicle (FHDV). To achieve this, the study uses real-world data from the Waymo Open Dataset to approximate the behaviors of human-driven vehicles (HDVs) through an inverse reinforcement learning (IRL) approach. The study then proposes a preference-based soft actor-critic (PbSAC) algorithm to optimize the speed of AVs in a three-vehicle car- following scenario, while also considering safety, efficiency, and string stability for both AV and FHDV in the reward function. To further improve the control algorithm, the study develops a preference-adjusting module that adaptively updates the weights of the reward function based on expert evaluation. Experimental results show that the proposed algorithm can significantly improve safety, efficiency, and string stability for both AV and FHDV.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, Vehicular Active and Passive Safety
Abstract: There is a gap in risk assessment of trajectories between the trajectory information coming from a traffic motion prediction module and what is actually needed. Closing this gap necessitates advancements in prediction beyond current practices. Existing prediction models yield joint predictions of agents' future trajectories with uncertainty weights or marginal Gaussian probability density functions (PDFs) for single agents. Although, these methods achieve high accurate trajectory predictions, they only provide little or no information about the dependencies of interacting agents. Since traffic is a process of highly interdependent agents, whose actions directly influence their mutual behavior, the existing methods are not sufficient to reliably assess the risk of future trajectories. This paper addresses that gap by introducing a novel approach to motion prediction, focusing on predicting agent-pair covariance matrices in a ``scene-centric'' manner, which can then be used to model Gaussian joint PDFs for all agent-pairs in the scene. We propose a model capable of predicting agent-pair Gaussian joint PDFs, leveraging an enhanced awareness of interactions. Utilizing the prediction results of our model, this work forms the foundation for comprehensive risk assessment with statistically based methods for analyzing agents' relations by their predicted joint PDFs.

Keywords: Automated Vehicles, Vehicular Active and Passive Safety, Advanced Driver Assistance Systems (ADAS)
Abstract: In the rapidly evolving field of autonomous driving, reliable prediction is pivotal for vehicular safety. However, trajectory predictions often deviate from actual paths, particularly in complex and challenging environments, leading to significant errors. To address this issue, our study introduces a novel method for Dynamic Occupancy Set (DOS) prediction, it effectively combines advanced trajectory prediction networks with a DOS prediction module, overcoming the shortcomings of existing models. It provides a comprehensive and adaptable framework for predicting the potential occupancy sets of traffic participants. The innovative contributions of this study include the development of a novel DOS prediction model specifically tailored for navigating complex scenarios, the introduction of precise DOS mathematical representations, and the formulation of optimized loss functions that collectively advance the safety and efficiency of autonomous systems. Through rigorous validation, our method demonstrates marked improvements over traditional models, establishing a new benchmark for safety and operational efficiency in intelligent transportation systems.

Keywords: Automated Vehicles
Abstract: Trajectory prediction plays a pivotal role in the autonomous driving systems. Existing methods generally employ agent-centric or scene-centric approaches to represent driving scenarios. However, these methods introduce significant redundant computations or pose losses, resulting in suboptimal prediction efficiency and accuracy. To tackle these problems, a novel multi-target trajectory prediction model, named Hierarchical Scene TRansformer (HSTR), is introduced. The driving scene is decomposed into two independent components by HSTR: global and local. In the global part, global interaction information is established and shared among all predicted agents, thereby reducing redundant computations. In the local part, an individual reference frame is established for each vehicle to eliminate the impact of pose variations and extract temporal features. Moreover, an adaptive anchor point generation method is proposed to address the challenge of capturing future modalities for vehicles. This method dynamically generates corresponding anchor points based on different driving scenarios to guide the prediction of trajectories across various modalities. The model performance is verified on the argoverse1 and argoverse2 datasets, and the experimental results demonstrate that competitive performance is achieved by HSTR in terms of efficiency and precision compared to the state-of-the-art methods.

Keywords: Automated Vehicles, Cooperative Vehicles, Vehicle Control and Motion Planning
Abstract: Cooperative platooning is a promising method for improving energy efficiency and traffic throughput on interstates. Ensuring collision avoidance is particularly difficult in platooning due to the small desired inter-vehicle spacing. We propose a safety protocol that can be applied to arbitrary controllers in platooning to prevent collisions in a provably correct manner while still realizing a small distance to the pre-ceding vehicle. Our protocol intervenes as rarely and smoothly as possible, and its safety is ensured even if communication fails. In addition, we propose a safety protocol for consensus techniques where the vehicles of the platoon successively agree on a common braking limit. Our safety protocols are evaluated on various scenarios using the CommonRoad benchmark suite.

Keywords: Automated Vehicles, End-To-End (E2E) Autonomous Driving, Vehicle Control and Motion Planning
Abstract: Traditional vehicle trajectory prediction models widely exist the generalization problem towards unknown scenarios. In this paper, we address the generalization via the following ways. A conditional variational autoencoder based on invariance penalty is adopted to predict trajectory endpoints. In addition, we propose a domain division method to enhance the performance of the invariance principle and design the maneuver-related subtasks to reconstruct the consistent features of trajectories. The experiment is carried out on the INTERACTION dataset, which is well employed in the study of trajectory domains. Compared to the SOTA models, the mADE at 3s decreases from 1.16 to 0.53. The ablation study demonstrates the effectiveness of each module in our model. The results show that our method achieves excellence performance when generalized to unknown domains.

Keywords: Automated Vehicles, Functional Safety in Intelligent Vehicles, Vehicle Control and Motion Planning
Abstract: Predicting the future motion of surrounding vehicles is a crucial enabler for safe autonomous driving. The field of motion prediction has seen large progress recently with State-of-the-Art (SotA) models achieving impressive results on large-scale public benchmarks. However, recent work revealed that learning-based methods are prone to predict off-road trajectories in challenging scenarios. These can be created by perturbing existing scenarios with additional turns in front of the target vehicle while the motion history is left unchanged. We argue that this indicates that SotA models do not consider the map information sufficiently and demonstrate how this can be solved by representing the model inputs and outputs in a Frenet frame defined by lane centreline sequences. To this end, we present a general wrapper that leverages a Frenet representation of the scene, and that can be applied to SotA models without changing their architecture. We demonstrate the effectiveness of this approach in a comprehensive benchmark comprising two SotA motion prediction models. Our experiments show that this reduces the off-road rate in challenging scenarios by more than 90%, without sacrificing average performance. Code and supplementary material are available under: https://mh0797.github.io/stayontrack/.

Keywords: Automated Vehicles, Simulation and Real-World Testing Methodologies, Vehicle Control and Motion Planning
Abstract: Current validation methods often rely on recorded data and basic functional checks, which may not be sufficient to encompass the scenarios an autonomous vehicle might encounter. In addition, there is a growing need for complex scenarios with changing vehicle interactions for comprehensive validation. This work introduces a novel synchronous multi-agent simulation framework for autonomous vehicles in interactive scenarios. Our approach creates an interactive scenario and incorporates publicly available edge-case scenarios wherein simulated vehicles are replaced by agents navigating to predefined destinations. We provide a platform that enables the integration of different autonomous driving planning methodologies and includes a set of evaluation metrics to assess autonomous driving behavior. Our study explores different planning setups and adjusts simulation complexity to test the framework’s adaptability and performance. Results highlight the critical role of simulating vehicle interactions to enhance autonomous driving systems. Our setup offers unique insights for developing advanced algorithms for complex driving tasks to accelerate future investigations and developments in this field.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Existing approaches to trajectory planning for autonomous racing employ sampling-based methods, generating numerous jerk-optimal trajectories and selecting the most favorable feasible trajectory based on a cost function penalizing deviations from an offline-calculated racing line. While successful on oval tracks, these methods face limitations on complex circuits due to the simplistic geometry of jerk-optimal edges failing to capture the complexity of the racing line. Additionally, they only consider two-dimensional tracks, potentially neglecting or surpassing the actual dynamic potential. In this paper, we present a sampling-based local trajectory planning approach for autonomous racing that can maintain the lap time of the racing line even on complex race tracks and consider the race track's three-dimensional effects. In simulative experiments, we demonstrate that our approach achieves lower lap times and improved utilization of dynamic limits compared to existing approaches. We also investigate the impact of online racing line generation, in which the time-optimal solution is planned from the current vehicle state for a limited spatial horizon, in contrast to a closed racing line calculated offline. We show that combining the sampling-based planner with the online racing line generation can significantly reduce lap times in multi-vehicle scenarios.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Our work introduces a module for assessing the trajectory safety of autonomous vehicles in dynamic environments marked by high uncertainty. We focus on occluded areas and occluded traffic participants with limited information about surrounding obstacles. To address this problem, we propose a software module that handles blind spots (BS) created by static and dynamic obstacles in urban environments. We identify potential occluded traffic participants, predict their movement, and assess the ego vehicle’s trajectory using various criticality metrics. The method offers a straightforward and modular integration into motion planning algorithms. We present critical real-world scenarios to evaluate our module and apply our approach to a publicly available trajectory planning algorithm. Our results demonstrate that safe yet efficient driving with occluded road users can be achieved by incorporating safety assessments into the planning process.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Automated vehicles require carefully designed cost functions, which are challenging to specify due to the complexity of the behavior they need to cover. Inverse reinforcement learning is a principled methodology for deriving cost functions, but it requires high-quality expert demonstrations, which are expensive to obtain. Recently, scenario-based testing has emerged as a promising approach for validation of driving behavior. In this paper, we introduce a novel methodology that circumvents the need for costly expert driving demonstrations by harnessing scenario-based testing. Our Test-Driven Inverse Reinforcement Learning approach leverages Bayesian inference, utilizing the outcomes of scenario tests as observations to infer cost functions. We rigorously evaluate our method on simulated and real-world scenarios and demonstrate its ability to learn cost functions that successfully pass the respective scenario tests. We also show that the learned cost function generalizes well by also passing scenario tests from an unseen validation set and illustrate that few scenario tests are sufficient to learn meaningful cost functions. This innovative framework not only streamlines the cost function specification process but also offers a cost-effective and practical solution for advancing automated driving systems.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Motion prediction is a key factor towards the full deployment of autonomous vehicles. It is fundamental in order to ensure safety while navigating through highly interactive and complex scenarios. Lack of visibility due to an obstructed view or sensor range poses a great safety issue for autonomous vehicles. The inclusion of occlusion in interaction-aware approaches is not very well explored in the literature. In this work, the MultIAMP framework, which produces multimodal probabilistic outputs from the integration of a Dynamic Bayesian Network and Markov chains, is extended to tackle occlusions. The framework is evaluated with a state-of-the-art motion planner in two realistic use cases.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Real-time safety-critical control is essential for high-level autonomous driving. Existing methods usually formulate safety-critical control as a constrained optimal control problem (COCP), and suffer from high computational complexity of the underlying iterative optimization processes. To address the issue of complexity, this paper presents an explicit safety-critical control method called the Control Safety Function (CSF) approach, which can replace online optimization with an analytical control law, dramatically enhancing real-time control capabilities. The CSF is formulated as the weighted sum of Control Lyapunov Function (CLF) and Control Barrier Functions (CBFs), with the value of CSF increasing to infinity as the state approaches the boundary of safe sets. The explicit control law is then derived from the gradient of CSF and system dynamics. Different from existing explicit controllers that can only apply to systems of relative degree one, the CSF method provides an approach to enforce safety constraints to systems with high relative degree, making CSF especially suitable for autonomous driving. The CSF approach is evaluated in a vehicle path-tracking scenario with multiple obstacles, accompanied by a comparative analysis against the Model Predictive Control (MPC) method. Simulation results indicate that CSF achieves control accuracy comparable to MPC, with significant reduction in computation time - approximately 3.23 ms per step, which is about 94.0% faster. These results suggest that CSF is a promising approach for real-time safety-critical control of high-level autonomous driving.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles, Automotive Cyber Physical Systems
Abstract: Map-based methods for autonomous racing estimate the vehicle's location, which is used to follow a high-level plan. While map-based methods demonstrate high-performance results, they are limited by requiring a map of the environment. In contrast, mapless methods can operate in unmapped contexts since they directly process raw sensor data (often LiDAR) to calculate commands, but suffer from poor performance. In response, we propose the local map framework that uses easily extractable, low-level features to build local maps of the visible region that form the input to optimisation-based trajectory planners. Our local map generation extracts the visible racetrack boundaries and calculates a centre line and track widths used for planning. We evaluate our method for simulated F1Tenth autonomous racing using a trajectory optimisation and tracking strategy and a model predictive controller. Our method achieves lap times that are 8.8% faster than the Follow-The-Gap method and 3.22% faster than end-to-end neural networks due to the optimisation resulting in a faster speed profile. The local map planner is 3.28% slower than global methods that have access to an entire map of the track that can be used for planning. Critically, our approach enables high-speed autonomous racing on unmapped tracks, achieving performance similar to global methods without requiring a track map.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, Advanced Driver Assistance Systems (ADAS)
Abstract: Accurately predicting human driving behavior, particularly in highly interactive traffic scenarios, poses a significant challenge. In this work, we investigate the application of graph-based observations to Adversarial Imitation Learning (AIL) methods. Unlike conventional feature-based observations, this allows us to flexibly account for different road structures as well as a varying number of surrounding vehicles interacting with each other. We assess the method in a complex roundabout scenario from the INTERACTION dataset, employing several state-of-the-art AIL methods. The results indicate that our proposed approach successfully yields realistic driver models, applicable for accurate predictions of human driving behavior.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, Battery Management Systems and State-of-Charge (SoC) Estimation
Abstract: In this paper, we are discussing and presenting the concept of automated driving in production facilities. All activities are carried out within the publicly funded project E-Self with the project partners Ford, Kopernikus and the Institute of Automotive Engineering of the Technical University Braunschweig. The central aim of the project is the development and integration of automated driving at the end-of-line in the production at Ford Motor Company's manufacturing plant in Cologne. The driving function thereby is mostly based upon automated valet driving with an infrastructure-based perception and action planning. Especially for electric vehicles the state of charge of the battery is critical, since energy is needed for all testing and driving operations at end-of-line. Within this paper, we want to introduce the general concept of automated driving in production, the implementation possibilities and implementation in the scope of the project as well as discuss further potentials of this technology.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, End-To-End (E2E) Autonomous Driving
Abstract: In urban environments characterized by various dynamic variables such as moving vehicles, pedestrians, and bicycles, path planning for collision avoidance is essential for autonomous driving. Enhancing the stability of the route can be achieved by incorporating not only information about recognized surrounding objects but also dynamic information during the path planning process. Therefore, this study proposes a convolutional neural network-based local path planning technique for autonomous vehicles in urban environments using a dynamic occupancy grid map (DOGM). This approach ensures precision in path generation by considering the occupancy and speed of various objects. During the learning process, we implemented inverse reinforcement learning by combining trajectory information driven by expert intentions with environmental information obtained from DOGM through a combination of convolution layers. This demonstrates the feasibility of designing stable paths with low collision rates in urban areas. Particularly noteworthy is the superior performance achieved when DOGM is used as input data for deep learning, surpassing conventional algorithms.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, Human Factors for Intelligent Vehicles
Abstract: This paper presents a game-theoretic strategy for racing, where the autonomous ego agent seeks to block a racing opponent that aims to overtake the ego agent. After a library of trajectory candidates and an associated reward matrix are constructed, the optimal trajectory in terms of maximizing the cumulative reward over the planning horizon is determined based on the level-K reasoning framework. In particular, the level of the opponent is estimated online according to its behavior over a past window and is then used to determine the trajectory for the ego agent. Taking into account that the opponent may change its level and strategy during the decision process of the ego agent, we introduce a trajectory mixing strategy that blends the level-K optimal trajectory with a fail-safe trajectory. The overall algorithm was tested and evaluated in various simulated racing scenarios, which also includes human-in-the-loop experiments. Comparative analysis against the conventional level-K framework demonstrates the superiority of our proposed approach in terms of overtake-blocking success rates.

Keywords: Sensor Fusion for Localization, Automated Vehicles, SLAM (Simultaneous Localization and Mapping)
Abstract: The sideslip angle, crucial for vehicle safety and stability, is determined using both longitudinal and lateral velocities. However, measuring the lateral component often necessitates costly sensors, leading to its common estimation, a topic thoroughly explored in existing literature. This paper introduces LOP-UKF, a novel method for estimating vehicle lateral velocity by integrating Lidar Odometry with the Pacejka tire model predictions, resulting in a robust estimation via an Unscendent Kalman Filter (UKF). This combination represents a distinct alternative to more traditional methodologies, resulting in a reliable solution also in edge cases. We present experimental results obtained using the Dallara AV-21 across diverse circuits and track conditions, demonstrating the effectiveness of our method.

Keywords: Sensor Fusion for Localization, Integration of HD map and Onboard Sensors, Advanced Driver Assistance Systems (ADAS)
Abstract: Simultaneous localization and mapping is a prevalent method for localization in automated parking applications. However, it requires to access an area for exploring purposes before the automated parking function can be completely applied. As this is inconvenient for parking functions, where often unknown areas are entered, our approach proposes a radar-based localization method primarily for applications inside of buildings which have a ground plan available. This ground plan contains the walls, pillars and parking lots of the building in a two-dimensional, bird’s eye view perspective. Based on the ground plan, a synthesized point cloud is generated to be matched with the filtered radar point cloud via a Normal Distributions Transform algorithm. The measurements generated hereby are fused with odometry measurements by a factor graph. This architecture is capable of processing independent, asynchronous incoming data in parallel and can easily be extended e.g. by camera data. We outline our pipeline and show in experiments that it serves as a solid basis which competes with other state-of-the-art localization algorithms. Some drawbacks, e.g. the noisiness of the radar data in slowspeed or standstill situations, are discussed. Future work should incorporate camera data to further improve the robustness of this approach.

Keywords: Sensor Fusion for Localization, Sensor Signal Processing, Automated Vehicles
Abstract: The unloading of bulk carriers is often accompanied by high dynamic changes in the cargo hold, which in turn brings difficulties in perception and localization within the hold for the clearing operation. To address the high risk and low efficiency of manual observations, this paper introduces a robot-grabber collaborative localization method for highly dynamic clearing scenarios of the bulk carrier. Firstly, a collaborative localization system is constructed to enable unobstructed perception and autonomous localization of the cargo hold clearing robot and the grabber. The point cloud intensity and distance constraints are utilized to accurately determine the calibration parameters in this system, achieving unified localization of these collaborative equipment. Additionally, a multi-objective factor graph optimization based cooperative localization method is proposed to address the dynamic interference of bulk material and grabber in the cargo hold, taking into account the multi-end observation of the robot perception and system calibration, thereby obtaining robust and high-precision collaborative localization results. Finally, experiments are conducted in real port scenarios of bulk carrier clearing to validate the effectiveness of the proposed algorithm.

Keywords: Sensor Fusion for Localization, SLAM (Simultaneous Localization and Mapping)
Abstract: Accurate localization ability is a crucial component for autonomous robots. Assuming that there already exists LiDAR 3D points maps, it is cost-effective to localize the robot only with onboard camera compared to LiDAR. However, matching 2D visual information with 3D point cloud maps presents huge challenges due to different modalities, dimensions, noise and occlusion issues. To overcome it, we propose an end-to-end neural network-based solution, which estimates the 6-DoF pose of the camera in an existing LiDAR map with centi-meter accuracy. Given a query image, a pre-acquired point cloud and an initial pose, the cross-modality network will output a precise pose. By projecting the 3D point cloud onto the image plane, a depth image is acquired as seen from the initial pose. Subsequently, a cross-modality flow network establishes the correspondences of 2D pixels and projected points. Importantly, we leverage a robust probabilistic Perspective-n-Point (PnP) module, which are capable of fine-tuning 2D pairs and learning the pairs weight in an end-to-end manner. We conduct a comprehensive evaluation of our proposed algorithm using KITTI datasets. Furthermore, deploying the algorithm on the real-world parking lot scenario validates its strong practicality of the proposed algorithm. We highlight that this research offers a cost-effective and highly accurate solution that can be readily deployed in low-cost commercial vehicles.

Keywords: SLAM (Simultaneous Localization and Mapping), Automated Vehicles, Sensor Signal Processing
Abstract: As a fundamental task for intelligent robots, visual SLAM has made significant progress in recent years. How- ever, robust SLAM in weakly textured environments remains a challenging task. In this paper, we present a novel visual SLAM system named RWT-SLAM to address this problem. Unlike existing methods that use detector-based deep networks for interest point detection, we propose extracting distinctive features from a detector-free based network, namely LoFTR, to avoid the difficulty of manual annotations of feature points in weakly textured images. We generate multi-level feature vectors from LoFTR to form dense descriptors for each pixel in the input image. A keypoint localization component is then proposed to measure the saliency of the descriptors and select the distinctive pixels as keypoints. We integrate this new keypoint into the popular ORB-SLAM framework and compare it with the state-of-the-art methods. Extensive experiments on popular TUM RGB-D, OpenLORIS-Scene, as well as our own dataset are carried out. The results demonstrate the superior performance of our method in weakly textured environments.

Keywords: SLAM (Simultaneous Localization and Mapping), Cooperative Vehicles
Abstract: In this paper, we propose a centralized multi-robot stereo SLAM method based on a direct approach, with the aim of achieving Fast, Low-bandwidth, and Semi-dense mapping for collaborative applications, namely CMD-SLAM. In CMD-SLAM, each agent independently runs stereo-DSO as a front-end visual odometry(VO) and shares information with the central server via the TCP/IP protocol. To lower the bandwidth, we have designed a new communication strategy. Additionally, we employ a LiDAR descriptor-based place recognition method in the server's back-end. This method fully exploits the information from the 3D point cloud structure generated during the direct method odometry, achieving efficient loop closure detection. In optimization, to achieve a balance between accuracy and speed, we adopt a Sim(3) pose graph optimization (PGO) and employ some trick to reduce the time consumption. Finally, we evaluate CMD-SLAM on publicly available datasets and large-scale outdoor environments, comparing it with state-of-the-art approaches to demonstrate its effectiveness.

Keywords: SLAM (Simultaneous Localization and Mapping)
Abstract: In open-pit mines, long-term map management is crucial for route-planning of autonomous mining trucks during haulage. However, degraded scenarios with low-texture mountains may cause mapping discontinuity after rasterization. Moreover, the high-frequency environmental changes may lead to inter-session matching failures, resulting in significant mapping inaccuracies. To address the above issues, we propose a pose reference and information density based long-term mapping method for degraded scenarios. We introduce adaptive information compensation to ensure the even distribution of map information after the mapping process. Sequentially, we utilize poses corresponding to central and query sessions along with the information density model to achieve inter-session matching. Outdated information pruning is then employed to update the map. Furthermore, we evaluate the map quality assessing overlap and utilization rates under field tests, and the results suggest that our proposed methods yield over an 8% improvement in overlap rate compared to state-of-the-art methods.

Keywords: Integration of HD map and Onboard Sensors, SLAM (Simultaneous Localization and Mapping)
Abstract: In the realm of autonomous vehicles, shifting scenarios can lead to localization failures, due to challenges like GPS signal loss and structural degradation. To address these issues, our paper introduces a multi-sensor and high-definition (HD) map-based framework for resilient vehicle localization in challenging environments. The framework integrates GPS, LiDAR, and ultra-wideband (UWB) to provide GPS localization results, UWB range measurements, LiDAR odometry results, and range measurements from plane detection and HD maps. These observations are then transformed into diverse constraints of varying scales, which are further represented as factors. Enhanced by a self-calibration module, these factors are seamlessly fused within a factor graph framework to achieve robust vehicle localization. Specifically, in environments where GPS is available and structured, the proposed method fuses results from GPS positioning and LiDAR simultaneous localization and mapping (SLAM). In GPS-unavailable and structured scenarios, fusion involves LiDAR SLAM results and UWB ranging information. For GPS-unavailable and degenerated-structured environments, the fusion incorporates range measurements from LiDAR plane detection and UWB anchors. We validate the effectiveness of the proposed method through practical applications in diverse environments covering a total driving distance of approximately 20.54 km, including underground parking, tunnels, roads under urban viaducts, and typical urban roads.

Keywords: Integration of HD map and Onboard Sensors, Sensor Fusion for Localization, Perception Including Object Event Detection and Response (OEDR)
Abstract: Monocular online map segmentation is of great significance to mapless autonomous driving, and the core step is the View Transformation Module (VTM), which is used to transfer feature from the image perspective to the Bird-Eye-View (BEV). Most existing methods directly draw from the field of 3D object perception, either projecting 2D features into 3D space based on depth estimation, or projecting 3D coordinates into 2D images to query corresponding features, while ignoring the geometry and semantics from the ground surface. In this paper, we proposed a ground aware forward-backward view transformation module. The forward projection is used to generate the initial sparse BEV features and the geometric and semantic prior information of the ground surface. The backward module refines the BEV features based on the geometric and semantic priors, thereby improving the accuracy of map segmentation. In addition, the data partitioning of most previous related works has the problem of data leakage, so we repartitioned and experimented on the nuScense data set to conduct a fair evaluation. Experimental results demonstrate that our method achieves the highest accuracy on the test set. Code will be released at https://github.com/Brickzhuantou/MonoBEVseg.

Keywords: Automated Vehicles, Sensor Fusion for Localization, Integration of HD map and Onboard Sensors
Abstract: Although pre-built LiDAR maps have been widely used for location estimation, the high maintenance cost of pre-built maps hinders their expansion in practical application. Lightweight maps with global consistency and extensive coverage, such as OpenStreetMap (OSM), provide a low-cost potential alternative for prior reference. However, OSM exhibits significant differences in precision and structure with on-board perception, leading to incompatibility and poor performance with the existing descriptor matching method. Thus, this paper proposes a planar descriptor based on building boundary directions that is tailored for OSM-perception unification. Based on the proposed descriptor, OSM can be transformed into a prior map with the potential to replace pre-built maps, suitable for location estimation in urban scenes. To evaluate the effectiveness of replacing pre-built LiDAR maps, we conducted experiments in the urban environment sequences of KITTI. The results indicate that the proposed method achieves the best performance on location estimation accuracy utilizing OSM compared to other feature descriptors.

Keywords: Sensor Signal Processing, Automated Vehicles, Sensor Fusion for Localization
Abstract: In recent years, the rise in both natural and man-made disasters, along with armed conflicts and terrorist threats, has elevated the demand for Search and Rescue (SAR) missions worldwide. This paper underscores the critical necessity to enhance SAR capacity, safety, and capabilities, with a primary goal of reducing response times through the integration of robots into SAR operations. The examination of research on robotized SAR highlights deficiencies in both software and hardware, particularly focusing on perception systems for robotized SAR platforms.

Keywords: Sensor Signal Processing, Automated Vehicles, Sensor Fusion for Localization
Abstract: Integrating information from multiple modalities enhances the robustness of scene perception systems in autonomous vehicles, providing a more comprehensive and reliable sensory framework. However, the modality incompleteness in multi-modal segmentation remains under-explored. In this work, we establish a task called Modality-Incomplete Scene Segmentation (MISS), which encompasses both system-level modality absence and sensor-level modality errors. To avoid the predominant modality reliance in multi-modal fusion, we introduce a Missing-aware Modal Switch (MMS) strategy to proactively manage missing modalities during training. Utilizing bit-level batch-wise sampling enhances the model’s performance in both complete and incomplete testing scenarios. Furthermore, we introduce the Fourier Prompt Tuning (FPT) method to incorporate representative spectral information into a limited number of learnable prompts that maintain robustness against all MISS scenarios. Akin to fine-tuning effects but with fewer tunable parameters (1.1%). Extensive experiments prove the efficacy of our proposed approach, showcasing an improvement of 5.84% mIoU over the prior state-of-the-art parameter-efficient methods in modality missing. The source code will be publicly available.

Keywords: Automated Vehicles
Abstract: Autonomous vehicles rely on accurate trajectory prediction to inform decision-making processes related to navigation and collision avoidance. However, current trajectory prediction models show signs of overfitting, which may lead to unsafe or suboptimal behavior. To address these challenges, this paper presents a comprehensive framework that categorizes and assesses the definitions and strategies used in the literature on evaluating and improving the robustness of trajectory prediction models. This involves a detailed exploration of various approaches, including data slicing methods, perturbation techniques, model architecture changes, and post-training adjustments. In the literature, we see many promising methods for increasing robustness, which are necessary for safe and reliable autonomous driving.

Keywords: Sensor Signal Processing, Automated Vehicles
Abstract: Tracking by detection(TBD) method has achieved great improvements for its high efficiency, extensibility and portability, but it still struggles on computational efficiency. Many recently proposed methods improve performance by integrating appearance similarity and simply extract appearance feature for all the targets. This results redundant calculations as some targets can already be easily tracked without feature extraction, such as targets walking alone. In this work, we tackle the efficiency problem from a new perspective and propose AETrack, An Efficient approach for online multi-object Tracking(MOT), which integrates three association metrics through a novel cascaded matching strategy. Instead of simply computing all the association metrics for all tracklets, our matching strategy dynamically chooses and fuses the metrics for each tracklet considering both effectiveness and efficiency. Inference speed is boosted greatly and accuracy is still competitive. AETrack achieves 64.7 HOTA on MOT17 test set while running at 58 FPS and 62.8 HOTA on MOT20 at 52 FPS. Our code and models will be public soon.

Keywords: Automated Vehicles, Future Mobility and Smart City, Integration of Infrastructure and Intelligent Vehicles
Abstract: Automated docking technologies of marine boats have been enlightened by an increasing number of literature. This paper contributes to the literature by proposing a mathematical framework that automates ``trailer loading'' in the presence of wind disturbances, which is unexplored despite its importance for boat owners. The comprehensive pipeline of localization, system identification, and trajectory optimization is structured, along with techniques to enhance the reliability of performance. The performance of the proposed method was demonstrated with a commercial pontoon boat in Michigan, in 2023, securing an 80% success rate with the presence of perception errors and wind disturbance. This result indicates the strong potential of the proposed pipeline, effectively accommodating the wind effect.

Keywords: Future Mobility and Smart City
Abstract: This study analyzes user engagement with a micromobility sharing system tailored for students and staff offering a monthly mobility allowance. The system comprises a variety of vehicles, including city bikes, e-bikes, e-cargo bikes, e-mopeds, and e-scooters. Using K-Means clustering based on monthly trip frequency, we categorized users into five segments — non-users, one-time users, infrequent users, frequent users, and super users. Our results indicate a high inactivity rate among registered users, with low conversion from initial to regular use. The trend shows that many users do not progress to more frequent usage levels, with the majority being one-time or infrequent users. While infrequent users tend to favor e-bikes and e-scooters, the most active users—super users—are more likely to utilize a wider variety of vehicles. Key demographic data and trip patterns were significant in determining user engagement levels and predicting potential churn. These findings emphasize the importance of understanding user behavior to effectively tailor service offerings. The insights gained from this study can inform service enhancements aimed at stimulating user activity and reducing churn, providing actionable guidance for micromobility service providers to improve customer retention and service utilization.

Keywords: Future Mobility and Smart City, Integration of Infrastructure and Intelligent Vehicles, Automated Vehicles
Abstract: The challenges in implementing SAE Level 4/5 automated vehicles are manifold, with intersection navigation being a pervasive one. We analyze a novel road topology invented by a co-author of this paper, Xiayong Hu. The topology eliminates the need for traditional traffic control and cross-traffic at intersections, potentially improving the safety of autonomous driving systems. The topology, herein called the Zonal Road Topology, consists of unidirectional loops of road with traffic flowing either clockwise or counter-clockwise. Adjacent loops are directionally aligned with one another, allowing vehicles to transfer from one loop to another through a simple lane change. To evaluate the Zonal Road Topology, a one km² pilot-track near Changshu, China is currently being set aside for testing. In parallel, traffic simulations are being performed. To this end, we conduct a simulation-based comparison between the Zonal Road Topology and a traditional road topology for a generic Electric Vehicle (EV) using the Simulation for Urban MObility (SUMO) platform and MATLAB/Simulink. We analyze the topologies in terms of their travel efficiency, safety, energy usage, and capacity. Drive time, number of halts, progress rate, and other metrics are analyzed across varied traffic levels to investigate the advantages and disadvantages of the Zonal Road Topology. Our results indicate that vehicles on the Zonal Road Topology have a lower, more consistent drive time, make more progress, and halt less frequently, while using less energy on average. The Zonal Road Topology also has the capacity to support a greater amount of vehicles. These results become more prominent at higher traffic densities.

Keywords: Integration of HD map and Onboard Sensors
Abstract: Online High-definition map (HD-map) construction based on vehicle sensors has garnered widespread attention recently. While state-of-the-art methods achieve remarkable accuracy, most of them overlook the importance of inference speed and the inherent linear priors of map elements. Concretely, slow inference speed impacts the safety of autonomous vehicles, making it challenging for applications. Additionally, the absence of linear priors in map element predictions results in distorted or blurry outcomes. To address these issues, we propose E-MLP, an effortless online HD-map construction method that relies solely on camera sensors and incorporates the linear priors of map elements. Specifically, we first introduce a novel Principal Feature Analysis (PFA) module, designed to efficiently reduce the time cost of view transformation. Then, two thoughtfully crafted loss functions are introduced to incorporate the natural linear priors of map elements as constraints in the map construction process. Extensive experiments conducted on the nuScenes dataset revealed that, compared to the baseline method, our approach achieved a remarkable 34.9% increase in inference speed with virtually no loss in accuracy.

Keywords: Integration of HD map and Onboard Sensors, Automated Vehicles, Integration of Infrastructure and Intelligent Vehicles
Abstract: In high-definition maps (HD maps), the road lane centerline and lane topology graph play essential roles in navigation, planning, and decision-making. Existing research primarily focuses on extracting physical infrastructure, such as lane boundaries, neglecting lane centerline and topology reasoning, leaving it as an unsolved problem. To tackle these challenges, we introduce an automatic lane topology extraction method for HD maps, termed LaneDAG, which extracts vectorized centerlines and their topology from prebuilt lane lines and road boundaries in HD maps. It formulates centerline extraction as a set prediction problem and lane topology prediction as a directed acyclic graph (DAG) construction problem. A novel mechanism that fusing geometric and attention-based features in the DAG is proposed to model the topological relationship between centerlines. Experiments conducted on the Argoverse 2 dataset demonstrate the proposed method's superior performance compared to existing methods, showcasing its capability to extract lane centerlines and topology in HD maps automatically.

Keywords: Integration of Infrastructure and Intelligent Vehicles, Automated Vehicles, Verification and Validation Techniques
Abstract: Advanced Driver Assistance Systems (ADAS) play a pivotal role in modern road vehicles, enhancing safety. However, persistent challenges in managing ADAS systems and automated vehicles during dynamic traffic scenarios hinder the widespread adoption of ADAS and Automated Driving (AD) systems. Recognizing the susceptibility of perception sensors to weather and road hazards, along with their typical operational limitations, V2X communication becomes critically important to achieve higher levels of autonomy and robustness. In the EU-funded project ESRIUM, safety improvement is attained by developing a digital map capable of accurately identifying road damage and offering real-time recommendations for connected vehicles. In this paper, we report the implementation and road testing results for the infrastructure-assisted automated driving system developed within the ESRIUM project. These tests were conducted on real-life public roads and under typical driving conditions on the Austrian highway A2, showcasing the effectiveness of infrastructure-assisted AD vehicles in diverse traffic scenarios. These findings represent a significant advancement in validating automated vehicles on operational highways, emphasizing the vital role of infrastructure and V2X communication in enhancing ADAS/AD road safety and efficiency.

Keywords: Integration of Infrastructure and Intelligent Vehicles, Automotive Cyber Physical Systems, Automated Vehicles
Abstract: Intelligent intersection management has been a research hotspot in recent years within the domain of intelligent transportation systems (ITS). Existing studies exhibit a deficiency in explicitly addressing the trajectories of human driven vehicles (HDVs) under mixed traffic environment, and they fall short of harnessing the full potential of connected and automated vehicles (CAVs) and Vehicle-to-Infrastructure (V2I) technologies for the joint optimization of signal timing and phasing and vehicle trajectories. In this study, we propose a cooperative intersection management method designed for mixed traffic environment. Our approach jointly optimizes the green time durations, phase order of traffic signals, and vehicle trajectories. To account for the impact of traffic signals on HDVs, we incorporate it as a virtual leading vehicle within the optimal velocity model (OVM). The comprehensive model is formulated as a nonlinear programming problem and then converted into a mixed integer programming problem using the big-M method. We conduct simulations of the proposed method in various scenarios at different MPRs. The results reveal a significant reduction in average travel time compared to the actuated signal control, highlighting the enhanced efficiency of the intersection achieved through our proposed method.

Keywords: Integration of Infrastructure and Intelligent Vehicles, Cooperative Vehicles, Simulation and Real-World Testing Methodologies
Abstract: Floating Car Observers (FCOs) are an innovative method to collect traffic data by deploying sensor-equipped vehicles to detect and locate other vehicles. We demonstrate that even a small penetration rate of FCOs can identify a significant amount of vehicles at a given intersection. This is achieved through the emulation of detection within a microscopic traffic simulation. Additionally, leveraging data from previous moments can enhance the detection of vehicles in the current frame. Our findings indicate that, with a 20-second observation window, it is possible to recover up to 20% of vehicles that are not visible by FCOs in the current timestep. To exploit this, we developed a data-driven strategy, utilizing sequences of Bird's Eye View (BEV) representations of detected vehicles and deep learning models. This approach aims to bring currently undetected vehicles into view in the present moment, enhancing the currently detected vehicles. Results of different spatiotemporal architectures show that up to 41% of the vehicles can be recovered into the current timestep at their current position. This enhancement enriches the information initially available by the FCO, allowing an improved estimation of traffic states and metrics (e.g. density and queue length) for improved implementation of traffic management strategies. The code and dataset are available at: github.com/urbanAIthi/TFCO

Keywords: Automated Vehicles, Integration of Infrastructure and Intelligent Vehicles, Integration of HD map and Onboard Sensors
Abstract: High-definition angular marking maps (HDAM maps) are vital in large-scale environments with variable appearances. In these scenarios, unmanned ground vehicles (UGVs) can use angular markings for localization because they are easy to identify and informative for localization. However, creating such a marking map relies heavily on manual measurement and annotation, which is time-consuming and laborious. Although Inverse Perspective Mapping (IPM) offers a low-cost and automated alternative, its accuracy is compromised by vehicle motion and the arduous pre-calibration of the IPM matrix. To fill these gaps, we propose a pose-guided optimization framework for IPM. This framework enables the automated generation of HDAM maps, while concurrently refining the preliminary IPM matrix. We deployed the proposed method in two different automated ports, and the method yielded HDAM maps with near-centimeter precision. Moreover, the refined IPM matrix matched the accuracy of manual calibrations. The supplementary materials and videos are available at http://liuhongji.site/PGO-IPM/.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, Future Mobility and Smart City
Abstract: Trajectory prediction for all road participants constitutes a crucial module in decision-making processes. The primary challenge lies in comprehending the interactions between these agents. Conventional models typically rely on historical trajectories of agents to discern their interactions, often falling short of capturing nuanced expertise. This paper introduces VGA, a novel trajectory prediction model based on Virtual Interaction Force (VIF) and a Graph Attention Network. VGA considers the VIF between agents, employing force a feature vector expressing the impact intensity among agents to encapsulate their interactions. Notably, the model utilizes a graph module to transform raw input into an adjacency matrix. The map encoder and VIF encoder leverage an attention network to extract the agent-lane and agent-VIF relationships. Subsequently, the decoder derives features encompassing the agent-map and agent-VIF relationships, facilitating the prediction of future agent trajectories. Experimental results validate VGA's ability to yield precise predictions in multi-modal trajectory tasks across the Argoverse 2 Motion Forecast dataset and the Suzhou intersection trajectory dataset. Furthermore, the results of ablation experiment also verify that VIF encoder enhances prediction accuracy.

Keywords: Verification and Validation Techniques, Advanced Driver Assistance Systems (ADAS)
Abstract: Huge image data sets are the foundation for the development of the perception of automated driving systems. A large number of images is necessary to train robust neural networks that can cope with diverse situations. A sufficiently large data set contains challenging situations and objects. For testing the resulting functions, it is necessary that these situations and objects can be found and extracted from the data set. While it is relatively easy to record a large amount of unlabeled data, it is far more difficult to find demanding situations and objects. However, during the development of perception systems, it must be possible to access challenging data without having to perform lengthy and time-consuming annotations. A developer must therefore be able to search dynamically for specific situations and objects in a data set. Thus, we designed a method which is based on state-of-the-art neural networks to search for objects with certain properties within an image. For the ease of use, the query of this search is described using natural language. To determine the time savings and performance gains, we evaluated our method qualitatively and quantitatively on automotive data sets.

Keywords: Verification and Validation Techniques, Drone and Urban Air Mobility, Simulation and Real-World Testing Methodologies
Abstract: Systems of cooperating unmanned aerial vehicles(UAVs) can achieve requirements that individual UAVs cannot achieve. Particularly of interest to us are groups composed of UAVs that are cooperating in a distributed way, which would include some types of drone swarms. We refer to these groups as distributed cooperative unmanned aerial systems (DC-UAS). Cyber-physical systems must be tested before operational deployment, and while there has been substantial research into designing DC-UAS capabilities, there is a lack of systematic approaches for generating and executing test cases for system-level verification. Scenario-based testing (SBT) combined with search methods (SBT+search) is an approach that has been used to test autonomous cars and UAVs, but not in the context of distributed cooperative operations. In this paper, we show how the existing SBT+search approach to test an individual UAV can be adapted to conduct system-level testing of a DC-UAS, despite the additional properties inherent to a DC-UAS. Specifically, the adaptation involves incorporating the global task of the DC-UAS and information about the system’s individual UAVs into the approach’s search process. We then utilize a model of a decentralized drone swarm to concretely demonstrate the adapted approach, resulting in the generation of scenarios that challenge the example system’s ability to behave safely and perform global tasks. We also propose additional considerations when formulating the test case generation process. The result is a testing approach for DC-UAS that builds on existing approaches, producing a solution for test case generation and execution through the perspective of system-level verification rather than design.

Keywords: Verification and Validation Techniques, Perception Including Object Event Detection and Response (OEDR), Automated Vehicles
Abstract: Traffic Light Recognition (TLR) is vital for Autonomous Driving Systems as it supplies critical information at intersections. Modern TLRs leverage camera and geolocation data, incorporating complex pre-(post)-processing steps and multiple deep learning (DL) models for detecting, recognizing, and tracking traffic lights. While the adversarial robustness of standalone DL models has been extensively studied, the robustness of a modern TLR system, i.e., a complex software component with code and DL models, is rarely studied and hence requires research efforts.

In this work, we propose a novel testing framework (namely SITAR) targeting TLR modules from representative Level-4 ADS, such as Baidu Apollo and Autoware. We design a novel adversarial attack loss function to evaluate and improve the adversarial robustness of modern TLR systems. We applied SITAR on Apollo TLR and compared our novel loss function with state-of-the-art approaches that can effectively attack object detection and image recognition models. SITAR is shown to be effective and our novel loss function performs better than previous SOTAs with a 93% to 100% success rate with a maximum five-step iteration and eight pixels per perturbation.

Keywords: Verification and Validation Techniques, Sensor Signal Processing, Automated Vehicles
Abstract: In order to reliably validate autonomous driving functions, known risks must be taken into account and unknown risks must be identified. This work addresses this challenge by investigating risks at the level of object state estimations. The proposed methodology utilizes the differences between object state estimations from independent sensors, enabling the detection of relevant differences. This is a significant advantage, because sensor errors can be detected without ground truth. A deep autoencoder architecture is introduced to map the differences between state estimations into a latent space. The autoencoder contains Transformer and LSTM components to effectively process signals of varying lengths. The latent space is shaped using a k-means friendly design procedure, in order to find a suitable representation for anomaly detection. Detecting anomalies is a key component in the validation process of autonomous vehicles, contributing to the identification of unknown risks. The proposed approach is evaluated using real-world automotive sensor data from cameras and laser scanners in the publicly available nuScenes dataset. The results show that the generated latent space using the k-means friendly procedure is well suited for clustering differences between state estimations from these two sensors and thus for anomaly detection. In the framework specified in the safety standard ISO 21448 (SOTIF) the proposed methodology can play a key role for the detection of unknown risks on the perception level during the operation phase of autonomous vehicles.

Keywords: Automated Vehicles, Verification and Validation Techniques, Vehicle Control and Motion Planning
Abstract: The emergence of Multimodal Large Language Models ((M)LLMs) has ushered in new avenues in artificial intelligence, particularly for autonomous driving by offering enhanced understanding and reasoning capabilities. This paper introduces LimSim++, an extended version of LimSim designed for the application of (M)LLMs in autonomous driving. Acknowledging the limitations of existing simulation platforms, LimSim++ addresses the need for a long-term closed-loop infrastructure supporting continuous learning and improved generalization in autonomous driving. The platform offers extended-duration, multi-scenario simulations, providing crucial information for (M)LLM-driven vehicles. Users can engage in prompt engineering, model evaluation, and framework enhancement, making LimSim++ a versatile tool for research and practice. This paper additionally introduces a baseline (M)LLM-driven framework, systematically validated through quantitative experiments across diverse scenarios. The open-source resources of LimSim++ are available at: https://pjlab-adg.github.io/limsim-plus/.

Keywords: Verification and Validation Techniques, Simulation and Real-World Testing Methodologies, Automated Vehicles
Abstract: Safety assessment is one of the main challenges in deploying Automated Driving Systems (ADSs) on public roads. Scenario-based assessment is a common method to test such systems. Such scenario-based testing involves modeling the ADSs in a simulation environment to examine and evaluate their safety. Due to the complexity and uncertainty of the driving environment, the number of possible scenarios that ADSs can encounter is virtually infinite and there is a need for reduction of possible scenarios to a finite set. This research presents a generic framework to formulate a dissimilarity metric, which focuses on the comparison of driving scenarios on their most critical scenes, to reduce the number of possible scenarios into a finite and computationally manageable set.

Keywords: Verification and Validation Techniques, Simulation and Real-World Testing Methodologies, Automotive Cyber Physical Systems
Abstract: In this paper, we introduce the design of HackCar, a testing platform for replicating attacks and defenses on a generic automotive system without requiring access to a complete vehicle. This platform empowers security researchers to illustrate the consequences of attacks targeting an automotive system on a realistic platform, facilitating the development and testing of security countermeasures against both existing and novel attacks. The HackCar platform is built upon an F 1−10th model, to which various automotive-grade microcontrollers are connected through automotive communication protocols. This solution is crafted to be entirely modular, allowing for the creation of diverse test scenarios. Researchers and practitioners can thus develop innovative security solutions while adhering to the constraints of automotive-grade microcontrollers. We showcase our design by comparing it with a real, licensed, and unmodified vehicle. Additionally, we analyze the behavior of the HackCar in both an attack-free scenario and a scenario where an attack on in-vehicle communication is deployed.

Keywords: Verification and Validation Techniques, Simulation and Real-World Testing Methodologies, Functional Safety in Intelligent Vehicles
Abstract: Scenario-based testing has become a promising approach to overcome the complexity of real-world traffic for safety assurance of automated vehicles. Within scenario-based testing, a system under test is confronted with a set of predefined scenarios. This set shall ensure more efficient testing of an automated vehicle operating in an open context compared to real-world testing. However, the question arises if a scenario catalog can cover the open context sufficiently to allow an argumentation for sufficiently safe driving functions and how this can be proven. Within this paper, a methodology is proposed to argue a sufficient completeness of a scenario concept using a goal structured notation. Thereby, the distinction between completeness and coverage is discussed. For both, methods are proposed for a streamlined argumentation and regarding evidence. These methods are applied to a scenario concept and the inD dataset to prove the usability.

Keywords: Verification and Validation Techniques, Simulation and Real-World Testing Methodologies, Functional Safety in Intelligent Vehicles
Abstract: Scenario-based verification and validation (V&V) has emerged as the predominant approach for the performance evaluation of automated driving systems (ADSs). Many scenario-generation methods have been proposed to search for critical scenarios, i.e. disengagement or traffic rule violations. However, the widely adopted binary (pass/fail) criterion suffers from two main limitations, i.e., the difficulty of locating root causes and the lack of statistical guarantee of testing sufficiency. Recently, new scenario engineering approaches focusing on the intelligence of ADSs enlightened a promising pathway via dynamic driving task decomposition and function atom constraints. However, none of the state-of-the-art scenario description languages support such approaches. To fill this gap and facilitate further research into this promising direction, in this work, we propose a generic architecture to extend the existing scenario description languages for the intelligence testing of ADSs. The case study with WMG SDL demonstrates the capability and flexibility of the proposed extension design in defining intelligence function constraints.

Keywords: Verification and Validation Techniques, Simulation and Real-World Testing Methodologies
Abstract: Scenario-based testing is a promising method to develop, verify and validate automated driving systems (ADS) since pure on-road testing seems inefficient for complex traffic environments. A major challenge for this approach is the provision and management of a sufficient number of scenarios to test a system. The provision, generation, and management of scenario at scale is investigated in current research. This paper presents the scenario database scenario.center to process and manage scenario data covering the needs of scenario-based testing approaches comprehensively and automatically. Thereby, requirements for such databases are described. Based on those, a four-step approach is proposed. Firstly, a common input format with defined quality requirements is defined. This is utilized for detecting events and base scenarios automatically. Furthermore, methods for searchability, evaluation of data quality and different scenario generation methods are proposed to allow a broad applicability serving different needs. For evaluation, the methodology is compared to state-of-the-art scenario databases. Finally, the application and capabilities of the database are shown by applying the methodology to the inD dataset. A public demonstration of the database interface is provided at https://scenario.center.

Keywords: Verification and Validation Techniques, Simulation and Real-World Testing Methodologies
Abstract: The multi-vehicle coordinated lane change is one typical application of intelligent connected vehicle(ICV), which must be systematically and thoroughly verified before across-the-board commercial application. Existing evaluation frameworks face challenges in effectively verifying multi-vehicle coordinated lane change algorithm, whose decision-making process is more complex and needs to consider more complex surrounding environments. This complexity introduces the ``curse of dimensionality" into the verification process, adversely impacting verification efficiency.To address the aforementioned challenge, an efficient verification strategy with optimal feature subset selection is proposed in this study. Initially, the subset feature is defined by the integrals of position probability density function between host vehicle and surrounding vehicles across various decision-making phases of coordinated lane change algorithm. Following this, the optimal feature subset selection method is presented for verification in different decision-making phases of coordinated lane change algorithm. Subsequently, the verification strategy is delineated.Finally, the optimal feature subset selection verification strategy is implemented within a coordinated lane change scenario. A multi-start search algorithm is employed to explore the feasible domain of the multi-vehicle coordinated lane change algorithm. Verification through simulation is then executed, and its efficiency is compared with a widely used evaluation framework based on Test Matrix. Notably, the proposed strategy demonstrates a minimum efficiency improvement of 85%.The verification results underscore the effectiveness the proposed method in verification of phased multi-vehicle coordinated lane change decision-making algorithm.

Keywords: Verification and Validation Techniques, Simulation and Real-World Testing Methodologies
Abstract: Extensive research has been conducted to explore vehicle lane changes, while the study on cut-ins has not received sufficient attention. The existing studies have not addressed the fundamental question of why studying cut-ins is crucial, despite the extensive investigation into lane changes. To tackle this issue, it is important to demonstrate how cut-ins, as a special type of lane change, differ from other lane changes. In this paper, we explore to compare driving characteristics of cut-ins and other lane changes based on naturalistic driving data. The highD dataset is employed to conduct the comparison. We extract all lane-change events from the dataset and exclude events that are not suitable for our comparison. Lane-change events are then categorized into the cut-in events and other lane-change events based on various gap-based rules. Several performance metrics are designed to measure the driving characteristics of the two types of events. We prove the significant differences between the cut-in behavior and other lane-change behavior by using the Wilcoxon rank-sum test. The results suggest the necessity of conducting specialized studies on cut-ins, offering valuable insights for future research in this field.

Keywords: Verification and Validation Techniques, Vehicle Control and Motion Planning, Automated Vehicles
Abstract: The implementation of road user models that can reproduce a credible realistic behavior in a multi-agent simulation is still an open problem. We propose a data-driven approach to sift through trajectories from a specific scenario and separate them into semantic similar macro-maneuvers from raw observation. Cars and two different types of Vulnerable Road Users (VRU) will be considered: pedestrians and cyclists, in two environments, intersections and roundabouts. Different methods will be compared, two of then, the main focus of this work, also using the starting and ending points of the trajectory as a clustering feature together with a dissimilarity measure. The results reported here evaluate methods to obtain well-defined trajectory classes from raw data without the use of map information to cluster while also separating "eccentric" or incomplete trajectories from the ones that are representative of a macro-maneuver. The final clustering results can then be manually inspected and accepted or rejected according to the trajectories it contains to then be used to train models capable to extrapolate the observed behavior.

Keywords: Verification and Validation Techniques, Vehicular Active and Passive Safety, Sensor Signal Processing
Abstract: Visual-based object detection has become a crucial component in the realm of autonomous vehicles. However, conducting reliable testing for such systems remains unresolved. In this paper, we advocate for the application of causal inference to investigate the pivotal environmental factors influencing detection accuracy. Through the integration of diffusion models, we address the specialized conditional generalization of hazardous testing images. Our approach involves the construction of observational data to attribute key factors and fine-tune the diffusion model. Additionally, we introduce an optimal prompt words search method that strikes a balance between test coverage and level of challenge. Subsequently, leveraging these optimal prompts, we propose a cost-effective testing image generation through both “Text2Scene” and “Image2Scene” fashions. The experimental results indicate that, on the generalized dataset, the performance of object detection algorithms is the poorest, with the average detection accuracy decreasing from 0.81 to 0.285. Moreover, retraining object detection models on our generalized critical test cases can ultimately enhance algorithm performance, achieving a median accuracy improvement of up to 8.13%. Overall, our research proposes a novel approach to generalize test cases, thereby contributing to the advancement and deployment of safer autonomous vehicles.

Keywords: Verification and Validation Techniques
Abstract: During the transition from human-driving to autonomous-driving, Automated Driving Systems (ADS) have to deal with the crises caused by uncertain human-driving factors, especially in the junction-scenes which are flexible and complex. To validate the Safety of the Intended Functionality (SOTIF) in ADS with scenario-based tests, it is challenging to create reasonable hazardous-test-cases, which consider causing collision and avoiding liability simultaneously. Currently, parameter-search-based approaches with parameterized-maneuvers are widely utilized to explore test cases. However, in complicated scenes (i.e., junction-scenes), the trade-off between efficiency and fineness of maneuver-parameterization reduces the searching ability. To address the above challenges, we propose a cost-based controller with a designed state-transfer for collision to lead the agent vehicle (i.e., the vehicle causing events). The approach directly outputs continuous actions as the substitute for discrete parameterized-maneuvers, reducing the searching-state-space and enabling more detailed behaviors. In our real-time test, given different scenes in which Ego owns the right-of-way, our approach generates Ego-liable collision-cases with the success rates above 90%, higher than the ones using parameter-search-based method which are below 70%. More importantly, our method creates more hazardous cases with higher efficiency, which achieves 1.9-2.6 times of the impact-speed and takes only 18%-24% of time-consumption by contrast.

Keywords: Automated Vehicles, Cooperative Vehicles
Abstract: Cooperative driving is emerging as an effective way to improve traffic efficiency and safety, and has attracted considerable research attention. However, a major drawback of most existing studies is that they rely on ideal communication conditions and overlook the critical issue of communication failures in vehicles. These failures have the potential to disrupt traffic efficiency and introduce serious safety risks. In this paper, we propose a fault-tolerant cooperative driving strategy that systematically addresses the challenges posed by communication failures within a global planning and local gaming framework. In the global planning stage, the centralized controller coordinates the passing order of all vehicles to optimize traffic efficiency. In the local gaming stage, the faulty vehicle engages in a two-player cooperative game to decide the order with potentially conflicting vehicles. Simulation results show that our strategy enhances the fault tolerance of the system, ensuring driving safety while mitigating the impact of faults on traffic efficiency. This work provides insights for building a more robust and safe cooperative driving system under real-world communication challenges.

Keywords: Automated Vehicles, Sensor Signal Processing, Verification and Validation Techniques
Abstract: Currently, with the rapid growth of training datasets for autonomous driving systems, we are faced with a challenge: how to efficiently retrieve specific traffic scenes from massive amount of scene in multiple datasets. This challenge primarily stems from the heterogeneity of existing datasets, meaning these datasets contain different types of data, follow different data formats, and use different sensors for data collection. To address this issue, we present RSG-Search Plus, a universal traffic scene searching method based on Road Scene Graph and Large Language Models (LLMs). Our approach first transform datasets into scene graphs to exclude irrelevant details, then efficiently retrieving specific configurations among thousands of traffic scenes by matching isomorphic sub-graphs between input graph and road scene graph. Experimental results demonstrate that our graph searching method can accurately match the scenes described by input condition. Additionally, this method is easily adaptable to different datasets, significantly simplifying the scene search process.

Keywords: Automated Vehicles, Simulation and Real-World Testing Methodologies
Abstract: As autonomous driving technology matures, the safety and robustness of its key components, including trajectory prediction is vital. Although real-world datasets such as Waymo Open Motion provide recorded real scenarios, the majority of the scenes appear benign, often lacking diverse safety-critical situations that are essential for developing robust models against nuanced risks. However, generating safety-critical data using simulation faces severe simulation to real gap. Using real-world environments is even less desirable due to safety risks. In this context, we propose an approach to utilize existing real-world datasets by identifying safety-relevant scenarios naively overlooked, e.g., near misses and proactive maneuvers. Our approach expands the spectrum of safety-relevance, allowing us to study trajectory prediction models under a safety-informed, distribution shift setting. We contribute a versatile scenario characterization method, a novel scoring scheme for reevaluating a scene using counterfactual scenarios to find hidden risky scenarios, and an evaluation of trajectory prediction models in this setting. We further contribute a remediation strategy, achieving a 10% average reduction in predicted trajectories' collision rates. To facilitate future research, we release our code for this overall SafeShift framework to the public: github.com/cmubig/SafeShift

Keywords: Automated Vehicles, Simulation and Real-World Testing Methodologies
Abstract: In this work, we test the influence of several levels of communication and processing delay for traffic wave dissipation control. The approach uses Connected Automated Vehicles that are controlled in simulation through reinforcement learning and non-reinforcement learning controllers, and compares their performance with a pure human driving scenario that has no control delay. We measure the performances with respect to average traffic speed, traffic speed standard deviation, and percentage of compliance with a custom designed safety monitor. The work shows that reinforcement learned controllers can perform with almost no deterioration in performance with latencies of 1 s or less. Non-reinforcement learning controllers, which are not intentionally modeled with latency in mind, show rapid deterioration in performance with any unexpected latency, which shows that the motivating problem requires a solution that is robust to latency. The paper discusses the training and reward function modifications required in order to consider delay as part of the framework, and discusses how the results may be suitable for deployment on high-latency networks such as mobile phones, without the need for 5G deployment.

Keywords: Automated Vehicles, Simulation and Real-World Testing Methodologies, Verification and Validation Techniques
Abstract: With the focus on data-driven scenario-based methods for the safety assessment of autonomous driving systems, there is a need to generate scenario databases from naturalistic driving data. In this study, we look at the lane wandering and the cut-out scenario, two scenarios that have received little attention to date. The paper is structured as follows: first of all, we model the two scenarios and define parameters necessary to accurately describe them. After that we show how algorithms can be used to detect the scenarios automatically in a large naturalistic driving database recorded on German highways using vehicles equipped with pre-series sensors. Following that, we analyze selected scenario parameters in detail and discuss the findings of the scenario data analysis to derive recommendations for the safety assessment of autonomous driving systems. We were able to show that lane wandering scenarios may in many cases lead to safety critical situations due to the significant lateral velocities and that cut-out scenarios are especially critical if they involve a lane change to the left lane.

Keywords: Automated Vehicles, Vehicle Control and Motion Planning, Simulation and Real-World Testing Methodologies
Abstract: The on-ramp area is a very common bottleneck for trajectory planning of autonomous driving. Existing studies are mostly based on rule computation or behavior cloning of human drivers. This study proposes an optimization-based trajectory planning model to improve merging efficiency and safety. To combine safety and efficiency criteria, this research developed a risk-constrained model based on computation of risk map combining road and traffic characteristics based on dynamic interaction responses of the mainstream vehicle. Dynamic programming (DP) and quadratic programming (QP) are applied to optimize speed profile of merging. To evaluate the algorithm's performance, CARLA and SUMO are used to create on-ramp scenarios and traffic flow in the on-ramp area. The results reveal that the proposed bi-level risk-constrained optimization (BRO) model achieves a balance of safety and efficiency maneuvers as compared to the rule-based method. When compared to the non-optimization-based merging trajectory planning model, the efficiency of traffic flow is effectively improved by 23.3% in low-density scenarios and 28.3% in high-density traffic flow scenarios. The simulation results reveal that the proposed BRO method reduces the overall traffic delay at on-ramp merging area while improving safety performance.

Keywords: Automated Vehicles, Verification and Validation Techniques, Policy, Ethics, and Regulations
Abstract: Traffic rules formalization is crucial for verifying the compliance and safety of autonomous vehicles (AVs). However, manual translation of natural language traffic rules as formal specification requires domain knowledge and logic expertise, which limits its adaptation. This paper introduces TR2MTL, a framework that employs large language models (LLMs) to automatically translate traffic rules (TR) into metric temporal logic (MTL). It is envisioned as a human-in-loop system for AV rule formalization. It utilizes a chain-of-thought in-context learning approach to guide the LLM in step-by-step translation and generating valid and grammatically correct MTL formulas. It can be extended to various forms of temporal logic and rules. We evaluated the framework on a challenging dataset of traffic rules we created from various sources and compared it against LLMs using different in-context learning methods. Results show that TR2MTL is domain-agnostic, achieving high accuracy and generalization capability even with a small dataset. Moreover, the method effectively predicts formulas with varying degrees of logical and semantic structure in unstructured traffic rules.

Keywords: Cooperative Vehicles, Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Achieving safe lane changing is a crucial function of autonomous vehicles, with the complexity and uncertainty of interaction involved. Learning-based approaches and vehicle collaboration techniques can enhance vehicles' awareness of the dynamic environment, thereby enhancing the interactive capabilities. Therefore, this paper proposes a Multi-Agent Reinforcement Learning (MARL) approach to coordinate connected vehicles in reaching their respective lane changing targets. Vehicle state, scene elements, potential risk, and intention information are abstracted into highly expressive vectorized inputs. Based on this, a lightweight parameter-sharing network framework is designed to learn safe and robust cooperative lane changing policies. To address the challenges arising from multi-objects and multi-targets, a Prioritized Action Extrapolation (PAE) mechanism is employed to train the network. Through priority assignment and action extrapolation, the proposed MARL approach can optimize the decision sequence dynamically and enhance the interaction in multi-vehicle scenarios, thereby improving the vehicles' intention attainment rate. Simulated experiments in 2-lane and 3-lane scenarios have been conducted to verify the adaptability and performance of the proposed MARL method.

Keywords: Cooperative Vehicles, Automated Vehicles, Vehicle Control and Motion Planning
Abstract: Cooperative on-ramp merging control for connected automated vehicles (CAVs) has been extensively investigated. However, they did neglect the connected vehicle identification process, which is a must for CAV cooperations. In this paper, we introduced a connected vehicle identification system (VIS) into the on-ramp merging control process for the first time and proposed an evaluation framework to assess the impacts of VIS on on-ramp merging performance. First, the mixed-traffic cooperative merging problem was formulated. Then, a real-world merging trajectory dataset was processed to generate dangerous merging scenarios. Aiming at resolving the potential collision risks in mixed traffic where CAVs and traditional human-driven vehicles (THVs) coexist, we proposed on-ramp merging strategies for CAVs in different mixed traffic situations considering the connected vehicle identification process. The performances were evaluated via simulations. Results indicated that while safety was assured for all cases with CAVs, the cases with VIS had delayed initiation of cooperation, limiting the range of cooperative merging and leading to increased fuel consumption and acceleration variations.

Keywords: Drone and Urban Air Mobility, Simulation and Real-World Testing Methodologies, Future Mobility and Smart City
Abstract: Most urban logistics providers are able to deliver goods efficiently using conventional vehicles, except in areas that have low quality road infrastructure and steep road slopes, which pose a challenge for delivery. This lowers the quality of delivery services, creating pockets of underserved populations in urban areas, which is undesirable from an economic and social perspective. The flexibility of drones has made them an attractive option for last-mile delivery in this context. Optimizing the utilization of remaining energy and payload capacity during drone return trips is essential for maximizing overall utility. However, prior studies have overlooked the consideration of backhaul in the context of drone-only systems, as well as the assessment of energy consumption under wind conditions. This paper proposes a new model for the Drone Delivery Routing Problem with Backhaul (DDRPB), which incorporates both wind conditions and backhauling requirements, and is applicable for areas that have high delivery demand and return requests, but suffer from low delivery service quality. We introduce a linearized drone-wind-energy consumption model and design a Lagrangian relaxation heuristic algorithm to solve the resulting NP-hard optimization problem efficiently. A case study of an urban area with delivery challenges in Seoul, South Korea is presented. The results show that the proposed model effectively decreases overall costs.

Keywords: Sensor Signal Processing, Cooperative Vehicles, Automated Vehicles
Abstract: Collective Perception has attracted significant attention in recent years due to its advantage for mitigating occlusion and expanding the field-of-view, thereby enhancing reliability, efficiency, and, most crucially, decision-making safety.However, developing collective perception models is highly resource demanding due to extensive requirements of processing input data for many agents, usually dozens of images and point clouds for a single frame. This not only slows down the model development process for collective perception but also impedes the utilization of larger models. In this paper, we propose an agent-based training framework that handles the deep learning modules and agent data separately to have a cleaner data flow structure. This framework not only provides an API for flexibly prototyping the data processing pipeline and defining the gradient calculation for each agent, but also provides the user interface for interactive training, testing and data visualization. Training experiment results of four collective object detection models on the prominent collective perception benchmark OPV2V show that the agent-based training can significantly reduce the GPU memory consumption and training time while retaining inference performance. The framework and model implementations are available at url{https://github.com/YuanYunshuang/CoSense3D}

Keywords: Cooperative Vehicles, Automated Vehicles, Future Mobility and Smart City
Abstract: This research proposes a coordination method for multi-connected and automated vehicles (CAVs) and traffic signal. It aims at reducing stop-and-go maneuvers of CAVs and enhancing traffic efficiency. The proposed method has the following highlights: i) Adaptive to actual CAV and human-driven vehicle (HV) mixed traffic; ii) Jointly optimization of both vehicle trajectory and signal timing via formulating in the spatial domain; iii) Parallel distributed computing. Simulation test results demonstrate that the proposed coordinated control significantly outperforms the benchmark method. The proposed method reduces the average travel delay by 29.56%, enhances fuel efficiency by 18.87%, and reduces stop count by 87.10%. The proposed parallel distributed computing algorithm ensures a computation time basically within 10 milliseconds. It indicates that the proposed method is ready for real-time large-scale implementation.

Keywords: Teleoperation of Intelligent Vehicles, Automated Vehicles, Simulation and Real-World Testing Methodologies
Abstract: The advancement of automated shuttle buses has yet to reach a stage where they can operate fully autonomously on public roads. To address this limitation, academia and industry are exploring remote monitoring and control as an interim solution. Despite these efforts, certain scenarios, such as assisting a disabled person with boarding, remain challenging to manage remotely. Our study, conducted in a shuttle bus mock-up, explores the potential for passenger involvement in managing situations like this via collaboration with a remote control center or the vehicle itself. Our results indicate a general willingness of passengers to help in exceptional cases. However, there was a noticeable reluctance to deal with technical malfunctions, with participants showing low acceptance levels for such interventions. Interestingly, the study revealed that experiencing these critical situations led to a significant increase in trust toward automated driving technologies. These insights have important implications for developing Human-Machine Interfaces that effectively support passengers and remote operators in handling unusual situations.

Keywords: Wireless Power Transfer Systems for Mobility, Cooperative Vehicles, Teleoperation of Intelligent Vehicles
Abstract: In order to improve the load balancing degree, the bandwidth utilization of multi-function vehicle bus (MVB) and the real-time performance of vehicle communication system, an optimization design method of MVB scheduling table based on multi-strategy hybrid whale optimization algorithm (MHWOA) is proposed. The principle of MVB periodic information communication is briefly described. The constraints and optimization objectives of MVB scheduling table are defined. The periodic information scheduling table model is established. The MHWOA is used to optimize the bus scheduling table of multi-function vehicle. The comparison is made with the improved difference algorithm and genetic algorithm. The results show that the MHWOA can obtain more balanced solutions and higher bus utilization in the same search time.

Keywords: Sensor Signal Processing, Perception Including Object Event Detection and Response (OEDR), Sensor Fusion for Localization
Abstract: Online 3D multi-object tracking (MOT) has recently received significant research interests due to the expanding demand of 3D perception in advanced driver assistance systems (ADAS) and autonomous driving (AD). Among the existing 3D MOT frameworks for ADAS and AD, conventional point object tracking (POT) framework using the tracking-by-detection (TBD) strategy has been well studied and accepted for LiDAR and 4D imaging radar point clouds. In contrast, extended object tracking (EOT), another important framework which accepts the joint-detection-and-tracking (JDT) strategy, has rarely been explored for online 3D MOT applications. This paper provides the first systematical investigation of the EOT framework for online 3D MOT in real-world ADAS and AD scenarios. Specifically, the widely accepted TBD-POT framework, the recently investigated JDT-EOT framework, and our proposed TBD-EOT framework are compared via extensive evaluations on two open source 4D imaging radar datasets: View-of-Delft and TJ4DRadSet. Experiment results demonstrate that the conventional TBD-POT framework remains preferable for online 3D MOT with high tracking performance and low computational complexity, while the proposed TBD-EOT framework has the potential to outperform it in certain situations. However, the results also show that the JDT-EOT framework encounters multiple problems and performs inadequately in evaluation scenarios. After analyzing the causes of these phenomena based on various evaluation metrics and visualizations, we provide possible guidelines to improve the performance of these MOT frameworks on real-world data. These provide the first benchmark and important insights for the future development of 4D imaging radar-based online 3D MOT.

Keywords: Verification and Validation Techniques, Simulation and Real-World Testing Methodologies, Automated Vehicles
Abstract: The selection of relevant test scenarios for the scenario-based testing and safety validation of automated driving systems (ADSs) remains challenging. An important aspect of the relevance of a scenario is the challenge it poses for an ADS. Existing methods for calculating the challenge of a scenario aim to express the challenge in terms of a metric value. Metric values are useful to select the least or most challenging scenario. However, they fail to provide human-interpretable information on the cause of the challenge which is critical information for the efficient selection of relevant test scenarios. Therefore, this paper presents the Challenge Description Method that mitigates this issue by analyzing scenarios and providing a description of their challenge in terms of the minimum required lane changes and their difficulty. Applying the method to different highway scenarios showed that it is capable of analyzing complex scenarios and providing easy-to-understand descriptions that can be used to select relevant test scenarios.

Keywords: Integration of HD map and Onboard Sensors, Sensor Signal Processing
Abstract: To reduce the expensive labor costs of manually labeling autonomous driving datasets, an alternative is to automatically label the datasets using an offline perception system. However, objects might be temporarily occluded. Such occlusion scenarios in the datasets are common yet underexplored in offline auto labeling. In this work, we propose an offline tracking model that focuses on occluded object tracks. It leverages the concept of object permanence, which means objects continue to exist even if they are not observed anymore. The model contains three parts: a standard online tracker, a re-identification (ReID) module that associates tracklets before and after occlusion, and a track completion module that completes the fragmented tracks. The Re-ID module and the track completion module use the vectorized lane map as a prior to refine the tracking results with occlusion. The model can effectively recover the occluded object trajectories. It significantly improves the original online tracking result, demonstrating its potential to be applied in offline auto labeling as a useful plugin to improve tracking by recovering occlusions.

Keywords: Perception Including Object Event Detection and Response (OEDR), Sensor Fusion for Localization, Sensor Signal Processing
Abstract: Leveraging inputs over multiple consecutive frames has been shown to benefit 3D object detection. However, existing approaches often demonstrate unsatisfactory scaling with increasing temporal histories. In this work, we propose SF3D, a late fusion module which addresses this issue by better modeling temporal relationships via a two-stream factorization. Concretely, SF3D operates on an input sequence of consecutive bird's-eye view (BEV) features, which is partitioned into ``short-term'' and ``long-term'' frames. A more heavily parameterized short-term branch using adapters and deformable attention aggregates features closer to the current timestep. In parallel, a long-term branch composed of efficiently implemented global convolution layers aggregates a larger window of temporally distant historical features. This two-stream paradigm allows SF3D to effectively consume near-term information, while scaling to efficiently leverage longer historical windows. We show that SF3D works with arbitrary upstream BEV encoders and downstream detectors, achieving improvements over recent state-of-the-art on the Waymo Open and nuScenes benchmarks.

Keywords: Perception Including Object Event Detection and Response (OEDR), End-To-End (E2E) Autonomous Driving
Abstract: In autonomous vehicles, understanding the surrounding 3D environment of the ego vehicle in real-time is essential. A compact way to represent scenes while encoding geometric distances and semantic object information is via 3D semantic occupancy maps. State of the art 3D mapping methods leverage transformers with cross-attention mechanisms to elevate 2D vision-centric camera features into the 3D domain. However, these methods encounter significant challenges in real-time applications due to their high computational demands during inference. This limitation is particularly problematic in autonomous vehicles, where GPU resources must be shared with other tasks such as localization and planning. In this paper, we introduce an approach that extracts features from front-view 2D camera images and LiDAR scans, then employs a sparse convolution network (Minkowski Engine), for 3D semantic occupancy prediction. Given that outdoor scenes in autonomous driving scenarios are inherently sparse, the utilization of sparse convolution is particularly apt. By jointly solving the problems of 3D scene completion of sparse scenes and 3D semantic segmentation, we provide a more efficient learning framework suitable for real-time applications in autonomous vehicles. We also demonstrate competitive accuracy on the nuScenes dataset.

Keywords: Advanced Driver Assistance Systems (ADAS), Automated Vehicles, Pedestrian Protection
Abstract: The prediction of road users' future motion is a critical task in supporting advanced driver-assistance systems (ADAS). It plays an even more crucial role for autonomous driving (AD) in enabling the planning and execution of safe driving maneuvers. Based on our previous work, Context-Aware Scene Prediction Network (CASPNet), an improved system, CASPNet++, is proposed. In this work, we focus on further enhancing the interaction modeling and scene understanding to support the joint prediction of all road users in a scene using spatiotemporal grids to model future occupancy. Moreover, an instance-based output head is introduced to provide multi-modal trajectories for agents of interest. In extensive quantitative and qualitative analysis, we demonstrate the scalability of CASPNet++ in utilizing and fusing diverse environmental input sources such as HD maps, Radar detection, and Lidar segmentation. Tested on the urban-focused prediction dataset nuScenes, CASPNet++ reaches state-of-the-art performance. The model has been deployed in a testing vehicle, running in real-time at 20 Hz with moderate computational resources alongside a machine learning-based perception system.

Keywords: Advanced Driver Assistance Systems (ADAS), Automated Vehicles
Abstract: Multi-task learning is effective for object detection and segmentation, which are closely related to each other and necessary for automated driving. However, there is a problem with the learning process in conventional multi-task learning models. In multi-task learning, common features among downstream tasks are first extracted by a backbone network. Then, these features are used for different downstream tasks. Since the required feature is different depending on the downstream task, it is necessary to extract features suitable for each downstream task. In this paper, we propose a multi-tasking model that introduces BiFPN feature fusion method for automated driving tasks and the Next-ViT model utilizing CNN and Transformer to extract features. From the evaluation experiments of automated driving tasks, we confirmed that the proposed method improves the accuracy of multi-task learning.

Keywords: Advanced Driver Assistance Systems (ADAS), Automated Vehicles
Abstract: State-of-the-art L3+ systems for automated driving rely on high-definition (HD) maps in order to cover their safety and performance requirements. HD maps require constant maintenance effort to keep them up-to-date as well as costly solutions for a highly precise global localization of the ego-vehicle. An alternative is to generate a local HD map that depends on the environment sensors only and that does not require a global localization. This contribution presents a lightweight real-time approach for generating local HD maps in highway scenarios. For proof-of-concept a system integration into an L3+ AD stack was conducted. The gathered offline and online results proof the feasibility of the approach.

Keywords: Advanced Driver Assistance Systems (ADAS), Functional Safety in Intelligent Vehicles, Future Mobility and Smart City
Abstract: The proliferation of shared micro-mobility services, including electric scooters (e-scooters), plays an important role in modern urban travel. Despite the growing popularity of e-scooters, their interactions with motor vehicles or pedestrians can lead to potential traffic accidents. In particular, the vehicle e-scooter interaction (VEI) at intersections is of utmost importance to study, where the actions and intentions of e-scooter riders can vary greatly depending on the dynamic traffic situations. Moreover, due to the unique moving characteristics of e-scooters, drivers must maintain adequate awareness of the environment to mitigate unforeseen collision risks. In this paper, we aim to provide a novel risk analysis methodology to identify potential risky factors in the VEI process. Qualitative and quantitative analyses of various traffic situations are introduced to validate the functionality of the proposed system. The backtracking process algorithm (BPA) is also applied to demonstrate the effectiveness of the designed framework.

Keywords: Advanced Driver Assistance Systems (ADAS), Future Mobility and Smart City, Automated Vehicles
Abstract: Electric scooters (e-scooters) have become increasingly popular for intermodal transportation across major US cities, raising safety concerns for both motorists and non-motorists. To develop E-scooter AEB (Autonomous Emergency Braking) systems for intelligent vehicles, it is important to understand the association between e-scooter crash characteristics including facility type, crash severity, and motorist/non-motorist maneuvers. This paper investigates e-scooter crashes to map variable associations. The findings reveal that less severe crashes may be highly underreported. The most critical facilities for e-scooter safety are intersection crosswalks and travel lanes, with the former resulting in more severe injuries. Observing crash trajectories, right-turn crashes are found to be most common at intersections, while only a few intersection crashes result from left-turning vehicles. The findings highlight the need for dedicated safety design and policies for e-scooters, especially for vehicles at intersection crosswalks. Since data on e-scooter crashes is limited, future studies should focus on gathering larger samples from a wider geographic area to investigate and quantify predictor-crash relationships and develop diagnostic and predictive models using statistics and data-driven approaches.

Keywords: Advanced Driver Assistance Systems (ADAS), Human Factors for Intelligent Vehicles, Automated Vehicles
Abstract: Despite the widespread implementation, the Adaptive Cruise Control (ACC) systems still fall short in delivering a satisfactory human-friendly experience, primarily due to the heterogeneity in driving experience preferences and the unpredictable, heterogeneous nature of human driving behaviors. To address this critical gap, we introduce an innovative driving style-aware car-following model that effectively captures the varying cut-in tendencies of adjacent vehicles by utilizing the Max-Ent Inverse Reinforcement Learning (IRL) method. A distinct reward function is developed to replicate human driving behavior, which can achieve a harmonious equilibrium between efficiency, safety, and comfort. The efficacy of this model is rigorously validated through a comprehensive analysis on car-following episodes extracted from the NGSIM I-80 dataset. A novel human-friendly metric is utilized for evaluating the performance of the proposed model in comparison to standard benchmarks. The results demonstrably favor our approach, showing notable enhancements in efficiency, safety, and comfort. Additionally, the model's versatility is confirmed by its ability to accommodate a wide spectrum of driving styles, as evidenced by the diverse weights learned from different driving styles. These findings highlight the significant potential of our model in advancing ACC technology for more human-oriented vehicular systems that align closely with the natural driving instincts and preferences of humans.

Keywords: Advanced Driver Assistance Systems (ADAS), Human Factors for Intelligent Vehicles, Perception Including Object Event Detection and Response (OEDR)
Abstract: To advance driver monitoring and assistance systems, it is important to understand how drivers allocate their attention, in other words, where do they tend to look and why. Traditionally, factors affecting human visual attention have been divided into bottom-up (involuntary attraction to salient regions) and top-down (driven by the demands of the task being performed). Although both play a role in directing drivers' gaze, most of the existing models for drivers' gaze prediction apply techniques developed for bottom-up saliency and do not consider influences of the drivers' actions explicitly. Likewise, common driving attention benchmarks lack relevant annotations for drivers' actions and the context in which they are performed. Therefore, to enable analysis and modeling of these factors for drivers' gaze prediction, we propose the following: 1) we correct the data processing pipeline used in DR(eye)VE to reduce noise in the recorded gaze data; 2) we then add per-frame labels for driving task and context; 3) we benchmark a number of baseline and SOTA models for saliency and driver gaze prediction and use new annotations to analyze how their performance changes in scenarios involving different tasks; and, lastly, 4) we develop a novel model that modulates drivers' gaze prediction with explicit action and context information. While reducing noise in the DR(eye)VE gaze data improves results of all models, we show that using task information in our model boosts performance even further compared to bottom-up models on the cleaned up data: both overall (by 24% KLD and 89% NSS) and on scenarios that involve performing safety-critical maneuvers and crossing intersections (by up to 10--30% KLD). Extended annotations and code are available at https://github.com/ykotseruba/SCOUT.

Keywords: Advanced Driver Assistance Systems (ADAS), Human Factors for Intelligent Vehicles, Perception Including Object Event Detection and Response (OEDR)
Abstract: Driving is a visuomotor task, i.e., there is a connection between what drivers see and what they do. While some models of drivers' gaze account for top-down effects of drivers' actions, the majority learn only bottom-up correlations between human gaze and driving footage. The crux of the problem is lack of public data with annotations that could be used to train top-down models and evaluate how well models of any kind capture effects of task on attention. As a result, top-down models are trained and evaluated on private data and public benchmarks measure only the overall fit to human data.

In this paper, we focus on data limitations by examining four large-scale public datasets, DR(eye)VE, BDD-A, MAAD, and LBW, used to train and evaluate algorithms for drivers' gaze prediction. We define a set of driving tasks (lateral and longitudinal maneuvers) and context elements (intersections and right-of-way) known to affect drivers' attention, augment the datasets with annotations based on the said definitions, and analyze the characteristics of data recording and processing pipelines w.r.t. capturing what the drivers see and do. In sum, the contributions of this work are: 1) quantifying biases of the public datasets, 2) examining performance of the SOTA bottom-up models on subsets of the data involving non-trivial drivers' actions, 3) linking shortcomings of the bottom-up models to data limitations, and 4) recommendations for future data collection and processing. The new annotations and code for reproducing the results are available at https://github.com/ykotseruba/SCOUT.

Keywords: Advanced Driver Assistance Systems (ADAS), Human Factors for Intelligent Vehicles, Simulation and Real-World Testing Methodologies
Abstract: Advancements in Intelligent Transportation Systems (ITS) enhance driver safety, comfort, and traffic flow. However, because human drivers remain integral to operating human-driven vehicles, their perception and attention are of significant importance in real-world traffic conditions. Previous studies have highlighted the increased accident susceptibility of novice drivers, but there has been limited research on how experience and perception affect each other. This study analyzes driver behavior in real-world traffic conditions based on driver experience, time of day, traffic congestion, and route familiarity. Our findings indicate that novice drivers exhibit lower spatial awareness, while intermediate drivers demonstrate improved awareness in moderate traffic. Conversely, experienced drivers consistently perform well across all conditions. Familiarity with routes enhances performance for all drivers, with experienced drivers adapting more quickly in unfamiliar situations. The study also contributes to existing data by providing continuous driver fixations in various driving scenarios and gaze-enhanced saliency maps, gaze-focused object detection, and saliency ranking to advance driver perception and autonomous vehicles.

Keywords: Advanced Driver Assistance Systems (ADAS), Human Factors for Intelligent Vehicles, Software-Defined Vehicle for Intelligent Vehicles
Abstract: In the domain of road safety, evaluating and enhancing driver attention is crucial for reducing the frequency of road accidents. This research paper proposes a novel methodology for assessing driver attention by leveraging multi-camera data. Our approach considers a dual-camera setup capturing the face and body parts from the driver's side, focusing on four key tasks: distraction detection, gaze direction analysis, fatigue detection, and hands-on-wheel monitoring. We utilize facial landmarks for fatigue detection, specifically targeting the mouth and eye regions. SqueezeNet, a lightweight convolutional neural network, is employed to discern signs of driver fatigue. Gaze direction analysis uses the same network but focuses solely on eye landmark features. Distraction activities are identified by extracting optical flow features and using them as input to the robust I3D network. Further, hands-on-wheel detection is achieved by extracting hand landmarks, followed by using a 3D CNN model. We propose a driver attention score to consolidate these tasks into a unified measure of driver attention. This score is a holistic representation of the driver's attentiveness, combining insights from distraction detection, gaze direction analysis, fatigue detection, and hands-on-wheel monitoring. Our methodology is validated on a driver monitoring dataset (DMD), where training and application demonstrate the effectiveness of the proposed approach in assessing and quantifying driver attention.

Keywords: Sensor Signal Processing, Advanced Driver Assistance Systems (ADAS), Automated Vehicles
Abstract: Recognition of surrounding traffic participants is important for the safe driving of automated vehicles. Methods using distance measurement sensors from LiDAR and MWR, and image information from cameras have been mainly developed. In recent years, 4D imaging radar has been developed as a next-generation MWR. It can measure three-dimensional position with relative velocity in irradiation direction. In this study, we developed 3D object detection model using 4D imaging radar and evaluated the recognition performance from a practical point of view. The developed model is based on a simple object detection pipeline using point features in BEV. Our model integrates the road structure feature using a predefined image map created using LiDAR without annotation. In addition, we also developed semi-supervised label generation using LiDAR in order to train the model for sparse point clouds. The evaluations show the recognition of vehicles is improved by adding velocity features and image map information. The results on our dataset show the F-value for the Car class was improved by +18%, and the F-value for the Large Car class was improved by +40% by introducing measured velocity and irradiation direction vector. Then, the precision for the Car class was improved by +16%, and the precision for the Large Car class was improved by +30% by introducing an image map that suppressed false positives with a low processing cost.

Keywords: Automated Vehicles, Advanced Driver Assistance Systems (ADAS), Perception Including Object Event Detection and Response (OEDR)
Abstract: The development of autonomous vehicles has been accelerating, marked by a variety of competitions that challenge teams with diverse missions. Recently,racing-based autonomous driving competitions have gained prominence. Notably, the 2023 Hyundai Motor Group Autonomous Driving Challenge (HMG ADC) stands out as a manufacturer-operated event with a racing concept. This competition was distinctive, featuring mass-produced vehicles on race track with multiple vehicles simultaneously. In this paper, we explore the AutoKU team's participation in the HMG ADC, highlighting their system, which is designed for two types of driving: solo and multi-vehicle racing. We detail the use of an identical mass-produced Hyundai IONIQ 5 vehicle equipped for autonomous driving without any performance modifications. The paper will discuss AutoKU's approach and performance in solo and multi-vehicle races, showcasing their strategies and achievements in this innovative autonomous racing challenge. (Video:https://youtu.be/wLtmUkahnYA?si=AjqH6hYe10O94laq).

Keywords: Advanced Driver Assistance Systems (ADAS), Human Factors for Intelligent Vehicles, Vehicle Control and Motion Planning
Abstract: In this paper, we delve into the pedestrian behavior understanding problem from the perspective of three different tasks: intention estimation, action prediction, and event risk assessment. We first define the tasks and discuss how these tasks are represented and annotated in two widely used pedestrian datasets, JAAD and PIE. We then propose a new benchmark based on these definitions, available annotations, and three new classes of metrics, each designed to assess different aspects of the model performance.

We apply the new evaluation approach to examine four SOTA prediction models on each task and compare their performance w.r.t. metrics and input modalities. In particular, we analyze the differences between intention estimation and action prediction tasks by considering various scenarios and contextual factors. Lastly, we examine model agreement across these two tasks to show their complementary role. The proposed benchmark reveals new facts about the role of different data modalities, the tasks, and relevant data properties. We conclude by elaborating on our findings and proposing future research directions.

Keywords: Cooperative Vehicles, Advanced Driver Assistance Systems (ADAS), Perception Including Object Event Detection and Response (OEDR)
Abstract: Cooperative perception is a technique that enhances the on-board sensing and perception of automated vehicles by fusing data from multiple sources, such as other vehicles, roadside infrastructure, cloud/edge servers, among others. It can improve the performance of automated driving in complex scenarios, like unsignalled roundabouts or intersections where the visibility and awareness of other road users are limited. Motion Prediction (MP) is a key component of cooperative perception, as it enables the estimation and prediction of microscopic traffic states, such as the positions and speeds of all vehicles. It relies on information from other agents and their relationships among them, so the information provided by external sources is valuable because it enhances the understanding of the scene.

In this paper, we present improved MP through Vehicle to Vehicle (V2V) communication. We have trained Hierarchical Vector Transformer (HiVT) to be a map-less solution that can be used in road domains. With this model, we have implemented and compared two association methods to evaluate our framework on a real V2V dataset (V2V4Real). Our evaluation concludes that our V2V MP improves performance due to better scene understanding over a single-vehicle MP.

Keywords: Advanced Driver Assistance Systems (ADAS), Human Factors for Intelligent Vehicles
Abstract: Recognizing distractions on the road is crucial to reduce traffic accidents. Video-based networks are typically used, but are limited by their computational cost and are vulnerable to viewpoint changes. In this paper, we propose a novel approach for pose-based driver action classification using speech recognition networks, which is lighter and more viewpoint invariant that video-based one. We leverage the similarity in the encoding of information between audio and pose data, representing poses as key points over time. Our architecture is based on Squeezeformer, an efficient attention-based speech recognition network. We introduce a selection of data augmentation techniques to enhance generalization. Experiments on the Drive&Act dataset demonstrate superior performance compared to state-of-the-art methods. Additionally, we explore the integration of object information and the impact of viewpoint changes. Our results highlight the effectiveness and robustness of speech recognition networks in pose-based action classification.

Keywords: Vehicle Control and Motion Planning, Automated Vehicles
Abstract: Determining the optimal cost function parameters of Model Predictive Control (MPC) to optimize multiple control objectives is a challenging and time-consuming task. Multi-objective Bayesian Optimization (BO) techniques solve this problem by determining a Pareto optimal parameter set for an MPC with static weights. However, a single parameter set may not deliver the most optimal closed-loop control performance when the context of the MPC operating conditions changes during its operation, urging the need to adapt the cost function weights at runtime. Deep Reinforcement Learning (RL) algorithms can automatically learn context-dependent optimal parameter sets and dynamically adapt for a Weights-varying MPC (WMPC). However, learning cost function weights from scratch in a continuous action space may lead to unsafe operating states. To solve this, we propose a novel approach limiting the RL action space within a safe learning space that we represent by a catalog of pre-optimized feasible BO Pareto-optimal weight sets. We conceive an RL agent not to learn in a continuous space but to select the most optimal discrete actions, each corresponding to a single set of Pareto optimal weights, by proactively anticipating upcoming control tasks in a context-dependent manner. This approach introduces a two-step optimization: (1) safety-critical with BO and (2) performance-driven with RL. Hence, even an untrained RL agent guarantees a safe and optimal performance. Simulation results demonstrate that an untrained RL-WMPC shows Pareto-optimal closed-loop behavior and training the RL-WMPC helps exhibit a performance beyond the Pareto-front. The code used in this research is publicly accessible as open-source software: https://github.com/bzarr/TUM-CONTROL

Keywords: Vehicle Control and Motion Planning, Automotive Cyber Physical Systems, Automated Vehicles
Abstract: High-level autonomous vehicles and domain-based electronic and electrical (E/E) architectures are important development directions of the intelligent automobile industry. The domain-centralized E/E architecture has become the potential upgrade to the autonomous vehicle benefitting from its powerful software updates, cabling reduction, and functional integration. Aiming at the efficient motion control of the autonomous vehicle equipped with domain-centralized E/E architecture, a novel control framework with algorithms improvement is proposed in this paper, which contains the multi-hops loop delay analysis to solve the control stability problem caused by the heterogeneous topology loop delay of domain-centralized E/E architecture. In this framework, the motion controller is generated through the combination of modified double reinforcement learning algorithm and multi-steps predictive control method, and the loop delay is integrated into the controller optimization. Through the virtual driving environment simulation and real world scenario, the results show that the proposed framework achieves better performance in terms of path tracking and obstacles avoidance, and the stability of control strategies to loop delay is also guaranteed.

Keywords: Vehicle Control and Motion Planning, Cooperative Vehicles, Automated Vehicles
Abstract: Multi-agent cooperative motion planning offers the potential to improve safety and the overall traffic flow. However, many approaches for multi-agent driving do not incorporate traffic rules or do not generalize to arbitrary scenarios. To address these open problems, we propose a novel method to negotiate individual rule-compliant driving corridors and independently plan trajectories for each controlled agent in them. We incorporate predictions into the conflict negotiation process to enable decision-making over long time horizons. Our approach is applicable to arbitrary scenarios, including mixed cooperative and non-cooperative traffic participants, as demonstrated through our numerical experiments.

Keywords: Vehicle Control and Motion Planning, Cooperative Vehicles, Automated Vehicles
Abstract: On-ramp merging problem focuses on vehicle safety and traffic efficiency. It can be considered as a hierarchical planning scenario with free flow zone, preparation zone and merging zone. Previous researches consider Reinforcement Learning (RL) as a potential solution due to its general learning ability. However, flat-RL tends to consider the on-ramp merging problem as a whole, neglecting its hierarchical property and causing limited improvement. Instead, with temporal abstraction, Option-based Hierarchical Reinforcement Learning (HRL) is capable to solve complicated problem by using task decomposition, giving a hint to adapt various zones in on-ramp merging problem. We hence propose an HRL-based Macro-Micro Adaptive framework (HRL-MMA). In this end-to-end framework, a Macro-Micro Adaptive Layer (MMAL) provides both macroscopic traffic information and microscopic vehicle information to the framework. The macroscopic information aims to help the master of the framework to choose options of different capacities, while the latter guarantees the safety of merging. Extensive experiments involve both the state-of-the-art baselines and several variants of the proposed framework. Compared with the IDM model, the proposed HRL-MMA framework has a 46.98% increase on the network average velocity and a 59.16% improvement on the emergency braking rate, largely ameliorating the safety of the merging problem.

Keywords: Vehicle Control and Motion Planning, Cooperative Vehicles, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications
Abstract: This paper presents a cooperative intelligent driving (CID) scheme to optimally control a vehicle's speed in multi-lane traffic, smooth its flow, and influence others to improve their performance. Under the scheme, lane-wise traffic speeds along the road, in the form of a road-speed profile (RSP), are dynamically estimated using information from connected vehicles that broadcast their states. The driving decision under the scheme is computed in a model predictive control (MPC) framework that optimizes the vehicle's acceleration to equalize traffic speeds across the lanes in a cooperative approach besides attaining the objective of safe and smooth driving. The optimization problem in the scheme is solved using a real-time computation method. The scheme is assessed by implementing it on a small portion of vehicles in typical freeway traffic affected by lane blocks or merging flows using the AIMSUN traffic simulator. It is found that a fraction of cooperative intelligent vehicles can relieve bottlenecks, harmonize the flow over lanes, and significantly improve overall traffic performance.

Keywords: Vehicle Control and Motion Planning, Cooperative Vehicles, Vehicular Active and Passive Safety
Abstract: This paper investigates the multi-agent system leader following consensus problem and its application in the intelligent transportation systems (ITS) field. The leader agent provides the desired reference trajectory, and the other follower agents operate cooperatively with the leader under the predefined motion state constraints. The considered agents are second-order nonlinear systems with parameter uncertainties and unknown disturbances. To achieve cooperative operation of the system, a state-constraints multi-agent cooperative adaptive control (SMCAC) method is given for the follower agent. The barrier Lyapunov function (BLF) is constructed to analyze the performance of the method in terms of error convergence and state constraints. The proposed method is then applied to the control of a multi-train system under the train-to-train communication topology. Numerical simulations on a five-train system are given to demonstrate the theoretical analysis.

Keywords: Vehicle Control and Motion Planning, End-To-End (E2E) Autonomous Driving, Advanced Driver Assistance Systems (ADAS)
Abstract: Robust and accurate path tracking for au- tonomous vehicle navigation is a complex task, especially when it comes to managing system uncertainties such as inertia, slippage, and action delays. Although model-based controllers are efficient, their performance can be limited by such uncertainties and by the complexity of the gain tuning process. To address this, our study evaluates the effectiveness of four strategies using reinforcement learning (RL) with a controller, to provide either - steering correction, full gain tuning, gain correction, or end-to-end learning without any controller - to improve trajectory tracking. These methods are trained on geometric controllers (Pure Pursuit, Stanley) and model predictive controllers (Romea, EBSF). Our results show that all RL methods improve tracking at high speeds, with steering correction proving the most consistently effective in all cases.

Keywords: Vehicle Control and Motion Planning, End-To-End (E2E) Autonomous Driving, Automated Vehicles
Abstract: This paper addresses motion forecasting in multi-agent environments, pivotal for ensuring safety of autonomous vehicles. Traditional and recent data-driven marginal trajectory prediction methods struggle to properly learn non-linear agent-to-agent interactions. We present SSL-Interactions that proposes pretext tasks to enhance interaction modeling for trajectory prediction. We introduce four interaction-aware pretext tasks to encapsulate various aspects of agent interactions: range gap prediction, closest distance prediction, direction of movement prediction, and type of interaction prediction. We further propose an approach to curate interaction-heavy scenarios from datasets. This curated data has two advantages: it provides a stronger learning signal to the interaction model, and facilitates generation of pseudo-labels for interaction-centric pretext tasks. We also propose three new metrics specifically designed to evaluate predictions in interactive scenes. Our empirical evaluations indicate SSL-Interactions outperforms state-of-the-art motion forecasting methods quantitatively with up to 8% improvement, and qualitatively, for interaction-heavy scenarios.

Keywords: Vehicle Control and Motion Planning, Integration of Infrastructure and Intelligent Vehicles, Advanced Driver Assistance Systems (ADAS)
Abstract: A motion planner is established to realize piloting automated driving on unstructured roads. It has the following features: i) improved adaptivity to over-the-horizon driving environment, ii) enhanced compatibility with unstructured roads, and iii) guaranteed computational efficiency for real-time application. The performance of the proposed motion planner was evaluated in a software-in-the-loop simulation platform. section. The evaluation includes: i) unstructured roads compatibility validation, and ii) validation of adaptivity to traffic events. Experiment results showed that applying the planner can enhance adaptivity to over-the-horizon traffic events on unstructured roads. The average travel efficiency enhancement is about 12.18% and the average perceived risk reduction is about 57.19%.

Keywords: Vehicle Control and Motion Planning, Vehicular Active and Passive Safety, Automated Vehicles
Abstract: To improve the handling performance of four-in-wheel-motor-drive (4IWMD) and four-wheel-steering (4WS) vehicles, an integrated control scheme based on active rear steering (ARS) and torque coordination is developed in this study. Considering the tracking of reference states, and the constraints of motor torque and steering angle, the integrated control scheme is designed through model predictive control (MPC). The active rear steering and direct yaw moment generated by the in-wheel motors can assist the vehicle in tracking the desired yaw rate to improve the handling performance. By utilizing the anti-dive forces of the vehicle suspensions, roll motion can be directly controlled via torque coordination without using active suspension. Simulation on a single lane change maneuver is performed, and the results demonstrate that the proposed controller can effectively improve the handling performance and ensure roll stability.

Keywords: Vehicle Control and Motion Planning, Vehicular Active and Passive Safety, Eco-Driving and Energy-Efficient Vehicles
Abstract: Articulated Heavy Vehicles (AHVs) play a crucial role in today's transportation, offering significant commercial and environmental advantages. However, challenges like jackknifing and trailer swing in AHVs highlight the need for focused research. This paper introduces innovative yaw stability algorithms designed to tackle these concerns, employing advanced control allocation techniques, including distributed control allocation. A power loss minimization algorithm with four different methods to maintain yaw stability is tested with real test vehicles. In this framework, the algorithms ensure that control actions stay within predetermined safe limits, contributing significantly to the overall safety and efficiency of AHVs.

Keywords: Vehicle Control and Motion Planning, Verification and Validation Techniques, Automated Vehicles
Abstract: Compliance with the rules of the road is crucial for the safe operation of autonomous vehicles. Previous work has shown that one can expedite rule-compliant motion planning by constraining the search space based on the reachable states of the vehicle. We propose an algorithm to overapproximate the states that a vehicle can reach while adhering to a linear temporal logic specification. By integrating model checking into reachability analysis, we can exclude many non-compliant states early. Moreover, we only have to semantically split the reachable set when necessary to decide the validity of the specification. This significantly reduces the computation time compared to existing approaches. We benchmark our approach in recorded real-world scenarios to demonstrate its real-time capability.

Keywords: Vehicle Control and Motion Planning
Abstract: Motion planning for autonomous vehicles in complex, real-world urban scenarios is a fundamental challenge in autonomous driving. To this end, we present MIMP, a Modular and Interpretable Motion Planning framework tailored for operation in such complex scenarios. Our approach consists of three key modules: trajectory generation, trajectory scoring with a trainable cost volume, and a safety filter. In the trajectory generation module, a wide range of driving behaviors is covered by generating a large set of potentially viable trajectories for the ego vehicle. To score these trajectories, we use a deep learning model, which learns a spatio-temporal cost volume to assess all trajectories in real-time. Finally, a safety filter module ensures safety in a deterministic and verifiable manner by checking for compliance with the drivable area and the absence of collisions with other agents in their future positions, obtained with a simple projection module. The trajectory with the lowest cost that passes the safety filter is the final plan, without any additional adjustments. Our results in closed-loop testing closely match those of other top-performing methods on the nuPlan benchmark and outperform them in most challenging scenarios. We emphasize the simplicity of our three building blocks, demonstrating the potential of an elegant and straightforward approach for motion planning.

Keywords: Vehicle Control and Motion Planning
Abstract: We propose a hierarchical architecture designed for scalable real-time Model Predictive Control (MPC) in complex, multi-modal traffic scenarios. This architecture comprises two key components: 1) RAID-Net, a novel attention-based Recurrent Neural Network that predicts relevant interactions along the MPC prediction horizon between the autonomous vehicle and the surrounding vehicles using Lagrangian duality, and 2) a reduced Stochastic MPC problem that eliminates irrelevant collision avoidance constraints, enhancing computational efficiency. Our approach is demonstrated in a simulated traffic intersection with interactive surrounding vehicles, showcasing a 12x speed-up in solving the motion planning problem. A video demonstrating the proposed architecture in multiple complex traffic scenarios can be found here: https://youtu.be/-pRiOnPb9_c. GitHub: https://github.com/MPC-Berkeley/hmpc_raidnet

Keywords: Perception Including Object Event Detection and Response (OEDR), Advanced Driver Assistance Systems (ADAS), Automated Vehicles
Abstract: Provident detection of other road users at night has the potential for increasing road safety. For this purpose, humans intuitively use visual cues, such as light cones and light reflections emitted by other road users to be able to react to oncoming traffic at an early stage. Computer vision methods can imitate this behavior by predicting the appearance of vehicles based on light reflections caused by the vehicle's headlights. Since current object detection algorithms are mainly based on detecting directly visible objects annotated via bounding boxes, the detection and annotation of light reflections without sharp boundaries is challenging. For this reason, the extensive open-source PVDN (Provident Vehicle Detection at Night) dataset was published that includes traffic scenarios at night with light reflections annotated via keypoints. In this paper, we explore a generic approach to annotate objects without clear boundaries, such as light reflections, by combining sparse keypoint annotations of humans with the concept of Boolean map saliency. With that, we create context-aware saliency maps that capture unsharp object boundaries, such as of light reflections. We show that this approach allows for an automated derivation of different object representations, such as bounding boxes, so that detection models can be trained and the problem of providently detecting vehicles at night can be tackled from a different perspective. Our approach makes it possible to derive bounding boxes with superior quality compared to previous approaches and to develop better object detection algorithms. With this paper, we provide a powerful method to study the problem of detecting objects with unsharp boundaries and, in particular, to investigate the detection of vehicles at night before they

Keywords: Perception Including Object Event Detection and Response (OEDR), Sensor Signal Processing, Automated Vehicles
Abstract: This paper describes a method to detect generic dynamic objects for automated driving. First, a LiDAR-based dynamic grid is generated online. Second, a deep learning-based detector is trained on the dynamic grid to infer the presence of dynamic objects of any type, which is a prerequisite for safe automated vehicles in arbitrary, edge-case scenarios. The Rotation-equivariant Detector (ReDet) originally designed for oriented object detection on aerial images was chosen due to its high detection performance. Experiments are conducted based on real sensor data and the benefits in comparison to classic dynamic cell clustering strategies are highlighted. The false positive object detection rate is strongly reduced by the proposed approach.

Keywords: Perception Including Object Event Detection and Response (OEDR), Sensor Signal Processing, Automated Vehicles
Abstract: Autonomous vehicles today are highly dependent on high-definition maps of the area they navigate in. This creates economic barriers in the way of the massive deployment of this technology, which can be overcome by estimating the drivable area onboard the autonomous vehicle. This task has been exhaustively explored using RGB cameras while LiDAR-only approaches are less common. In this paper, we propose LOGIC: a LiDAR-Only Geometric-Intensity Channel-based method for drivable area estimation. Our approach first obtains proposals of the drivable area by leveraging different pointcloud analysis procedures, including geometrical features, intensity evaluation and relations between neighbouring points. Then, these proposals are modeled as probabilistic drivability estimations and fused over time on a grid. This way of proceeding allows for a comprehensive analysis of the LiDAR data while also producing robust estimations. In addition, grid-level fusion enables the potential accommodation of additional navigable area detection methods or sensor inputs. Our method is able to match the performance of state-of-the-art methods without training or case-by-case parameter tuning while being tested on over 37000 LiDAR frames in the Waymo Open Dataset.

Keywords: Perception Including Object Event Detection and Response (OEDR), Sensor Signal Processing, Automated Vehicles
Abstract: Current autonomous driving technologies are being rolled out in geo-fenced areas with well-defined operation conditions such as time of operation, area, weather conditions and road conditions. In this way, challenging conditions as adverse weather, slippery road or densely-populated city centers can be excluded. In order to lift the geo-fenced restriction and allow a more dynamic availability of autonomous driving functions, it is necessary for the vehicle to autonomously perform an environment condition assessment in real time to identify when the system cannot operate safely and either stop operation or require the resting passenger to take control. In particular, adverse-weather challenges are a fundamental limitation as sensor performance degenerates quickly, prohibiting the use of sensors such as cameras to locate and monitor road signs, pedestrians or other vehicles. To address this issue, we train a deep learning model to identify outdoor weather and dangerous road conditions, enabling a quick reaction to new situations and environments. We achieve this by introducing an improved taxonomy and label hierarchy for a state-of-the-art adverse-weather dataset, relabelling it with a novel semi-automated labeling pipeline. Using the novel proposed dataset and hierarchy, we train RECNet, a deep learning model for the classification of environment conditions from a single RGB frame. We outperform baseline models by relative 16% in F1-Score, while maintaining a real-time capable performance of 20 Hz.

Keywords: Perception Including Object Event Detection and Response (OEDR), Sensor Signal Processing, Automated Vehicles
Abstract: The fusion of multimodal sensor data streams such as camera images and LiDAR point clouds plays an important role in the operation of autonomous vehicles (AVs). Robust perception across a range of adverse weather and lighting conditions is generally required for AVs to be deployed widely. While multi-sensor fusion networks have been previously developed for perception in sunny and clear weather conditions, these methods show a significant degradation in performance under night-time and poor weather conditions. In this paper, we propose a simple yet effective technique called ContextualFusion to incorporate the domain knowledge about cameras and LiDARs behaving differently across lighting and weather variations into 3D object detection models. Specifically,we design a Gated Convolutional Fusion (GatedConv) approach for the fusion of sensor streams based on the operational context. To aid our evaluation, we use the open-source simulator CARLA to create a multimodal adverse-condition dataset calledAdverseOp3D to address the shortcomings of existing datasets being biased towards daytime and good-weather conditions.Our ContextualFusion approach yields an mAP improvement of 6.2% over state-of-the-art methods on our context-balanced synthetic dataset. Finally, our method enhances state-of-the-art3D objection performance at night on the real-world NuScenes dataset with a significant mAP improvement of 11.7%.

Keywords: Perception Including Object Event Detection and Response (OEDR), Sensor Signal Processing, Simulation and Real-World Testing Methodologies
Abstract: This paper proposes a 3D Structural Convolutional Network (3D-SCN) for 3D convolutional encoding layers in LiDAR-based self-driving applications. The 3D-SCN leverages novel convolutional kernels that incorporate cosine similarity and Euclidean distance metrics to adeptly capture geometric characteristics from LiDAR datasets. This design is specifically crafted to maintain feature invariance amidst the disparities in regional data and sensor-specific channel variations. Experiment conducted on various LiDAR-based point cloud datasets demonstrate that the proposed 3D-SCN (3D Structural Convolutional Network) shows consistent performance across different LiDAR sensor specifications, even when trained on a specific dataset. To further validate its effectiveness and enhance the diversity of the LiDAR domain, we introduce the PanKyo dataset, which includes a comprehensive set of samples with 32, 64, and 128 channel domain differences. The results presented underscore the efficacy of the 3D-SCN in enhancing performance and robustness for LiDAR-based 3D recognition tasks in the context of self-driving applications.

Keywords: Perception Including Object Event Detection and Response (OEDR), Simulation and Real-World Testing Methodologies, Automated Vehicles
Abstract: To ensure safe operation of autonomous vehicles in complex urban environments, complete perception of the environment is necessary. However, due to environmental conditions, sensor limitations, and occlusions, this is not always possible from a single point of view. To address this issue, collective perception is an effective method. Realistic and large-scale datasets are essential for training and evaluating collective perception methods. This paper provides the first comprehensive technical review of collective perception datasets in the context of autonomous driving. The survey analyzes existing V2V and V2X datasets, categorizing them based on different criteria such as sensor modalities, environmental conditions, and scenario variety. The focus is on their applicability for the development of connected automated vehicles. This study aims to identify the key criteria of all datasets and to present their strengths, weaknesses, and anomalies. Finally, this survey concludes by making recommendations regarding which dataset is most suitable for collective 3D object detection, tracking, and semantic segmentation.

Keywords: Perception Including Object Event Detection and Response (OEDR), Simulation and Real-World Testing Methodologies, Pedestrian Protection
Abstract: No augmented application is possible without animated humanoid avatars. At the same time, generating human replicas from real-world monocular hand-held or robotic sensor setups is challenging due to the limited availability of views. Previous work showed the feasibility of virtual avatars but required the presence of 360' views of the targeted subject. To address this issue, we propose HINT, a NeRF-based algorithm able to learn a detailed and complete human model from limited viewing angles. We achieve this by introducing a symmetry prior, regularization constraints and training cues from large human datasets. In particular, we introduce a sagittal plane symmetry prior to the appearance of the human, directly supervise the density function of the human model using explicit 3D body modeling, and leverage a co-learned human digitization network as additional supervision for the unseen angles. As a result, our method can reconstruct complete humans even from a few viewing angles, increasing performance by more than 15% PSNR compared to previous state-of-the-art algorithms.

Keywords: Perception Including Object Event Detection and Response (OEDR), Simulation and Real-World Testing Methodologies
Abstract: Autonomous Vehicles (AVs) are expected to take safe and efficient decisions. Hence, AVs need to be robust to real world situations and especially to cope with open world setting i.e. the ability to handle novelty such as unseen objects. Classical object detection models are trained to recognize a predefined set of classes but struggle to generalize well to novel classes at inference stage. Open-Set Object Detection (OSOD) aims to address the challenge of correctly detecting objects from unknown classes. However, autonomous driving systems possess specific open-set characteristics that are not yet covered by OSOD methods. Indeed, a detection error could lead to catastrophic events, emphasizing the importance of prioritizing the quality of box detection over quantity. Also the specific characteristics of objects encountered in road scenes could be leveraged to improve their detection in the open-world setting. In this vein, we introduce a new definition of objects of interest for autonomous driving perception, enabling the proposition of an AV specialized open-set object detector coined ADOS. The proposed model uses a new score, learnt with the background ground truth of the semantic segmentation. This On Road Object score measures whether the object is on drivable areas, enhancing the selection of unknown detection. Experimental evaluations are conducted on simulated and real world datasets and reveal that our method outperforms the baseline approaches in unknown object detection settings with the same detection performance on known objects as the closed-set object detector.

Keywords: Perception Including Object Event Detection and Response (OEDR), Smart Infrastructure, Sensor Signal Processing
Abstract: Navigating intersections poses a significant challenge for autonomous vehicles (AVs) due to the limitations of LiDAR-based perception caused by occlusion. Collaborative perception has recently attracted a large research interest thanks to the ability to enhance AVs' perception via deep information fusion with intelligent roadside units (RSU), thus minimizing the impact of occlusion. While significant advancement has been made, the data-hungry nature of these methods creates a major hurdle for their real-world deployment, particularly due to the need for annotated RSU data. Manually annotating the vast amount of RSU data required for training is prohibitively expensive, given the sheer number of intersections and the effort involved in annotating point clouds. We address this challenge by devising a label-efficient object detection method for RSU based on unsupervised object discovery. Our introduces two new modules including object discovery based on a spatial temporal aggregation of point clouds and refinement to increase object discovery performance. Furthermore, we demonstrate that fine-tuning on a small portion of annotated data allows our object discovery models to narrow the performance gap with, or even surpass, fully supervised models. Extensive experiments are carried out in simulated and real-world datasets to evaluate our method. The code will be made publicly available upon acceptance of the paper.

Keywords: Perception Including Object Event Detection and Response (OEDR), Vehicle Control and Motion Planning, Verification and Validation Techniques
Abstract: This paper investigates the integration of Large Vision Language Models (LVLMs) with multi-sensor information, including visual and localization data from cameras and LiDAR data to a holistic understanding of traffic videos. Traffic scene understanding is a challenging problem. With complex interaction between the road actors, infrastructure, and traffic rules, it is often difficult to answer questions related to road safety, pedestrian safety, safe maneuvering characteristics, and human factors. Typical processes use a single task-oriented neural network model and combine them through semantic and symbolic reasoning. These processes often suffer from reasoning bias and incompleteness. In recent years, LVLMs have opened new avenues to perceive spatiotemporal information. These models can leverage the large knowledge base from the world and summarize spatiotemporal information effectively. The interactive nature of most of these systems allows humans to directly interact in a visual question-answering mode.

In this paper, we have extensively tested the capabilities of such LVLMs to answer key transportation research questions from videos captured through front cameras. We have curated an extensive set of multiple-choice questions to evaluate the performance of these LVLMs. Our results show that LVLMs have abilities to understand various transportation-related aspects to a great extent. Furthermore, we have shown that the addition of supplementary modalities to the VQA settings helps improve the performance of LVLMs. With the addition of 3D trajectories of surrounding objects with the 2D video frames, we observed a significant increase in MCQ performance related to vehicle-to-vehicle interaction tasks.

Keywords: Perception Including Object Event Detection and Response (OEDR)
Abstract: Recently, 3D object detection techniques based on the fusion of camera and LiDAR sensor modalities have received much attention due to their complementary capabilities. However, prevalent multi-modal models are relatively homogeneous in terms of feature fusion strategies, making their performance being strictly limited to the detection results of one of the modalities. While the latest data-level fusion models based on virtual point clouds do not make further use of image features, resulting in a large amount of noise in depth estimation. To address the above issues, this paper integrates the advantages of data-level and feature-level sensor fusion, and proposes MLF3D, a 3D object detection based on multi-level fusion. MLF3D generates virtual point clouds to realize the data-level fusion, and implements feature-level fusion through two key designs: VIConv3D and ASFA. VIConv3D reduces the noise problem and realizes deep interactive enhancement of features through cross-modal fusion, noise sensing, and cross-space fusion. ASFA refines the bounding box by adaptively fusing cross-layer spatial semantic information. Our MLF3D achieves 92.91%, 87.71% AP and 85.25% AP in easy, medium and hard scenarios on the KITTI's 3D Car Detection Leaderboard, realizing excellent performance.

Keywords: Perception Including Object Event Detection and Response (OEDR)
Abstract: Large visual models have recently demonstrated their promising performance on zero-shot transfer. However, so far, none of the existing methods explicitly possess the ability to perform zero-shot transfer on parking-slot detection, which results in current deep-learning based methods relying on training data sets, and methods based on traditional computer vision exhibiting poor robustness. In this paper, we propose a large visual model-based parking-slot detection method, which utilizes a large visual model (segment anything) to segment an around-view image and infer parking-slots by analyzing the relationship of marking-points in masks. In addition, we classify real-world parking-slots into two categories, line-based and area-based. The proposed method employs a two-stage approach which has a manually designed post-processing step without training. Multiple experiments have been carried out on public benchmarks, and our method demonstrates the capability for zero-shot transfer. The code will be released at https://github.com/Zhai0123/SAM-PS.

Keywords: Perception Including Object Event Detection and Response (OEDR)
Abstract: Automotive radar has been extensively utilized in cars for many years as an essential sensor, primarily due to its robustness in extreme weather conditions, its capacity to measure Doppler information in the surrounding environment, and its cost-effectiveness. Recently, developments in radar technologies and the availability of open-source radar data sets have attracted more attention to radars and using them for perception tasks in deep learning based autonomous driving. However, annotating radar data for large-scale autonomous driving perception tasks is challenging, i.e., it is difficult for humans to label this data and often requires a semi-automatic approach that involves projecting labels from other sensors, such as cameras and LiDARs. The lack of high-quality labeled data has limited the performance of radar perception models. In this paper, we propose MVRAE, a Multi-View Radar AutoEncoder, which employs self-supervised learning to learn meaningful representations from multi-view radar data without any labels. Our approach is based on the intuition that a good representation for multi-view radar data, which includes range-angle, range-Doppler, and angle-Doppler views, should enable the reconstruction of one view solely from the representations of the other two views. Experimental results demonstrate that our proposed self-supervised method, that can be used as a pre-training step for autonomous driving task, allows the model to learn meaningful representations from unlabeled radar data and achieves enhanced label efficiency for downstream tasks, such as radar semantic segmentation. To the best of our knowledge, MVRAE is the first work that employs self-supervised learning and conducts systematic experiments with multi-view radar data.

Keywords: Human Factors for Intelligent Vehicles, Policy, Ethics, and Regulations, Automated Vehicles
Abstract: Connected and automated vehicles (CAVs) are poised to transform the transport system. However, significant uncertainties remain about their impact, particularly regarding concerns that this advanced technology might exacerbate injustices, such as safety disparities for vulnerable road users (VRUs). Therefore, understanding the potential conflicts of this technology with societal values such as justice and safety is crucial for responsible implementation. To date, no research has focused on what safety and justice in transport mean in the context of CAV deployment and how the potential benefits of CAVs can be harnessed without exacerbating the existing vulnerabilities and injustices VRUs face. This paper addresses this gap by exploring car drivers’ and pedestrians’ perceptions of safety and justice issues that CAVs might exacerbate using an existing theoretical framework. Employing a qualitative approach, the study delves into the nuanced aspects of these concepts. Interviews were conducted with 30 participants (40% pedestrians) in Queensland, Australia, aged between 18 and 79. These interviews were recorded, transcribed, organised, and analysed using reflexive thematic analysis. Three main themes emerged from the participants’ discussions: (1) CAVs as a safety problem for VRUs, (2) CAVs as a justice problem for VRUs, and (3) CAVs as an alignment with societal values problem. Participants emphasised the safety challenges CAVs pose for VRUs, highlighting the need for thorough evaluation and regulatory oversight. Concerns were also raised about CAVs potentially marginalising vulnerable groups within society. Participants advocated for inclusive discussions and a justice-oriented approach to designing a comprehensive transport system to address these concerns.

Keywords: Smart Infrastructure, Advanced Driver Assistance Systems (ADAS)
Abstract: Stopping at a red light at intersections is one of the main contributors to travel time delays in urban traffic. Drivers often stop as close as possible to their lead vehicle, which may cause additional time delay because they should wait before accelerating to secure a safe distance. In this paper, we present an infrastructure-based optimal spacing system that guides vehicles to maintain the optimal spacing when coming to a stop. To achieve this, we formulate an optimization problem to compute the optimal spacing between vehicles such that all vehicles can enter the intersection as soon as possible when the light turns green. To solve the problem efficiently, we decompose it into sub-problems involving only two vehicles and sequentially solve them. We validate through simulations that our approach effectively reduces travel time delay.

Keywords: Smart Infrastructure, Integration of Infrastructure and Intelligent Vehicles, Automated Vehicles
Abstract: The number of electric vehicles is on the rise. To meet the growing demand for charging, automated charging services are being developed. However, many of these services are currently limited in compatibility to either low-speed chargers or come with a high cost per service. This study introduces a new automated charging system that incorporates the design and operation of a mobile manipulator. Experiments conducted on the entire process demonstrate high reliability and compatibility with high-speed chargers. By implementing this system, it is anticipated that the cost of the automated charging services will be reduced.

Keywords: Smart Infrastructure, Integration of Infrastructure and Intelligent Vehicles, Cooperative Vehicles
Abstract: Road infrastructure has become a key enabler for achieving full autonomous driving. However, research on this topic is still awaiting answers, especially at the experimentation side. For this reason, we present AUTODRAITEC, a novel AI-based system that is deployed on the road infrastructure to control the driving of Connected and Autonomous Vehicles (CAVs). The system deploys a hybrid machine learning approach comprised of a supervised learning classifier to characterize the behaviors of human drivers, with a deep reinforcement learning policy to provide speed recommendations for CAVs. This new architecture aims to enhance the situational awareness for autonomous driving systems, and improve the explainability of AI actions through the understanding of others human drivers behaviors. Beside simulation evaluation, a Proof of Concept (PoC) of the system is presented. Using a 1:18 scale testbed that faithfully replicates real-world driving scenarios, we demonstrate that AUTODRAITEC consistently succeeds in avoiding accidents, enhancing safety distance and efficiency, while preserving the traffic flow rate. The presented solution is also scalable to different driving use cases.

Keywords: Smart Infrastructure, Integration of Infrastructure and Intelligent Vehicles, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications
Abstract: Infrastructure-assisted cooperative intelligent transportation systems (C-ITS) leverage roadside intelligent infrastructure and vehicular network technology to facilitate information exchange among traffic participants, enhancing road safety, efficiency, and sustainability. However, this requires not only the massive deployment of infrastructure, including advanced sensors, communication devices, and computing units at various levels, but also the involvement of various stakeholders in the development of functions and real-road testing. In this paper, we present our test field - First Mile, as an open innovation lab for C-ITS in Ingolstadt, Germany, offering an open environment for world-wide stakeholders to conduct research and testing in C-ITS. In particular, we equip a 3.5 km area with 22 roadside intelligent masts and 89 sensors, achieving dense deployment on public roads. Our design includes a protocol stack tailored for different C-ITS stations and services, conforming to European communication standards. Furthermore, we conduct quantitative analyzes of key performance metrics, such as radio signal quality and End-to-End delay, to assess the efficacy of First-Mile in different data processing pipelines. Finally, we delve into the future prospects of large-scale C-ITS deployment, guided by extensive and prolonged measurement studies.

Keywords: Smart Infrastructure, Perception Including Object Event Detection and Response (OEDR), Sensor Fusion for Localization
Abstract: This research focuses on solving challenges related to identifying unfamiliar object categories in the realm of Open-Set Object Detection (OSOD) using infrastructure sensors. Traditional camera-based OSOD systems struggle to generate proposals for dissimilar novel classes due to a lack of feature similarity. This research introduces a novel approach named Fusion Object Detector (FOD), which emphasizes the localization and identification of semantically dissimilar unknown objects through a multimodal fusion architecture involving infrastructure-mounted cameras and LiDARs. FOD leverages a camera-based closed-set object detector for the identification of known class objects, while simultaneously utilizing clusters derived from fused LiDAR point clouds for the detection of unknown class objects. This research work also presents a novel dataset named Thermal camera and LiDAR in Infrastructure Dataset (TLID). TLID comprises fused sensor measurements from multiple thermal cameras and LiDARs mounted in three urban crossings of Ingolstadt city and at CARISSMA outdoor test track. The proposed methodology is evaluated using both an in-house dataset and a publicly available infrastructure dataset for the task of OSOD. The results quantify the importance of multimodal sensor information for the task of identifying dissimilar unknown objects.

Keywords: Smart Infrastructure, Sensor Fusion for Localization, Future Mobility and Smart City
Abstract: 4D mmWave radar sensors are suitable for roadside perception in city-scale Intelligent Transportation Systems (ITS) due to their long sensing range, weatherproof functionality, simple mechanical design, and low manufacturing cost. In this work, we investigate radar-based ITS for scalable traffic analysis. Localization of these radar sensors at city scale is a fundamental task in ITS. For flexible sensor setups, it requires even more effort. To address this task, we propose a self-localization approach that matches two descriptions of the "road": the one from the geometry of the motion trajectories of cumulatively observed vehicles, and the other one from the aerial laser scan. An Iterative Closest Point (ICP) algorithm is used to register the motion trajectory in the road section of the laser scan. The resulting estimate of the transformation matrix represents the sensor pose in a global reference frame. We evaluate the results and show that the method outperforms other map-based radar localization methods, especially for the orientation estimation. Beyond the localization result, we project radar sensor data onto a city-scale laser scan and generate a scalable occupancy heat map as a traffic analysis tool. This is demonstrated using two radar sensors monitoring an urban area in the real world.

Keywords: Smart Infrastructure, Sensor Fusion for Localization, Integration of Infrastructure and Intelligent Vehicles
Abstract: This work presents the design and validation of a distributed multi-sensor object tracking algorithm designed to integrate heterogeneous sensory data from multiple static acquisition stations. The primary challenge addressed is the accurate tracking of targets in complex urban environments, where occlusions and the dynamic nature of traffic frequently hinder detection and tracking efforts. This challenge is particularly relevant in multimodal exchange areas, where vehicular traffic merges with heavy pedestrian and bicycle flow. We also address the scenario of delayed detection, which can easily occur when data from multiple stations are combined or when intensive data processing is performed. Our algorithm ensures high coverage and accuracy by maintaining dual Extended Kalman Filter states for each object, thus allowing for the assimilation of delayed detections and preserving optimal filter estimates at all times. The results of the proposed pipeline, tested using a digital twin of the Milano Bovisa Campus, demonstrate its efficacy, achieving high tracking precision across various scenarios and sensor combinations. Moreover, the results highlight the advantages of a distributed multi-sensor acquisition system compared to a single central station.

Keywords: Smart Infrastructure, Sensor Fusion for Localization, Simulation and Real-World Testing Methodologies
Abstract: Most vision-based sensing and localization infrastructure today employ conventional area scanning cameras due to the high information density and cost efficiency offered by them. While the information-rich two-dimensional images provided by such sensors make it easier to detect and classify traffic objects with the help of deep neural networks, their accurate localization in the three-dimensional real world also calls for a reliable calibration methodology, that maintains accuracy not just during installation, but also under continuous operation over time. In this paper, we propose a camera calibration architecture that extracts and uses corresponding targets from high definition maps, augment it with an efficient stabilization mechanism in order to compensate for the errors arising out of fast transient vibrations and slow orientational drifts. Finally, we evaluate its performance on a real-world test site.

Keywords: Smart Infrastructure, Sensor Signal Processing
Abstract: Cooperative vehicle infrastructure system (CVIS) plays a crucial role in achieving fully autonomous driving. However, Conducting research on infrastructure-side monocular 3D object detection is challenging due to the significant discrepancy in calibration parameters of cameras mounted on different infrastructures. This discrepancy can create ambiguity for detection algorithms. To address this issue, our approach focuses on directly regress 8 vertices of 3D bounding box at image level to mitigate the impact of calibration parameters. During the training and inference process, our method do not need any calibration parameter. The 3D pose and position parameters are obtained after post-processing. We proposed a simple post-processing algorithm to calculate 3D parameters from 8 image-level vertices. And since background from the view of infrastructure remains unchanged, we propose using Gaussian Mixture Model (GMM) branch to generate moving-objects-sensitive (MOS) features. This approach enhances the recognition of objects, leading to our method being termed GMMNet. GMMNet achieves a high mean average precision (mAP) on the DAIR-V2X-I dataset, surpassing other start-of-the-art methods by a significant margin. Furthermore, GMMNet exhibits a greater generalization ability.

Keywords: Smart Infrastructure, Sensor Signal Processing
Abstract: Roadside infrastructure has enjoyed widespread adoption for various tasks such as traffic surveillance, traffic monitoring, control of traffic flow, and prioritization of public transit and emergency vehicles. As automated driving functions and vehicle communications continue to be researched, cooperative and connected driving scenarios can now be realized. Cooperative driving, however, imposes stringent environmental perception and model requirements. In particular, road users, including pedestrians and cyclists, must be reliably detected and accurately localized. Furthermore, the perception framework must have low latency to provide up-to-date information. In this work, we present a refined, camera-based reference point detector design that does not rely on annotated infrastructure datasets and incorporates fusion with cost-effective radar sensor data to increase system reliability, if available. The reference point detector design is realized with box and instance segmentation object detector models to extract object ground points. In parallel, objects are extracted from radar target data through a clustering pipeline and fused with camera object detections. To demonstrate the real-world applicability of our approaches for cooperative driving scenarios, we provide an extensive evaluation of data from a real test site.

Keywords: Smart Infrastructure, Verification and Validation Techniques, Future Mobility and Smart City
Abstract: The lack of trust in algorithms is usually an issue when using Reinforcement Learning (RL) agents for control in real-world domains such as production plants, autonomous vehicles, or traffic-related infrastructure, partly due to the lack of verifiability of the model itself. In such scenarios, Petri nets (PNs) are often available for flowcharts or process steps, as they are versatile and standardized. In order to facilitate integration of RL models and as a step towards increasing AI trustworthiness, we propose an approach that uses PNs with three main advantages over typical RL approaches: Firstly, the agent can now easily be modeled with a combined state including both external environmental observations and agent-specific state information from a given PN. Secondly, we can enforce constraints for state-dependent actions through the inherent PN model. And lastly, we can increase trustworthiness by verifying PN properties through techniques such as model checking. We test our approach on a typical four-way intersection traffic light control setting and present our results, beating cycle-based baselines.

Keywords: Automated Vehicles
Abstract: This paper presents a comparison of the effect of change in ego-vehicle in two different pipelines of imitation learning for autonomous driving (AD) between direct control-based and waypoint-based pipelines. Control-based pipeline involves predicting control signals directly to control the car, whereas waypoint-based pipeline predicts future trajectory of the car and uses a controller module to generate the control signals from the predicted waypoints. In this study, CIL++ was used for control-based method whereas TransFuser was used for waypoint-based method. In our experiments, we used CARLA simulator and deployed both imitation learning models, without retraining or re-tuning the controller parameters, on various cars different from the car used during training. We used Town05 from CARLA's Leaderboard benchmark to evaluate the performance based on driving score, the main metric used in the benchmark. Based on the experiment results, TransFuser is more robust in adapting to different ego-vehicles than CIL++. TransFuser performed better when deployed to different vehicles. However, the performance still suffered when there was a significant change in the car classes. The source code of this work is made publicly available at https://github.com/ailabspace/Comparison-of-Autonomous-Driving-IL-Pipeline-for-Ego-Vehicle-Changes.

Keywords: Sensor Signal Processing, Automated Vehicles, Smart Infrastructure
Abstract: The curation of large-scale datasets is still costly and requires much time and resources. Data is often manually labeled, and the challenge of creating high-quality datasets remains. In this work, we fill the research gap using active learning for multi-modal 3D object detection. We propose ActiveAnno3D, an active learning framework to select data samples for labeling that are of maximum informativeness for training. We explore various continuous training methods and integrate the most efficient method regarding computational demand and detection performance. Furthermore, we perform extensive experiments and ablation studies with BEVFusion and PV-RCNN on the nuScenes and TUM Traffic Intersection (TUMTraf-I) dataset. We show that we can achieve almost the same performance with PV-RCNN and the entropy-based query strategy when using only half of the training data (77.25 mAP compared to 83.50 mAP) of the TUMTraf-I dataset. BEVFusion achieved an mAP of 64.31 when using half of the training data and 52.88 mAP when using the complete nuScenes dataset. We integrate our active learning framework into the proAnno labeling tool to enable AI-assisted data selection and labeling and minimize the labeling costs. Finally, we provide code, weights, and visualization results on our website: https://active3d-framework.github.io/active3d-framework.

Keywords: Functional Safety in Intelligent Vehicles, Automotive Cyber Physical Systems, Integration of Infrastructure and Intelligent Vehicles
Abstract: Cooperative Intelligent Transportation System (C-ITS) is a prime example of Cyber-Physical System (CPS)-Internet of Things (IoT), in that the mechanics are combined with electronic, information technologies, and Vehicle-to-everything (V2X) communication. In that respect, C-ITS inherits the unique properties of CPS-IoT, known as open connectivity and runtime adaptivity. This means that C-ITS can change its behavior or structure at runtime, which creates unknowns and uncertainties during system development. Moreover, IT solutions, such as Artificial Intelligence (AI)/Machine Learning (ML), Open Source Software (OSS), and Commercial-off-the-shelf (COTS) components, are used in automotive systems to foster innovation. However, these solutions often lack the guarantees that are necessary for safety-relevant applications. All aspects considered, this paper identifies the gaps between industry and research by examining state-of-the-art safety standards and literature and analyzing the limitations in safeguarding C-ITS using the Shuttle2X project as an example.

Keywords: Functional Safety in Intelligent Vehicles, Future Mobility and Smart City, Security and Privacy
Abstract: Over the last year, significant advancements have been made in the realms of large language models (LLMs) and multi-modal large language models (MLLMs), particularly in their application to autonomous driving. These models have showcased remarkable abilities in processing and interacting with complex information. In autonomous driving (AD), LLMs and MLLMs are extensively used, requiring access to sensitive vehicle data such as precise locations, images, and road conditions. This data is transmitted to an LLM-based inference cloud for advanced analysis. However, concerns arise regarding data security, as the protection against data and privacy breaches primarily depends on the LLM's inherent security measures, without additional scrutiny or evaluation of the LLM's inference outputs. Despite its importance, the security aspect of LLMs in autonomous driving remains underexplored. Addressing this gap, our research introduces a novel security framework for autonomous vehicles, utilizing a multi-agent LLM approach. This framework is designed to safeguard sensitive information associated with autonomous vehicles from potential leaks, while also ensuring that LLM outputs adhere to driving regulations and align with human values. It includes mechanisms to filter out irrelevant queries and verify the safety and reliability of LLM outputs. Utilizing this framework, we evaluated the security, privacy, and cost aspects of eleven large language model-driven autonomous driving cues. Additionally, we performed QA tests on these driving prompts, which successfully demonstrated the framework's efficacy.

Keywords: Functional Safety in Intelligent Vehicles, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Security and Privacy
Abstract: Connected and automated vehicles already are the main force in realizing the vision of intelligent transportation in smart cities. However, enabling broadband connectivity for vehicles brings up new threats of spreading fake information. By broadcasting false sharing data, an attack vehicle has the ability to cause nearby vehicles to get confused or possibly collide in catastrophic accidents. The present study presents a resilient fusion-based multimodal abnormal detection technique, referred to as FMAD. FMAD facilitates a fusion model based on Dempster-Shafer's theory to strengthen confidence in the final detection assessment of detection results from multiple vehicles. FMAD can determine whether a vehicle is spreading false maneuver information with up to 96.18 percent accuracy of confidence. Meanwhile, our method outperforms all existing approaches in terms of the reliability of the detection decision.

Keywords: Functional Safety in Intelligent Vehicles, Vehicular Active and Passive Safety, Automated Vehicles
Abstract: Worldwide efforts have been made to conduct various demonstration projects on real roads to accelerate the commercialization of autonomous vehicles (AVs). However, traffic safety issues associated with mixed traffic streams need to be systematically addressed to facilitate the adoption of AVs. The objective of this study is to evaluate driving safety of AV based on analyzing real-world AV data obtained from a real-world autonomous mobility testbed in Seoul, Korea. In addition, this study attempts to identify potential risk factors including road infrastructure and traffic conditions. Behavior characteristics and driving safety were evaluated using data collected from AVs that drove on actual roads in autonomous driving (AD) mode and manual driving (MD) mode. This study also proposed a driving risk indicator (RDI) for autonomous vehicle based on take-over triggered events to evaluate driving safety. Potential risk factors were identified by modeling a binomial logistic regression using the road facility characteristics data. Statistically significant independent variables were defined as potential risk factors. 'Illegal parking', 'dedicated turning lanes', and ‘turning movements at the intersection' were identified as potential risk factors in the AV. The results of this study are expected to provide a foundation for the improvement of infrastructure and to establish policies to enhance traffic safety for AVs.

Keywords: Automated Vehicles, Functional Safety in Intelligent Vehicles, Sensor Signal Processing
Abstract: In the rapidly evolving landscape of Automated Driving Systems (ADS), the dynamic nature of driving environments poses significant challenges to traditional static risk assessment models. Addressing this, our paper introduces an approach focused on predicting severity and controllability ratings using context-specific risk features. We have developed a model that leverages the strengths of Random Forest (RF) and Gradient Boosting Decision Trees (GBDT) to adeptly navigate the complexities of diverse driving conditions.

Central to our approach is the effective processing and analysis of data derived from risk-specific contexts, converting this information into actionable risk features through well-established mathematical formulations. This model excels in contextual understanding, providing a more detailed and accurate risk assessment compared to traditional methods. Unlike subjective approaches, our model offers an objective and data-driven analysis. A key implementation of our model is showcased in an Adaptive Cruise Control (ACC) system, particularly in highway lane-following scenarios under various driving conditions. This application highlights the model's capability to dynamically assess risks and prioritize factors, thereby substantially improving Dynamic Risk Assessment (DRA) in ADS.

In essence, our research successfully bridges the gap between static risk models and the dynamic nature of driving environments. By integrating ensemble learning models like RF and GBDT, we present an advanced and dynamic tool for risk assessment in ADS, marking an improvement in the field of automotive safety.

Keywords: Automated Vehicles, Functional Safety in Intelligent Vehicles, Verification and Validation Techniques
Abstract: The Operational Design Domain (ODD) consists of operating conditions under which an Automated Driving System (ADS) feature is intended to be deployed and should satisfy safety and performance requirements. Creating human-interpretable and monitorable ODD specifications for ADS features, comprising black-box and non-deterministic Machine Learning (ML) components, is complicated owing to the unknown impact of possibly infinite operational contexts on system requirement fulfillment. Furthermore, these ML components may be updated to address unforeseen operational contexts encountered after feature deployment, thus necessitating further updates to the ODD. This paper proposes a novel approach for online ODD identification, i.e., discovering operating conditions wherein the ADS feature satisfies system requirements, using fuzzy behavior oracles. Our data-driven approach involves human-interpretable representation of operational contexts, facilitating the semi-automatic generation of conditional ODD statements and updates to ODD post-feature deployment. The feasibility of our approach is validated with a case study on a Lane Change Assist ADS feature, which exhibits a 55% improvement in scalability, allowing its deployment in a broader ODD.

Keywords: Sensor Signal Processing, Functional Safety in Intelligent Vehicles, Automated Vehicles
Abstract: In automated driving, the safety and robustness of the overall system are among the most important key challenges today. To tackle these safety and robustness challenges, the monitoring and self-assessment of all modules in the automated system is necessary. Tracking surrounding objects as part of the environmental perception is a key module in automated systems. Thus, this work presents a novel overall concept and framework for self-assessment in multi-object tracking based on the subjective logic theory. The self-assessment concept is comprehensively discussed and evaluated by simulations and real-world data of the KITTI dataset, showing the relevance of this proposed method.

Keywords: Vehicular Active and Passive Safety, Functional Safety in Intelligent Vehicles, Automated Vehicles
Abstract: Navigating unsignalized intersections in urban environments poses a complex challenge for self-driving vehicles, where issues such as view obstructions, unpredictable pedestrian crossings, and diverse traffic participants demand a great focus on crash prevention. In this paper, we propose a novel state representation for Reinforcement Learning (RL) agents centered around the information perceivable by an autonomous agent, enabling the safe navigation of previously uncharted road maps.

Our approach surpasses several baseline models by a significant margin in terms of safety and energy consumption metrics. These improvements are achieved while maintaining a competitive average travel speed. Our findings pave the way for more robust and reliable autonomous navigation strategies, promising safer and more efficient urban traffic environments.

Keywords: Automated Vehicles, Simulation and Real-World Testing Methodologies, Functional Safety in Intelligent Vehicles
Abstract: The transition from advanced driver assistance systems to highly automated vehicles proves to be difficult, as the driver is no longer a safety fallback for the latter systems. One of the main challenges is formed by edge cases in the encountered driving scenarios that trigger functional insufficiencies in automated driving (AD) systems. Functional insufficiencies, for the sake of understanding, may be viewed as an inappropriate understanding of or response to a scenario in an AD system, which in turn causes dangerous vehicle behavior. Prior research suggests that using an architecture capable of including redundant heterogeneous AD systems as separate channels, such as the Safety Shell, can mitigate some of these functional insufficiencies. However, this benefit has only been evaluated in limited and deterministic simulation environments. To overcome this, our objectives in this paper are to (i) develop an experimental method for extensive testing of such architectures, and (ii) to assess the suitability of the Safety Shell architecture to handle edge cases with this new method. Using the developed experimental setup we observe a significant safety and availability increase of the Safety Shell compared to the included individual AD channels in the tested scenarios. Finally, our study provides insight into the requirements for the evaluated AD channels.

Keywords: Security and Privacy
Abstract: Trajectory prediction is critical for the safe planning and navigation of automated vehicles. The trajectory prediction models based on the neural networks are vulnerable to adversarial attacks. Previous attack methods have achieved high attack success rates but overlook the adaptability to realistic scenarios and the concealment of the deceits. To address this problem, we propose a speed-adaptive stealthy adversarial attack method named SA-Attack. This method searches the sensitive region of trajectory prediction models and generates the adversarial trajectories by using the vehicle-following method and incorporating information about forthcoming trajectories. Our method has the ability to adapt to different speed scenarios by reconstructing the trajectory from scratch. Fusing future trajectory trends and curvature constraints can guarantee the smoothness of adversarial trajectories, further ensuring the stealthiness of attacks. The empirical study on the datasets of nuScenes and Apolloscape demonstrates the attack performance of our proposed method. Finally, we also demonstrate the adaptability and stealthiness of SA-Attack for different speed scenarios. Our code is available at the repository: https://github.com/eclipse-bot/SA-Attack.

Keywords: Security and Privacy
Abstract: Highly connected and automated driving technologies have ushered digital transformation and flexibility to modern cars. However, the vehicle's attack surface has significantly expanded due to increased connectivity. To address this problem, automotive manufacturers are adopting more secure practices driven by standards and regulations. In addition to the deployed cryptographically strong security measures in automotive, we need an Intrusion Detection and Prevention System (IDPS) that actively monitors the vehicle for intrusions, prevents them, and provides notification, as required by UN Regulation No. 155. In this work, we aim to identify the current limitations of the existing automotive approaches and contribute to an advanced IDPS solution. We propose architectural changes that improve reliability and form a framework to propose reactions in a safety-related automotive context. We evaluate our proposed architecture with regard to performance and security design. With the proposed changes to the IDPS architecture, our aim is to integrate a dynamic and adaptive strategy for IDPS, enhancing resilience against emerging threats and vulnerabilities.

Keywords: Security and Privacy, Software-Defined Vehicle for Intelligent Vehicles, Automotive Cyber Physical Systems
Abstract: To satisfy the bandwidth requirement of emerging automotive applications such as autonomous driving, high bandwidth automotive Ethernet is increasingly employed as the backbone of in-vehicle networks. Besides its high bandwidth, Ethernet-based systems have much more complex software architectures that are more prone to security vulnerabilities than conventional networks. This study presents such an attack scenario that can obstruct safety-critical data flows between networked computer systems by generating garbage packets that are crammed into the Linux transmit queues. As a solution, we provide a hidden transmit queue (named Band-S) that can be used only by applications authorized by a trusted execution environment (TEE) of automotive application processors. We implemented the system based on the open source OP-TEE package and proved its feasibility for evading the cram attack by malicious applications.

Keywords: Vehicular Active and Passive Safety, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Security and Privacy
Abstract: Higher levels of automation and connectivity can improve the performance of existing Advanced Driver Assistance Systems (ADAS) / Automated Driving Systems (ADS) or enable new Connected, Cooperative, and Automated Mobility (CCAM) applications. But this will also introduce new cybersecurity risks. An example of an ADAS which is becoming mandatory in all vehicles sold in Europe is the Intelligent Speed Assistance (ISA). This system will receive information from traffic signs, High-Definition (HD) maps and Infrastructure-to-Vehicle (I2V) communication in order to set the recommended/mandatory speed. In this research, we design a Misbehaviour Detection System (MDS) for ISA, in the scope of the NIST cybersecurity cycle, capable of securing the network (for I2V communication) and protecting the integrity of the data used and shared for ISA. During the operation of the system, the MDS assigns a Trust Score (for tracking misbehaviours) and a Validity Score (for tracking reputation) to each of the ISA input signals.

Keywords: Automated Vehicles
Abstract: Sensor calibration is crucial for autonomous driving, providing the basis for accurate localization and consistent data fusion. Enabling the use of high-accuracy GNSS sensors, this work focuses on the antenna lever arm calibration. We propose a globally optimal multi-antenna lever arm calibration approach based on motion measurements. For this, we derive an optimization method that further allows the integration of a-priori knowledge. Globally optimal solutions are obtained by leveraging the Lagrangian dual problem and a primal recovery strategy. Generally, motion-based calibration for autonomous vehicles is known to be difficult due to cars' predominantly planar motion. Therefore, we first describe the motion requirements for a unique solution and then propose a planar motion extension to overcome this issue and enable a calibration based on the restricted motion of autonomous vehicles. Last we present and discuss the results of our thorough evaluation. Using simulated and augmented real-world data, we achieve accurate calibration results and fast run times that allow online deployment.

Keywords: Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Cooperative Vehicles, End-To-End (E2E) Autonomous Driving
Abstract: Semantic communication is a key paradigm in future 6G systems, designed to revolutionize the physical layer of traditional communication, in order to enhance efficiency of data transmission. Fueled by advancements in deep learning architectures, image processing has made significant strides in segmentation and scenes analysis for autonomous vehicles (AVs). Motivated by these advancements, we present an innovative outlook on communication systems, by leveraging semantic data. Thus, we introduce a novel semantic end-to-end communication system, named SEECAD, specifically designed for image transmission in autonomous driving environments. SEECAD is based on a theoretical model, aligning with the semantic level concepts and leveraging a shared knowledge base to efficiently transmit meaningful image data. The semantic encoder and decoder of SEECAD are built upon deep learning architecture, empowered by Low-Density Parity-Check (LDPC) codes. This integration serves to minimize semantic error transmission and enhance the segmentation accuracy at the receiver. Our proposed semantic communication approach was extensively evaluated in various wireless image transmission scenarios over an AWGN channel, using different QAM modulations (4QAM and 16QAM). Our experimental results demonstrated that the proposed SEECAD achieves accurate and effective image transmission in noisy environments.

Keywords: Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Integration of Infrastructure and Intelligent Vehicles, Integration of Onboard Systems and Cloud-Based Services
Abstract: Future automated mobility is expected to rely on Cooperative Perception (CP) applications to achieve high degrees of reliable autonomy. CP applications are characterized by the exchange of large sensor data objects, such as camera frames or LIDAR point clouds. The high mobility of nodes in combination with the safety-critical nature of CP data, demands architectures enabling continuous connectivity for reliable transmission of large data objects. This work presents a user-centric architecture combining a cell-free Radio-Access-Network (RAN) with a centralized backbone management. By monitoring multiple available connections through an ultra-lightweight heartbeat-based protocol, the handover procedure can be reduced to a low-latency backbone reconfiguration. Thus, the need for redundant data transmission to achieve loss-free and continuous data transmissions during handover is avoided.

Keywords: Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Software-Defined Vehicle for Intelligent Vehicles, Integration of Infrastructure and Intelligent Vehicles
Abstract: V2X (Vehicle-to-everything) communication has garnered much attention in propelling Internet of Vehicles (IoV) to sought shelter for many mission-critical edge-based applications in Intelligent Transportation Systems (ITS). The goal of achieving end-to-end latency (E2E) for on-road video analytics has become an essential critique to ensure the timely realization of computation-intensive tasks. By adopting the edge services (ES) along with the deployment of better application configuration, the co-optimization of video analytics accuracy and E2E latency can be achieved. However, there are certain challenges to this, such as poor application configuration, variable network conditions, erratic movement of the vehicles, which compromise the E2E latency, and passive strategies of congestion control that fail to avoid the oversubscription of the available bandwidth. To address the key challenges discussed, we propose a Fleet management scheme (FMS), a traffic video stream orchestrator in this work, which introduces Synergetic Service placement and Cost Minimization algorithm (SSPCM) to provision accurate streaming analytics and also advocates a Bandwidth Reappropriation algorithm for priority-based streams at the edge node level. SSPCM is solved based on Lyapunov optimization and operates online without needing future information, and attains a verifiable performance bound on the Long-term low-latency (LTLL) constraint violation. Extensive evaluations using realistic data reveal the superior performance of the proposed scheme in balancing accuracy with E2E latency.

Keywords: Security and Privacy, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications
Abstract: Pseudonyms are widely used in Cooperative Intelligent Transport Systems (C-ITS) to protect the location privacy of vehicles. However, the unlinkability nature of pseudonyms also enables Sybil attacks, where a malicious vehicle can pretend to be multiple vehicles at the same time. In this paper, we propose a novel protocol called zero-knowledge Proof of Distinct Identity (zk-PoDI,) which allows a vehicle to prove that it is not the owner of another pseudonym in the local area, without revealing its actual identity. Zk-PoDI is based on the Diophantine equation and zk-SNARK, and does not rely on any specific pseudonym design or infrastructure assistance. We show that zk-PoDI satisfies all the requirements for a practical Sybil-resistance pseudonym system, and it has low latency, adjustable difficulty, moderate computation overhead, and negligible communication cost. We also discuss the future work of implementing and evaluating zk-PoDI in a realistic city-scale simulation environment.

Keywords: Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications
Abstract: The impact of brake safety is particularly serious in Intelligent vehicles, and braking performance can be improved by introducing Vehicle to Vehicle (V2V) communications. This paper proposes a methodology for an optimisation analysis on vehicle transmitter power and channel delay following the IEEE 802.11p standard to determine the minimum required V2V communications distance for maintaining a safe connected brake in a rural Line-of-sight driving scenario. We have built up a methodology on simulating the maximum delay from Packet Error Rates to analyse under the worst situation. After being optimised for different vehicle initial speeds, the minimum required transmitter power for a smooth and safe brake action is determined.

Keywords: Teleoperation of Intelligent Vehicles, Vehicle Control and Motion Planning, Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications
Abstract: In this paper, we demonstrate how the network latency, the longitudinal velocity and the path curvature affect performance of the teleoperated driving (ToD). The performance of a ToD system is studied analytically through stability analysis of a dimensionless vehicle dynamics model with a scaled delay, which integrates the end-to-end (E2E) latency and the longitudinal velocity of the vehicle. We also establish a numerical simulation framework for ToD while incorporating a stochastic latency in the control loop arising from vehicle-to-network-to-vehicle (V2N2V) communication through a wireless network. The stochasticity of the latency mostly comes from the network scalability challenges to support high video bitrates, which also leads to packet drops. We provide simulation results of teleoperating a vehicle in a realistic parking lot scenario and demonstrate the effects of speed, curvature and stochastic latency on the maneuver performance.

Keywords: Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Cooperative Vehicles
Abstract: Collective Perception will play a crucial role for ensuring vehicular safety in the near future, enabling the sharing of local perceived objects with other Intelligent Transport System Stations (ITS-Ss). However, at the beginning of the roll-out, low market penetration rates are expected. This paper proposes and evaluates an application layer multi-hop Collective Perception Service (CPS) for vehicular ad-hoc networks. The goal is to improve the environmental awareness ratio in scenarios with low CPS market penetration. In such scenarios, the CPS service without forwarding enabled struggles to achieve complete awareness. A decentralized application layer forwarding algorithm is presented that shares perceived object information across multiple hops while maintaining a low age of information. The proposed approach is compared against standard CPS with no forwarding and CPS with geographically-scoped (GBC) multi-hop forwarding. Simulations according to standards of the European Telecommunications Standards Institute (ETSI) demonstrate that the application layer forwarding achieves near 100% awareness at 10% penetration rate versus 92% for standard CPS. The awareness improvement comes with moderate channel load, unlike GBC forwarding which quickly saturates the channel. The median age of information remains below 80 ms for the proposed scheme, enabling real-time CPS operation. Our application layer multi-hop approach effectively improves environmental awareness during initial CPS deployment while aligning with latency and channel load requirements.

Keywords: Vehicle-To-Everything (V2X) and Cellular V2X (C-V2X) Communications, Simulation and Real-World Testing Methodologies, Future Mobility and Smart City
Abstract: Adaptive Traffic Signal Control (ATSC) is an improvement over fixed timing-based or sensor-based signal management. Having real-time learning-based adaptive signals that can optimize the timings of the signal phases of the intersection reduces waiting time and emissions. However, connected vehicles-based adaptive traffic signal requires a large amount of data to be exchanged between vehicles and the infrastructure. Cellular Vehicle to Everything (CV2X) enables us to build adaptive traffic signals using connected vehicle data. In this paper, we investigate the performance of 5G-CV2X for an adaptive traffic signal as well as for wireless signaling of traffic signal status using the 5.9 GHz ITS band and 24 GHz mmWave. Our results indicate that 5G CV2X with 5.9GHz frequency is capable of supporting CV2X-based adaptive traffic signals in heavy vehicular traffic with an average delay of about 4.25 ms for vehicle-to-infrastructure communication, 2.03 ms for infrastructure-to-vehicle communication and 42.71 ms for vehicle-to-vehicle communication. Using 24GHz mmWave further reduces the communication delay to the sub-millisecond range. Nevertheless, the performance gain is at the cost of additional infrastructure due to mmWave's shorter range.

Keywords: Policy, Ethics, and Regulations
Abstract: This paper proposes a methodology for allocating courier workloads to courier drivers in a digital-based courier platform with heterogeneous user characteristics. The framework proposed in this paper is inspired by the envy-free cake-cutting problem, which captures the individual-level unfairness arising from the preferences of heterogeneous drivers. In this study, we incorporate this unfairness characteristic and stated preferences data into an optimization model, which we call the envy-based workload balancing vehicle routing problem (EWB-VRP). The objective of this optimization model is to assign delivery zones by explicitly considering drivers' income, workload, and preferences. Compared to existing models, our results show that our model significantly reduces envy among drivers while keeping travel times close to the system optimum. Our proposed research framework has the potential to improve the overall fairness of the courier logistics sector by incorporating individual preferences and drivers' skills in choosing efficient routes.

Keywords: Policy, Ethics, and Regulations, Sensor Signal Processing
Abstract: With the development in sensors, computing resources and deep neural networks, the safety mechanisms of vehicles are continuously developed towards the fully automated driving system (FADS). One of the most crucial aspects of this technology is the environmental perception. Most of the existing works focus on recognizing specific targets in the scene, and often overlook the holistic information for sufficient use by FADS. In this paper, we adopt a multi-task learning approach to achieve more comprehensive recognition of environmental information. On the other hand, due to the increasing prominence of traffic issues in urban areas, the problem of illegal parking has gained more attention. Thus, a vision system to recognize illegal parking in road scenes based on environmental perception is proposed. We also collect an illegal parking dataset and make it available publicly for related research. Source code and dataset are available publicly.