Lan, Tianxiang (Technische Universitaet Braunschweig), Schade, Nick (Technische Universität Braunschweig), Pannek, Jürgen (Institute for Intermodal Transportation and Logistic System, Tec)

Assessing Reliability of Virtual Testing for Deep Learning Models in Autonomous Vehicles

Scheduled for presentation during the Poster Session "Validation, simulation, and virtual testing I" (WeAT14), Wednesday, September 25, 2024, 10:30−12:30, Foyer

2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC), September 24- 27, 2024, Edmonton, Canada

This information is tentative and subject to change. Compiled on December 26, 2024

Abstract

Testing of autonomous vehicles is a crucial step to ensure safety, but it can be costly and time-consuming. Virtual testing can be a solution to reduce these costs and time limitations. However, it is necessary to evaluate the reliability of virtual testing before it can be used as a replacement for specific field tests. Especially deep learning (DL) models in automated driving systems (ADS) can be sensitive to the data they encounter. In this work, we propose a cascade method to evaluate the reliability of virtual testing for DL models in autonomous driving. This method considers both the scenario level and the model level to identify the origins of the reality gap concerning scenario modeling, the fidelity of the simulation environment, and the model performance. We show an exemplary case study to demonstrate the method, which focuses on the object tracking task in autonomous driving. Based on the case study, we identify several challenges at different stages of the virtual testing pipeline. The result of this work can be used to improve the reliability of virtual testing for DL models in autonomous driving via calibrating the pipeline according to the identified challenges.