Bauer, Erik (ETH Zurich), Yin, Yue (University Kassel), Ayeb, Mohamed (University of Kassel), Krause, Christoph (BMW), Brabetz, Ludwig (University of Kassel)

BEAM: Bilevel Evaluation and Analysis of Multi-Object-Tracking under Latency

Scheduled for presentation during the Regular Session "Sensing, Vision, and Perception III" (ThAT5), Thursday, September 26, 2024, 11:30−11:50, Salon 13

2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC), September 24- 27, 2024, Edmonton, Canada

This information is tentative and subject to change. Compiled on December 26, 2024

Abstract

Environmental perception, particularly multi-object-tracking, is a crucial part of autonomous vehicles. However, system-level disturbances like latencies and jittering sampling rates can highly degrade the performance of multi-object-tracking (MOT) systems, which in turn compromises the functionality and safety of the vehicle. In this work, we propose BEAM, a novel framework for evaluating MOT performance with respect to system-level disturbances such as perception latency. BEAM utilizes a bilevel evaluation scheme: first, a tracking state error distribution is computed for both the disturbed and undisturbed system. The relative change of the two distributions is measured using the Jensen-Shannon-divergence, from which we distill an easily interpretable evaluation score. In extensive experiments, we evaluate a state-of-the-art MOT tracking system on a real-world dataset (KITTI) both with and without a spatio-temporal latency compensator, injecting different perception latencies. Comparing BEAM to current MOT evaluation metrics, we show that our proposed framework is able to provide meaningful evaluation scores under latency where other metrics begin to fail. With our work, we present a novel disturbance-focused evaluation framework which explicitly evaluates both state precision and robustness against adverse system-level conditions. By introducing BEAM, we aim to contribute to more robust, safer perception systems through disturbance-focused evaluation.