Xu, Zixuan (Korea Advanced Institute of Science and Technology (KAIST)), Chen, Tiantian (KAIST), Chen, Sikai (University of Wisconsin-Madison)

A LLM-based Multimodal Warning System for Driver Assistance

Scheduled for presentation during the Invited Session "Learning-powered and Knowledge-driven Autonomous Driving I" (ThAT1), Thursday, September 26, 2024, 11:30−11:50, Salon 1

2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC), September 24- 27, 2024, Edmonton, Canada

This information is tentative and subject to change. Compiled on October 8, 2024

Abstract

Advanced driver assistance systems (ADAS) play a crucial role in enhancing road safety by providing timely warnings and assistance. However, drivers exhibit varying hazard perception abilities and interaction needs due to their distinct characteristics, necessitating personalized warning systems to improve user experience and acceptance. Despite most advanced systems today are able to offer personalization, they remain static based on user setting rather than updating automatically. Large Language Models (LLMs), known for their exceptional knowledge acquisition, reasoning, and human-machine interaction skills, offer a promising solution for developing more customized systems. In light of this, we developed a LLM-based Multimodal Warning system (LLM-MW), which offers personalized warnings through multimodal interaction. By interpreting the traffic environment and drivers' profiles, the system can plan multimodal warning contents and perform warnings. Our experiment results show that the LLM-MW provides a level of personalization for different drivers in various scenarios.