ITSC 2025 Paper Abstract

Paper FR-LM-T44.1

Yang, Ye (Shanghai DianJi University), Li, Donghe (Xi'an Jiaotong University), Sun, Li (Xi’an Jiaotong University), Liu, Sen (Shanghai Dianji University), Cao, Zhuobin (Xi'an Jiaotong university), Chen, Shitao (Xi'an Jiaotong University, Xi'an, China), Yang, Qingyu (Xi'an Jiaotong University)

Detection and Correction of Driver Mode Confusion Using LLM-Based Semantic Feedback in SAE Level 2 Automation

Scheduled for presentation during the Regular Session "S44a-Human Factors and Human Machine Interaction in Automated Driving" (FR-LM-T44), Friday, November 21, 2025, 10:30−10:50, Currumbin

2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC), November 18-21, 2025, Gold Coast, Australia

This information is tentative and subject to change. Compiled on October 18, 2025

Keywords Driver Behavior Monitoring and Feedback Systems for Semi-autonomous Vehicles, Human-Machine Interaction Systems for Enhanced Driver Assistance and Safety, User-Centric HMI Design for Autonomous Vehicle Control Systems

Abstract

This research presents a novel real-time framework for detecting and correcting Driver Mode Confusion (DMC) in SAE Level 2 automated driving systems. Existing systems rely solely on static indicators (e.g., dashboard lights) to communicate system states, failing to assess or rectify drivers' internal cognitive states, resulting in delayed takeovers and safety-critical incidents. Our framework integrates three interconnected modules: System Mode Monitoring, Driver Mode Confusion Detection, and Controlled Semantic Prompt Generation. The methodology fuses vehicle telemetry, visual behavior (gaze direction, hand position, head pose), and verbal semantic features through a lightweight classifier to identify cognitive misalignment, subsequently employing large language models to generate adaptive semantic feedback. Experimental results demonstrate 86% accuracy and an F1-score of 0.711 on the test set, validating the critical importance of multimodal integration, particularly linguistic features. This research addresses a fundamental gap in human-automation interaction by transitioning from passive status indication to active cognitive calibration, thereby enhancing the safety and reliability of partially automated driving systems where responsibility is distributed between human and machine.