Shihab, Ibne (Iowa State University), bhagat, sudesh (iowa state university), Sharma, Anuj (Iowa State Univeristy), Wood, Jonathan (Iowa State University)

Accuracy Is Not Agreement: Expert-Aligned Evaluation of Crash Narrative Classification Models

Scheduled for presentation during the Regular Session "S36b-Behavior Modeling and Decision-Making in Traffic Systems" (FR-EA-T36), Friday, November 21, 2025, 14:50−14:50, Surfers Paradise 3

2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC), November 18-21, 2025, Gold Coast, Australia

This information is tentative and subject to change. Compiled on October 18, 2025

Abstract

This study investigates the relationship between deep learning (DL) model accuracy and expert agreement in classifying crash narratives. We evaluate five DL models—including BERT variants, USE, and a zero-shot classifier—against expert labels and narratives, and extend the analysis to four large language models (LLMs): GPT-4, LLaMA 3, Qwen, and Claude. Our findings reveal an inverse relationship: models with higher technical accuracy often show lower agreement with human experts, while LLMs demonstrate stronger expert alignment despite lower accuracy. We use Cohen’s Kappa and Principal Component Analysis (PCA) to quantify and visualize model-expert agreement, and employ SHAP analysis to explain misclassifications. Results show that expert-aligned models rely more on contextual and temporal cues than location-specific keywords. These findings suggest that accuracy alone is insufficient for safety-critical NLP tasks. We argue for incorporating expert agreement into model evaluation frameworks and highlight the potential of LLMs as interpretable tools in crash analysis pipelines.