Ewecker, Lukas (Porsche), Klugmann, Christopher (Quality-Match GmbH), Kondermann, Daniel (Universitaet Heidelberg), Schwager, Robin (Dr. Ing. h.c. F. Porsche AG), Villmann, Thomas (University of Applied Sciences Mittweida, Saxon Institute for Co)

Are Gamers Good Annotators? a Comparative Study of Gaming Crowds and Professional Annotators

Scheduled for presentation during the Poster Session "Transportation Data Analysis and Calibration" (FrAT17), Friday, September 27, 2024, 10:30−12:30, Foyer

2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC), September 24- 27, 2024, Edmonton, Canada

This information is tentative and subject to change. Compiled on October 7, 2024

Abstract

Annotating image data at scale and at a high quality using professional annotators is expensive and time-consuming. This paper investigates a new cost-efficient and scalable source of annotators. We propose to leverage large pools of online gamers—so-called gaming crowds—to annotate images. Therefore, we investigate the use of gaming crowds for annotating images, addressing the growing need for labeled datasets in computer vision and machine learning. While gaming crowds offer a cost-efficient and readily available annotator pool, their performance varies across tasks. Our experiments on a custom image dataset reveal that, for certain categorical object annotation tasks, gaming crowds can match professional annotators in quality, albeit with more noise. Challenges arise in complex tasks, leading to higher ambiguity and reduced agreement with professionals, particularly in finer distinctions. Key insights gleaned from the study underscore the importance of task design in ensuring clarity and minimizing ambiguity, as well as the need for strategies to identify and mitigate the impact of malicious annotators while fostering sustained user engagement throughout the annotation process. A comprehensive analysis discusses challenges and opportunities in gaming crowdsourcing for image annotation. Despite its potential, refining gaming crowdsourcing is essential for its successful integration into large-scale image annotation, benefiting computer vision and machine learning applications.